A common operation in CLI applications is interacting with the file system, particularly traversing it recursively to find files. Go provides an elegant mechanism for this through the filepath.Walk function in the path/filepath package.

In this lesson, we'll create a utility called walker that traverses a directory structure and collects statistics about it.

The filepath Package

The path/filepath package provides functions for manipulating file name paths in a way that's compatible with the target operating system. Besides cross-platform path manipulation, it also offers the Walk function, which traverses a file tree starting from a specified root.

Basic Usage of filepath.Walk

Let's start with a basic implementation that just prints all the files found:

package main

import (
    "fmt"
    "os"
    "path/filepath"
)

func main() {
    // Default to current directory, but allow specifying a different one
    rootDir := "."
    if len(os.Args) > 1 {
        rootDir = os.Args[1]
    }

    // Walk the directory tree
    filepath.Walk(rootDir, func(path string, info os.FileInfo, err error) error {
        // Print each file path
        fmt.Println(path)
        return nil
    })
}

When we run this code, it will list all files and directories within the specified directory, recursively.

Collecting Statistics

Let's enhance our program to collect statistics about the files and directories:

package main

import (
    "fmt"
    "os"
    "path/filepath"
)

func main() {
    // Default to current directory, but allow specifying a different one
    rootDir := "."
    if len(os.Args) > 1 {
        rootDir = os.Args[1]
    }

    // Initialize counters
    totalFiles := 0
    totalDirs := 0
    var totalBytes int64 = 0

    // Walk the directory tree
    filepath.Walk(rootDir, func(path string, info os.FileInfo, err error) error {
        // Skip errors
        if err != nil {
            return nil
        }

        // Skip the root directory itself
        if path == "." {
            return nil
        }

        if info.IsDir() {
            totalDirs++
        } else {
            totalFiles++
            totalBytes += info.Size()
        }
        
        return nil
    })

    // Print summary
    fmt.Println("Summary:")
    fmt.Println("  Total files:       ", totalFiles)
    fmt.Println("  Total directories: ", totalDirs)
    fmt.Println("  Total bytes:       ", totalBytes)
}

Now we're counting the number of files, directories, and the total size in bytes. Note that we skip the root directory itself in the count.

Filtering by File Extension

Let's add the ability to filter files by their extension:

package main

import (
    "flag"
    "fmt"
    "os"
    "path/filepath"
)

func main() {
    // Add extension flag
    var desiredExt string
    flag.StringVar(&desiredExt, "ext", "", "Filter files by extension (e.g., .go)")
    flag.Parse()

    // Default to current directory, but allow specifying a different one
    rootDir := "."
    if flag.NArg() > 0 {
        rootDir = flag.Arg(0)
    }

    // Initialize counters
    totalFiles := 0
    totalDirs := 0
    var totalBytes int64 = 0

    // Walk the directory tree
    filepath.Walk(rootDir, func(path string, info os.FileInfo, err error) error {
        // Skip errors
        if err != nil {
            return nil
        }

        // Skip the root directory itself
        if path == "." {
            return nil
        }

        if info.IsDir() {
            totalDirs++
        } else {
            // Check extension if filter is set
            fileExt := filepath.Ext(path)
            if desiredExt != "" && fileExt != desiredExt {
                return nil
            }
            
            totalFiles++
            totalBytes += info.Size()
        }
        
        return nil
    })

    // Print summary
    fmt.Println("Summary:")
    fmt.Println("  Total files:       ", totalFiles)
    fmt.Println("  Total directories: ", totalDirs)
    fmt.Println("  Total bytes:       ", totalBytes)
}

Now we can run the program with -ext=.go to only count Go files.

Using the io/fs Package

In more recent versions of Go, the io/fs package provides a flexible filesystem interface that can be used with different implementations. Let's refactor our code to use this approach:

package main

import (
    "flag"
    "fmt"
    "io/fs"
    "os"
    "path/filepath"
)

func main() {
    // Add extension flag
    var desiredExt string
    flag.StringVar(&desiredExt, "ext", "", "Filter files by extension (e.g., .go)")
    flag.Parse()

    // Default to current directory, but allow specifying a different one
    rootDir := "."
    if flag.NArg() > 0 {
        rootDir = flag.Arg(0)
    }

    // Create a filesystem from the root directory
    dirFS := os.DirFS(rootDir)

    // Initialize counters
    totalFiles := 0
    totalDirs := 0
    var totalBytes int64 = 0

    // Walk the directory tree using fs.WalkDir
    fs.WalkDir(dirFS, ".", func(path string, entry fs.DirEntry, err error) error {
        // Skip errors
        if err != nil {
            return nil
        }

        // Skip the root directory itself
        if path == "." {
            return nil
        }

        if entry.IsDir() {
            totalDirs++
        } else {
            // Check extension if filter is set
            fileExt := filepath.Ext(path)
            if desiredExt != "" && fileExt != desiredExt {
                return nil
            }
            
            // Get file info to access the size
            info, err := entry.Info()
            if err != nil {
                return err
            }
            
            totalFiles++
            totalBytes += info.Size()
        }
        
        return nil
    })

    // Print summary
    fmt.Println("Summary:")
    fmt.Println("  Total files:       ", totalFiles)
    fmt.Println("  Total directories: ", totalDirs)
    fmt.Println("  Total bytes:       ", totalBytes)
}

By using os.DirFS and fs.WalkDir, we've made our code more flexible. The fs.FS interface can be implemented by various file systems, including:

1. Physical filesystems (via os.DirFS)
2. In-memory filesystems
3. Embedded filesystems (using the embed package)
4. Network filesystems
5. Custom implementations

Key Benefits of the fs Package

The fs package provides significant advantages:

1. Abstraction: Decouples code from the underlying file system
2. Testability: Makes it easier to write tests using mock filesystems
3. Flexibility: Works with different filesystem implementations
4. Compatibility: Many standard library functions now accept fs.FS parameters

Summary

This lesson covered:

1. Using filepath.Walk to traverse a directory tree
2. Collecting statistics about files and directories
3. Filtering files by extension
4. Using the more flexible io/fs package and fs.WalkDir

These techniques are valuable for many CLI applications that need to interact with the filesystem, such as:

• Build tools
• File searching utilities
• Code analysis tools
• Backup software
• Static site generators

In the next lesson, we'll examine file locks, a topic introduced in the previous module.