Command Line Applications in Go

Locked video

Please purchase the course to watch this video.

Powerful Command Line Applications

Embedding Files

The embed package in Go, introduced in version 1.16, allows developers to easily include external files—like configuration files, images, or databases—into their CLI applications’ binaries. This dramatically simplifies the distribution process, eliminating the need for hardcoded data strings or online downloads, thus enhancing data integrity and ease of use. By utilizing the embed directive, developers can seamlessly embed files into their binary at compile time, ensuring that necessary resources are bundled with the application, even if the original files are removed. Furthermore, the embed package supports not just individual files but entire directories through its fs type, which integrates smoothly with Go's file handling methods. This capability is particularly beneficial for providing configuration setups or static assets in applications, offering a significant advantage in context like web servers and CLI tools. The lesson encourages practical engagement with the embed package to enhance understanding of its functionality within application development.

Sometimes, when distributing your CLI application to other users, you may wish to include an external file with your application's binary, such as:

Default configuration files
Images or static assets
SQLite database files
Trained AI models
Template files
Documentation or help text

In the past, this used to be quite difficult to achieve with Go, requiring you to either:

Hardcode the data into string variables (tedious and error-prone)
Download the data from the internet when your application starts up (requires network connectivity)

Since Go 1.16, however, we now have the ability to embed files into our application binary through the use of the embed package, which provides access to files embedded in the running Go program.

The Old Way: Hardcoding Data

Before the embed package, you might have had to do something like this:

package main

import "fmt"

// Hardcoded data - not practical for large files!
var data = `
name: "My Application"
version: "1.0.0"
database:
  host: "localhost"
  port: 5432
settings:
  debug: false
  timeout: 30s
`

func main() {
    fmt.Println("Configuration:")
    fmt.Println(data)
}

Problems with Hardcoding:

Tedious for large files - Imagine copying megabytes of data
Binary data is impossible - Can't hardcode images, SQLite databases, etc.
Error-prone - Easy to miss bytes or introduce typos
Maintenance nightmare - Hard to update embedded content

Project Setup

Let's start with a simple project structure:

my-cli-app/
├── main.go
└── data.yaml

data.yaml

yaml

name: "My Application"
version: "1.0.0"
database:
  host: "localhost"
  port: 5432
settings:
  debug: false
  timeout: "30s"

Using the Embed Package

Step 1: Import the Embed Package

package main

import (
    _ "embed"  // Note: blank import
    "fmt"
)

Important: We use a blank import (_) because we're not directly referencing the embed package in our code - it's used via the //go:embed directive.

Step 2: Embed a Single File

package main

import (
    _ "embed"
    "fmt"
)

//go:embed data.yaml
var data string

func main() {
    fmt.Println("Embedded configuration:")
    fmt.Println(data)
}

How It Works:

//go:embed data.yaml - This is a directive that tells the compiler to embed the file
Directives are like preprocessor macros - they instruct the toolchain to perform actions
The directive applies to the next package variable (in this case, data)
The file contents are embedded as a string in this example

Testing the Embedding

bash

# Run the application
go run main.go
# Output: Shows the contents of data.yaml

# Build the application  
go build -o myapp

Now let's test that the file is truly embedded:

bash

# Remove the original file
rm data.yaml

# The binary still works!
./myapp
# Output: Still shows the embedded data!

# Move the binary to a different location
mv myapp /tmp/
cd /tmp/
./myapp
# Output: Still works because data is embedded at compile time!

Important Restrictions

1. Files Must Exist at Compile Time

bash

# This will fail if data.yaml doesn't exist
rm data.yaml
go build
# Error: pattern data.yaml: no matching files found

2. Only Files in Current Directory or Below

//go:embed ../data.yaml  // ❌ Invalid - can't go up directories
//go:embed /etc/passwd    // ❌ Invalid - absolute paths not allowed
//go:embed data.yaml      // ✅ Valid - current directory
//go:embed configs/app.yaml // ✅ Valid - subdirectory

The embedding is relative to where the Go file is located, and you can only embed files within that directory or its subdirectories.

Embedding as Byte Slices

You can also embed files as byte slices, which is useful for binary data:

package main

import (
    _ "embed"
    "fmt"
)

//go:embed data.yaml
var data []byte

func main() {
    fmt.Printf("Data as bytes: %v\n", data)
    fmt.Printf("Data as string: %s\n", string(data))
    fmt.Printf("File size: %d bytes\n", len(data))
}

This is particularly useful for:

Binary files (images, databases, executables)
When you need the raw bytes for processing
Calculating file sizes or checksums

Embedding Directories with embed.FS

For more complex scenarios, you can embed entire directories using the embed.FS type:

Project Structure:

my-cli-app/
├── main.go
└── data/
    ├── config.yaml
    ├── schema.sql
    └── templates/
        ├── help.txt
        └── welcome.txt

Embedding a Directory:

package main

import (
    "embed"
    "fmt"
    "io/fs"
)

//go:embed data
var dataFS embed.FS

func main() {
    // List contents of the embedded directory
    entries, err := dataFS.ReadDir("data")
    if err != nil {
        panic(err)
    }
    
    fmt.Println("Embedded files and directories:")
    for _, entry := range entries {
        fmt.Printf("- %s (dir: %t)\n", entry.Name(), entry.IsDir())
    }
    
    // Read a specific file
    configData, err := dataFS.ReadFile("data/config.yaml")
    if err != nil {
        panic(err)
    }
    
    fmt.Println("\nconfig.yaml contents:")
    fmt.Println(string(configData))
}

Key Features of embed.FS:

Implements fs.FS interface - Compatible with all io/fs functions
Read-only file system - You can't write to embedded files
Preserves directory structure - Maintains the original file organization
Works with fs.WalkDir - Can traverse the entire embedded tree

Advanced: Walking Through Embedded File Systems

You can use fs.WalkDir to traverse the entire embedded file system:

package main

import (
    "embed"
    "fmt"
    "io/fs"
    "path/filepath"
)

//go:embed data
var dataFS embed.FS

func main() {
    fmt.Println("Walking through embedded file system:")
    
    err := fs.WalkDir(dataFS, "data", func(path string, d fs.DirEntry, err error) error {
        if err != nil {
            return err
        }
        
        // Get relative path from root
        relPath := filepath.ToSlash(path)
        
        if d.IsDir() {
            fmt.Printf("📁 %s/\n", relPath)
        } else {
            // Get file info
            info, err := d.Info()
            if err != nil {
                return err
            }
            fmt.Printf("📄 %s (%d bytes)\n", relPath, info.Size())
        }
        
        return nil
    })
    
    if err != nil {
        panic(err)
    }
}

Practical Use Cases for CLI Applications

1. Default Configuration Files

package main

import (
    _ "embed"
    "fmt"
    "os"
    "gopkg.in/yaml.v3"
)

//go:embed default-config.yaml
var defaultConfigData string

type Config struct {
    Database struct {
        Host string `yaml:"host"`
        Port int    `yaml:"port"`
    } `yaml:"database"`
    App struct {
        Name    string `yaml:"name"`
        Version string `yaml:"version"`
    } `yaml:"app"`
}

func loadConfig(filename string) (*Config, error) {
    var config Config
    
    // Try to load user config first
    if _, err := os.Stat(filename); os.IsNotExist(err) {
        // File doesn't exist, use embedded default
        fmt.Println("Config file not found, using default configuration")
        return parseConfig(defaultConfigData)
    }
    
    // Load user config
    data, err := os.ReadFile(filename)
    if err != nil {
        return nil, err
    }
    
    return parseConfig(string(data))
}

func parseConfig(data string) (*Config, error) {
    var config Config
    err := yaml.Unmarshal([]byte(data), &config)
    return &config, err
}

2. Help Text and Templates

//go:embed templates
var templates embed.FS

func showHelp(command string) {
    helpFile := fmt.Sprintf("templates/help/%s.txt", command)
    helpText, err := templates.ReadFile(helpFile)
    if err != nil {
        fmt.Printf("No help available for command: %s\n", command)
        return
    }
    fmt.Println(string(helpText))
}

3. SQLite Schema Files

//go:embed schema.sql
var schemaSQL string

func initializeDatabase(db *sql.DB) error {
    _, err := db.Exec(schemaSQL)
    return err
}

Real-World Example: CLI Application with Embedded Assets

Here's a complete example of a CLI app that uses embedded files:

package main

import (
    "embed"
    "flag"
    "fmt"
    "io/fs"
    "os"
    "path/filepath"
    "strings"
)

//go:embed assets
var assetsFS embed.FS

func main() {
    var command = flag.String("cmd", "help", "Command to run")
    flag.Parse()
    
    switch *command {
    case "help":
        showHelp()
    case "init":
        initProject()
    case "list":
        listAssets()
    default:
        fmt.Printf("Unknown command: %s\n", *command)
        showHelp()
    }
}

func showHelp() {
    helpText, err := assetsFS.ReadFile("assets/help.txt")
    if err != nil {
        fmt.Println("Help file not found")
        return
    }
    fmt.Println(string(helpText))
}

func initProject() {
    configTemplate, err := assetsFS.ReadFile("assets/templates/config.yaml")
    if err != nil {
        fmt.Println("Config template not found")
        return
    }
    
    err = os.WriteFile("config.yaml", configTemplate, 0644)
    if err != nil {
        fmt.Printf("Error creating config file: %v\n", err)
        return
    }
    
    fmt.Println("Project initialized with default config.yaml")
}

func listAssets() {
    fmt.Println("Embedded assets:")
    fs.WalkDir(assetsFS, "assets", func(path string, d fs.DirEntry, err error) error {
        if err != nil {
            return err
        }
        
        if !d.IsDir() {
            // Remove 'assets/' prefix for cleaner output
            cleanPath := strings.TrimPrefix(path, "assets/")
            fmt.Printf("  %s\n", cleanPath)
        }
        return nil
    })
}

Benefits of File Embedding

Single Binary Distribution - No need to distribute separate files
No Runtime Dependencies - Files are always available
Performance - No disk I/O for accessing embedded files
Security - Files can't be modified after compilation
Simplicity - Users don't need to manage multiple files

Best Practices

Keep embedded files small - Large files increase binary size
Use compression - Consider compressing data before embedding for large files
Separate user data from embedded data - Let users override embedded defaults
Version your embedded assets - Include version info in embedded files
Document embedded files - Make it clear what files are embedded in your application

Homework Assignment

Download the file system provided in the lesson resources and complete the following tasks:

Task Requirements:

Download the provided file system (link in lesson description)
Use the embed package to load the entire file system into your application
Use fs.WalkDir to walk through the embedded file system
List all directories and files with their sizes and types

Expected Output Format:

Walking embedded file system:
📁 data/
📁 data/configs/
📄 data/configs/app.yaml (245 bytes)
📄 data/configs/database.yaml (156 bytes)
📁 data/templates/
📄 data/templates/welcome.txt (89 bytes)
📄 data/schema.sql (1,234 bytes)

Bonus Challenges:

Add file extension filtering - Only show certain file types
Calculate total size - Show the total size of all embedded files
Search functionality - Find files matching a pattern
Extract functionality - Allow extracting embedded files to disk

This homework will give you a good understanding of how to use the embed package effectively, as well as making use of it with other Go standard library functions.

Once you're complete, that covers how you can use the embed package within your own code, and we'll move on to the next lesson where we're going to start taking a look at how we can actually build cross-platform code to support multiple operating systems.

Embedded File System Walker

Completed

Pending

Download the file system provided in the lesson resources and implement a complete embedded file system walker.

Requirements:

Download the provided file system (link in lesson description)
Use the embed package to load the entire file system into your application
Use fs.WalkDir to walk through the embedded file system
List all directories and files with their sizes and types

Expected Output Format:

Walking embedded file system:
📁 data/
📁 data/configs/
📄 data/configs/app.yaml (245 bytes)
📄 data/configs/database.yaml (156 bytes)
📁 data/templates/
📄 data/templates/welcome.txt (89 bytes)
📄 data/schema.sql (1,234 bytes)

Bonus Challenges:

Add file extension filtering - Only show certain file types
Calculate total size - Show the total size of all embedded files
Search functionality - Find files matching a pattern
Extract functionality - Allow extracting embedded files to disk

How would you rate this lesson?

Full Course

The Old Way: Hardcoding Data

Problems with Hardcoding:

Project Setup

data.yaml

Using the Embed Package

Step 1: Import the Embed Package

Step 2: Embed a Single File

How It Works:

Testing the Embedding

Important Restrictions

1. Files Must Exist at Compile Time

2. Only Files in Current Directory or Below

Embedding as Byte Slices

Embedding Directories with embed.FS

Project Structure:

Embedding a Directory:

Key Features of embed.FS:

Advanced: Walking Through Embedded File Systems

Practical Use Cases for CLI Applications

1. Default Configuration Files

2. Help Text and Templates

3. SQLite Schema Files

Real-World Example: CLI Application with Embedded Assets

Benefits of File Embedding

Best Practices

Homework Assignment

Task Requirements:

Expected Output Format:

Bonus Challenges:

Embedded File System Walker

Get Support