Executing Commands in Go

Introduction

A common operation when it comes to building command line applications is to execute other commands or scripts that are available on your system. We actually saw a couple of examples of this in the last module, when we were adding in our end-to-end tests.

The first example is within the TestMain function, where we're building our Go application using the exec.Command function, passing in the command of go with the arguments of build, -o and then our binary name. In addition to this, we were also calling the command for each test, passing in our inputs and capturing the output in order to understand whether or not our application was working.

Use Cases for Executing Commands

Whilst end-to-end testing of a CLI application is one use for executing commands, there's actually many other reasons to do so in the context of a command line application:

• Make use of existing tooling to provide additional functionality without implementing it yourself
• Use tools like ffmpeg for video encoding
• Automate scripts for deployment
• Monitor system tools and resources

Real-World Example

For this course, I created my own CLI application based on the knowledge I learned. This application:

1. Transcodes videos into variable bitrates for streaming
2. Creates an ID for each video
3. Uploads them to a Cloudflare bucket
4. Stores the upload in the database

This was achieved in part by using the exec command:

// Example of calling ffprobe to get video framerate
ffprobeCmd := exec.Command("ffprobe", "-v", "error", "-select_streams", "v:0", 
                         "-show_entries", "stream=r_frame_rate", "-of", 
                         "default=noprint_wrappers=1:nokey=1", videoPath)

// Example of calling ffmpeg for transcoding
ffmpegCmd := exec.Command("ffmpeg", "-i", inputPath, 
                        "-c:v", "libx264", "-crf", "23", 
                        "-preset", "medium", "-c:a", "aac", "-b:a", "128k",
                        "-hls_time", "10", "-hls_list_size", "0",
                        "-f", "hls", outputPath)

All of this was achieved by executing other commands within my CLI application, which made it a lot easier to leverage existing functionality that was available on my system, specifically ffmpeg.

Using the os/exec Package

When it comes to executing commands in Go, there are a few different ways to do so. However, the simplest way by far is to use the os/exec package, which is the one we used inside of our end-to-end tests.

From the documentation:

Package exec runs external commands. It wraps the os.StartProcess function to make it easier to remap standard input and standard output, connect IO with pipes and do other adjustments.

When it comes to executing commands, this is the package you're going to want to use, as it provides a number of niceties on top of the StartProcess command already.

Basic Command Execution

Here's a simple example of executing the ls -la command:

package main

import (
	"bytes"
	"fmt"
	"log"
	"os/exec"
)

func main() {
	// Create command
	cmd := exec.Command("ls", "-la")
	
	// Create buffer for standard output
	var stdout bytes.Buffer
	cmd.Stdout = &stdout
	
	// Run the command
	err := cmd.Run()
	if err != nil {
		log.Fatalf("Failed to run command: %s", err)
	}
	
	// Print the output
	fmt.Printf("Command: %s\nOutput:\n%s", cmd.String(), stdout.String())
}

Running this produces very similar output to directly using ls -la in the terminal.

Simplified Output Capture with Output() Method

While using bytes.Buffer works, Go provides a simpler approach with the Output() method:

package main

import (
	"fmt"
	"log"
	"os/exec"
)

func main() {
	// Create command
	cmd := exec.Command("ls", "-la")
	
	// Run command and capture output
	stdout, err := cmd.Output()
	if err != nil {
		log.Fatalf("Failed to run command: %s", err)
	}
	
	// Print the output
	fmt.Printf("Command: %s\nOutput:\n%s", cmd.String(), string(stdout))
}

We can simplify further by directly calling Output():

stdout, err := exec.Command("ls", "-la").Output()
if err != nil {
	log.Fatalf("Failed to run command: %s", err)
}
fmt.Printf("Output:\n%s", string(stdout))

Improving End-to-End Tests

Let's apply this to our end-to-end tests:

// Before:
var stdout bytes.Buffer
command.Stdout = &stdout
err := command.Run()
if err != nil {
	return "", fmt.Errorf("failed to run command: %w", err)
}
output := stdout.String()

// After:
output, err := command.Output()
if err != nil {
	return "", fmt.Errorf("failed to run command: %w", err)
}
return string(output), nil

This is much simpler and does the exact same thing.

Capturing Standard Error

We can also capture standard error output. Let's see an example with the wc command:

package main

import (
	"fmt"
	"log"
	"os/exec"
)

func main() {
	// Create a file with some words
	exec.Command("bash", "-c", "echo \"one two three four five\" > words.txt").Run()
	
	// Create command that will generate an error
	cmd := exec.Command("wc", "words.txt", "no_exist.txt")
	
	// Run command and try to capture output
	stdout, err := cmd.Output()
	if err != nil {
		fmt.Println("Failed to run command:", err)
	}
	
	// Print the output
	fmt.Println(string(stdout))
}

When running this, we'll get the error message and partial output, but we're missing the standard error message about the missing file.

Using CombinedOutput() for Both stdout and stderr

To capture both standard output and standard error:

package main

import (
	"fmt"
	"os/exec"
)

func main() {
	// Create a file with some words
	exec.Command("bash", "-c", "echo \"one two three four five\" > words.txt").Run()
	
	// Create command that will generate an error
	cmd := exec.Command("wc", "words.txt", "no_exist.txt")
	
	// Run command and capture combined output
	output, err := cmd.CombinedOutput()
	if err != nil {
		fmt.Println("Failed to run command:", err)
	}
	
	// Print the combined output
	fmt.Println(string(output))
}

The CombinedOutput() method returns both standard output and standard error in a single byte slice.

Managing Separate stdout and stderr Streams

One drawback of CombinedOutput() is that it merges both stdout and stderr, so you lose the information about which is which. For more control, you can keep them separate:

package main

import (
	"bytes"
	"fmt"
	"os/exec"
)

func main() {
	// Create command
	cmd := exec.Command("wc", "words.txt", "no_exist.txt")
	
	// Create buffer for standard error
	var stderr bytes.Buffer
	cmd.Stderr = &stderr
	
	// Run command and capture stdout
	stdout, err := cmd.Output()
	
	// Print outputs
	fmt.Println("Standard Output:")
	fmt.Println(string(stdout))
	
	if err != nil {
		fmt.Println("\nStandard Error:")
		fmt.Println(stderr.String())
	}
}

This approach keeps stdout and stderr separate, allowing more control over how you handle each stream.

Conclusion

This lesson provides a basic overview of how to execute commands in Go using the os/exec package. In the next lessons, we'll explore:

1. How to pass input to commands
2. Managing long-running processes
3. Capturing output asynchronously/concurrently
4. Advanced process management for command line applications