Command Line Applications in Go

Locked video

Please purchase the course to watch this video.

Filesystem & Networking

Testing HTTP Requests

09:20

Testing code that interacts with HTTP services is crucial for ensuring reliability, yet traditional testing methods can be problematic due to dependencies on external networks which may not always be available. This lesson introduces effective strategies for testing HTTP requests without relying on third-party services, highlighting the use of the net/http/httptest package in Go. By creating a local, controlled test server, developers can validate that their requests are functioning correctly without the risk of flaky tests or rate limits. Key techniques discussed include using integration tests to send actual requests to a custom server, validating the response, and asserting the properties of HTTP requests. This approach empowers developers to conduct comprehensive tests that simulate real-world scenarios while maintaining full control over the testing environment.

No links available for this lesson.

When it comes to writing code that works with an HTTP service, at some point you're going to want to be able to test that your code works correctly. However, knowing how to write tests that interface with HTTP can somewhat be elusive, given the fact that you don't actually want your tests to be making network requests. Instead, you want to be controlling the environment that your test runs in, so that you can test all of the different strategies that come back, and you're not dependent on a third-party dependency, which could or couldn't break.

The Problem with External Dependencies

Here's a simple function that makes an HTTP GET request:

package main

import (
    "context"
    "fmt"
    "io"
    "log"
    "net/http"
    "os"
    "os/signal"
)

// getRequest makes a GET request to the specified URL and returns the response body
func getRequest(ctx context.Context, url string) ([]byte, error) {
    // Create a new HTTP request
    req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
    if err != nil {
        return nil, fmt.Errorf("failed to create request: %w", err)
    }
    
    // Send the request
    resp, err := http.DefaultClient.Do(req)
    if err != nil {
        return nil, fmt.Errorf("failed to send request: %w", err)
    }
    defer resp.Body.Close()
    
    // Read the response body
    body, err := io.ReadAll(resp.Body)
    if err != nil {
        return nil, fmt.Errorf("failed to read response body: %w", err)
    }
    
    return body, nil
}

func main() {
    // Create a context that cancels on interrupt signal
    ctx, cancel := signal.NotifyContext(context.Background(), os.Interrupt)
    defer cancel()
    
    // Make a request to httpbin.org
    data, err := getRequest(ctx, "https://httpbin.org/get")
    if err != nil {
        log.Fatal(err)
    }
    
    // Print the response
    fmt.Println(string(data))
}

If we wanted to test this function, we could write:

func TestGetRequest(t *testing.T) {
    ctx := context.Background()
    data, err := getRequest(ctx, "https://httpbin.org/get")
    if err != nil {
        t.Fatal("Failed to make request:", err)
    }
    
    // Now what? How do we verify the response?
}

But this approach has several problems:

It makes actual HTTP requests to an external service
The test will fail if there's no internet connection
We can't easily test error scenarios
We have limited control over the response

Using httptest for Better Testing

Go's standard library provides the net/http/httptest package, which allows us to create a local HTTP server for testing. This approach:

Doesn't require internet connectivity
Gives us complete control over the response
Allows us to inspect the requests we receive
Makes tests faster and more reliable

Let's rewrite our test using httptest:

func TestGetRequest(t *testing.T) {
    // Create a test server
    ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        // Write a response
        fmt.Fprintln(w, "Hello, world")
    }))
    defer ts.Close()
    
    // Make a request to the test server
    ctx := context.Background()
    data, err := getRequest(ctx, ts.URL)
    if err != nil {
        t.Fatal("Error should not exist:", err)
    }
    
    // Verify the response
    want := "Hello, world\n"
    if want != string(data) {
        t.Fatalf("Expected %q, got %q", want, string(data))
    }
}

This test:

Creates a local HTTP server that responds with "Hello, world"
Makes a request to that server using our getRequest function
Verifies that the response matches what we expect

Testing Request Properties

Beyond testing the response, we often want to verify that our client is making the correct request. The httptest server gives us access to the incoming request, allowing us to check:

HTTP method
Headers
URL parameters
Request body
etc.

Let's enhance our test to check these properties:

func TestGetRequest(t *testing.T) {
    // Create a test server that verifies request properties
    ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        // Check the HTTP method
        if r.Method != http.MethodGet {
            t.Logf("Expected method %s, got %s", http.MethodGet, r.Method)
            t.Fail()
        }
        
        // Check for a specific header
        userAgent := r.Header.Get("User-Agent")
        if !strings.Contains(userAgent, "Go") {
            t.Logf("Expected User-Agent to contain 'Go', got %q", userAgent)
            t.Fail()
        }
        
        // Write a response
        fmt.Fprintln(w, "Hello, world")
    }))
    defer ts.Close()
    
    // Make a request to the test server
    ctx := context.Background()
    data, err := getRequest(ctx, ts.URL)
    if err != nil {
        t.Fatal("Error should not exist:", err)
    }
    
    // Verify the response
    want := "Hello, world\n"
    if want != string(data) {
        t.Fatalf("Expected %q, got %q", want, string(data))
    }
}

Note: Use t.Log and t.Fail() instead of t.Fatal() in the handler function. This is because t.Fatal() will try to stop test execution immediately, which can cause problems when called from a different goroutine (like the server handler).

Testing Different HTTP Scenarios

One of the most powerful aspects of using httptest is the ability to test different HTTP scenarios:

Testing Success Responses

func TestGetRequestSuccess(t *testing.T) {
    ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        w.WriteHeader(http.StatusOK)
        fmt.Fprintln(w, `{"status": "success"}`)
    }))
    defer ts.Close()
    
    data, err := getRequest(context.Background(), ts.URL)
    if err != nil {
        t.Fatal(err)
    }
    
    if !strings.Contains(string(data), "success") {
        t.Fatalf("Expected success response, got: %s", data)
    }
}

Testing Error Responses

func TestGetRequestServerError(t *testing.T) {
    ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        w.WriteHeader(http.StatusInternalServerError)
        fmt.Fprintln(w, `{"error": "server error"}`)
    }))
    defer ts.Close()
    
    _, err := getRequest(context.Background(), ts.URL)
    if err == nil {
        t.Fatal("Expected error but got nil")
    }
}

Testing Timeouts

func TestGetRequestTimeout(t *testing.T) {
    ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        // Simulate a delay
        time.Sleep(100 * time.Millisecond)
        fmt.Fprintln(w, "Delayed response")
    }))
    defer ts.Close()
    
    // Create a context with a short timeout
    ctx, cancel := context.WithTimeout(context.Background(), 50*time.Millisecond)
    defer cancel()
    
    _, err := getRequest(ctx, ts.URL)
    if err == nil {
        t.Fatal("Expected timeout error but got nil")
    }
}

Designing for Testability

To make HTTP client code more testable, consider these design principles:

Dependency Injection: Accept the base URL as a parameter or field

type Client struct {
    BaseURL string
    HTTPClient *http.Client
}

func NewClient(baseURL string) *Client {
    return &Client{
        BaseURL: baseURL,
        HTTPClient: &http.Client{Timeout: 10 * time.Second},
    }
}

func (c *Client) GetData() ([]byte, error) {
    resp, err := c.HTTPClient.Get(c.BaseURL + "/data")
    // ...
}

By using the httptest package and applying good design principles, we can write reliable tests for our HTTP-related code that don't depend on external services, run quickly, work offline, and provide consistent results.

No homework tasks for this lesson.

How would you rate this lesson?

Full Course