About

This post documents how to stream a response body when making HTTP requests with the Go programming language and its standard library.

Background

The Go programming language provides a rich standard library that includes the net/http package for the implementation of HTTP clients and servers.

This post focuses on how to make HTTP requests with the net/http package, and in particular it focuses on how to stream a response body in chunks rather than reading the entire body into memory at once.

This can be useful in a number of scenarios including when a client wants to implement a progess bar for a large download, or when a client wants to eagerly process data as it arrives rather than wait to download the entire response body into memory at once.

It is also a pattern you could borrow if you find yourself implementing an interface that implements a stream of some sort and it is a pattern that is both common and idiomatic in Go.

Experiment

Context

Our experiment will implement a function that downloads the response body in chunks, and also prints the number of bytes downloaded so far at each iteration.

The http.Get method returns a http.Response object that represents a HTTP response, and among its fields are a Body field that returns a object that implements the io.ReadCloser interface.

When a request is made, the response body is not read into memory and instead it can be read into memory all at once using io.ReadAll or it can be read in smaller chunks using the Read method defined by the io.ReadCloser interface.

The response object also provides a ContentLength field that holds the total size of the response body in bytes but sometimes it might not be set, and in that case, its value will be -1. If you plan to implement a progress bar, this field is essential but in this experiment we won't need to use it:

package main

import (
  "fmt"
  "io"
  "net/http"
)

func main() {
  res, err := http.Get("https://www.openbsd.org/images/puffy78.gif")
  if err != nil {
    panic(err)
  }
  _, err = download(res.Body)
  if err != nil {
    panic(err)
  }
}

func download(body io.ReadCloser) ([]byte, error) {
  defer body.Close()
  chunk := make([]byte, 2048)
  buffer := make([]byte, 0, 2048*100)
  read := 0
  for {
    n, err := body.Read(chunk)
    if err == io.EOF {
      break
    } else if err != nil {
      return buffer, err
    } else {
      buffer = append(buffer, chunk[:n]...)
      read += n
    }
    fmt.Printf("\033[0K\rread %d bytes", read)
  }
  fmt.Println()
  return buffer, nil
}

Explanation

• res, err := http.Get("https://www.openbsd.org/images/puffy78.gif")
  This line sends a GET request to the specified URL and returns a response object.

• _, err = download(res.Body)
  This line calls the download function to read the response body in chunks and display progress.

• defer body.Close()
  This line ensures the response body is closed after reading is complete.

• chunk := make([]byte, 2048)
  This line creates a buffer to hold each chunk of data read from the response body.

• buffer := make([]byte, 0, 2048*100)
  This line initializes a zero-length buffer with an initial capacity of 200KB, and it will accumulate the downloaded data. It will be expanded if/when the buffer overflows.

• n, err := body.Read(chunk)
  This line reads up to 2048 bytes from the response body into the chunk buffer. The n variable returns the number of bytes that were read (might be less than 2048).

• buffer = append(buffer, chunk[:n]...)
  This line appends the newly read chunk to the buffer, growing it as needed, and appending only the valid portion of the chunk (up to n bytes).

• read += n
  This line increments the total number of bytes read so far.

• fmt.Printf("\033[0K\rread %d bytes", read)
  This line prints the current download progress, updating the same line in the terminal.

• fmt.Println()
  This line prints a newline after the download is complete for better output formatting.

• if err == io.EOF { break }
  This line checks for the end of the response body and exits the loop when all data has been read.

Conclusion

Play demo