About

This post shows how to stream an HTTP response body in Go using the standard library.

Background

Go’s standard library includes net/http, which gives you both an HTTP client and server. This post focuses on streaming a response body in chunks instead of reading everything into memory at once. That is useful for large downloads (where you want a progress bar) or when you want to process data as it arrives, and it is a common, idiomatic pattern for stream-like interfaces in Go.

Experiment

Overview

Our experiment implements a function that reads the response body in chunks and prints how many bytes have been downloaded so far. http.Get returns an http.Response. Its Body field is an io.ReadCloser we can read from and close when we are done.

By default, the body is streamed. You can pull it all at once with io.ReadAll, or read smaller chunks using Read. The response also has a ContentLength field with the total size in bytes, but it is sometimes -1 when the server does not send it. If you want a progress bar, you will need it, but this experiment does not rely on it:

package main

import (
  "fmt"
  "io"
  "net/http"
)

func main() {
  res, err := http.Get("https://www.openbsd.org/images/puffy78.gif")
  if err != nil {
    panic(err)
  }
  _, err = download(res.Body)
  if err != nil {
    panic(err)
  }
}

func download(body io.ReadCloser) ([]byte, error) {
  defer body.Close()
  chunk := make([]byte, 2048)
  buffer := make([]byte, 0, 2048*100)
  read := 0
  for {
    n, err := body.Read(chunk)
    if err == io.EOF {
      break
    } else if err != nil {
      return buffer, err
    } else {
      buffer = append(buffer, chunk[:n]...)
      read += n
    }
    fmt.Printf("\033[0K\rread %d bytes", read)
  }
  fmt.Println()
  return buffer, nil
}

Explanation

• res, err := http.Get("https://www.openbsd.org/images/puffy78.gif")
  Sends a GET request and returns a response.

• _, err = download(res.Body)
  Reads the body in chunks and prints progress.

• defer body.Close()
  Closes the body when we are done.

• chunk := make([]byte, 2048)
  Allocates a 2KB buffer for each read.

• buffer := make([]byte, 0, 2048*100)
  Preallocates space for the downloaded data, growing if needed.

• n, err := body.Read(chunk)
  Reads up to 2KB and returns how many bytes were actually read.

• buffer = append(buffer, chunk[:n]...)
  Appends only the bytes we just read.

• read += n
  Keeps a running total.

• fmt.Printf("\033[0K\rread %d bytes", read)
  Updates the same terminal line with progress.

• fmt.Println()
  Adds a final newline after the loop.

• if err == io.EOF { break }
  Stops when we hit the end of the stream.

Conclusion

Play demo