每个部分读取超过4096字节的字节。

Question

I'm trying to process a multipart file upload in small chunks to avoid storing the entire file in memory. The following function seems to solve this, however when passing a []byte as the destination for the part.Read() method, it reads the part in chunks of 4096 bytes instead of in chunks of the destination size (len([]byte)).

When opening a local file and Read()'ing it into a []byte of the same size, it uses the entire space available as expected. Thus I think it's something specific to the part.Reader(). However, I'm unable to find anything about a default or max size for that function.

For reference, the function is as follows:

func ReceiveFile(w http.ResponseWriter, r *http.Request) {
  reader, err := r.MultipartReader()
  if err != nil {
    panic(err)
  }
  if reader == nil {
    panic("Wrong media type")
  }
  buf := make([]byte, 16384)
  fmt.Println(len(buf))
  for {
    part, err := reader.NextPart()
    if err == io.EOF {
      break
    }
    if err != nil {
      panic(err)
    }
    var n int
    for {
      n, err = part.Read(buf)
      if err == io.EOF {
        break
      }
      if err != nil {
        panic(err)
      }
      fmt.Printf("Read %d bytes into buf
", n)
      fmt.Println(len(buf))
    }
    n, err = part.Read(buf)
    fmt.Printf("Finally read %d bytes into buf
", n)
    fmt.Println(len(buf))
  }

Answer 1

The part reader does not attempt to fill the caller's buffer as allowed by the io.Reader contract.

The best way to handle this depends on the requirements of the application.

If you want to slurp the part into memory, then use ioutil.ReadAll:

for {
    part, err := reader.NextPart()
    if err == io.EOF {
      break
    }
    if err != nil {
      // handle error
    }
    p, err := ioutil.ReadAll(part)
    if err != nil {
      // handle error
    }
    // p is []byte with the contents of the part
}

If you want to copy the part to the io.Writer w, then use io.Copy:

for {
    part, err := reader.NextPart()
    if err == io.EOF {
      break
    }
    if err != nil {
      // handle error
    }
    w := // open a writer
    _, err := io.Copy(w, part)
    if err != nil {
      // handle error
    }
}

If you want to process fixed size chunks, then use io.ReadFull:

 buf := make([]byte, chunkSize)
 for {
    part, err := reader.NextPart()
    if err == io.EOF {
      break
    }
    if err != nil {
      // handle error
    }
    _, err := io.ReadFull(part, buf)
    if err != nil {
      // handle error
      // Note that ReadFull returns an error if it cannot fill buf
    }
    // process the next chunk in buf
}

If the application data is structured in some other way than fix sized chunks, then bufio.Scanner might be of help.