base64解码器（io.Reader实现）的不良行为

Question

I have tried, within a for loop, to re-declare/assign a base64 decoder and used the os.Seek function to go back to the beginning of the file at the end of the loop before this, in order for the called function (in this test case PrintBytes) to be able to process the file from beginning to end time and time again throughout the for loop.

Here is my (I'm sure terribly un-idiomatic) code, which fails to read the 2nd byte into the []byte of length 2 and capacity 2 during the second iteration of the main for loop in main():

package main

import (
    "encoding/base64"
    "io"
    "log"
    "net/http"
    "os"
)

var (
    remote_file string = "http://cryptopals.com/static/challenge-data/6.txt"
    local_file  string = "secrets_01_06.txt"
)

func main() {
    f, err := os.Open(local_file)
    if err != nil {
        DownloadFile(local_file, remote_file)
        f, err = os.Open(local_file)
        if err != nil {
            log.Fatal(err)
        }
    }
    defer f.Close()

    for blocksize := 1; blocksize <= 5; blocksize++ {
        decoder := base64.NewDecoder(base64.StdEncoding, f)
        PrintBytes(decoder, blocksize)
        _, err := f.Seek(0, 0)
        if err != nil {
            log.Fatal(err)
        }
    }
}

func PrintBytes(reader io.Reader, blocksize int) {
    block := make([]byte, blocksize)
    for {
        n, err := reader.Read(block)
        if err != nil && err != io.EOF {
            log.Fatal(err)
        }
        if n != blocksize {
            log.Printf("n=%d\tblocksize=%d\tbreaking...", n, blocksize)
            break
        }
        log.Printf("%x\tblocksize=%d", block, blocksize)
    }
}

func DownloadFile(local string, url string) {
    f, err := os.Create(local)
    if err != nil {
        log.Fatal(err)
    }
    defer f.Close()

    resp, err := http.Get(url)
    if err != nil {
        log.Fatal(err)
    }
    defer resp.Body.Close()

    _, err = io.Copy(f, resp.Body)
    if err != nil {
        log.Fatal(err)
    }
}

The output from this code can be viewed here https://gist.github.com/tomatopeel/b8e2f04179c7613e2a8c8973a72ec085

It is this behaviour that I don't understand: https://gist.github.com/tomatopeel/b8e2f04179c7613e2a8c8973a72ec085#file-bad_reader_log-L5758

I was expecting it to simply read the file 2 bytes at a time into the 2-byte slice, from beginning to end. For what reason does it only read 1 byte here?

Answer 1

It is not the problem of encoding/base64. When using io.Reader, it's not guaranteed that number of bytes read exactly equal to the buffer size (i.e. blocksize in your example code). The documentation states:

Read reads up to len(p) bytes into p. It returns the number of bytes read (0 <= n <= len(p)) and any error encountered. Even if Read returns n < len(p), it may use all of p as scratch space during the call. If some data is available but not len(p) bytes, Read conventionally returns what is available instead of waiting for more.

In your example, change PrintBytes to

func PrintBytes(reader io.Reader, blocksize int) {
    block := make([]byte, blocksize)
    for {
        n, err := reader.Read(block)
        //Process the data if n > 0, even when err != nil
        if n > 0 {
            log.Printf("%x\tblocksize=%d", block[:n], blocksize)
        }

        //Check for error
        if err != nil {
            if err != io.EOF {
                log.Fatal(err)
            } else if err == io.EOF {
                break
            }
        } else if n == 0 {
            //Considered as nothing happened
            log.Printf("WARNING: read return 0,nil")
        }
    }
}

Update:

Correct usage of io.Reader, modify code to always process the data if n > 0 even when error occurs.