douyanpeng0748
2016-08-20 04:44
浏览 597
已采纳

从Golang中的io.Reader到io.Writer读取/复制一定数量的字节,或者如果超过一定字节数限制,则返回错误?

I have an io.Reader in Golang and I want to double-check that the size of its data is below a predetermined maximum before or while running io.Copy() to save it to disk using io.Writer.

Since the file data in io.Reader could theoretically be quite large, I want to minimize memory usage and processing here if avoidable.
I don't think there's a function that's like io.CopyLessThanOrEqualToThisManyBytesOrReturnError(), but I did notice that io.ReadFull() can do the opposite to return an error if not enough bytes are there to fill the provided buffer.

Does anyone have a solution to this?


EDIT:
To clarify, copying a fraction of the data is not OK. It either needs to fail if it's over the threshold, or work if it's under.

图片转代码服务由CSDN问答提供 功能建议

我在Golang中有一个 io.Reader ,我想仔细检查一下 在运行 io.Copy()以使用 io.Writer 将其保存到磁盘之前或之时,其数据大小低于预定的最大值。

由于 io.Reader 中的文件数据理论上可能很大,因此,如果可以避免的话,我想在这里将内存使用和处理减到最少。
我不认为有一个类似于 io.CopyLessThanOrEqualToThisManyBytesOrReturnError(),但是我确实注意到,如果没有足够的字节来填充提供的缓冲区, io.ReadFull()可以相反地返回错误。

有人对此有解决方案吗?


EDIT:
\ n为明确起见,复制一部分数据是不正确的。 如果它超过阈值,则需要失败,否则,则无法工作。

  • 写回答
  • 好问题 提建议
  • 关注问题
  • 收藏
  • 邀请回答

2条回答 默认 最新

  • dongtong0796 2016-08-20 05:28
    已采纳

    Since io.Reader interface not knows anything about size or length of underlying data, there is only one solution for this problem:

    You may use one buffer with maximum size nMax (predetermined maximum)+1 and in every call to Your CopyLessThanOrEqualToThisManyBytesOrReturnError function, inside this function read input and buffer it, and check for this buffer length, if it is less than or equal to nMax then do io.Write, otherwise return error:

    const nMax = 5 // your predetermined maximum
    
    func CopyLessThanOrEqualToThisManyBytesOrReturnError(r io.Reader, w io.Writer) error {
        var buf = make([]byte, nMax+1)
        nRead, e := io.ReadFull(r, buf)
        if nRead > 0 && nRead <= nMax {
            w.Write(buf[:nRead])
            return nil
        }
        if nRead > nMax {
            return fmt.Errorf("there is more data")
        }
        return e
    }
    

    Like this working sample code unsing string:

    package main
    
    import (
        "fmt"
        "io"
        "os"
        "strings"
    )
    
    const nMax = 5 // your predetermined maximum
    
    func CopyLessThanOrEqualToThisManyBytesOrReturnError(r io.Reader, w io.Writer) error {
        var buf = make([]byte, nMax+1)
        nRead, e := io.ReadFull(r, buf)
        if nRead > 0 && nRead <= nMax {
            w.Write(buf[:nRead])
            return nil
        }
        if nRead > nMax {
            return fmt.Errorf("there is more data")
        }
        return e
    }
    
    func main() {
        r := strings.NewReader("123456789")
        err := CopyLessThanOrEqualToThisManyBytesOrReturnError(r, os.Stdout)
        if err != nil {
            fmt.Println(err) // there is more data
        }
    
        r = strings.NewReader("123
    ")
        err = CopyLessThanOrEqualToThisManyBytesOrReturnError(r, os.Stdout) // 123
        if err != nil {
            fmt.Println(err)
        }
    
        r = strings.NewReader("")
        err = CopyLessThanOrEqualToThisManyBytesOrReturnError(r, os.Stdout)
        if err != nil {
            fmt.Println(err) // EOF
        }
    }
    

    output:

    there is more data
    123
    EOF
    

    Working sample code, using files:

    package main
    
    import (
        "fmt"
        "io"
        "os"
    )
    
    const nMax = 5 // your predetermined maximum
    
    func CopyLessThanOrEqualToThisManyBytesOrReturnError(r io.Reader, w io.Writer) error {
        var buf = make([]byte, nMax+1)
        nRead, e := io.ReadFull(r, buf)
        if nRead > 0 && nRead <= nMax {
            w.Write(buf[:nRead])
            return nil
        }
        if nRead > nMax {
            return fmt.Errorf("there is more data")
        }
        return e
    }
    
    func main() {
        r, err := os.Open("input.bin")
        if err != nil {
            panic(err)
        }
        defer r.Close()
    
        w, err := os.Create("output.bin")
        if err != nil {
            panic(err)
        }
        defer w.Close()
    
        err = CopyLessThanOrEqualToThisManyBytesOrReturnError(r, w)
        if err != nil {
            fmt.Println(err)
        }
        fmt.Println("Done.")
    }
    

    Working sample code, using []byte:

    package main
    
    import (
        "bytes"
        "fmt"
        "io"
    )
    
    const nMax = 5 // your predetermined maximum
    
    func CopyLessThanOrEqualToThisManyBytesOrReturnError(r io.Reader, w io.Writer) error {
        var buf = make([]byte, nMax+1)
        nRead, e := io.ReadFull(r, buf)
        if nRead > 0 && nRead <= nMax {
            w.Write(buf[:nRead])
            return nil
        }
        if nRead > nMax {
            return fmt.Errorf("there is more data")
        }
        return e
    }
    
    func main() {
        bs := []byte{1, 2, 3, 4, 5}
        r := bytes.NewReader(bs)
    
        w := &bytes.Buffer{}
    
        err := CopyLessThanOrEqualToThisManyBytesOrReturnError(r, w)
        if err != nil {
            fmt.Println(err)
        }
    
        fmt.Println("Done.")
        fmt.Println(w.Bytes())
    }
    

    output:

    Done.
    [1 2 3 4 5]
    
    已采纳该答案
    评论
    解决 无用
    打赏 举报
  • douyong4842 2016-08-20 05:06

    You can use io.CopyN: https://golang.org/pkg/io/#CopyN which will return a nil error if it successfully copies exactly N bytes, or else an io.EOF (or possibly another error) if less than N bytes.

    For example:

    func copyMax(dst io.Writer, src io.Reader, n int64) error
        _, err := io.CopyN(dst, src, n)
        if err != nil {
            // Don't care if there's less available
            return nil
        }
    
        nextByte := make([]byte, 1)
        nRead, _ := io.ReadFull(src, nextByte)
        if nRead > 0 {
            // There's too much data
            return errors.New("Too much data")
        }
    
        return nil
    }
    
    评论
    解决 无用
    打赏 举报

相关推荐 更多相似问题