在io.ReadCloser中查找字符串而无需进行大量分配

Question

I have a large io.ReadCloser that I got from an http.Request in my HTTP handler func. I need to proxy the request to another server, but first I want to find a string in the body matching a regex like Title: (\w+). This is hard -- copying the whole body into a new buffer to operate on takes up way too much memory, and I've tried using regexp.FindReaderSubmatchIndex but it only gives me the index of the result, not the actual string.

What's the best way to do this? Tokenizers and JSON decoders and such seem to work on io streams, and this is a really simple use case for that. Can someone please point me in the right direction?

Answer 1

Here's my solution. I placed a pipe between the response body and its reader, and wrapped the reader with an io.TeeReader so it would write to the pipe as I read from it. I wrapped that in a bufio.Scanner and scanned lines. After I was done scanning lines, I was sure to consume the remainder of the body (with io.Copy(ioutil.Discard, body)) so that the rest of the body would be written to the pipe.

if request.Body == nil {
    proxy(request)
    return
}

// The body is *not* nil,
// so we're going to process it line-by-line.

bodySrc := request.Body             // Original io source of the request body.
pr, pw := io.Pipe()                 // Pipe between bodySrc and request.Body.
body := io.TeeReader(bodySrc, pw)   // When you read from body, it will read from bodySrc and writes to the pipe.
request.Body = ioutil.NopCloser(pr) // The other end of the pipe is request.Body. That's what proxy() will read.

go func() {

    scanner = bufio.NewScanner(body)
    for scanner.Scan() {
        x := scanner.Bytes()
        if processLine(x) {
            break
        }
    }

    // We're done with the body,
    // so consume the rest of it and close the source and the pipe.
    io.Copy(ioutil.Discard, body)
    bodySrc.Close()
    pw.Close()

}()

// As proxy reads request.Body, it's actually keeping up
// with the scanning done in the above goroutine.
proxy(request)