继续保存获取扫描文件错误：http：无效在封闭的正文上读取

Question

When I read 1000 records file, I keep get error message every 10~20 records:

scan file error: http: invalid Read on closed Body

Here is my code

func parser(resp http.ResponseWriter, req *http.Request){

var count int

//....some of my code..

resp.Header().Set("Content-Type", "text/plain")
    scanner := bufio.NewScanner(req.Body)

    ctx := context.Background()
    for scanner.Scan() {
        itemID := scanner.Text()
        category := api.SearchAPI.FindCategory(itemID, lang, ctx)
        _, _ = fmt.Fprintf(resp, "%v,%v 
", itemID, category)
        count++
    }

    if err := scanner.Err(); err != nil {
        logger.Errorf("scan file error: %v", err)
        http.Error(resp, err.Error(), http.StatusBadRequest)
        return
    }

   //.....
}

Answer 1

Looks like your server is closing the connection. Check for any timeouts you have and why the request takes so long. You could process the scanner.Text() asynchronously so your scanning is not blocked for the searchAPI to respond, and the request body is not open for too long.

    resp.Header().Set("Content-Type", "text/plain")
    scanner := bufio.NewScanner(req.Body)

    ctx := context.Background()
    for scanner.Scan() {
        itemID := scanner.Text()
        go func(itemID string){
           category := api.SearchAPI.FindCategory(itemID, lang, ctx)
           _, _ = fmt.Fprintf(resp, "%v,%v 
", itemID, category)
           count++ //ENSURE YOU HAVE AN ATOMIC COUNTER INCREMENT, OR INCREMENT AFTER itemID IS READ
        }(itemID)
    }

    if err := scanner.Err(); err != nil {
        logger.Errorf("scan file error: %v", err)
        http.Error(resp, err.Error(), http.StatusBadRequest)
        return
    }

   //.....
}

Alternately you could simply collect all itemIDs in a slice, close the request body and then process them one by one.

resp.Header().Set("Content-Type", "text/plain")
    scanner := bufio.NewScanner(req.Body)

    ctx := context.Background()
    itemIDs := make([]string, 0)
    for scanner.Scan() {
        itemID := scanner.Text()
        itemIDs = append(itemIDs, itemID)
    }

    if err := scanner.Err(); err != nil {
        logger.Errorf("scan file error: %v", err)
        http.Error(resp, err.Error(), http.StatusBadRequest)
        return
    }

    for _, itemID := range itemIDs {
        category := api.SearchAPI.FindCategory(itemID, lang, ctx)
        _, _ = fmt.Fprintf(resp, "%v,%v 
", itemID, category)
        count++
    }

   //.....
}