Disclaimer: I'm not sure if the title is my actual problem, but it's the only reason that makes sense for what's below.
Scenario
I am uploading large videos to my Golang server. I have an 8GB file that takes roughly 15 minutes locally and works well. However, on my staging server (Google Cloud) this file takes a little over an hour to upload and this is where the problem hits.
Problem
My client is waiting on the file to finish uploading in order to call another API endpoint, but only on the staging server does it fail. So it seems like when my server is looking to write the response, then the connection won't allow it because the client shows a 502 error and a CORS error, which I know is Chrome's way of saying "We don't know, check your server".
Code
Inside my SrcHandler is this:
defer func(begin time.Time) {
pr, pw := io.Pipe()
defer pw.Close()
if err := r.ParseMultipartForm(32 << 20); err != nil {
http.Error(w, err.Error(), http.StatusBadRequest)
fmt.Printf("%v
", err)
return
}
srcFile, handler, err := r.FormFile("video")
if err != nil {
http.Error(w, "INVALID_FILE", http.StatusBadRequest)
return
}
defer srcFile.Close()
// Copy uploaded file to *PipeWriter
go func() {
defer srcFile.Close()
defer pw.Close()
io.Copy(pw, srcFile)
}()
// Prep fileName
var srcFileName = strings.Replace(handler.Filename, " ", "_", -1)
var extension = filepath.Ext(srcFileName)
srcFileName = srcFileName[0 : len(srcFileName)-len(extension)]
// Create temp file to write too
f, err := ioutil.TempFile("./tmp/pending", srcFileName+"_")
if err != nil {
log.Fatal(err)
}
defer f.Close()
// Copy *PipeReader to temp file
io.Copy(f, pr)
contentType, err := getFileContentType("./" + f.Name())
if err != nil {
fmt.Printf("Content Type Error: %v
", err)
return
}
fmt.Printf("ContentType = %v
", contentType)
if contentType != "video/x-flv" && contentType != "video/mp4" &&
contentType != "video/quicktime" && contentType != "video/x-msvideo" &&
contentType != "video/x-ms-wmv" && contentType != "video/webm" {
http.Error(w, "INVALID_FILE_TYPE", http.StatusBadRequest)
os.Remove(f.Name())
return
}
fmt.Printf("Upload path -> %v
", f.Name())
upload := UploaderResponse{srcFileName, "./" + f.Name()}
resp, err := json.Marshal(upload)
if err != nil {
fmt.Printf("Error using Marshal %v
", err)
}
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusOK)
status, err2 := w.Write(resp)
if err2 != nil {
fmt.Printf("Error when writing response to app %v
", err2)
} else {
fmt.Printf("Status from uploading: %v
", status)
}
_ = u.logger.Log("upload.Handler Duration ", time.Since(begin))
}(time.Now())
The code gets down to the last line, u.logger.Log() which actually shows output. So the video is uploaded correctly...only responding to the client fails. Does anyone have ideas?
Disclaimer 2: I just found out that w.Write returns an error, so I'm waiting on a file to finish uploading to see if I get an errors logged. Will respond with an update.
Update 1
I uploaded a new file and the w.Write() call works, returning a 91 integer. I'm not sure what the issue may be now. Could it be a time-out somewhere I don't know of. FYI, my server read and write timeouts are both 6 hours.
If it helps, here are the logs. It should be easy to follow the code with the outputs:
