Golang net / http上传大文件，未定义错误

Question

I'm having an error that is rather hard to debug when uploading large files to a server made using golang's default net/http package. The upload code looks like this:

    uploadForm.onsubmit = () => {
        const formData = new FormData(uploadForm);
        const isPublic : boolean = (<HTMLInputElement>document.getElementById('public_switch')).checked;
        formData.append('file', (<HTMLInputElement>document.getElementById('file')).files[0]);
        formData.append('compression', (<HTMLInputElement>document.getElementById('compression')).value);
        formData.append('public', String(isPublic));
        const xhr = new XMLHttpRequest();
        xhr.open("POST", "/upload/");
        xhr.send(formData);
        xhr.onreadystatechange = function() {
            console.log(xhr.responseText);
        }
    }

I have a server written in golang which I start as follows:

var server = &http.Server{
    Addr:         ":" + Configuration.Port,
    ReadTimeout:  300 * time.Second,
    WriteTimeout: 300 * time.Second,
    ReadHeaderTimeout: 300 * time.Second,
    MaxHeaderBytes: 500000000}

http.HandleFunc("/upload/", uploadFile)

server.ListenAndServe()

Finally I accept the and parse the file using the following code

func uploadFile(w http.ResponseWriter, r *http.Request) {
    //Parsing the upload arguments into the values we shall be working with
    r.ParseMultipartForm(5000000000000000)

    file, _, err := r.FormFile("file")
    //etc

Now, the code itself fails at 'r.FormFile("file")' with the very descriptive error message: "multipart: NextPart: EOF"

Is there some sort of setting on file limit or timeouts which I might not be setting either in the go code or in javascript ? The file I'm trying to upload is ~1.7GB so clearly fits within the limits supported by http.

Any idea how I could debug this issue a bit better without having to delve into FormFile or capture the request itself ? The code works just fine with smaller files ( a few Mb's).

Answer 1

OK, so, to answer my own question in case this stuff comes up again.

The problem is poorly javascript side and I'm afraid it has to do with the fact that the web apis for form&files aren't really up to scruff.

This is the solution I came up with javascript side:

const uploadFile = (form_id: string) => {
    const uploadForm: HTMLFormElement = <HTMLFormElement>document.getElementById(form_id);
    document.getElementById("submit_form").addEventListener("click", function(e) {
        e.preventDefault()

        let reader: FileReader = new FileReader();
        reader.readAsArrayBuffer((<HTMLInputElement>document.getElementById('file')).files[0]);

        reader.onload = function(evt) {

            const formData = new FormData(uploadForm);
            const isPublic: boolean = (<HTMLInputElement>document.getElementById('public_switch')).checked;
            formData.append('file', (<any>evt.target).result);
            formData.append('compression', (<HTMLInputElement>document.getElementById('compression')).value);
            formData.append('public', String(isPublic));
            const xhr = new XMLHttpRequest();
            xhr.open("POST", "/upload/");
            xhr.send(formData);
            xhr.onreadystatechange = function() {
                console.log(xhr.responseText + '  
 status is: ' + xhr.statusText);
            }

        };
    });

}

Quite simple really, but it took a long time to find since in the typical javascipt-community spirit all online examples are cluttered with shit.

The handling of large files should be done using a FileReader object which is basically called as such:

 let reader = new FileReader();
        reader.readAsArrayBuffer(dom_element_with_file_input.files[0]);

Then its used much like any other async web api:

    reader.onload = function(e) {
const your_file = e.target.result
    }

The variable 'your_file' is then used for whatever you want to do with the file. In my case it was appending to form data which is done like this:

        reader.onload = function(e) {
    const your_file = e.target.result
const formData = new FormData(dom_element_of_type_form);
formData.append('my_file', your_file);
formData.append('other_stuff', 'a string with metadata');
        }

I'm leaving this as an example since I could not actually find any so post with a clear example of the file Reader API, expect for ones cluttered in literally hundreds of lines of code. The example I followed almost to the T is the one here:

https://developer.mozilla.org/en-US/docs/Using_files_from_web_applications. But I used 'readAsArrayBuffer' instead for speed purposes.

One last note is that the reader API seems to be very slow and consumes a lot of ressource, which means that for arbitrary file sizes browser may simply crash, I had this problem with chromioum and I still can't fix it, so I'm open to suggestions, if I find a fix for browser crashes (or at least a way to handle them gracefully) I will update the question and mark it as the answer, in the meanwhile if anyone find a duplicate of my answer or a way to fix the browser crashes please tell me.