dongzhui2636
2014-03-03 23:20
浏览 344

使用mgo将上载的文件存储在MongoDB GridFS中,而不保存到内存

noob Golang and Sinatra person here. I have hacked a Sinatra app to accept an uploaded file posted from an HTML form and save it to a hosted MongoDB database via GridFS. This seems to work fine. I am writing the same app in Golang using the mgo driver.

Functionally it works fine. However in my Golang code, I read the file into memory and then write the file from memory to the MongoDB using mgo. This appears much slower than my equivalent Sinatra app. I get the sense that the interaction between Rack and Sinatra does not execute this "middle" or "interim" step.

Here's a snippet of my Go code:

func uploadfilePageHandler(w http.ResponseWriter, req *http.Request) {
  // Capture multipart form file information
  file, handler, err := req.FormFile("filename")
  if err != nil {
    fmt.Println(err)
  }

  // Read the file into memory
  data, err := ioutil.ReadAll(file)
  // ... check err value for nil

  // Specify the Mongodb database
  my_db := mongo_session.DB("... database name...")

  // Create the file in the Mongodb Gridfs instance
  my_file, err := my_db.GridFS("fs").Create(unique_filename)
  // ... check err value for nil

  // Write the file to the Mongodb Gridfs instance
  n, err := my_file.Write(data)
  // ... check err value for nil

  // Close the file
  err = my_file.Close()
  // ... check err value for nil

  // Write a log type message
  fmt.Printf("%d bytes written to the Mongodb instance
", n)

  // ... other statements redirecting to rest of user flow...
}

Question:

  • Is this "interim" step needed (data, err := ioutil.ReadAll(file))?
  • If so, can I execute this step more efficiently?
  • Are there other accepted practices or approaches I should be considering?

Thanks...

图片转代码服务由CSDN问答提供 功能建议

noob Golang和Sinatra人在这里。 我入侵了Sinatra应用程序,以接受从HTML表单发布的上传文件,并通过GridFS将其保存到托管的MongoDB数据库中。 这似乎很好。 我正在使用mgo驱动程序在Golang中编写相同的应用程序。

功能正常。 但是在我的Golang代码中,我将文件读入内存,然后使用mgo将文件从内存写入MongoDB。 这似乎比我同等的Sinatra应用要慢得多。 我感觉到Rack和Sinatra之间的交互不会执行此“中间”或“临时”步骤。

以下是我的Go代码的片段: \ n

  func uploadfilePageHandler(w http.ResponseWriter,req * http.Request){
 //捕获多部分表单文件信息
 file,handler,err:=  req.FormFile(“ filename”)
 if err!= nil {
 fmt.Println(err)
} 
 
 //将文件读入内存
数据,err:= ioutil.ReadAll(file  )
 // ...检查nil的err值
 
 //指定Mongodb数据库
 my_db:= mongo_session.DB(“ ...数据库名称...”)
 
 //创建 Mongodb Gridfs实例中的文件
 my_file,err:= my_db.GridFS(“ fs”)。Create(unique_filename)
 // ...检查err值是否为nil 
 
 //将文件写入 Mongodb Gridfs实例
n,err:= my_file.Write(data)
 // ...检查err值是否为nil 
 
 //关闭文件
 err = my_file.Close()
 //。  ..检查err值是否为nil 
 
 //写一个日志类型mess  age 
 fmt.Printf(“%d个字节写入Mongodb实例
”,n)
 
 // ...其他重定向到其他用户流的语句... 
} 
   
 
 

问题

  • 是否需要此“临时”步骤(数据, 错误:= ioutil.ReadAll(file))?
  • 如果可以,我是否可以更有效地执行此步骤?
  • 我是否应该考虑其他可接受的做法或方法?

    谢谢...

  • 写回答
  • 好问题 提建议
  • 关注问题
  • 收藏
  • 邀请回答

2条回答 默认 最新

  • dongxuan1660 2014-03-05 14:28
    已采纳

    No, you should not read the file entirely in memory at once, as that will break when the file is too large. The second example in the documentation for GridFS.Create avoids this problem:

    file, err := db.GridFS("fs").Create("myfile.txt")
    check(err)
    messages, err := os.Open("/var/log/messages")
    check(err)
    defer messages.Close()
    err = io.Copy(file, messages)
    check(err)
    err = file.Close()
    check(err)
    

    As for why it's slower than something else, hard to tell without diving into the details of the two approaches used.

    已采纳该答案
    评论
    解决 无用
    打赏 举报
  • drrvnbwle80177811 2015-11-28 20:43

    Once you have the file from multipartForm, it can be saved into GridFs using below function. I tested this against huge files as well ( upto 570MB).

    //....code inside the handlerfunc
    for _, fileHeaders := range r.MultipartForm.File {
            for _, fileHeader := range fileHeaders {
                file, _ := fileHeader.Open()            
                if gridFile, err := db.GridFS("fs").Create(fileHeader.Filename); err != nil {
                    //errorResponse(w, err, http.StatusInternalServerError)
                    return
                } else {
                    gridFile.SetMeta(fileMetadata)
                    gridFile.SetName(fileHeader.Filename)
                    if err := writeToGridFile(file, gridFile); err != nil {
                        //errorResponse(w, err, http.StatusInternalServerError)
                        return
                    }
    
    func writeToGridFile(file multipart.File, gridFile *mgo.GridFile) error {
        reader := bufio.NewReader(file)
        defer func() { file.Close() }()
        // make a buffer to keep chunks that are read
        buf := make([]byte, 1024)
        for {
            // read a chunk
            n, err := reader.Read(buf)
            if err != nil && err != io.EOF {
                return errors.New("Could not read the input file")
            }
            if n == 0 {
                break
            }
            // write a chunk
            if _, err := gridFile.Write(buf[:n]); err != nil {
                return errors.New("Could not write to GridFs for "+ gridFile.Name())
            }
        }
        gridFile.Close()
        return nil
    }
    
    评论
    解决 无用
    打赏 举报

相关推荐 更多相似问题