duanche8554
duanche8554
2017-02-28 13:15
浏览 21
已采纳

Bigquery.go导出作业比WebGUI慢得多

I'm using the bigquery.go library.

While investigating some of the performance I have found that my export (.csv to GCS) job (and export job only) started from the client takes about 60seconds on average while the same job started from the WebGUI takes about 20seconds. What could be the reason for this?

The code is the following:

time1 := time.Now()
job_extract, err := extractor.Run(ctx)
if err != nil {
    return err
}

status, err = job_extract.Wait(ctx)
if err != nil {
    return err
}

if status.Err() != nil {
    log.Fatalf("Job failed with error %v", status.Err())
    return status.Err()
}

time2 := time.Since(time1)

图片转代码服务由CSDN问答提供 功能建议

我正在使用bigquery.go库。

正在调查一些 在性能方面,我发现从客户端启动的导出(.csv至GCS)作业(仅导出作业)平均大约需要60秒,而从WebGUI启动的同一作业大约需要20秒。

代码如下:

  time1:= time.Now()
job_extract,  err:= extractor.Run(ctx)
if err!= nil {
 return err 
} 
 
status,err = job_extract.Wait(ctx)
if err!= nil {
 return err 
}  
 
if status.Err()!= nil {
 log.Fatalf(“作业失败,错误%v”,status.Err())
返回status.Err()
} 
 
time2:  = time.Since(time1)
   
 
  • 点赞
  • 写回答
  • 关注问题
  • 收藏
  • 邀请回答

1条回答 默认 最新

  • donglang8008
    donglang8008 2017-02-28 14:01
    已采纳

    The WEB UI usually has a polling mechanism to check when a job has finished, so you might see longer times. Usually the files exported to GCS appear sooner than the job actually finishes in the WebUI.

    To make sure and see exact timing, please use the cli tool to obtain the most recent jobs:

    bq  ls -j -a --max_results=15
    

    running this would display a table with job ids and the respective timing.

                   jobId                Job Type    State      Start Time      Duration
     --------------------------------- ---------- --------- ----------------- ----------
      bquijob_1864e679_15a84d8878a      query      SUCCESS   28 Feb 07:11:06   0:00:04
      bquijob_770b512_15a84d8122c       query      FAILURE   28 Feb 07:10:35   0:00:00
      bquijob_de0df03_15a84d6a4fa       query      FAILURE   28 Feb 07:09:02   0:00:00
      bquijob_52c4f7d7_15a84d660e6      query      FAILURE   28 Feb 07:08:44   0:00:00
      bquijob_76a2c1be_15a84d5e769      query      FAILURE   28 Feb 07:08:13   0:00:00
      bquijob_7f51dde5_15a84d55afb      query      SUCCESS   28 Feb 07:07:41   0:00:08
      bquijob_34f25864_15a84d50503      query      SUCCESS   28 Feb 07:07:18   0:00:08
      job_Ca0cuRTAjY7MEHAs7vTJMxtVYTs   query      SUCCESS   28 Feb 07:00:47   0:00:09
      job_hHfmcdwyBsPsYF5dDvvOdR1Rmd0   load       SUCCESS   28 Feb 07:00:26   0:00:20
      job_mkiLf_mFHLKSplGJOtg-XDKzvv4   load       SUCCESS   28 Feb 02:52:50   0:00:02
      job_3RsPvttxWwv3SzVoOI9Cv_2yWtA   query      SUCCESS   27 Feb 21:18:40   0:00:08
      job_JLsqJO0NEIlKNac6jkDWbwneGMg   extract    SUCCESS   27 Feb 11:35:04   0:00:17
      job_KOS7vKX4aX0FNbK6dibE7cxzcQA   query      SUCCESS   27 Feb 11:33:44   0:00:37
      bquijob_44046bec_15a802f703a      query      SUCCESS   27 Feb 09:27:48   0:00:07
      job_2qQ6YSWeXaP2y2doONQJsIoga3c   query      SUCCESS   27 Feb 08:53:20   0:00:06
    

    You would be able this way to check the extract job DURATION. In case you validate indeed it's a problem, please post such a table back in your question as sooner/later a Google engineer will check it. But without proper details, we can just assume your measurement is wrong.

    点赞 评论

相关推荐