Bigquery.go导出作业比WebGUI慢得多

Question

I'm using the bigquery.go library.

While investigating some of the performance I have found that my export (.csv to GCS) job (and export job only) started from the client takes about 60seconds on average while the same job started from the WebGUI takes about 20seconds. What could be the reason for this?

The code is the following:

time1 := time.Now()
job_extract, err := extractor.Run(ctx)
if err != nil {
    return err
}

status, err = job_extract.Wait(ctx)
if err != nil {
    return err
}

if status.Err() != nil {
    log.Fatalf("Job failed with error %v", status.Err())
    return status.Err()
}

time2 := time.Since(time1)

Answer 1

The WEB UI usually has a polling mechanism to check when a job has finished, so you might see longer times. Usually the files exported to GCS appear sooner than the job actually finishes in the WebUI.

To make sure and see exact timing, please use the cli tool to obtain the most recent jobs:

bq  ls -j -a --max_results=15

running this would display a table with job ids and the respective timing.

               jobId                Job Type    State      Start Time      Duration
 --------------------------------- ---------- --------- ----------------- ----------
  bquijob_1864e679_15a84d8878a      query      SUCCESS   28 Feb 07:11:06   0:00:04
  bquijob_770b512_15a84d8122c       query      FAILURE   28 Feb 07:10:35   0:00:00
  bquijob_de0df03_15a84d6a4fa       query      FAILURE   28 Feb 07:09:02   0:00:00
  bquijob_52c4f7d7_15a84d660e6      query      FAILURE   28 Feb 07:08:44   0:00:00
  bquijob_76a2c1be_15a84d5e769      query      FAILURE   28 Feb 07:08:13   0:00:00
  bquijob_7f51dde5_15a84d55afb      query      SUCCESS   28 Feb 07:07:41   0:00:08
  bquijob_34f25864_15a84d50503      query      SUCCESS   28 Feb 07:07:18   0:00:08
  job_Ca0cuRTAjY7MEHAs7vTJMxtVYTs   query      SUCCESS   28 Feb 07:00:47   0:00:09
  job_hHfmcdwyBsPsYF5dDvvOdR1Rmd0   load       SUCCESS   28 Feb 07:00:26   0:00:20
  job_mkiLf_mFHLKSplGJOtg-XDKzvv4   load       SUCCESS   28 Feb 02:52:50   0:00:02
  job_3RsPvttxWwv3SzVoOI9Cv_2yWtA   query      SUCCESS   27 Feb 21:18:40   0:00:08
  job_JLsqJO0NEIlKNac6jkDWbwneGMg   extract    SUCCESS   27 Feb 11:35:04   0:00:17
  job_KOS7vKX4aX0FNbK6dibE7cxzcQA   query      SUCCESS   27 Feb 11:33:44   0:00:37
  bquijob_44046bec_15a802f703a      query      SUCCESS   27 Feb 09:27:48   0:00:07
  job_2qQ6YSWeXaP2y2doONQJsIoga3c   query      SUCCESS   27 Feb 08:53:20   0:00:06

You would be able this way to check the extract job DURATION. In case you validate indeed it's a problem, please post such a table back in your question as sooner/later a Google engineer will check it. But without proper details, we can just assume your measurement is wrong.