导入到BigQuery中之前的CSV中的日期格式

Question

I have developed a Golang code that takes the csv file from Google cloud storage than import it to the Big Query table.

Everything is ok except that my CSV contains a DATE column in format of "2017-06-14 00:49:52 PDT". This cause the issue that CSV file can not be imported in the Big Query since the format must be "2017-06-14". I can not manually edit it in the CSV before uploading to the Google storage because it is about a very huge files (that changes every day).

Is there any option to update the CSV hosted on a storage using go lang and leave only "2017-06-14" value for this column (DATE) before executing the rest of code which import it to the Big Query, or any other solution?

Thank you in advance!

package storagetobigquery

import (
    "cloud.google.com/go/bigquery"
    "github.com/gin-gonic/gin"
    "google.golang.org/appengine"
)

// StoragetoBigquery function
func StoragetoBigquery(c *gin.Context) {
    ctx := appengine.NewContext(c.Request)

    client, err := bigquery.NewClient(ctx, "MY PROJECT ID")

    gcsRef := bigquery.NewGCSReference("PATH TO THE GOOGLE STORAGE CSV FILE")
    gcsRef.SourceFormat = bigquery.CSV
    gcsRef.AutoDetect = true
    gcsRef.SkipLeadingRows = 1


    loader := client.Dataset("DATASET NAME").Table(TABLE NAME).LoaderFrom(gcsRef)
    loader.WriteDisposition = bigquery.WriteTruncate

    job, err := loader.Run(ctx)
    if err != nil {
        panic(err)
    }
    status, err := job.Wait(ctx)
    if err != nil {
        panic(err)
    }

    if status.Err() != nil {
        panic(status.Err)
    }
}

Answer 1

If you have to use the Go client library, before changing the format you need to get the object from GCS. Then, re-upload to import it to BQ as you're doing. There is no method documented in https://godoc.org/cloud.google.com/go/storage to update the object in GCS directly, only its metadata.