GAE / Go：数据存储迭代器太慢

Question

Iteration to datastore query result in GAE/Go is very slow.

q := datastore.NewQuery("MyStruct")
gaeLog.Infof(ctx, "run")                                                 // (1)
it := client.Run(ctx, q)
list := make([]MyStruct, 0, 10000)
gaeLog.Infof(ctx, "start mapping")                                       // (2)
for {
    var m MyStruct
    _, err := it.Next(&m)
    if err == iterator.Done {
        break
    }
    if err != nil {
        gaeLog.Errorf(ctx, "datastore read error : %s ", err.Error())
        <some error handling>
        break
    }
    list = append(list , m)
}
gaeLog.Infof(ctx, "end mapping. count : %d", len(list))                  // (3)

The result is below.

18:02:11.283 run                             // (1)
18:02:11.291 start mapping                   // (2)
18:02:15.741 end mapping. count : 2400       // (3)

It takes about 4.5 seconds between (2) and (3), just only 2400 record. It is very slow.

How can I improve performance?

[Update]

I added the query in above code q := datastore.NewQuery("MyStruct"). I tried to retrieve all the entities in the kind MyStruct. This kind has 2400 entities.

Answer 1

I was using cloud.google.com/go/datastore and found it is slow. I migrated to use google.golang.org/appengine/datastore.

The result is as follows, less than 1 second.

13:57:46.216 run
13:57:46.367 start mapping
13:57:47.063 end mapping. count : 2400