PutItem进入DynamoDB的速度很慢

Question

I have an EC2 instance which opens up a json file, reads each line and does a putItem operation into two tables.

Without the putItem operation, Golang parses a 67k line file in about 3 seconds.

With the putItem operation, it is processing 10k items every 5 minutes. The put operation into dynamodb is not being throttled. The WCUs and RCUs have been set accordingly. So Is there a reason that the putItem operation is holding up the code?

I assume that Golang is waiting for each put operation to succeed?

Still quite unsure, if anyone has done a mass insert into dynamodb using golang, then it would be helpful if you shedded some light on how you circumvented this.

Answer 1

The slowness comes from the fact that each insert has to do a full HTTP round trip to dynamo.

10k items in 5 mins is about 30ms per item, which is expected of an HTTP trip.

You can use a batch update, docs here

In the docs, you have BatchWriteItemInput, which takes a map[string][]*WriteRequest.

type BatchWriteItemInput struct {
    // ... trimmed

    RequestItems map[string][]*WriteRequest

    ...
}

The WriteRequest model is a little odd, because its used for both Delete and Put operations, just ignore the DeleteRequest *DeleteRequest field.

It's important to note that there are some limits on the batch operation:

• You can't change the same item more than once in a batch request
• You must have more than 2 and fewer than 25 items in a batch
• Each item cannot be bigger than 400KB and the total batch cannot be bigger than 16MB.