如何将多个并发请求分配给AWS Lambda函数？

Question

I want to build a cronjob like system that gets all users from database and make multiple (I mean lots of) concurrent requests for each of them and make some executions and save the result to db. It will run every hour in every day 7/24.

I came up with the solution that:

1. Gets all users from db (that's the easy part)
2. Dynamically creates lambda functions and distributes all users to these functions
3. Each lambda function makes concurrent requests and executions (handling results and saving them to db)
4. Communicate these functions with SNS when needed

So, does my approach make sense for this situation?

The most important thing here is scaling (that's why I thought to distribute all users to lambda functions, for limiting concurrent requests and resources), how we can come with an scalable and efficient idea for exponentially increased user count?

Or any other suggestions?

Answer 1

Here is my solution:

if 100 concurrent lambdas are not enough for your need, create a ticket to increase your limit, you will be charged what will you use.

However, still you can't determine that how many lambdas will be required in future. It is not necessary to process each user in a separate lambda, instead you can invoke a lambda with a chunk of user data. e.g. Let's say, your max. lambda limit is 100 and there are 1000 users then you can do something (i don't know go, here is a python code which may not be 100% syntactically correct)

users = get_users_fromdb()   # users = [1,2,3,... 1000]
number_of_users = len(users)
chunk_size = number_of_users / 100   # 100 is your lambda limit
for i in range(0, number_of_users, chunk_size)
    # e.g. chunk_users_data = [1,2,3 ... 10]
    chunk_users_data = users[i * chunk_size : (i + 1) * chunk_size ]
    invoke_lambda_to_process_users_chunk_data()

Here is what you can do in other lambda

users = event.get('users')
for user in users:
    try:
       process_user(user)
    except Exception as e:
        print(e)  # handle exception / error if you want

Update:

By default, 100 is limit for concurrent running lambdas. If you have 100K users, IMO, you should go for a support case to increase your account's concurrent lambda limit to 1000 or more. I am working on lambda and we have 10K limit. One more thing to keep in mind that it is not sure that your one lambda invocation will be able to process all users in a chunk, so add some logic to reinvoke with remaining users before timeout. A lambda can run upto max. of 5 minutes. YOu can get remaining time from context object in milli seconds.