如何使用gearman每秒运行50k个工作

Question

According to Gearman website

"A 16 core Intel machine is able to process upwards of 50k jobs per second."

I have load balancer that moves traffic to 4 different machines. Each machine has 8 cores. I want to have the ability to run 13K jobs per machine, per second (it's definitely more then 50K jobs).

Each job takes between 0.02 - 0.8 MS.

How many workers do I need to open for this type of performance?

What is the steps that I need to take to open these amount of workers?

Answer 1

Depending on what kind of processing you're doing, this will require a little experimentation and load testing. Before you start, make sure you have a way to reboot the server without SSH, as you can easily peg the CPU. Follow these steps to find the optimum number of workers:

1. Begin by adding a number of workers equal to the number of cores minus one. If you have 8 cores, start with 7 workers (hopefully leaving a core free for doing things like SSH).
2. Run top and observe the load average. The load average should not be higher than the number of cores. For 8 cores, a load average of 7 or above would indicate you have too many workers. A lower load average means you can try adding another worker.
3. If you added another worker in step 2, observe the load average again. Also observe the increase in RAM usage.
4. If you repeat the above steps, eventually you will either run out of CPU or RAM.

When doing parallel processing, keep in mind that you could run into a point of diminishing returns. Read about Amdahl's law for more information.