We're running a very time-sensitive web application (response time has to be below 100ms), with a lot of requests (about 200k requests per minute at peak). The architecture is really simple: a load balancer, several web servers, running apache and php, and a database running MySQL.
We also need to be able to generate statistics based on those requests.
About a year ago when we were serving a tenth of our current traffic volume, we developed some bash/python scripts to periodically dump the logs from mysql, trasnfer them to another server, import them again and run the statistics there, the idea being to have the production servers do as little as possible so that we could have a low response time.
As you might imagine, that solution didn't scale very well, and at the moment, the stat server is barely keeping up. We need a way to generate statistics in real time.
Do you have any experiences with this kind of a setup? Our idea at the moment is to have the web servers call the stats servers in real time on each request.
The two main problems are:
- how should we approach this so that the response time doesn't increase too much
- the stats server or server has to handle all the requests from all the webheads / or it needs to be able to scale horizontally.
