doufendi9063
doufendi9063
2011-06-21 16:40

有效处理“大数据”的技术(读取,写入,查询,聚合)

已采纳

Building a proof of concept analytics dashboard feature for a web application, allowing users to view who has been visiting their profile and viewing the content they have posted. Think about.me style dashboard.

Currently writing data points (subject, verb, object, timestamp tuple format) into a MySQL table (which is rapidly approaching a million records) as users perform relevant actions in the application. This writing happens out of sync using a cURL + cron job + message queue mechanism.

Not sure how to proceed for the actual collation, aggregation and presentation of this data on the analytics dashboard in a way that won't tie up far too many resources on my server.

Looking for tips, war stories and technologies that will simplify my life with this stuff.

  • 点赞
  • 写回答
  • 关注问题
  • 收藏
  • 复制链接分享
  • 邀请回答

1条回答

  • doutan8775 doutan8775 10年前

    Why not just start with the select queries and see how it performs?
    If and when you run into slowness you can always:

    • Tweak your queries, use memory tables, get creative with triggers, blackhole tables and whatnot.
    • Install a slave server and select from the slave.
    • Install a blackhole master and attach multiple slaves to that

    I recommend that you read High performance MySQL: http://oreilly.com/catalog/9780596003067

    点赞 9 评论 复制链接分享