打开/处理580GB错误日志文件的建议

Question

A friend asked me to help him with a simple php script I placed on their Wordpress site. In doing so, I came across a few glaring issues I thought I could help him with. Then I came across the beast.

A 580 gigabyte error log file.

This is a simple blog, granted with millions of views a month and gigabytes of pictures, but shouldn't he do some maintenance on the thing? I know I could tail -f the log file in most circumstances but I think it may make planets collide trying to open a text file that large.

Question

Should I tell him to just wipe it and start monitoring it fresh from here on out or is there a proper way to review even, say, the last 10 days worth of logs? I haven't spent enough time on their server (just browsed FTP) to see what they are running.

Answer 1

This gives you the last 100 lines:

tail -n 100 logfile > last-100-lines.log

In terms of 'date', you could also:

cat |grep '2012-12' > all-dec.log

Not, the exact term to grep for depends on what the log file looks like. Both operations will take a little while to run. If this application is disk-intense, you could run into problems by saturating the disk since this monster is pretty large and it's very unlikely you have 600+ GB of RAM and that log file is cached in memory. ;-)

Anyway, for kicks, try bzip2 --best logfile and then download it and then evaluate it on localhost.

It's not pretty to work with files that large, but it should be do-able.