使用PHP（XPath），PHP / Python（Regex）或Python（XPath）从html中提取信息

Question

I have approx. 40k+ html documents where I need to extract information from. I have tried to do so using PHP+Tidy(because most files are not well-formed)+DOMDocument+XPath but it is extremely slow.... I am advised to use regexp but the html files are not marked up semantically (table based layout, with meaning-less tag/classes used everywhere) and I don't know where i should start...

Just being curious, is using regexp (PHP/Python) faster than using Python's XPath library? Is Xpath library for Python generally faster than PHP's counterpart?

Answer 1

If speed is a requirement have a look at lxml. lxml is a pythonic binding for the libxml2 and libxslt C libraries. Using the C libraries is much faster than any pure php or python version.

There are some impressive benchmarks from Ian Bicking:

In Conclusion

I knew lxml was fast before I started these benchmarks, but I didn’t expect it to be quite this fast.

Parsing Results:

Parsing Resutls http://1.2.3.9/bmi/blog.ianbicking.org/wp-content/uploads/images/parsing-results.png