weixin_39810441 2020-11-29 16:45
浏览 0

Update most dependencies to recent versions

This change upgrades Heritrix 3 to Java 8 (and back to 7 - see later comments), and updates almost all dependencies under heritrix-common to the latest versions. This includes Kryo and the Berkeley DB JE used for state management. These later versions appear to be less bugger and more robust, but this as the changes run deep into the core of the crawler they will require much testing. (i.e. don't merge this, at least not yet!)

The new BDB-JE has better support for alerting to GC issues and looks out for Thread.interrupt() events, after which it will consider the DB environment tainted. Therefore interrupts should be avoided in the code and threads should be shut down properly instead.

The OneLineSimpleLayout class has also been moved into heritrix-commons so it can be used.

For reasons that are not clear, the FetchStats class needed a new constructor so Kryo could rebuild it, passing in a comparator.

The main outstanding issue is that the org.archive.crawler.selftest.StatisticsSelfTest test class fails (blocking any later tests).

该提问来源于开源项目：internetarchive/heritrix3

  • 写回答

5条回答 默认 最新

  • weixin_39810441 2020-11-29 16:45
    关注

    On my laptop, the StatisticsSelfTest throws these errors and then hangs...

    
SEVERE: Fatal exception in ToeThread #7
java.lang.ArrayIndexOutOfBoundsException: -1
    at com.esotericsoftware.kryo.util.IntArray.pop(IntArray.java:175)
    at com.esotericsoftware.kryo.Kryo.reference(Kryo.java:874)
    at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:539)
    at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:753)
    at org.archive.bdb.KryoBinding.entryToObject(KryoBinding.java:96)
    at org.archive.crawler.frontier.BdbMultipleWorkQueues.get(BdbMultipleWorkQueues.java:267)
    at org.archive.crawler.frontier.BdbWorkQueue.peekItem(BdbWorkQueue.java:103)
    at org.archive.crawler.frontier.WorkQueue.peek(WorkQueue.java:174)
    at org.archive.crawler.frontier.WorkQueueFrontier.findEligibleURI(WorkQueueFrontier.java:629)
    at org.archive.crawler.frontier.AbstractFrontier.next(AbstractFrontier.java:450)
    at org.archive.crawler.framework.ToeThread.run(ToeThread.java:134)

Sep 08, 2017 9:29:21 PM org.archive.crawler.framework.ToeThread recoverableProblem
SEVERE: Problem java.lang.ClassCastException: java.util.HashMap cannot be cast to java.lang.String occured when trying to process 'http://127.0.0.1:7777/a.html' at step ABOUT_TO_BEGIN_PROCESSOR in 

java.lang.ClassCastException: java.util.HashMap cannot be cast to java.lang.String
    at org.archive.modules.CrawlURI.inheritFrom(CrawlURI.java:1588)
    at org.archive.modules.CrawlURI.createCrawlURI(CrawlURI.java:1619)
    at org.archive.modules.CrawlURI.markPrerequisite(CrawlURI.java:1833)
    at org.archive.crawler.prefetch.PreconditionEnforcer.considerRobotsPreconditions(PreconditionEnforcer.java:231)
    at org.archive.crawler.prefetch.PreconditionEnforcer.innerProcessResult(PreconditionEnforcer.java:178)
    at org.archive.modules.Processor.process(Processor.java:142)
    at org.archive.modules.ProcessorChain.process(ProcessorChain.java:131)
    at org.archive.crawler.framework.ToeThread.run(ToeThread.java:148)

    And frankly I don't know what's going on, and why that files when preceding self-tests work okay.

    评论

报告相同问题？