2 valiant303 valiant303 于 2016.02.17 13:09 提问

nutch自定义要抓取内容

刚接触nutch,环境是VirtualBox虚拟机安装centos6.5 64位 ,在CentOS下使用svn从官网检出nutch 2.3 。初步的需求就是,根据我自定义的url,通过输入某些关键词(或html标签、或者正则表达式),来把匹配的网页内容抓取下来。后续再进行分析(后话)
我还在学习中,发现nutch2.3版本中,已经用bin/crawl命令取代了老版本的 bin/nutch crawl ,参数列表几乎完全都变了.
我尝试了如下操作:
这是2.3版本的命令参数: bin/crawl
Usage: crawl []

然后我使用:bin/crawl urls/ MyFirstCrawl http://localhost:8080/solr 6 其中:
urls:是我建立的抓取文件所在的上级目录(结构:urls/urls.txt,urls.txt中存了要抓取的页面url)
MyFirstCrawl:自定义的crawl名称
solrUrl:这个地址是随便填写的
然后如下错误:
/home/release-2.3/runtime/local/bin/nutch generate -D mapred.reduce.tasks=2 -D mapred.child.java.opts=-Xmx1000m -D mapred.reduce.tasks.speculative.execution=false -D mapred.map.tasks.speculative.execution=false -D mapred.compress.map.output=true -topN 50000 -noNorm -noFilter -adddays 0 -crawlId MyFirstCrawl -batchId 1455677914-18990
GeneratorJob: starting at 2016-02-17 10:58:35
GeneratorJob: Selecting best-scoring urls due for fetch.
GeneratorJob: starting
GeneratorJob: filtering: false
GeneratorJob: normalizing: false
GeneratorJob: topN: 50000
java.util.NoSuchElementException
at java.util.TreeMap.key(TreeMap.java:1221)
at java.util.TreeMap.firstKey(TreeMap.java:285)
at org.apache.gora.memory.store.MemStore.execute(MemStore.java:125)
at org.apache.gora.query.impl.QueryBase.execute(QueryBase.java:73)
at org.apache.gora.mapreduce.GoraRecordReader.executeQuery(GoraRecordReader.java:68)
at org.apache.gora.mapreduce.GoraRecordReader.nextKeyValue(GoraRecordReader.java:110)
at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:531)
at org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java:67)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:364)
at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:223)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
GeneratorJob: finished at 2016-02-17 10:58:38, time elapsed: 00:00:02
GeneratorJob: generated batch id: 1455677914-18990 containing 0 URLs
Generate returned 1 (no new segments created)
Escaping loop: no more URLs to fetch now

请问这个错误代表什么?我该怎么样调整。
另外,如果想要完成我开头说的抓取需求 我该做怎样的配置才能实现?

1个回答

devmiao
devmiao   Ds   Rxr 2016.02.18 23:11
Csdn user default icon
上传中...
上传图片
插入图片
准确详细的回答,更有利于被提问者采纳,从而获得C币。复制、灌水、广告等回答会被删除,是时候展现真正的技术了!