dongxichan8627 2011-05-31 10:39
浏览 123

Java:HtmlUnit效率关联PHP CURL?

I have a spider class which on a user request spiders websites for content. Each search results in loading about 30 websites, spidering them for the information and then standardizing this information.

I have written this in PHP using CURL, since PHP is lacking multitasking I would like to switch to Java (I am aware of the multi process curl which does not suit my demand). I need a http client which can POST/GET, receive and set cookies as well as modify HTTP headers.

I have found HtmlUnit which seems nifty but also exceeds my demand, and since the package is relatively big and I will have many hundread requests a minute I don't want to have an overkill solution slowing down my servers.

Do you think this would be an issue and do you have other suggestions to replace CURL in Java? Should I use the Java CURL binding? This is a question of efficiency and server load.

  • 写回答

3条回答 默认 最新

  • douyue8191 2011-05-31 11:30

    Perhaps take a look at Apache Http Client ?

    You can create a HttpClient per thread and use that to do your requests

    while (running) {
    HttpClient client = new DefaultHttpClient();
    HttpGet GET = new HttpGet("");
    HttpResponse response = client.execute(GET);
    // do stuff with response

    Even better, if you re-use the HttpClient between requests, it will remember the cookies sent back on previous responses, and automatically apply them to your next request. In that sense a single HttpClient models a http conversation.

    So if you did

      // cookies received in response
      // the second get will send the cookies back received from GET1 response.

    You could then take a look at Java's ExecutorService that will make it easy to place spider jobs and have multiple threads running.

    本回答被题主选为最佳回答 , 对您是否有帮助呢?



  • ¥15 关于#stm32#的问题:/* User can add his own implementation to report the HAL error return state */
  • ¥15 imageware粗糙度表面
  • ¥15 python使用pulp线性优化时报错
  • ¥15 为什么我的uibot导入py模块出错呀。py文件放在了uibot里对应的python文件夹了,卸了重安也不行
  • ¥15 开源或低价数据中台哪个最好
  • ¥15 arduino编程出现字符串疑似覆盖现象
  • ¥15 我的b站在没有碰到屏幕的情况下偶尔会自动跳出进度条,就像在屏幕上点了一下一样,但我并没有点。而且视频进度并没有变。这可能是什么原因造成的?
  • ¥30 STK matlab python仿真
  • ¥15 关于IMageEnView 图标定位问题
  • ¥20 求解答(matlab)