doumeitang572461 2011-12-27 21:55
浏览 42
已采纳

卷曲超过100K的请求

I am using curl_multi_exec to process over 100K requests. I do 100 requests at a time because curl_multi_exec can only handle 10- requests at a time to eventually get to 100K requests. We've added multiple servers to this system to spread the load [we are using load balancing]. What is the best way to have curl handle 100K requests and make use of these additional servers? What's the downside (other than time) of handling that many requests on one server? How can I use the additional servers to help handle those requests?

To elaborate- basically, we are using curl to send out over 100K requests to third party servers. The problem with only using 1 server is that there is a memory limit in the number of requests 1 server can handle. So we decided to add additional servers, but we are not sure how to design this system to use curl to handle that many requests..

Thanks!

  • 写回答

1条回答 默认 最新

  • douxie1957 2011-12-27 22:09
    关注

    Don't obsess about CURL. It's simply a tool. You're focusing on the wrong level of design.

    What you need to consider is how to spread this workload amongst multiple servers. A simple design would have one central database listing all your urls, and a method for clients to "check out" a url (or set of urls) to chug away on.

    Once you've got that working, the curl portion will be the easiest part of it all.

    本回答被题主选为最佳回答 , 对您是否有帮助呢?
    评论

报告相同问题？

  • 卷曲超过100K的请求 php
    2011-12-27 21:55
    回答 1 已采纳 Don't obsess about CURL. It's simply a tool. You're focusing on the wrong level of design. What y
  • 循环多卷曲请求PHP的最佳方法 php
    2015-08-17 09:34
    回答 3 已采纳 This is fairly easy with a library like GuzzleHttp. I don't know what your object structure or val
  • PHP杀死正在进行的多卷曲请求 http nginx php
    2017-08-03 19:52
    回答 1 已采纳 well, you should be able to cancel them at will with a CURLOPT_PROGRESSFUNCTION, have a global var
  • streams:PHP包装器用于多卷曲
    2021-05-10 14:12
    “ streams”是用于多卷曲PHP包装器 安装 安装 ： curl -sS https://getcomposer.org/installer | php 添加流依赖项： php composer.phar require 'pulyavin/streams:0.*' 用法 use pulyavin\streams\ Stream ; ...
  • Wowza Cloud API：卷曲请求 php
    2017-10-27 09:00
    回答 1 已采纳 You should check for curl errors after $result = curl_exec($ch);. // Check for errors and display
  • 使用PHP卷曲伪造发布请求 - 拒绝 php
    2012-05-19 19:16
    回答 3 已采纳 I have now fixed it, the problem was with PHP curl and following redirects, it seems that it doesn
  • 卷曲php没有加载网站样式 php
    2017-08-09 12:58
    回答 1 已采纳 Because curl only gets html (source code) of specified url and i think styles are addressed relati
  • curl:一个友好PHP卷曲包装器
    2021-05-14 01:22
    适用于PHP的curl包装类一个PHP包装器类，用于简化。如何使用以下代码显示了如何使用此类 $ http = new dcai\curl;// enable cache$ http = new dcai\curl( array ( 'cache' => true ));// enable cookie$ ...
  • 卷曲请求超时 php
    2014-10-08 20:32
    回答 1 已采纳 Quoting from here: That error is specific to mod_fcgid, not PHP itself. If you're using the curre
  • 使用cookie卷曲到Golang HTTP请求 php
    2016-08-14 06:09
    回答 1 已采纳 To save a cookie: // do whatever is needed to login and get the cookie response, err := http.Post
  • PHP和Curl：从多页卷曲请求中获取所有结果 php
    2017-10-23 09:51
    回答 1 已采纳 make an array that holds all results you've recieved thus far. then start on the first page. fetc
  • php循环里请求,PHP循环卷曲请求一个接一个
    2021-04-26 11:39
    Leung Rick的博客 循环应该已经以这种方式工作,因为你正在使用阻塞cURL接口而不是cURL Multi接口.$ch = curl_init();for ($i = 1; $i <= 10; $i++){curl_setopt($ch, CURLOPT...$res = curl_exec($ch);// Code checking $res is no...
  • 卷曲请求什么都不返回 php
    2014-01-10 20:02
    回答 1 已采纳 Thank you very much for your interest to help. There was a weird issue. I had space in curl_
  • 多卷曲：使用cURL库发出单个或异步请求的简单HTTP客户端
    2021-02-17 09:37
    多网址 一个简单的HTTP客户端，可以使用cURL库发出单个或异步...可以设置基本URL，以便将来与请求相关的方法不必指定完整的端点。 例如： use Bayfront\MultiCurl\Client; $client = new Client('https://jsonplacehol
  • curlcurl:卷曲cURL
    2021-05-24 16:05
    卷曲cURL 这是一个帮助程序类，它通过将cURL卷曲成一行来使代码更清晰。 使用Curl cUrl，您不必像单独进行连接那样初始化或关闭连接。 只需包装选项和kaboom！ 在2.0版中，所有选项都可以链式加载。 ＃＃用法需要...
  • easycurl:易卷曲
    2021-06-24 23:36
    易卷曲 EasyCurl 一个非常简单的 cURL 类。 设置 推荐人和用户代理选项。 包括默认值。 require "easycurl.class.php" ; $ curl = new EasyCurl ; $ curl -> referer = "http://google.com" ; $ curl -> user_...
  • userinfo:卷曲http
    2021-06-18 06:49
    用户信息 - curl -X GET http://byoung@bigbluehat.com/十多年来，我一直在浏览器地址栏中输入电子邮件地址。 现在，我想要回一些东西。用户 URI 方案mailto和（最近的） acct提供了通过电子邮件地址识别实体（人、...
  • composer-curl-plugin:作曲家卷曲插件
    2021-05-31 01:06
    作曲家卷曲插件安装安装插件 $ composer global require ngyuki/composer-curl-plugin:dev-master 例子安装 symfony $ composer require symfony/symfony: \* 无插件更新$ rm -fr ~ /.composer/cache/repo/$ ...
  • braintree_php:Braintree PHP库
    2021-04-07 05:29
    卷曲 dom 杂凑 的openssl xmlwriter 需要PHP版本> = 7.3。 Braintree PHP SDK已针对PHP 7.3和7.4以及8.0版进行了测试。 PHP核心开发社区已经发布了适用于PHP 5.4-7.2，并且不再接收安全更新。 因此，Braintree...
  • spider:PHP 中的灵活蜘蛛
    2021-06-14 08:45
    卷曲（请求管道） 依赖关系 请参阅composer.json 。 用法 use ddliu\spider\ Spider ; use ddliu\spider\ Pipe \ NormalizeUrlPipe ; use ddliu\spider\ Pipe \ RequestPipe ; use ddliu\spider\ Pipe \ ...
  • 没有解决我的问题, 去提问

悬赏问题

  • ¥15 想问一下树莓派接上显示屏后出现如图所示画面，是什么问题导致的
  • ¥100 嵌入式系统基于PIC16F882和热敏电阻的数字温度计
  • ¥15 cmd cl 0x000007b
  • ¥20 BAPI_PR_CHANGE how to add account assignment information for service line
  • ¥500 火焰左右视图、视差（基于双目相机）
  • ¥100 set_link_state
  • ¥15 虚幻5 UE美术毛发渲染
  • ¥15 CVRP 图论 物流运输优化
  • ¥15 Tableau online 嵌入ppt失败
  • ¥100 支付宝网页转账系统不识别账号