我写了一个爬虫,用线程池执行,报502的错误
如何解决?
for (int i = 1; i <= 402; i++) {
int finalI = i;
threadPoolExecutor.execute(() -> {
try {
DownloadSource.download("https://xxxxxxx/page/"+ finalI);
System.out.println("--------------------------------第"+ finalI +"页 加载完毕------------------------------------------");
} catch (Exception e) {
e.printStackTrace();
}
});
}
报错:
org.jsoup.HttpStatusException: HTTP error fetching URL. Status=502, URL=https://xxxxx
at org.jsoup.helper.HttpConnection$Response.execute(HttpConnection.java:762)
at org.jsoup.helper.HttpConnection$Response.execute(HttpConnection.java:707)
at org.jsoup.helper.HttpConnection.execute(HttpConnection.java:297)
at org.jsoup.helper.HttpConnection.get(HttpConnection.java:286)
at com.DownloadSource.download(DownloadSource.java:25)
at com.DownloadSource.lambda$main$0(DownloadSource.java:53)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)