大家好，我现在遇到一个问题，手里有50，000多url，需要一一下载下来，由于都是国外的网址，因此相当慢，现在想弄成多线程抓取网页，不过我的代码无法正常运行，请达人不吝指教，多谢了，以下是我的代码：
[code="java"]
import ......

public class Down2011CaseMeshTread extends Thread {

public static int count = 0;  
public static List<String> docDoiList = getCaseDoiList2();  
private static URL url;    
private static String doi;  
static int BUFFER_SIZE = 1024*10;  

public Down2011CaseMeshTread(String doi) throws MalformedURLException{  
    String urlStr = "http://www.iteye.com/" + doi;  
    this.url = new URL(urlStr);  
    this.doi = doi;  
}  

public static Connection getConnection() throws Exception {  
       //连接数据库的 
  }  

public static String getDocNameByDoi(String docDoi){  
       //获得docDoi对应的文件名，就是下载下来之后文件的存储名字
}  

public void Test() throws IOException, InterruptedException{  

    StringBuffer sb = null;  
    BufferedReader in = null;  
    BufferedWriter out = null;  
    try {  
         sb = new StringBuffer();  
         int ch =0;  
         URLConnection conn = (HttpURLConnection)url.openConnection();  
         conn.setRequestProperty("User-Agent", "Mozilla/4.76");  
         conn.setDoOutput(true);  
         conn.setConnectTimeout(1000*60*10);  
         in = new BufferedReader(new InputStreamReader(url.openStream()));  
         FileOutputStream fo = new FileOutputStream("/home/" + getDocNameByDoi(doi));  
         OutputStreamWriter writer = new OutputStreamWriter(fo, "utf-8");  
         out = new BufferedWriter(writer);  
         while (!in.ready())  
         {  
              Thread.sleep(500); // wait for stream to be ready.  
         }  
         char[] buffer = new char[BUFFER_SIZE];  
         int charsRead = 0;  
         while ( (charsRead = in.read(buffer, 0, BUFFER_SIZE)) != -1 ) {  
              out.write(buffer, 0, charsRead);  
         }  
     out.close();  
     in.close();  
    }catch(Exception e){  
        e.printStackTrace();  
    }  
}  

public void run(){  
    try {  
        Test();  
        } catch (IOException e) {  
        e.printStackTrace();  
    } catch (InterruptedException e) {  
        e.printStackTrace();  
    }  
}  
    //应该是run()这里的问题，但我不知道改怎么改？？？？  

public static List<String> getCaseDoiList2(){  
       //获得url链表，一共五万多 
}  

public static void main(String args[]) throws MalformedURLException{  
    for(String docDoi : docDoiList){  
            Down2011CaseMeshTread down = new Down2011CaseMeshTread(docDoi);  
            down.start();  
    }  
}

}

[/code]

情况就是这么个情况，
请问这个该怎么改阿，非常感谢～

这个程序的问题是，开始执行特别快，大概一分钟能下两三百，但下到大约800的时候就报错
先是读到异常文件终止“EOF”
但这时还可以断断续续的下载
[code]
java.io.IOException: Premature EOF
at sun.net.www.http.ChunkedInputStream.fastRead(ChunkedInputStream.java:252)
at sun.net.www.http.ChunkedInputStream.read(ChunkedInputStream.java:680)
at java.io.FilterInputStream.read(FilterInputStream.java:133)
at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:2582)
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:282)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:324)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:176)
at java.io.InputStreamReader.read(InputStreamReader.java:184)
at java.io.BufferedReader.read1(BufferedReader.java:202)
at java.io.BufferedReader.read(BufferedReader.java:278)
at org.Down2011CaseMeshTread.Test(Down2011CaseMeshTread.java:138)
at org.Down2011CaseMeshTread.run(Down2011CaseMeshTread.java:150)
[/code]

然后再报连接超时错误，这个时候下载就都中断了
[code]
java.net.ConnectException: Connection timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:327)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:193)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:180)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:384)
at java.net.Socket.connect(Socket.java:546)
at java.net.Socket.connect(Socket.java:495)
at sun.net.NetworkClient.doConnect(NetworkClient.java:178)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:409)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:530)
at sun.net.www.http.HttpClient.(HttpClient.java:240)
at sun.net.www.http.HttpClient.New(HttpClient.java:321)
at sun.net.www.http.HttpClient.New(HttpClient.java:338)
at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:935)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:876)
at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:801)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1139)
at java.net.URL.openStream(URL.java:1029)
at org.Down2011CaseMeshTread.Test(Down2011CaseMeshTread.java:127)
at org.Down2011CaseMeshTread.run(Down2011CaseMeshTread.java:150)
[/code]

这个是连接的错误
估计是我没有设定线程的数量，所以在存在大量线程请求连接的时候出了问题，
但怎么修改我还不知道哦~

java如何抓取不同网站不同网页的内容

sunflower_open

3个回复