java.io.IOException: Server returned HTTP response code: 502 for URL:XXXXXX

Question

我在从系统数据库中拿到20W张图片的url，然后下载到本地的时候部分图片下载出现了这个错误：
java.io.IOException: Server returned HTTP response code: 502 for URL:XXXXX
代码：

public static String downLoadFromUrl(String urlStr, String fileName, String savePath) {
        JSONObject response = new JSONObject();
        JSONArray failds = new JSONArray();
        try {

            URL url = new URL(urlStr);
            HttpURLConnection conn = (HttpURLConnection) url.openConnection();
            // 设置超时间为3秒
            conn.setConnectTimeout(10 * 1000);
            // 防止屏蔽程序抓取而返回403错误
            conn.setRequestProperty("User-Agent", "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt)");

            // 得到输入流
            InputStream inputStream = conn.getInputStream();
            // 获取自己数组
            byte[] getData = readInputStream(inputStream);

            // 文件保存位置
            File saveDir = new File(savePath);
            if (!saveDir.exists()) {
                saveDir.mkdirs();
                logger.info("创建文件夹:" + saveDir.exists());
            }
            File file = new File(saveDir + File.separator + fileName);
            FileOutputStream fos = new FileOutputStream(file);
            fos.write(getData);
            if (fos != null) {
                fos.close();
            }
            if (inputStream != null) {
                inputStream.close();
            }
            return saveDir + File.separator + fileName;
        } catch (Exception e) {
            logger.error(">>>>>>image download faild,image url:" + urlStr + ",fileName:"+ fileName);
            failds.add(urlStr);
            e.printStackTrace();
        }
        response.put("faild",failds);
        return response.toString();

    }

    public InputStream getConnectionStream(String fullUrl) throws IOException {
        HttpClient client = new DefaultHttpClient();
        HttpGet request = new HttpGet(fullUrl);
        HttpResponse response = client.execute(request);
        InputStream is = response.getEntity().getContent();

        return is;
    }

    public static byte[] readInputStream(InputStream inputStream) throws IOException {
        byte[] buffer = new byte[1024];
        int len = 0;
        ByteArrayOutputStream bos = new ByteArrayOutputStream();
        while ((len = inputStream.read(buffer)) != -1) {
            bos.write(buffer, 0, len);
        }
        bos.close();
        return bos.toByteArray();
    }

下载图片就是循环调用这块代码，我应该如何优化才能避免这个错？

Answer 1

其实主要原因应该是你20w个请求循环发送，循环的效率很快，其实可以看做发送量多于服务器可以处理的数据量。这样就会造成请求阻塞，部分连接超时未响应。你需要在发送下载请求的时候进行延迟请求。可以考虑手动写一个连接池限定同时发送连接数量。或者如果你不考虑下载时间的问题可以请求响应以后再去请求一个新的，这样效率会下降，但能保证请求能正常响应。