weixin_37812800
weixin_37812800
采纳率0%
2019-04-10 10:17 阅读 672

hadoop 通过javaApi 上传文件远程主机强迫关闭了一个现有的连接,上传的文件为0字节

现在通过java Api来上传本地文件到hdfs中,
本地环境为win10,hdfs环境为centos7,
hdfs通过kerberos来添加验证,上传过程中,小文件可以上传成功,大文件上传后,报错

Exception in thread "main" java.io.IOException: 远程主机强迫关闭了一个现有的连接。
    at sun.nio.ch.SocketDispatcher.read0(Native Method)
    at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:43)
    at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
    at sun.nio.ch.IOUtil.read(IOUtil.java:197)
    at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
    at org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:57)
    at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
    at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
    at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
    at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:118)
    at java.io.FilterInputStream.read(FilterInputStream.java:83)
    at java.io.FilterInputStream.read(FilterInputStream.java:83)
    at org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:2280)
    at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1347)
    at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1266)
    at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:449)

无论大文件,小文件上传到hdfs中后,都为0字节

图片说明

以下为实现代码

    public static final String USER_KEY = "hdfs@TDH";
    public static final String KEY_TAB_PATH = "C:\\Users\\xx\\Desktop\\keytab";

    static Configuration conf = new Configuration();

    static {
        System.setProperty("java.security.krb5.conf", "C:\\Users\\xx\\Desktop\\krb5.conf");
        conf.set("hadoop.security.authentication", "kerberos");
        conf.set("fs.defaultFS", "hdfs://xx.xx.xx.xx:8020");
        //conf.set("dfs.client.use.datanode.hostname", "true");

        try {
            UserGroupInformation.setConfiguration(conf);
            UserGroupInformation.loginUserFromKeytab(USER_KEY, KEY_TAB_PATH);
        } catch (IOException e) {
            e.printStackTrace();
        }
    }

    public static void main(String[] args) throws IOException {
        FileSystem fs = FileSystem.get(conf);
        InputStream fis = new FileInputStream(new File("C:\\Users\\xx\Desktop\\a.zip"));
        OutputStream fos = fs.create(new Path("/tmp/b.exe"));
        IOUtils.copyBytes(fis,fos,1024);
    }

  • 点赞
  • 写回答
  • 关注问题
  • 收藏
  • 复制链接分享

2条回答 默认 最新

  • daisuo2599 daisuo2599 2019-04-12 09:59

    你这个应该是使用kerberos验证失败了,应该还需要设置一个登陆信息配置:java.security.auth.login.config

    点赞 评论 复制链接分享
  • weixin_40089657 weixin_40089657 2020-04-08 23:19

    认证是过了,权限问题

    点赞 评论 复制链接分享

相关推荐