qq_45278010 2022-04-21 01:37 采纳率: 66.7%
浏览 599
已结题

配置hive on spark测试，创建表，insert测试效果报错，如何解决？

问题遇到的现象和发生背景

hive on spark 测试插入数据：
create table student(id int, name string);
insert into table student values(1,'abc');

[atguigu@hadoop102 spark]$ hive
which: no hbase in (/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/module/jdk1.8.0_212/bin:/opt/module/hadoop-3.1.3/bin:/opt/module/hadoop-3.1.3/sbin:/opt/module/kafka/bin:/opt/module/hive/bin:/opt/module/spark/bin:/home/atguigu/.local/bin:/home/atguigu/bin)
Hive Session ID = d9cd91a7-1373-4a2f-a7a7-ba9f21474163

Logging initialized using configuration in jar:file:/opt/module/hive/lib/hive-common-3.1.2.jar!/hive-log4j2.properties Async: true
Hive Session ID = ab22e7b8-7224-4482-a30f-e528476272a1
hive (default)> insert into table student values(1,'abc');
Query ID = atguigu_20220421005402_5de0119d-385d-48ad-afb4-5acc32f5aafd
Total jobs = 1
Launching Job 1 out of 1
In order to change the average load for a reducer (in bytes):
  set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
  set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
  set mapreduce.job.reduces=<number>
Failed to monitor Job[-1] with exception 'java.lang.IllegalStateException(Connection to remote Spark driver was lost)' Last known state = SENT
Failed to execute spark task, with exception 'java.lang.IllegalStateException(RPC channel is closed.)'
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.spark.SparkTask. RPC channel is closed.
hive (default)>

yarn有任务，但是没完成

img

查看任务日志：

Name:    Hive on Spark (sessionId = b6539e92-dd16-4ab8-88d2-943f923226fc)
Application Type:    SPARK
Application Tags:    
Application Priority:    0 (Higher Integer value indicates higher priority)
YarnApplicationState:    FAILED
Queue:    default
FinalStatus Reported by AM:    FAILED
Started:    星期四 四月 21 00:54:14 +0800 2022
Launched:    星期四 四月 21 00:54:15 +0800 2022
Finished:    星期四 四月 21 00:56:11 +0800 2022
Elapsed:    1mins, 57sec
Tracking URL:    History
Log Aggregation Status:    SUCCEEDED
Application Timeout (Remaining Time):    Unlimited
Diagnostics:    
Application application_1650469616253_0002 failed 1 times (global limit =2; local limit is =1) due to AM Container for appattempt_1650469616253_0002_000001 exited with exitCode: 13
Failing this attempt.Diagnostics: [2022-04-21 00:56:11.775]Exception from container-launch.
Container id: container_1650469616253_0002_01_000001
Exit code: 13
[2022-04-21 00:56:11.809]Container exited with a non-zero exit code 13. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
ls.wrapException(NetUtils.java:755)
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1549)
at org.apache.hadoop.ipc.Client.call(Client.java:1491)
at org.apache.hadoop.ipc.Client.call(Client.java:1388)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118)
at com.sun.proxy.$Proxy15.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:904)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
at com.sun.proxy.$Proxy16.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1661)
at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1577)
at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1574)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1589)
at org.apache.spark.deploy.history.EventLogFileWriter.requireLogBaseDirAsDirectory(EventLogFileWriters.scala:77)
at org.apache.spark.deploy.history.SingleEventLogFileWriter.start(EventLogFileWriters.scala:221)
at org.apache.spark.scheduler.EventLoggingListener.start(EventLoggingListener.scala:81)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:572)
at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
at org.apache.hive.spark.client.RemoteDriver.<init>(RemoteDriver.java:161)
at org.apache.hive.spark.client.RemoteDriver.main(RemoteDriver.java:536)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:728)
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:700)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:804)
at org.apache.hadoop.ipc.Client$Connection.access$3800(Client.java:421)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1606)
at org.apache.hadoop.ipc.Client.call(Client.java:1435)
... 32 more
2022-04-21 00:56:11,568 WARN rpc.RpcDispatcher: [DriverProtocol] Closing RPC channel with 1 outstanding RPCs.
2022-04-21 00:56:11,591 INFO yarn.ApplicationMaster: Deleting staging directory hdfs://hadoop102:9820/user/atguigu/.sparkStaging/application_1650469616253_0002
2022-04-21 00:56:11,643 INFO util.ShutdownHookManager: Shutdown hook called
2022-04-21 00:56:11,644 INFO util.ShutdownHookManager: Deleting directory /opt/module/hadoop-3.1.3/data/nm-local-dir/usercache/atguigu/appcache/application_1650469616253_0002/spark-a9d09613-a501-4c2f-abe9-684a8fbf8ca4
[2022-04-21 00:56:11.810]Container exited with a non-zero exit code 13. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
ls.wrapException(NetUtils.java:755)
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1549)
at org.apache.hadoop.ipc.Client.call(Client.java:1491)
at org.apache.hadoop.ipc.Client.call(Client.java:1388)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118)
at com.sun.proxy.$Proxy15.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:904)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
at com.sun.proxy.$Proxy16.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1661)
at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1577)
at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1574)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1589)
at org.apache.spark.deploy.history.EventLogFileWriter.requireLogBaseDirAsDirectory(EventLogFileWriters.scala:77)
at org.apache.spark.deploy.history.SingleEventLogFileWriter.start(EventLogFileWriters.scala:221)
at org.apache.spark.scheduler.EventLoggingListener.start(EventLoggingListener.scala:81)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:572)
at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
at org.apache.hive.spark.client.RemoteDriver.<init>(RemoteDriver.java:161)
at org.apache.hive.spark.client.RemoteDriver.main(RemoteDriver.java:536)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:728)
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:700)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:804)
at org.apache.hadoop.ipc.Client$Connection.access$3800(Client.java:421)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1606)
at org.apache.hadoop.ipc.Client.call(Client.java:1435)
... 32 more
2022-04-21 00:56:11,568 WARN rpc.RpcDispatcher: [DriverProtocol] Closing RPC channel with 1 outstanding RPCs.
2022-04-21 00:56:11,591 INFO yarn.ApplicationMaster: Deleting staging directory hdfs://hadoop102:9820/user/atguigu/.sparkStaging/application_1650469616253_0002
2022-04-21 00:56:11,643 INFO util.ShutdownHookManager: Shutdown hook called
2022-04-21 00:56:11,644 INFO util.ShutdownHookManager: Deleting directory /opt/module/hadoop-3.1.3/data/nm-local-dir/usercache/atguigu/appcache/application_1650469616253_0002/spark-a9d09613-a501-4c2f-abe9-684a8fbf8ca4
For more detailed output, check the application tracking page: http://hadoop103:8088/cluster/app/application_1650469616253_0002 Then click on links to logs of each attempt.
. Failing the application.
Unmanaged Application:    false
Application Node Label expression:    <Not set>
AM container Node Label expression:    <DEFAULT_PARTITION>
Application Metrics
Total Resource Preempted:    <memory:0, vCores:0>
Total Number of Non-AM Containers Preempted:    0
Total Number of AM Containers Preempted:    0
Resource Preempted from Current Attempt:    <memory:0, vCores:0>
Number of Non-AM Containers Preempted from Current Attempt:    0
Aggregate Resource Allocation:    179968 MB-seconds, 117 vcore-seconds
Aggregate Preempted Resource Allocation:    0 MB-seconds, 0 vcore-seconds
操作环境、软件版本等信息

hive/conf/hive-site.xml 文件：

<property>
    <name>spark.yarn.jars</name>
    <value>hdfs://hadoop102:9820/spark-jars/*</value>
</property>

<!--Hive执行引擎-->
<property>
    <name>hive.execution.engine</name>
    <value>spark</value>
</property>

<!--Hive和Spark连接超时时间-->
<property>
    <name>hive.spark.client.connect.timeout</name>
    <value>10000ms</value>
</property>
</configuration>

hive/conf/spark-defaults.conf 文件：

spark.master                               yarn
spark.eventLog.enabled                   true
spark.eventLog.dir                        hdfs://hadoop102:8020/spark-history
spark.executor.memory                    1g
spark.driver.memory                                        1g

hadoop-3.1.3/etc/hadoop/core-site.xml 文件


   <property>
        <name>fs.defaultFS</name>
        <value>hdfs://hadoop102:9820</value>
</property>

hadoop-3.1.3/etc/hadoop/yarn-site.xml 文件

    <!-- yarn容器允许分配的最大最小内存 -->
    <property>
        <name>yarn.scheduler.minimum-allocation-mb</name>
        <value>512</value>
    </property>
    <property>
        <name>yarn.scheduler.maximum-allocation-mb</name>
        <value>4096</value>
    </property>

    <!-- yarn容器允许管理的物理内存大小 -->
    <property>
        <name>yarn.nodemanager.resource.memory-mb</name>
        <value>4096</value>
    </property>

    <!-- 关闭yarn对物理内存和虚拟内存的限制检查 -->
    <property>
        <name>yarn.nodemanager.pmem-check-enabled</name>
        <value>false</value>
    </property>
    <property>
        <name>yarn.nodemanager.vmem-check-enabled</name>
        <value>false</value>
    </property>
尝试过的解决方法
我想要达到的结果

使用insert into 语句，成功向student表里插入数据

  • 写回答

2条回答 默认 最新

  • 数仓白菜白 2022-04-21 07:20
    关注

    你在b站视频下面看看评论区，评论区下面的前几个有大佬提供了解决思路。我当时这里报错了，我用了评论区给的方法成功了。
    如果评论区的方法没有解决，说一个最不好听的方法，你把hive和spark重新装一遍。我由于之前用过hive ，我在hive on spakr的时候和你一样，select *可以使用 ，如果粘上一点会有mr任务的sql他就报错

    本回答被题主选为最佳回答 , 对您是否有帮助呢?
    评论
查看更多回答(1条)

报告相同问题？

问题事件

  • 系统已结题 4月30日
  • 已采纳回答 4月22日
  • 创建了问题 4月21日

悬赏问题

  • ¥15 一个服务器已经有一个系统了如果用usb再装一个系统，原来的系统会被覆盖掉吗
  • ¥15 使用esm_msa1_t12_100M_UR50S蛋白质语言模型进行零样本预测时，终端显示出了sequence handled的进度条，但是并不出结果就自动终止回到命令提示行了是怎么回事：
  • ¥15 前置放大电路与功率放大电路相连放大倍数出现问题
  • ¥30 关于<main>标签页面跳转的问题
  • ¥80 部署运行web自动化项目
  • ¥15 腾讯云如何建立同一个项目中物模型之间的联系
  • ¥30 VMware 云桌面水印如何添加
  • ¥15 用ns3仿真出5G核心网网元
  • ¥15 matlab答疑 关于海上风电的爬坡事件检测
  • ¥88 python部署量化回测异常问题