zhangzongpu 2021-05-26 21:06 采纳率: 66.7%
浏览 177
已采纳

HDP Spark插入HIVE报错

平台采用HDP搭建,使用HWC链接和操作hive。在插入hive表过程中报错(非必现,但错误出现频繁),报错信息如下图:

26-05-2021 10:31:53 CST DwdSumUser INFO - ====已存在用户处理dealAddDF====
26-05-2021 10:31:57 CST DwdSumUser INFO - 21/05/26 10:31:57 WARN TaskSetManager: Stage 245 contains a task of very large size (493 KB). The maximum recommended task size is 100 KB.
26-05-2021 10:31:57 CST DwdSumUser INFO - 21/05/26 10:31:57 WARN TaskSetManager: Stage 246 contains a task of very large size (519 KB). The maximum recommended task size is 100 KB.
26-05-2021 10:32:05 CST DwdSumUser INFO - +-------+---------+---------+--------+----+------+-------------+-------------+------------------+------------------+------------------+------------------+---+----+
26-05-2021 10:32:05 CST DwdSumUser INFO - |user_id|device_id|chip_type|province|city|county|register_date|register_hour|latest_active_date|latest_online_date|latest_active_hour|latest_online_hour| dt|hour|
26-05-2021 10:32:05 CST DwdSumUser INFO - +-------+---------+---------+--------+----+------+-------------+-------------+------------------+------------------+------------------+------------------+---+----+
26-05-2021 10:32:05 CST DwdSumUser INFO - +-------+---------+---------+--------+----+------+-------------+-------------+------------------+------------------+------------------+------------------+---+----+
26-05-2021 10:32:05 CST DwdSumUser INFO - 
26-05-2021 10:32:25 CST DwdSumUser INFO - 21/05/26 10:32:25 WARN HiveConnection: Failed to connect to node-01.hdp:10500
26-05-2021 10:32:25 CST DwdSumUser INFO - 21/05/26 10:32:25 ERROR Utils: Unable to read HiveServer2 configs from ZooKeeper
26-05-2021 10:32:25 CST DwdSumUser INFO - 21/05/26 10:32:25 ERROR HiveWarehouseDataSourceReader: Unable to read table schema
26-05-2021 10:32:25 CST DwdSumUser INFO - Exception in thread "main" java.lang.RuntimeException: java.io.IOException: java.sql.SQLException: Could not open client transport for any of the Server URI's in ZooKeeper: java.net.SocketException: Connection reset
26-05-2021 10:32:25 CST DwdSumUser INFO - 	at com.hortonworks.spark.sql.hive.llap.HiveWarehouseDataSourceReader.readSchema(HiveWarehouseDataSourceReader.java:129)
26-05-2021 10:32:25 CST DwdSumUser INFO - 	at org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation$.apply(DataSourceV2Relation.scala:56)
26-05-2021 10:32:25 CST DwdSumUser INFO - 	at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:224)
26-05-2021 10:32:25 CST DwdSumUser INFO - 	at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:164)
26-05-2021 10:32:25 CST DwdSumUser INFO - 	at com.hortonworks.spark.sql.hive.llap.HiveWarehouseSessionImpl.executeQueryInternal(HiveWarehouseSessionImpl.java:130)
26-05-2021 10:32:25 CST DwdSumUser INFO - 	at com.hortonworks.spark.sql.hive.llap.HiveWarehouseSessionImpl.executeSmart(HiveWarehouseSessionImpl.java:191)
26-05-2021 10:32:25 CST DwdSumUser INFO - 	at com.hortonworks.spark.sql.hive.llap.HiveWarehouseSessionImpl.executeQuery(HiveWarehouseSessionImpl.java:112)
26-05-2021 10:32:25 CST DwdSumUser INFO - 	at com.bsi.spark.DwdSumUser$$anonfun$main$1.apply$mcVI$sp(DwdSumUser.scala:185)
26-05-2021 10:32:25 CST DwdSumUser INFO - 	at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:160)
26-05-2021 10:32:25 CST DwdSumUser INFO - 	at com.bsi.spark.DwdSumUser$.main(DwdSumUser.scala:59)
26-05-2021 10:32:25 CST DwdSumUser INFO - 	at com.bsi.spark.DwdSumUser.main(DwdSumUser.scala)
26-05-2021 10:32:25 CST DwdSumUser INFO - 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
26-05-2021 10:32:25 CST DwdSumUser INFO - 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
26-05-2021 10:32:25 CST DwdSumUser INFO - 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
26-05-2021 10:32:25 CST DwdSumUser INFO - 	at java.lang.reflect.Method.invoke(Method.java:498)
26-05-2021 10:32:25 CST DwdSumUser INFO - 	at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
26-05-2021 10:32:25 CST DwdSumUser INFO - 	at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:904)
26-05-2021 10:32:25 CST DwdSumUser INFO - 	at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:198)
26-05-2021 10:32:25 CST DwdSumUser INFO - 	at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:228)
26-05-2021 10:32:25 CST DwdSumUser INFO - 	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:137)
26-05-2021 10:32:25 CST DwdSumUser INFO - 	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
26-05-2021 10:32:25 CST DwdSumUser INFO - Caused by: java.io.IOException: java.sql.SQLException: Could not open client transport for any of the Server URI's in ZooKeeper: java.net.SocketException: Connection reset
26-05-2021 10:32:25 CST DwdSumUser INFO - 	at org.apache.hadoop.hive.llap.LlapBaseInputFormat.getSplits(LlapBaseInputFormat.java:320)
26-05-2021 10:32:25 CST DwdSumUser INFO - 	at com.hortonworks.spark.sql.hive.llap.HiveWarehouseDataSourceReader.getTableSchema(HiveWarehouseDataSourceReader.java:109)
26-05-2021 10:32:25 CST DwdSumUser INFO - 	at com.hortonworks.spark.sql.hive.llap.HiveWarehouseDataSourceReader.readSchema(HiveWarehouseDataSourceReader.java:123)
26-05-2021 10:32:25 CST DwdSumUser INFO - 	... 20 more
26-05-2021 10:32:25 CST DwdSumUser INFO - Caused by: java.sql.SQLException: Could not open client transport for any of the Server URI's in ZooKeeper: java.net.SocketException: Connection reset
26-05-2021 10:32:25 CST DwdSumUser INFO - 	at shadehive.org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:344)
26-05-2021 10:32:25 CST DwdSumUser INFO - 	at shadehive.org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:107)
26-05-2021 10:32:25 CST DwdSumUser INFO - 	at java.sql.DriverManager.getConnection(DriverManager.java:664)
26-05-2021 10:32:25 CST DwdSumUser INFO - 	at java.sql.DriverManager.getConnection(DriverManager.java:247)
26-05-2021 10:32:25 CST DwdSumUser INFO - 	at org.apache.hadoop.hive.llap.LlapBaseInputFormat.getSplits(LlapBaseInputFormat.java:280)
26-05-2021 10:32:25 CST DwdSumUser INFO - 	... 22 more
26-05-2021 10:32:25 CST DwdSumUser INFO - Caused by: org.apache.thrift.transport.TTransportException: java.net.SocketException: Connection reset
26-05-2021 10:32:25 CST DwdSumUser INFO - 	at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
26-05-2021 10:32:25 CST DwdSumUser INFO - 	at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
26-05-2021 10:32:25 CST DwdSumUser INFO - 	at org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:178)
26-05-2021 10:32:25 CST DwdSumUser INFO - 	at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:305)
26-05-2021 10:32:25 CST DwdSumUser INFO - 	at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:38)
26-05-2021 10:32:25 CST DwdSumUser INFO - 	at shadehive.org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:431)
26-05-2021 10:32:25 CST DwdSumUser INFO - 	at shadehive.org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:312)
26-05-2021 10:32:25 CST DwdSumUser INFO - 	... 26 more
26-05-2021 10:32:25 CST DwdSumUser INFO - Caused by: java.net.SocketException: Connection reset
26-05-2021 10:32:25 CST DwdSumUser INFO - 	at java.net.SocketInputStream.read(SocketInputStream.java:210)
26-05-2021 10:32:25 CST DwdSumUser INFO - 	at java.net.SocketInputStream.read(SocketInputStream.java:141)
26-05-2021 10:32:25 CST DwdSumUser INFO - 	at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
26-05-2021 10:32:25 CST DwdSumUser INFO - 	at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
26-05-2021 10:32:25 CST DwdSumUser INFO - 	at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
26-05-2021 10:32:25 CST DwdSumUser INFO - 	at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
26-05-2021 10:32:25 CST DwdSumUser INFO - 	... 32 more
26-05-2021 10:32:27 CST DwdSumUser INFO - Process with id 118140 completed unsuccessfully in 803 seconds.
26-05-2021 10:32:27 CST DwdSumUser ERROR - Job run failed!
java.lang.RuntimeException: azkaban.jobExecutor.utils.process.ProcessFailureException: Process exited with code 1
	at azkaban.jobExecutor.ProcessJob.run(ProcessJob.java:312)
	at azkaban.execapp.JobRunner.runJob(JobRunner.java:830)
	at azkaban.execapp.JobRunner.doRun(JobRunner.java:607)
	at azkaban.execapp.JobRunner.run(JobRunner.java:568)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: azkaban.jobExecutor.utils.process.ProcessFailureException: Process exited with code 1
	at azkaban.jobExecutor.utils.process.AzkabanProcess.run(AzkabanProcess.java:125)
	at azkaban.jobExecutor.ProcessJob.run(ProcessJob.java:304)
	... 8 more
26-05-2021 10:32:27 CST DwdSumUser ERROR - azkaban.jobExecutor.utils.process.ProcessFailureException: Process exited with code 1 cause: azkaban.jobExecutor.utils.process.ProcessFailureException: Process exited with code 1
26-05-2021 10:32:27 CST DwdSumUser INFO - Finishing job DwdSumUser at 1621996347500 with status FAILED
FirstPrevious0NextLast
  • 写回答

7条回答

      报告相同问题?

      相关推荐 更多相似问题

      悬赏问题

      • ¥15 内存管理的一段代码不是很理解
      • ¥20 打开anaconda时卡在Loading applications无法进入界面
      • ¥15 网页超时时间设置失效
      • ¥15 有关绿色信贷毕业论文的问题
      • ¥30 关于#机器人#的问题,如何解决?
      • ¥15 求MATLAB函数ScalarLayerDisplay的代码
      • ¥15 安卓如何自动执行检测到的NFC标签,无需再点确认
      • ¥15 pyHM库mouse模块的ValueError错误
      • ¥15 python opencv 摄像头 传感器
      • ¥30 eMMC&Android&C&Linux