平台采用HDP搭建，使用HWC链接和操作hive。在插入hive表过程中报错（非必现，但错误出现频繁），报错信息如下图：
26-05-2021 10:31:53 CST DwdSumUser INFO - ====已存在用户处理dealAddDF====
26-05-2021 10:31:57 CST DwdSumUser INFO - 21/05/26 10:31:57 WARN TaskSetManager: Stage 245 contains a task of very large size (493 KB). The maximum recommended task size is 100 KB.
26-05-2021 10:31:57 CST DwdSumUser INFO - 21/05/26 10:31:57 WARN TaskSetManager: Stage 246 contains a task of very large size (519 KB). The maximum recommended task size is 100 KB.
26-05-2021 10:32:05 CST DwdSumUser INFO - +-------+---------+---------+--------+----+------+-------------+-------------+------------------+------------------+------------------+------------------+---+----+
26-05-2021 10:32:05 CST DwdSumUser INFO - |user_id|device_id|chip_type|province|city|county|register_date|register_hour|latest_active_date|latest_online_date|latest_active_hour|latest_online_hour| dt|hour|
26-05-2021 10:32:05 CST DwdSumUser INFO - +-------+---------+---------+--------+----+------+-------------+-------------+------------------+------------------+------------------+------------------+---+----+
26-05-2021 10:32:05 CST DwdSumUser INFO - +-------+---------+---------+--------+----+------+-------------+-------------+------------------+------------------+------------------+------------------+---+----+
26-05-2021 10:32:05 CST DwdSumUser INFO -
26-05-2021 10:32:25 CST DwdSumUser INFO - 21/05/26 10:32:25 WARN HiveConnection: Failed to connect to node-01.hdp:10500
26-05-2021 10:32:25 CST DwdSumUser INFO - 21/05/26 10:32:25 ERROR Utils: Unable to read HiveServer2 configs from ZooKeeper
26-05-2021 10:32:25 CST DwdSumUser INFO - 21/05/26 10:32:25 ERROR HiveWarehouseDataSourceReader: Unable to read table schema
26-05-2021 10:32:25 CST DwdSumUser INFO - Exception in thread "main" java.lang.RuntimeException: java.io.IOException: java.sql.SQLException: Could not open client transport for any of the Server URI's in ZooKeeper: java.net.SocketException: Connection reset
26-05-2021 10:32:25 CST DwdSumUser INFO - at com.hortonworks.spark.sql.hive.llap.HiveWarehouseDataSourceReader.readSchema(HiveWarehouseDataSourceReader.java:129)
26-05-2021 10:32:25 CST DwdSumUser INFO - at org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation$.apply(DataSourceV2Relation.scala:56)
26-05-2021 10:32:25 CST DwdSumUser INFO - at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:224)
26-05-2021 10:32:25 CST DwdSumUser INFO - at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:164)
26-05-2021 10:32:25 CST DwdSumUser INFO - at com.hortonworks.spark.sql.hive.llap.HiveWarehouseSessionImpl.executeQueryInternal(HiveWarehouseSessionImpl.java:130)
26-05-2021 10:32:25 CST DwdSumUser INFO - at com.hortonworks.spark.sql.hive.llap.HiveWarehouseSessionImpl.executeSmart(HiveWarehouseSessionImpl.java:191)
26-05-2021 10:32:25 CST DwdSumUser INFO - at com.hortonworks.spark.sql.hive.llap.HiveWarehouseSessionImpl.executeQuery(HiveWarehouseSessionImpl.java:112)
26-05-2021 10:32:25 CST DwdSumUser INFO - at com.bsi.spark.DwdSumUser$$anonfun$main$1.apply$mcVI$sp(DwdSumUser.scala:185)
26-05-2021 10:32:25 CST DwdSumUser INFO - at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:160)
26-05-2021 10:32:25 CST DwdSumUser INFO - at com.bsi.spark.DwdSumUser$.main(DwdSumUser.scala:59)
26-05-2021 10:32:25 CST DwdSumUser INFO - at com.bsi.spark.DwdSumUser.main(DwdSumUser.scala)
26-05-2021 10:32:25 CST DwdSumUser INFO - at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
26-05-2021 10:32:25 CST DwdSumUser INFO - at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
26-05-2021 10:32:25 CST DwdSumUser INFO - at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
26-05-2021 10:32:25 CST DwdSumUser INFO - at java.lang.reflect.Method.invoke(Method.java:498)
26-05-2021 10:32:25 CST DwdSumUser INFO - at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
26-05-2021 10:32:25 CST DwdSumUser INFO - at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:904)
26-05-2021 10:32:25 CST DwdSumUser INFO - at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:198)
26-05-2021 10:32:25 CST DwdSumUser INFO - at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:228)
26-05-2021 10:32:25 CST DwdSumUser INFO - at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:137)
26-05-2021 10:32:25 CST DwdSumUser INFO - at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
26-05-2021 10:32:25 CST DwdSumUser INFO - Caused by: java.io.IOException: java.sql.SQLException: Could not open client transport for any of the Server URI's in ZooKeeper: java.net.SocketException: Connection reset
26-05-2021 10:32:25 CST DwdSumUser INFO - at org.apache.hadoop.hive.llap.LlapBaseInputFormat.getSplits(LlapBaseInputFormat.java:320)
26-05-2021 10:32:25 CST DwdSumUser INFO - at com.hortonworks.spark.sql.hive.llap.HiveWarehouseDataSourceReader.getTableSchema(HiveWarehouseDataSourceReader.java:109)
26-05-2021 10:32:25 CST DwdSumUser INFO - at com.hortonworks.spark.sql.hive.llap.HiveWarehouseDataSourceReader.readSchema(HiveWarehouseDataSourceReader.java:123)
26-05-2021 10:32:25 CST DwdSumUser INFO - ... 20 more
26-05-2021 10:32:25 CST DwdSumUser INFO - Caused by: java.sql.SQLException: Could not open client transport for any of the Server URI's in ZooKeeper: java.net.SocketException: Connection reset
26-05-2021 10:32:25 CST DwdSumUser INFO - at shadehive.org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:344)
26-05-2021 10:32:25 CST DwdSumUser INFO - at shadehive.org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:107)
26-05-2021 10:32:25 CST DwdSumUser INFO - at java.sql.DriverManager.getConnection(DriverManager.java:664)
26-05-2021 10:32:25 CST DwdSumUser INFO - at java.sql.DriverManager.getConnection(DriverManager.java:247)
26-05-2021 10:32:25 CST DwdSumUser INFO - at org.apache.hadoop.hive.llap.LlapBaseInputFormat.getSplits(LlapBaseInputFormat.java:280)
26-05-2021 10:32:25 CST DwdSumUser INFO - ... 22 more
26-05-2021 10:32:25 CST DwdSumUser INFO - Caused by: org.apache.thrift.transport.TTransportException: java.net.SocketException: Connection reset
26-05-2021 10:32:25 CST DwdSumUser INFO - at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
26-05-2021 10:32:25 CST DwdSumUser INFO - at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
26-05-2021 10:32:25 CST DwdSumUser INFO - at org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:178)
26-05-2021 10:32:25 CST DwdSumUser INFO - at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:305)
26-05-2021 10:32:25 CST DwdSumUser INFO - at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:38)
26-05-2021 10:32:25 CST DwdSumUser INFO - at shadehive.org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:431)
26-05-2021 10:32:25 CST DwdSumUser INFO - at shadehive.org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:312)
26-05-2021 10:32:25 CST DwdSumUser INFO - ... 26 more
26-05-2021 10:32:25 CST DwdSumUser INFO - Caused by: java.net.SocketException: Connection reset
26-05-2021 10:32:25 CST DwdSumUser INFO - at java.net.SocketInputStream.read(SocketInputStream.java:210)
26-05-2021 10:32:25 CST DwdSumUser INFO - at java.net.SocketInputStream.read(SocketInputStream.java:141)
26-05-2021 10:32:25 CST DwdSumUser INFO - at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
26-05-2021 10:32:25 CST DwdSumUser INFO - at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
26-05-2021 10:32:25 CST DwdSumUser INFO - at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
26-05-2021 10:32:25 CST DwdSumUser INFO - at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
26-05-2021 10:32:25 CST DwdSumUser INFO - ... 32 more
26-05-2021 10:32:27 CST DwdSumUser INFO - Process with id 118140 completed unsuccessfully in 803 seconds.
26-05-2021 10:32:27 CST DwdSumUser ERROR - Job run failed!
java.lang.RuntimeException: azkaban.jobExecutor.utils.process.ProcessFailureException: Process exited with code 1
at azkaban.jobExecutor.ProcessJob.run(ProcessJob.java:312)
at azkaban.execapp.JobRunner.runJob(JobRunner.java:830)
at azkaban.execapp.JobRunner.doRun(JobRunner.java:607)
at azkaban.execapp.JobRunner.run(JobRunner.java:568)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: azkaban.jobExecutor.utils.process.ProcessFailureException: Process exited with code 1
at azkaban.jobExecutor.utils.process.AzkabanProcess.run(AzkabanProcess.java:125)
at azkaban.jobExecutor.ProcessJob.run(ProcessJob.java:304)
... 8 more
26-05-2021 10:32:27 CST DwdSumUser ERROR - azkaban.jobExecutor.utils.process.ProcessFailureException: Process exited with code 1 cause: azkaban.jobExecutor.utils.process.ProcessFailureException: Process exited with code 1
26-05-2021 10:32:27 CST DwdSumUser INFO - Finishing job DwdSumUser at 1621996347500 with status FAILED
FirstPrevious0NextLast
