book19897264 2022-09-02 16:58 采纳率: 0%
浏览 74
已结题

sparksql无法访问多分隔符的外部表

问题遇到的现象和发生背景

sparksql无法访问多分隔符的外部表

问题相关代码,请勿粘贴截图

测试代码

import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.security.UserGroupInformation
import org.apache.spark.sql.{DataFrame, SparkSession}
import java.io.IOException
object TEST {
  def main(args: Array[String]): Unit = {

    System.setProperty("hadoop.home.dir","C:\\winutil")
   doKerberosAuth(new Configuration())

val spark = SparkSession
  .builder()
  .appName("TEST")
  .master("local[*]")
  .enableHiveSupport()
  .getOrCreate()
    import spark.implicits._

    spark.sql("add jar C:\\DataLake\\TEST_yarnclient\\src\\main\\resources\\hive-contrib.jar")


    var a = "select * from udm_rdz_lat.rdz_la_zbnfpf_lat"
    var b = "select * from udm_rdz_source.rdz_la_zbnfpf"
    val aa: DataFrame = spark.sql(a)
    aa.show()
    val bb: DataFrame = spark.sql(b)
    bb.show()
  spark.close()
  }

  def doKerberosAuth(conf: Configuration): Unit = {
    import scala.collection.immutable._
    System.setProperty("java.security.krb5.conf", "C:\\DataLake\\TEST_yarnclient\\src\\main\\resources\\krb5.conf")
    conf.set("hadoop.security.authentication", "Kerberos")
    UserGroupInformation.setConfiguration(conf)
    try {
      UserGroupInformation.loginUserFromKeytab("test@QAALICO.CORP", "C:\\DataLake\\TEST_yarnclient\\src\\main\\resources\\udm.keytab")
    }
    catch {
      case e: IOException => e.printStackTrace()
    }
  }
}

pom文件

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>org.example</groupId>
    <artifactId>TEST</artifactId>
    <version>1.0-SNAPSHOT</version>

    <dependencies>
                <dependency>
                    <groupId>org.scala-lang</groupId>
                    <artifactId>scala-library</artifactId>
                    <version>2.11.12</version>
                </dependency>
                <dependency>
                    <groupId>org.scala-lang</groupId>
                    <artifactId>scala-compiler</artifactId>
                    <version>2.11.12</version>
                </dependency>

    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-sql_2.11</artifactId>
        <version>2.3.2</version>
        <scope>provided</scope>

    </dependency>

        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-hive_2.11</artifactId>
            <version>2.3.2</version>
            <!--<scope>provided</scope> -->
        </dependency>

        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-hive-thriftserver_2.11</artifactId>
            <version>2.3.2</version>
        </dependency>


        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-yarn_2.10</artifactId>
            <version>2.2.0</version>
            <scope>provided</scope>

        </dependency>

<!--hive-contrib.jar-->
        <dependency>
            <groupId>aaaa</groupId>
            <artifactId>aaaa-bbbb</artifactId>
            <version>1.1.1</version>
        </dependency>



    </dependencies>



    <properties>
        <maven.compiler.source>8</maven.compiler.source>
        <maven.compiler.target>8</maven.compiler.target>
    </properties>

</project>

运行结果及报错内容

主要报错内容:
ERROR Executor: Exception in task 0.0 in stage 2.0 (TID 3)
java.lang.NoSuchMethodError: org.apache.hadoop.hive.serde2.lazy.LazyStruct.parseMultiDelimit([B[BI)V

以下是详细日志:
log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
22/09/02 16:36:38 INFO SparkContext: Running Spark version 2.3.2
22/09/02 16:36:38 INFO SparkContext: Submitted application: TEST
22/09/02 16:36:38 INFO SecurityManager: Changing view acls to: zilong,test
22/09/02 16:36:38 INFO SecurityManager: Changing modify acls to: zilong,test
22/09/02 16:36:38 INFO SecurityManager: Changing view acls groups to:
22/09/02 16:36:38 INFO SecurityManager: Changing modify acls groups to:
22/09/02 16:36:38 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(zilong, test); groups with view permissions: Set(); users with modify permissions: Set(zilong, test); groups with modify permissions: Set()
22/09/02 16:36:39 INFO Utils: Successfully started service 'sparkDriver' on port 63485.
22/09/02 16:36:39 INFO SparkEnv: Registering MapOutputTracker
22/09/02 16:36:39 INFO SparkEnv: Registering BlockManagerMaster
22/09/02 16:36:39 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
22/09/02 16:36:39 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
22/09/02 16:36:39 INFO DiskBlockManager: Created local directory at C:\Users\zilong\AppData\Local\Temp\1\blockmgr-284035c7-994f-4819-b05c-b81ab636b5c8
22/09/02 16:36:39 INFO MemoryStore: MemoryStore started with capacity 1980.0 MB
22/09/02 16:36:39 INFO SparkEnv: Registering OutputCommitCoordinator
22/09/02 16:36:39 INFO Utils: Successfully started service 'SparkUI' on port 4040.
22/09/02 16:36:39 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://a01a01b02.alico.corp:4040
22/09/02 16:36:39 INFO Executor: Starting executor ID driver on host localhost
22/09/02 16:36:39 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 63526.
22/09/02 16:36:39 INFO NettyBlockTransferService: Server created on a01a01b02.alico.corp:63526
22/09/02 16:36:39 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
22/09/02 16:36:39 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, a01a01b02.alico.corp, 63526, None)
22/09/02 16:36:39 INFO BlockManagerMasterEndpoint: Registering block manager a01a01b02.alico.corp:63526 with 1980.0 MB RAM, BlockManagerId(driver, a01a01b02.alico.corp, 63526, None)
22/09/02 16:36:39 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, a01a01b02.alico.corp, 63526, None)
22/09/02 16:36:39 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, a01a01b02.alico.corp, 63526, None)
22/09/02 16:36:39 INFO SharedState: loading hive config file: file:/C:/DataLake/TEST_yarnclient/target/classes/hive-site.xml
22/09/02 16:36:39 INFO SharedState: spark.sql.warehouse.dir is not set, but hive.metastore.warehouse.dir is set. Setting spark.sql.warehouse.dir to the value of hive.metastore.warehouse.dir ('/warehouse/tablespace/managed/hive').
22/09/02 16:36:39 INFO SharedState: Warehouse path is '/warehouse/tablespace/managed/hive'.
22/09/02 16:36:40 INFO StateStoreCoordinatorRef: Registered StateStoreCoordinator endpoint
22/09/02 16:36:40 INFO HiveUtils: Initializing HiveMetastoreConnection version 1.2.1 using Spark classes.
22/09/02 16:36:41 WARN HiveConf: HiveConf of name hive.tez.cartesian-product.enabled does not exist
22/09/02 16:36:41 WARN HiveConf: HiveConf of name hive.metastore.warehouse.external.dir does not exist
22/09/02 16:36:41 WARN HiveConf: HiveConf of name hive.server2.webui.use.ssl does not exist
22/09/02 16:36:41 WARN HiveConf: HiveConf of name hive.heapsize does not exist
22/09/02 16:36:41 WARN HiveConf: HiveConf of name hive.server2.webui.port does not exist
22/09/02 16:36:41 WARN HiveConf: HiveConf of name hive.materializedview.rewriting.incremental does not exist
22/09/02 16:36:41 WARN HiveConf: HiveConf of name hive.server2.webui.cors.allowed.headers does not exist
22/09/02 16:36:41 WARN HiveConf: HiveConf of name hive.driver.parallel.compilation does not exist
22/09/02 16:36:41 WARN HiveConf: HiveConf of name hive.tez.bucket.pruning does not exist
22/09/02 16:36:41 WARN HiveConf: HiveConf of name hive.hook.proto.base-directory does not exist
22/09/02 16:36:41 WARN HiveConf: HiveConf of name hive.load.data.owner does not exist
22/09/02 16:36:41 WARN HiveConf: HiveConf of name hive.execution.mode does not exist
22/09/02 16:36:41 WARN HiveConf: HiveConf of name hive.vectorized.adaptor.usage.mode does not exist
22/09/02 16:36:41 WARN HiveConf: HiveConf of name hive.service.metrics.codahale.reporter.classes does not exist
22/09/02 16:36:41 WARN HiveConf: HiveConf of name hive.strict.managed.tables does not exist
22/09/02 16:36:41 WARN HiveConf: HiveConf of name hive.create.as.insert.only does not exist
22/09/02 16:36:41 WARN HiveConf: HiveConf of name hive.optimize.dynamic.partition.hashjoin does not exist
22/09/02 16:36:41 WARN HiveConf: HiveConf of name hive.server2.webui.enable.cors does not exist
22/09/02 16:36:41 WARN HiveConf: HiveConf of name hive.metastore.db.type does not exist
22/09/02 16:36:41 WARN HiveConf: HiveConf of name hive.txn.strict.locking.mode does not exist
22/09/02 16:36:41 WARN HiveConf: HiveConf of name hive.metastore.transactional.event.listeners does not exist
22/09/02 16:36:41 WARN HiveConf: HiveConf of name hive.tez.input.generate.consistent.splits does not exist
22/09/02 16:36:41 INFO metastore: Trying to connect to metastore with URI thrift://a01.alico.corp:9083
22/09/02 16:36:41 INFO metastore: Connected to metastore.
22/09/02 16:36:45 WARN DomainSocketFactory: The short-circuit local reads feature cannot be used because UNIX Domain sockets are not available on Windows.
22/09/02 16:36:46 INFO SessionState: Created local directory: C:/Users/ZHANGS1/AppData/Local/Temp/1/f98a580-2abc-4c1d-8fd0-ebb28e3a6dd2_resources
22/09/02 16:36:46 INFO SessionState: Created HDFS directory: /tmp/hive/test/f98a580-2abc-4c1d-8fd0-ebb28e3a6dd2
22/09/02 16:36:46 INFO SessionState: Created local directory: C:/Users/ZHANGS
1/AppData/Local/Temp/1/zilong/f98a580-2abc-4c1d-8fd0-ebb28e3a6dd2
22/09/02 16:36:46 INFO SessionState: Created HDFS directory: /tmp/hive/test/f98a580-2abc-4c1d-8fd0-ebb28e3a6dd2/_tmp_space.db
22/09/02 16:36:46 INFO HiveClientImpl: Warehouse location for Hive client (version 1.2.2) is /warehouse/tablespace/managed/hive
22/09/02 16:36:48 INFO JniBasedUnixGroupsMapping: Error getting groups for test: Unknown error.
22/09/02 16:36:48 WARN UserGroupInformation: No groups available for user test
22/09/02 16:36:48 WARN UserGroupInformation: No groups available for user test
22/09/02 16:36:48 INFO SessionState: Added [/C:/DataLake/TEST_yarnclient/src/main/resources/hive-contrib.jar] to class path
22/09/02 16:36:48 INFO SessionState: Added resources: [C:\DataLake\TEST_yarnclient\src\main\resources\hive-contrib.jar]
22/09/02 16:36:48 INFO SparkContext: Added JAR C:\DataLake\TEST_yarnclient\src\main\resources\hive-contrib.jar at spark://a01a01b02.alico.corp:63485/jars/hive-contrib.jar with timestamp 1662107808686
22/09/02 16:36:49 INFO CodeGenerator: Code generated in 312.7428 ms
22/09/02 16:36:50 INFO CodeGenerator: Code generated in 66.424 ms
22/09/02 16:36:50 INFO CodeGenerator: Code generated in 29.4999 ms
22/09/02 16:36:50 INFO ContextCleaner: Cleaned accumulator 0
22/09/02 16:36:50 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 387.9 KB, free 1979.6 MB)
22/09/02 16:36:50 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 37.5 KB, free 1979.6 MB)
22/09/02 16:36:50 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on a01a01b02.alico.corp:63526 (size: 37.5 KB, free: 1980.0 MB)
22/09/02 16:36:50 INFO SparkContext: Created broadcast 0 from
22/09/02 16:36:50 INFO PerfLogger:
22/09/02 16:36:50 INFO deprecation: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
22/09/02 16:36:50 INFO OrcInputFormat: FooterCacheHitRatio: 0/0
22/09/02 16:36:50 INFO PerfLogger: </PERFLOG method=OrcGetSplits start=1662107810801 end=1662107810925 duration=124 from=org.apache.hadoop.hive.ql.io.orc.ReaderImpl>
22/09/02 16:36:50 INFO SparkContext: Starting job: show at TEST.scala:34
22/09/02 16:36:50 INFO DAGScheduler: Registering RDD 5 (show at TEST.scala:34)
22/09/02 16:36:50 INFO DAGScheduler: Got job 0 (show at TEST.scala:34) with 1 output partitions
22/09/02 16:36:50 INFO DAGScheduler: Final stage: ResultStage 1 (show at TEST.scala:34)
22/09/02 16:36:50 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 0)
22/09/02 16:36:50 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 0)
22/09/02 16:36:50 INFO DAGScheduler: Submitting ShuffleMapStage 0 (MapPartitionsRDD[5] at show at TEST.scala:34), which has no missing parents
22/09/02 16:36:51 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 20.6 KB, free 1979.6 MB)
22/09/02 16:36:51 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 8.4 KB, free 1979.6 MB)
22/09/02 16:36:51 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on a01a01b02.alico.corp:63526 (size: 8.4 KB, free: 1980.0 MB)
22/09/02 16:36:51 INFO SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:1039
22/09/02 16:36:51 INFO DAGScheduler: Submitting 2 missing tasks from ShuffleMapStage 0 (MapPartitionsRDD[5] at show at TEST.scala:34) (first 15 tasks are for partitions Vector(0, 1))
22/09/02 16:36:51 INFO TaskSchedulerImpl: Adding task set 0.0 with 2 tasks
22/09/02 16:36:51 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, executor driver, partition 0, ANY, 8027 bytes)
22/09/02 16:36:51 INFO TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, localhost, executor driver, partition 1, ANY, 8027 bytes)
22/09/02 16:36:51 INFO Executor: Running task 1.0 in stage 0.0 (TID 1)
22/09/02 16:36:51 INFO Executor: Running task 0.0 in stage 0.0 (TID 0)
22/09/02 16:36:51 INFO Executor: Fetching spark://a01a01b02.alico.corp:63485/jars/hive-contrib.jar with timestamp 1662107808686
22/09/02 16:36:51 INFO TransportClientFactory: Successfully created connection to a01a01b02.alico.corp/10.165.63.245:63485 after 39 ms (0 ms spent in bootstraps)
22/09/02 16:36:51 INFO Utils: Fetching spark://a01a01b02.alico.corp:63485/jars/hive-contrib.jar to C:\Users\zilong\AppData\Local\Temp\1\spark-7917d05d-461f-4f96-8b42-03d7a2ce30d5\userFiles-8b4db58f-bd2b-42c8-b6a3-8f81ef07a2e6\fetchFileTemp3333765197903670028.tmp
22/09/02 16:36:51 INFO Executor: Adding file:/C:/Users/zilong/AppData/Local/Temp/1/spark-7917d05d-461f-4f96-8b42-03d7a2ce30d5/userFiles-8b4db58f-bd2b-42c8-b6a3-8f81ef07a2e6/hive-contrib.jar to class loader
22/09/02 16:36:51 INFO HadoopRDD: Input split: hdfs://a01.alico.corp:8020/warehouse/tablespace/external/hive/test_rdz_lat.db/rdz_la_zbnfpf_lat/part-00001-74822f38-c2a5-496b-86fa-a8e4836e4bde-c000.snappy.orc:0+2361
22/09/02 16:36:51 INFO HadoopRDD: Input split: hdfs://a01.alico.corp:8020/warehouse/tablespace/external/hive/test_rdz_lat.db/rdz_la_zbnfpf_lat/part-00000-74822f38-c2a5-496b-86fa-a8e4836e4bde-c000.snappy.orc:0+2363
22/09/02 16:36:52 INFO OrcRawRecordMerger: min key = null, max key = null
22/09/02 16:36:52 INFO OrcRawRecordMerger: min key = null, max key = null
22/09/02 16:36:52 INFO ReaderImpl: Reading ORC rows from hdfs://a01.alico.corp:8020/warehouse/tablespace/external/hive/test_rdz_lat.db/rdz_la_zbnfpf_lat/part-00001-74822f38-c2a5-496b-86fa-a8e4836e4bde-c000.snappy.orc with {include: [true], offset: 0, length: 9223372036854775807}
22/09/02 16:36:52 INFO ReaderImpl: Reading ORC rows from hdfs://a01.alico.corp:8020/warehouse/tablespace/external/hive/test_rdz_lat.db/rdz_la_zbnfpf_lat/part-00000-74822f38-c2a5-496b-86fa-a8e4836e4bde-c000.snappy.orc with {include: [true], offset: 0, length: 9223372036854775807}
22/09/02 16:36:52 INFO CodeGenerator: Code generated in 21.6888 ms
22/09/02 16:36:52 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0). 1481 bytes result sent to driver
22/09/02 16:36:52 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 1168 ms on localhost (executor driver) (1/2)
22/09/02 16:36:52 INFO Executor: Finished task 1.0 in stage 0.0 (TID 1). 1438 bytes result sent to driver
22/09/02 16:36:52 INFO TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 1213 ms on localhost (executor driver) (2/2)
22/09/02 16:36:52 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool
22/09/02 16:36:52 INFO DAGScheduler: ShuffleMapStage 0 (show at TEST.scala:34) finished in 1.334 s
22/09/02 16:36:52 INFO DAGScheduler: looking for newly runnable stages
22/09/02 16:36:52 INFO DAGScheduler: running: Set()
22/09/02 16:36:52 INFO DAGScheduler: waiting: Set(ResultStage 1)
22/09/02 16:36:52 INFO DAGScheduler: failed: Set()
22/09/02 16:36:52 INFO DAGScheduler: Submitting ResultStage 1 (MapPartitionsRDD[9] at show at TEST.scala:34), which has no missing parents
22/09/02 16:36:52 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 15.5 KB, free 1979.5 MB)
22/09/02 16:36:52 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 5.6 KB, free 1979.5 MB)
22/09/02 16:36:52 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on a01a01b02.alico.corp:63526 (size: 5.6 KB, free: 1979.9 MB)
22/09/02 16:36:52 INFO SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:1039
22/09/02 16:36:52 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 1 (MapPartitionsRDD[9] at show at TEST.scala:34) (first 15 tasks are for partitions Vector(0))
22/09/02 16:36:52 INFO TaskSchedulerImpl: Adding task set 1.0 with 1 tasks
22/09/02 16:36:52 INFO TaskSetManager: Starting task 0.0 in stage 1.0 (TID 2, localhost, executor driver, partition 0, ANY, 7754 bytes)
22/09/02 16:36:52 INFO Executor: Running task 0.0 in stage 1.0 (TID 2)
22/09/02 16:36:52 INFO ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
22/09/02 16:36:52 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 7 ms
22/09/02 16:36:52 INFO Executor: Finished task 0.0 in stage 1.0 (TID 2). 1674 bytes result sent to driver
22/09/02 16:36:52 INFO TaskSetManager: Finished task 0.0 in stage 1.0 (TID 2) in 43 ms on localhost (executor driver) (1/1)
22/09/02 16:36:52 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool
22/09/02 16:36:52 INFO DAGScheduler: ResultStage 1 (show at TEST.scala:34) finished in 0.066 s
22/09/02 16:36:52 INFO DAGScheduler: Job 0 finished: show at TEST.scala:34, took 1.473738 s

|HIST_ID|CLNTPFX|CLNTCOY| CLNTNUM|SEQNUM| CRTDATE|VALIDFLAG|BNFIDENT|USER_PROFILE| JOB_NAME| DATIME|ETL_DATATYPE| ETL_DATATIME| DT_DATATIME|BATCHID|CRTDATE_tr| log_id| test_etl_datatime|

| 12| CN| 9|00989323| 10|20180802| 2| 1| ZHANGS|QPADEV001G|2018-08-02 15:47:...| U|2022-06-16 16:14:00|2022-06-15 20:33:00| 0|2018-08-02|rdz_la_zbnfpf1655...|2022-06-20 15:59:13|

22/09/02 16:36:52 INFO CodeGenerator: Code generated in 14.412 ms
22/09/02 16:36:52 INFO CodeGenerator: Code generated in 28.6352 ms
22/09/02 16:36:52 INFO CodeGenerator: Code generated in 17.2243 ms
22/09/02 16:36:53 INFO MemoryStore: Block broadcast_3 stored as values in memory (estimated size 387.7 KB, free 1979.2 MB)
22/09/02 16:36:53 INFO MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 37.3 KB, free 1979.1 MB)
22/09/02 16:36:53 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on a01a01b02.alico.corp:63526 (size: 37.3 KB, free: 1979.9 MB)
22/09/02 16:36:53 INFO SparkContext: Created broadcast 3 from
22/09/02 16:36:53 INFO DFSClient: Created HDFS_DELEGATION_TOKEN token 1374659 for test on 10.164.29.130:8020
22/09/02 16:36:53 INFO TokenCache: Got dt for hdfs://a01.alico.corp:8020; Kind: HDFS_DELEGATION_TOKEN, Service: 10.164.29.130:8020, Ident: (HDFS_DELEGATION_TOKEN token 1374659 for test)
22/09/02 16:36:53 INFO FileInputFormat: Total input paths to process : 1
22/09/02 16:36:53 INFO SparkContext: Starting job: show at TEST.scala:37
22/09/02 16:36:53 INFO DAGScheduler: Registering RDD 15 (show at TEST.scala:37)
22/09/02 16:36:53 INFO DAGScheduler: Got job 1 (show at TEST.scala:37) with 1 output partitions
22/09/02 16:36:53 INFO DAGScheduler: Final stage: ResultStage 3 (show at TEST.scala:37)
22/09/02 16:36:53 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 2)
22/09/02 16:36:53 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 2)
22/09/02 16:36:53 INFO DAGScheduler: Submitting ShuffleMapStage 2 (MapPartitionsRDD[15] at show at TEST.scala:37), which has no missing parents
22/09/02 16:36:53 INFO MemoryStore: Block broadcast_4 stored as values in memory (estimated size 19.6 KB, free 1979.1 MB)
22/09/02 16:36:53 INFO MemoryStore: Block broadcast_4_piece0 stored as bytes in memory (estimated size 8.2 KB, free 1979.1 MB)
22/09/02 16:36:53 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on a01a01b02.alico.corp:63526 (size: 8.2 KB, free: 1979.9 MB)
22/09/02 16:36:53 INFO SparkContext: Created broadcast 4 from broadcast at DAGScheduler.scala:1039
22/09/02 16:36:53 INFO DAGScheduler: Submitting 1 missing tasks from ShuffleMapStage 2 (MapPartitionsRDD[15] at show at TEST.scala:37) (first 15 tasks are for partitions Vector(0))
22/09/02 16:36:53 INFO TaskSchedulerImpl: Adding task set 2.0 with 1 tasks
22/09/02 16:36:53 INFO TaskSetManager: Starting task 0.0 in stage 2.0 (TID 3, localhost, executor driver, partition 0, ANY, 7950 bytes)
22/09/02 16:36:53 INFO Executor: Running task 0.0 in stage 2.0 (TID 3)
22/09/02 16:36:53 INFO HadoopRDD: Input split: hdfs://a01.alico.corp:8020/warehouse/tablespace/external/hive/test_rdz_source.db/rdz_la_zbnfpf/zbnfpf.txt:0+2880
22/09/02 16:36:53 INFO CodeGenerator: Code generated in 15.6463 ms
22/09/02 16:36:53 ERROR Executor: Exception in task 0.0 in stage 2.0 (TID 3)
java.lang.NoSuchMethodError: org.apache.hadoop.hive.serde2.lazy.LazyStruct.parseMultiDelimit([B[BI)V
at org.apache.hadoop.hive.contrib.serde2.MultiDelimitSerDe.doDeserialize(MultiDelimitSerDe.java:164)
at org.apache.hadoop.hive.serde2.AbstractEncodingAwareSerDe.deserialize(AbstractEncodingAwareSerDe.java:71)
at org.apache.spark.sql.hive.HadoopTableReader$$anonfun$fillObject$2.apply(TableReader.scala:435)
at org.apache.spark.sql.hive.HadoopTableReader$$anonfun$fillObject$2.apply(TableReader.scala:434)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:410)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:410)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$10$$anon$1.hasNext(WholeStageCodegenExec.scala:614)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)
at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:125)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
at org.apache.spark.scheduler.Task.run(Task.scala:109)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
22/09/02 16:36:53 WARN TaskSetManager: Lost task 0.0 in stage 2.0 (TID 3, localhost, executor driver): java.lang.NoSuchMethodError: org.apache.hadoop.hive.serde2.lazy.LazyStruct.parseMultiDelimit([B[BI)V
at org.apache.hadoop.hive.contrib.serde2.MultiDelimitSerDe.doDeserialize(MultiDelimitSerDe.java:164)
at org.apache.hadoop.hive.serde2.AbstractEncodingAwareSerDe.deserialize(AbstractEncodingAwareSerDe.java:71)
at org.apache.spark.sql.hive.HadoopTableReader$$anonfun$fillObject$2.apply(TableReader.scala:435)
at org.apache.spark.sql.hive.HadoopTableReader$$anonfun$fillObject$2.apply(TableReader.scala:434)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:410)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:410)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$10$$anon$1.hasNext(WholeStageCodegenExec.scala:614)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)
at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:125)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
at org.apache.spark.scheduler.Task.run(Task.scala:109)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

22/09/02 16:36:53 ERROR TaskSetManager: Task 0 in stage 2.0 failed 1 times; aborting job
22/09/02 16:36:53 INFO TaskSchedulerImpl: Removed TaskSet 2.0, whose tasks have all completed, from pool
22/09/02 16:36:53 INFO TaskSchedulerImpl: Cancelling stage 2
22/09/02 16:36:53 INFO DAGScheduler: ShuffleMapStage 2 (show at TEST.scala:37) failed in 0.127 s due to Job aborted due to stage failure: Task 0 in stage 2.0 failed 1 times, most recent failure: Lost task 0.0 in stage 2.0 (TID 3, localhost, executor driver): java.lang.NoSuchMethodError: org.apache.hadoop.hive.serde2.lazy.LazyStruct.parseMultiDelimit([B[BI)V
at org.apache.hadoop.hive.contrib.serde2.MultiDelimitSerDe.doDeserialize(MultiDelimitSerDe.java:164)
at org.apache.hadoop.hive.serde2.AbstractEncodingAwareSerDe.deserialize(AbstractEncodingAwareSerDe.java:71)
at org.apache.spark.sql.hive.HadoopTableReader$$anonfun$fillObject$2.apply(TableReader.scala:435)
at org.apache.spark.sql.hive.HadoopTableReader$$anonfun$fillObject$2.apply(TableReader.scala:434)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:410)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:410)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$10$$anon$1.hasNext(WholeStageCodegenExec.scala:614)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)
at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:125)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
at org.apache.spark.scheduler.Task.run(Task.scala:109)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

Driver stacktrace:
22/09/02 16:36:53 INFO DAGScheduler: Job 1 failed: show at TEST.scala:37, took 0.132460 s
Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 2.0 failed 1 times, most recent failure: Lost task 0.0 in stage 2.0 (TID 3, localhost, executor driver): java.lang.NoSuchMethodError: org.apache.hadoop.hive.serde2.lazy.LazyStruct.parseMultiDelimit([B[BI)V
at org.apache.hadoop.hive.contrib.serde2.MultiDelimitSerDe.doDeserialize(MultiDelimitSerDe.java:164)
at org.apache.hadoop.hive.serde2.AbstractEncodingAwareSerDe.deserialize(AbstractEncodingAwareSerDe.java:71)
at org.apache.spark.sql.hive.HadoopTableReader$$anonfun$fillObject$2.apply(TableReader.scala:435)
at org.apache.spark.sql.hive.HadoopTableReader$$anonfun$fillObject$2.apply(TableReader.scala:434)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:410)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:410)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$10$$anon$1.hasNext(WholeStageCodegenExec.scala:614)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)
at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:125)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
at org.apache.spark.scheduler.Task.run(Task.scala:109)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

Driver stacktrace:
at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1651)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1639)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1638)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1638)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:831)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:831)
at scala.Option.foreach(Option.scala:257)
at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:831)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1872)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1821)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1810)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:642)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2034)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2055)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2074)
at org.apache.spark.sql.execution.SparkPlan.executeTake(SparkPlan.scala:363)
at org.apache.spark.sql.execution.CollectLimitExec.executeCollect(limit.scala:38)
at org.apache.spark.sql.Dataset.org$apache$spark$sql$Dataset$$collectFromPlan(Dataset.scala:3278)
at org.apache.spark.sql.Dataset$$anonfun$head$1.apply(Dataset.scala:2489)
at org.apache.spark.sql.Dataset$$anonfun$head$1.apply(Dataset.scala:2489)
at org.apache.spark.sql.Dataset$$anonfun$52.apply(Dataset.scala:3259)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:77)
at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3258)
at org.apache.spark.sql.Dataset.head(Dataset.scala:2489)
at org.apache.spark.sql.Dataset.take(Dataset.scala:2703)
at org.apache.spark.sql.Dataset.showString(Dataset.scala:254)
at org.apache.spark.sql.Dataset.show(Dataset.scala:723)
at org.apache.spark.sql.Dataset.show(Dataset.scala:682)
at org.apache.spark.sql.Dataset.show(Dataset.scala:691)
at TEST$.main(TEST.scala:37)
at TEST.main(TEST.scala)
Caused by: java.lang.NoSuchMethodError: org.apache.hadoop.hive.serde2.lazy.LazyStruct.parseMultiDelimit([B[BI)V
at org.apache.hadoop.hive.contrib.serde2.MultiDelimitSerDe.doDeserialize(MultiDelimitSerDe.java:164)
at org.apache.hadoop.hive.serde2.AbstractEncodingAwareSerDe.deserialize(AbstractEncodingAwareSerDe.java:71)
at org.apache.spark.sql.hive.HadoopTableReader$$anonfun$fillObject$2.apply(TableReader.scala:435)
at org.apache.spark.sql.hive.HadoopTableReader$$anonfun$fillObject$2.apply(TableReader.scala:434)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:410)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:410)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$10$$anon$1.hasNext(WholeStageCodegenExec.scala:614)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)
at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:125)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
at org.apache.spark.scheduler.Task.run(Task.scala:109)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
22/09/02 16:36:53 INFO SparkContext: Invoking stop() from shutdown hook
22/09/02 16:36:53 INFO SparkUI: Stopped Spark web UI at http://a01a01b02.alico.corp:4040
22/09/02 16:36:53 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
22/09/02 16:36:53 INFO MemoryStore: MemoryStore cleared
22/09/02 16:36:53 INFO BlockManager: BlockManager stopped
22/09/02 16:36:53 INFO BlockManagerMaster: BlockManagerMaster stopped
22/09/02 16:36:53 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
22/09/02 16:36:53 WARN SparkEnv: Exception while deleting Spark temp dir: C:\Users\zilong\AppData\Local\Temp\1\spark-7917d05d-461f-4f96-8b42-03d7a2ce30d5\userFiles-8b4db58f-bd2b-42c8-b6a3-8f81ef07a2e6
java.io.IOException: Failed to delete: C:\Users\zilong\AppData\Local\Temp\1\spark-7917d05d-461f-4f96-8b42-03d7a2ce30d5\userFiles-8b4db58f-bd2b-42c8-b6a3-8f81ef07a2e6
at org.apache.spark.util.Utils$.deleteRecursively(Utils.scala:1074)
at org.apache.spark.SparkEnv.stop(SparkEnv.scala:103)
at org.apache.spark.SparkContext$$anonfun$stop$11.apply$mcV$sp(SparkContext.scala:1947)
at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1361)
at org.apache.spark.SparkContext.stop(SparkContext.scala:1946)
at org.apache.spark.SparkContext$$anonfun$2.apply$mcV$sp(SparkContext.scala:573)
at org.apache.spark.util.SparkShutdownHook.run(ShutdownHookManager.scala:216)
at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ShutdownHookManager.scala:188)
at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(ShutdownHookManager.scala:188)
at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(ShutdownHookManager.scala:188)
at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1992)
at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply$mcV$sp(ShutdownHookManager.scala:188)
at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(ShutdownHookManager.scala:188)
at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(ShutdownHookManager.scala:188)
at scala.util.Try$.apply(Try.scala:192)
at org.apache.spark.util.SparkShutdownHookManager.runAll(ShutdownHookManager.scala:188)
at org.apache.spark.util.SparkShutdownHookManager$$anon$2.run(ShutdownHookManager.scala:178)
at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54)
22/09/02 16:36:53 INFO SparkContext: Successfully stopped SparkContext
22/09/02 16:36:53 INFO ShutdownHookManager: Shutdown hook called
22/09/02 16:36:53 INFO ShutdownHookManager: Deleting directory C:\Users\zilong\AppData\Local\Temp\1\spark-7917d05d-461f-4f96-8b42-03d7a2ce30d5\userFiles-8b4db58f-bd2b-42c8-b6a3-8f81ef07a2e6
22/09/02 16:36:53 ERROR ShutdownHookManager: Exception while deleting Spark temp dir: C:\Users\zilong\AppData\Local\Temp\1\spark-7917d05d-461f-4f96-8b42-03d7a2ce30d5\userFiles-8b4db58f-bd2b-42c8-b6a3-8f81ef07a2e6
java.io.IOException: Failed to delete: C:\Users\zilong\AppData\Local\Temp\1\spark-7917d05d-461f-4f96-8b42-03d7a2ce30d5\userFiles-8b4db58f-bd2b-42c8-b6a3-8f81ef07a2e6
at org.apache.spark.util.Utils$.deleteRecursively(Utils.scala:1074)
at org.apache.spark.util.ShutdownHookManager$$anonfun$1$$anonfun$apply$mcV$sp$3.apply(ShutdownHookManager.scala:65)
at org.apache.spark.util.ShutdownHookManager$$anonfun$1$$anonfun$apply$mcV$sp$3.apply(ShutdownHookManager.scala:62)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
at org.apache.spark.util.ShutdownHookManager$$anonfun$1.apply$mcV$sp(ShutdownHookManager.scala:62)
at org.apache.spark.util.SparkShutdownHook.run(ShutdownHookManager.scala:216)
at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ShutdownHookManager.scala:188)
at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(ShutdownHookManager.scala:188)
at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(ShutdownHookManager.scala:188)
at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1992)
at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply$mcV$sp(ShutdownHookManager.scala:188)
at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(ShutdownHookManager.scala:188)
at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(ShutdownHookManager.scala:188)
at scala.util.Try$.apply(Try.scala:192)
at org.apache.spark.util.SparkShutdownHookManager.runAll(ShutdownHookManager.scala:188)
at org.apache.spark.util.SparkShutdownHookManager$$anon$2.run(ShutdownHookManager.scala:178)
at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54)
22/09/02 16:36:53 INFO ShutdownHookManager: Deleting directory C:\Users\zilong\AppData\Local\Temp\1\spark-7917d05d-461f-4f96-8b42-03d7a2ce30d5
22/09/02 16:36:53 ERROR ShutdownHookManager: Exception while deleting Spark temp dir: C:\Users\zilong\AppData\Local\Temp\1\spark-7917d05d-461f-4f96-8b42-03d7a2ce30d5
java.io.IOException: Failed to delete: C:\Users\zilong\AppData\Local\Temp\1\spark-7917d05d-461f-4f96-8b42-03d7a2ce30d5
at org.apache.spark.util.Utils$.deleteRecursively(Utils.scala:1074)
at org.apache.spark.util.ShutdownHookManager$$anonfun$1$$anonfun$apply$mcV$sp$3.apply(ShutdownHookManager.scala:65)
at org.apache.spark.util.ShutdownHookManager$$anonfun$1$$anonfun$apply$mcV$sp$3.apply(ShutdownHookManager.scala:62)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
at org.apache.spark.util.ShutdownHookManager$$anonfun$1.apply$mcV$sp(ShutdownHookManager.scala:62)
at org.apache.spark.util.SparkShutdownHook.run(ShutdownHookManager.scala:216)
at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ShutdownHookManager.scala:188)
at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(ShutdownHookManager.scala:188)
at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(ShutdownHookManager.scala:188)
at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1992)
at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply$mcV$sp(ShutdownHookManager.scala:188)
at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(ShutdownHookManager.scala:188)
at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(ShutdownHookManager.scala:188)
at scala.util.Try$.apply(Try.scala:192)
at org.apache.spark.util.SparkShutdownHookManager.runAll(ShutdownHookManager.scala:188)
at org.apache.spark.util.SparkShutdownHookManager$$anon$2.run(ShutdownHookManager.scala:178)
at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54)

进程已结束,退出代码为 1

我的解答思路和尝试过的方法

增加了可以解析多分隔符的包hive-contrib.jar
资源包里面的hive-site.xml也尝试过修改hive.aux.jars.path对应的hive-contrib.jar路径
也尝试过把hive-contrib.jar打入maven库

我想要达到的结果

以上我写了两个查询语句,一个是单分隔符的orc外部表(从日志上看可以正常读取数据)
一个是多分隔符的txt外部表,无法访问
有什么办法可以正常访问这个多分隔符外部表

  • 写回答

1条回答 默认 最新

  • 来灵 2022-09-06 13:00
    关注

    出差日志
    22/09/02 16:36:53 ERROR Executor: Exception in task 0.0 in stage 2.0 (TID 3)
    java.lang.NoSuchMethodError: org.apache.hadoop.hive.serde2.lazy.LazyStruct.parseMultiDelimit([B[BI)V
    at org.apache.hadoop.hive.contrib.serde2.MultiDelimitSerDe.doDeserialize(MultiDelimitSerDe.java:164)
    at org.apache.hadoop.hive.serde2.AbstractEncodingAwareSerDe.deserialize(AbstractEncodingAwareSerDe.java:71)
    at org.apache.spark.sql.hive.HadoopTableReader$$anonfun$fillObject$2.apply(TableReader.scala:435)
    at org.apache.spark.sql.hive.HadoopTableReader$$anonfun$fillObject$2.apply(TableReader.scala:434)
    at scala.collection.Iterator$$anon$11.next(Iterator.scala:410)
    at scala.collection.Iterator$$anon$11.next(Iterator.scala:410)
    at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
    at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
    at org.apache.spark.sql.execution.WholeStageCodegenExec


    22/09/02 16:36:53 ERROR ShutdownHookManager: Exception while deleting Spark temp dir: C:\Users\zilong\AppData\Local\Temp\1\spark-7917d05d-461f-4f96-8b42-03d7a2ce30d5\userFiles-8b4db58f-bd2b-42c8-b6a3-8f81ef07a2e6
    java.io.IOException: Failed to delete: C:\Users\zilong\AppData\Local\Temp\1\spark-7917d05d-461f-4f96-8b42-03d7a2ce30d5\userFiles-8b4db58f-bd2b-42c8-b6a3-8f81ef07a2e6
    at org.apache.spark.util.Utils$.deleteRecursively(Utils.scala:1074)
    at org.apache.spark.util.ShutdownHookManager

    评论

报告相同问题?

问题事件

  • 已结题 (查看结题原因) 9月16日
  • 修改了问题 9月5日
  • 创建了问题 9月2日

悬赏问题

  • ¥15 这个电路是如何实现路灯控制器的,原理是什么,怎么求解灯亮起后熄灭的时间如图?
  • ¥15 matlab数字图像处理频率域滤波
  • ¥15 在abaqus做了二维正交切削模型,给刀具添加了超声振动条件后输出切削力为什么比普通切削增大这么多
  • ¥15 ELGamal和paillier计算效率谁快?
  • ¥15 file converter 转换格式失败 报错 Error marking filters as finished,如何解决?
  • ¥15 Arcgis相交分析无法绘制一个或多个图形
  • ¥15 关于#r语言#的问题:差异分析前数据准备,报错Error in data[, sampleName1] : subscript out of bounds请问怎么解决呀以下是全部代码:
  • ¥15 seatunnel-web使用SQL组件时候后台报错,无法找到表格
  • ¥15 fpga自动售货机数码管(相关搜索:数字时钟)
  • ¥15 用前端向数据库插入数据,通过debug发现数据能走到后端,但是放行之后就会提示错误