book19897264 2022-09-02 16:58 采纳率: 0%
浏览 29
已结题

sparksql无法访问多分隔符的外部表

问题遇到的现象和发生背景

sparksql无法访问多分隔符的外部表

问题相关代码,请勿粘贴截图

测试代码

import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.security.UserGroupInformation
import org.apache.spark.sql.{DataFrame, SparkSession}
import java.io.IOException
object TEST {
  def main(args: Array[String]): Unit = {

    System.setProperty("hadoop.home.dir","C:\\winutil")
   doKerberosAuth(new Configuration())

val spark = SparkSession
  .builder()
  .appName("TEST")
  .master("local[*]")
  .enableHiveSupport()
  .getOrCreate()
    import spark.implicits._

    spark.sql("add jar C:\\DataLake\\TEST_yarnclient\\src\\main\\resources\\hive-contrib.jar")


    var a = "select * from udm_rdz_lat.rdz_la_zbnfpf_lat"
    var b = "select * from udm_rdz_source.rdz_la_zbnfpf"
    val aa: DataFrame = spark.sql(a)
    aa.show()
    val bb: DataFrame = spark.sql(b)
    bb.show()
  spark.close()
  }

  def doKerberosAuth(conf: Configuration): Unit = {
    import scala.collection.immutable._
    System.setProperty("java.security.krb5.conf", "C:\\DataLake\\TEST_yarnclient\\src\\main\\resources\\krb5.conf")
    conf.set("hadoop.security.authentication", "Kerberos")
    UserGroupInformation.setConfiguration(conf)
    try {
      UserGroupInformation.loginUserFromKeytab("test@QAALICO.CORP", "C:\\DataLake\\TEST_yarnclient\\src\\main\\resources\\udm.keytab")
    }
    catch {
      case e: IOException => e.printStackTrace()
    }
  }
}

pom文件

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>org.example</groupId>
    <artifactId>TEST</artifactId>
    <version>1.0-SNAPSHOT</version>

    <dependencies>
                <dependency>
                    <groupId>org.scala-lang</groupId>
                    <artifactId>scala-library</artifactId>
                    <version>2.11.12</version>
                </dependency>
                <dependency>
                    <groupId>org.scala-lang</groupId>
                    <artifactId>scala-compiler</artifactId>
                    <version>2.11.12</version>
                </dependency>

    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-sql_2.11</artifactId>
        <version>2.3.2</version>
        <scope>provided</scope>

    </dependency>

        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-hive_2.11</artifactId>
            <version>2.3.2</version>
            <!--<scope>provided</scope> -->
        </dependency>

        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-hive-thriftserver_2.11</artifactId>
            <version>2.3.2</version>
        </dependency>


        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-yarn_2.10</artifactId>
            <version>2.2.0</version>
            <scope>provided</scope>

        </dependency>

<!--hive-contrib.jar-->
        <dependency>
            <groupId>aaaa</groupId>
            <artifactId>aaaa-bbbb</artifactId>
            <version>1.1.1</version>
        </dependency>



    </dependencies>



    <properties>
        <maven.compiler.source>8</maven.compiler.source>
        <maven.compiler.target>8</maven.compiler.target>
    </properties>

</project>

运行结果及报错内容

主要报错内容:
ERROR Executor: Exception in task 0.0 in stage 2.0 (TID 3)
java.lang.NoSuchMethodError: org.apache.hadoop.hive.serde2.lazy.LazyStruct.parseMultiDelimit([B[BI)V

以下是详细日志:
log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
22/09/02 16:36:38 INFO SparkContext: Running Spark version 2.3.2
22/09/02 16:36:38 INFO SparkContext: Submitted application: TEST
22/09/02 16:36:38 INFO SecurityManager: Changing view acls to: zilong,test
22/09/02 16:36:38 INFO SecurityManager: Changing modify acls to: zilong,test
22/09/02 16:36:38 INFO SecurityManager: Changing view acls groups to:
22/09/02 16:36:38 INFO SecurityManager: Changing modify acls groups to:
22/09/02 16:36:38 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(zilong, test); groups with view permissions: Set(); users with modify permissions: Set(zilong, test); groups with modify permissions: Set()
22/09/02 16:36:39 INFO Utils: Successfully started service 'sparkDriver' on port 63485.
22/09/02 16:36:39 INFO SparkEnv: Registering MapOutputTracker
22/09/02 16:36:39 INFO SparkEnv: Registering BlockManagerMaster
22/09/02 16:36:39 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
22/09/02 16:36:39 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
22/09/02 16:36:39 INFO DiskBlockManager: Created local directory at C:\Users\zilong\AppData\Local\Temp\1\blockmgr-284035c7-994f-4819-b05c-b81ab636b5c8
22/09/02 16:36:39 INFO MemoryStore: MemoryStore started with capacity 1980.0 MB
22/09/02 16:36:39 INFO SparkEnv: Registering OutputCommitCoordinator
22/09/02 16:36:39 INFO Utils: Successfully started service 'SparkUI' on port 4040.
22/09/02 16:36:39 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://a01a01b02.alico.corp:4040
22/09/02 16:36:39 INFO Executor: Starting executor ID driver on host localhost
22/09/02 16:36:39 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 63526.
22/09/02 16:36:39 INFO NettyBlockTransferService: Server created on a01a01b02.alico.corp:63526
22/09/02 16:36:39 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
22/09/02 16:36:39 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, a01a01b02.alico.corp, 63526, None)
22/09/02 16:36:39 INFO BlockManagerMasterEndpoint: Registering block manager a01a01b02.alico.corp:63526 with 1980.0 MB RAM, BlockManagerId(driver, a01a01b02.alico.corp, 63526, None)
22/09/02 16:36:39 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, a01a01b02.alico.corp, 63526, None)
22/09/02 16:36:39 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, a01a01b02.alico.corp, 63526, None)
22/09/02 16:36:39 INFO SharedState: loading hive config file: file:/C:/DataLake/TEST_yarnclient/target/classes/hive-site.xml
22/09/02 16:36:39 INFO SharedState: spark.sql.warehouse.dir is not set, but hive.metastore.warehouse.dir is set. Setting spark.sql.warehouse.dir to the value of hive.metastore.warehouse.dir ('/warehouse/tablespace/managed/hive').
22/09/02 16:36:39 INFO SharedState: Warehouse path is '/warehouse/tablespace/managed/hive'.
22/09/02 16:36:40 INFO StateStoreCoordinatorRef: Registered StateStoreCoordinator endpoint
22/09/02 16:36:40 INFO HiveUtils: Initializing HiveMetastoreConnection version 1.2.1 using Spark classes.
22/09/02 16:36:41 WARN HiveConf: HiveConf of name hive.tez.cartesian-product.enabled does not exist
22/09/02 16:36:41 WARN HiveConf: HiveConf of name hive.metastore.warehouse.external.dir does not exist
22/09/02 16:36:41 WARN HiveConf: HiveConf of name hive.server2.webui.use.ssl does not exist
22/09/02 16:36:41 WARN HiveConf: HiveConf of name hive.heapsize does not exist
22/09/02 16:36:41 WARN HiveConf: HiveConf of name hive.server2.webui.port does not exist
22/09/02 16:36:41 WARN HiveConf: HiveConf of name hive.materializedview.rewriting.incremental does not exist
22/09/02 16:36:41 WARN HiveConf: HiveConf of name hive.server2.webui.cors.allowed.headers does not exist
22/09/02 16:36:41 WARN HiveConf: HiveConf of name hive.driver.parallel.compilation does not exist
22/09/02 16:36:41 WARN HiveConf: HiveConf of name hive.tez.bucket.pruning does not exist
22/09/02 16:36:41 WARN HiveConf: HiveConf of name hive.hook.proto.base-directory does not exist
22/09/02 16:36:41 WARN HiveConf: HiveConf of name hive.load.data.owner does not exist
22/09/02 16:36:41 WARN HiveConf: HiveConf of name hive.execution.mode does not exist
22/09/02 16:36:41 WARN HiveConf: HiveConf of name hive.vectorized.adaptor.usage.mode does not exist
22/09/02 16:36:41 WARN HiveConf: HiveConf of name hive.service.metrics.codahale.reporter.classes does not exist
22/09/02 16:36:41 WARN HiveConf: HiveConf of name hive.strict.managed.tables does not exist
22/09/02 16:36:41 WARN HiveConf: HiveConf of name hive.create.as.insert.only does not exist
22/09/02 16:36:41 WARN HiveConf: HiveConf of name hive.optimize.dynamic.partition.hashjoin does not exist
22/09/02 16:36:41 WARN HiveConf: HiveConf of name hive.server2.webui.enable.cors does not exist
22/09/02 16:36:41 WARN HiveConf: HiveConf of name hive.metastore.db.type does not exist
22/09/02 16:36:41 WARN HiveConf: HiveConf of name hive.txn.strict.locking.mode does not exist
22/09/02 16:36:41 WARN HiveConf: HiveConf of name hive.metastore.transactional.event.listeners does not exist
22/09/02 16:36:41 WARN HiveConf: HiveConf of name hive.tez.input.generate.consistent.splits does not exist
22/09/02 16:36:41 INFO metastore: Trying to connect to metastore with URI thrift://a01.alico.corp:9083
22/09/02 16:36:41 INFO metastore: Connected to metastore.
22/09/02 16:36:45 WARN DomainSocketFactory: The short-circuit local reads feature cannot be used because UNIX Domain sockets are not available on Windows.
22/09/02 16:36:46 INFO SessionState: Created local directory: C:/Users/ZHANGS1/AppData/Local/Temp/1/f98a580-2abc-4c1d-8fd0-ebb28e3a6dd2_resources
22/09/02 16:36:46 INFO SessionState: Created HDFS directory: /tmp/hive/test/f98a580-2abc-4c1d-8fd0-ebb28e3a6dd2
22/09/02 16:36:46 INFO SessionState: Created local directory: C:/Users/ZHANGS
1/AppData/Local/Temp/1/zilong/f98a580-2abc-4c1d-8fd0-ebb28e3a6dd2
22/09/02 16:36:46 INFO SessionState: Created HDFS directory: /tmp/hive/test/f98a580-2abc-4c1d-8fd0-ebb28e3a6dd2/_tmp_space.db
22/09/02 16:36:46 INFO HiveClientImpl: Warehouse location for Hive client (version 1.2.2) is /warehouse/tablespace/managed/hive
22/09/02 16:36:48 INFO JniBasedUnixGroupsMapping: Error getting groups for test: Unknown error.
22/09/02 16:36:48 WARN UserGroupInformation: No groups available for user test
22/09/02 16:36:48 WARN UserGroupInformation: No groups available for user test
22/09/02 16:36:48 INFO SessionState: Added [/C:/DataLake/TEST_yarnclient/src/main/resources/hive-contrib.jar] to class path
22/09/02 16:36:48 INFO SessionState: Added resources: [C:\DataLake\TEST_yarnclient\src\main\resources\hive-contrib.jar]
22/09/02 16:36:48 INFO SparkContext: Added JAR C:\DataLake\TEST_yarnclient\src\main\resources\hive-contrib.jar at spark://a01a01b02.alico.corp:63485/jars/hive-contrib.jar with timestamp 1662107808686
22/09/02 16:36:49 INFO CodeGenerator: Code generated in 312.7428 ms
22/09/02 16:36:50 INFO CodeGenerator: Code generated in 66.424 ms
22/09/02 16:36:50 INFO CodeGenerator: Code generated in 29.4999 ms
22/09/02 16:36:50 INFO ContextCleaner: Cleaned accumulator 0
22/09/02 16:36:50 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 387.9 KB, free 1979.6 MB)
22/09/02 16:36:50 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 37.5 KB, free 1979.6 MB)
22/09/02 16:36:50 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on a01a01b02.alico.corp:63526 (size: 37.5 KB, free: 1980.0 MB)
22/09/02 16:36:50 INFO SparkContext: Created broadcast 0 from
22/09/02 16:36:50 INFO PerfLogger:
22/09/02 16:36:50 INFO deprecation: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
22/09/02 16:36:50 INFO OrcInputFormat: FooterCacheHitRatio: 0/0
22/09/02 16:36:50 INFO PerfLogger: </PERFLOG method=OrcGetSplits start=1662107810801 end=1662107810925 duration=124 from=org.apache.hadoop.hive.ql.io.orc.ReaderImpl>
22/09/02 16:36:50 INFO SparkContext: Starting job: show at TEST.scala:34
22/09/02 16:36:50 INFO DAGScheduler: Registering RDD 5 (show at TEST.scala:34)
22/09/02 16:36:50 INFO DAGScheduler: Got job 0 (show at TEST.scala:34) with 1 output partitions
22/09/02 16:36:50 INFO DAGScheduler: Final stage: ResultStage 1 (show at TEST.scala:34)
22/09/02 16:36:50 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 0)
22/09/02 16:36:50 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 0)
22/09/02 16:36:50 INFO DAGScheduler: Submitting ShuffleMapStage 0 (MapPartitionsRDD[5] at show at TEST.scala:34), which has no missing parents
22/09/02 16:36:51 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 20.6 KB, free 1979.6 MB)
22/09/02 16:36:51 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 8.4 KB, free 1979.6 MB)
22/09/02 16:36:51 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on a01a01b02.alico.corp:63526 (size: 8.4 KB, free: 1980.0 MB)
22/09/02 16:36:51 INFO SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:1039
22/09/02 16:36:51 INFO DAGScheduler: Submitting 2 missing tasks from ShuffleMapStage 0 (MapPartitionsRDD[5] at show at TEST.scala:34) (first 15 tasks are for partitions Vector(0, 1))
22/09/02 16:36:51 INFO TaskSchedulerImpl: Adding task set 0.0 with 2 tasks
22/09/02 16:36:51 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, executor driver, partition 0, ANY, 8027 bytes)
22/09/02 16:36:51 INFO TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, localhost, executor driver, partition 1, ANY, 8027 bytes)
22/09/02 16:36:51 INFO Executor: Running task 1.0 in stage 0.0 (TID 1)
22/09/02 16:36:51 INFO Executor: Running task 0.0 in stage 0.0 (TID 0)
22/09/02 16:36:51 INFO Executor: Fetching spark://a01a01b02.alico.corp:63485/jars/hive-contrib.jar with timestamp 1662107808686
22/09/02 16:36:51 INFO TransportClientFactory: Successfully created connection to a01a01b02.alico.corp/10.165.63.245:63485 after 39 ms (0 ms spent in bootstraps)
22/09/02 16:36:51 INFO Utils: Fetching spark://a01a01b02.alico.corp:63485/jars/hive-contrib.jar to C:\Users\zilong\AppData\Local\Temp\1\spark-7917d05d-461f-4f96-8b42-03d7a2ce30d5\userFiles-8b4db58f-bd2b-42c8-b6a3-8f81ef07a2e6\fetchFileTemp3333765197903670028.tmp
22/09/02 16:36:51 INFO Executor: Adding file:/C:/Users/zilong/AppData/Local/Temp/1/spark-7917d05d-461f-4f96-8b42-03d7a2ce30d5/userFiles-8b4db58f-bd2b-42c8-b6a3-8f81ef07a2e6/hive-contrib.jar to class loader
22/09/02 16:36:51 INFO HadoopRDD: Input split: hdfs://a01.alico.corp:8020/warehouse/tablespace/external/hive/test_rdz_lat.db/rdz_la_zbnfpf_lat/part-00001-74822f38-c2a5-496b-86fa-a8e4836e4bde-c000.snappy.orc:0+2361
22/09/02 16:36:51 INFO HadoopRDD: Input split: hdfs://a01.alico.corp:8020/warehouse/tablespace/external/hive/test_rdz_lat.db/rdz_la_zbnfpf_lat/part-00000-74822f38-c2a5-496b-86fa-a8e4836e4bde-c000.snappy.orc:0+2363
22/09/02 16:36:52 INFO OrcRawRecordMerger: min key = null, max key = null
22/09/02 16:36:52 INFO OrcRawRecordMerger: min key = null, max key = null
22/09/02 16:36:52 INFO ReaderImpl: Reading ORC rows from hdfs://a01.alico.corp:8020/warehouse/tablespace/external/hive/test_rdz_lat.db/rdz_la_zbnfpf_lat/part-00001-74822f38-c2a5-496b-86fa-a8e4836e4bde-c000.snappy.orc with {include: [true], offset: 0, length: 9223372036854775807}
22/09/02 16:36:52 INFO ReaderImpl: Reading ORC rows from hdfs://a01.alico.corp:8020/warehouse/tablespace/external/hive/test_rdz_lat.db/rdz_la_zbnfpf_lat/part-00000-74822f38-c2a5-496b-86fa-a8e4836e4bde-c000.snappy.orc with {include: [true], offset: 0, length: 9223372036854775807}
22/09/02 16:36:52 INFO CodeGenerator: Code generated in 21.6888 ms
22/09/02 16:36:52 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0). 1481 bytes result sent to driver
22/09/02 16:36:52 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 1168 ms on localhost (executor driver) (1/2)
22/09/02 16:36:52 INFO Executor: Finished task 1.0 in stage 0.0 (TID 1). 1438 bytes result sent to driver
22/09/02 16:36:52 INFO TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 1213 ms on localhost (executor driver) (2/2)
22/09/02 16:36:52 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool
22/09/02 16:36:52 INFO DAGScheduler: ShuffleMapStage 0 (show at TEST.scala:34) finished in 1.334 s
22/09/02 16:36:52 INFO DAGScheduler: looking for newly runnable stages
22/09/02 16:36:52 INFO DAGScheduler: running: Set()
22/09/02 16:36:52 INFO DAGScheduler: waiting: Set(ResultStage 1)
22/09/02 16:36:52 INFO DAGScheduler: failed: Set()
22/09/02 16:36:52 INFO DAGScheduler: Submitting ResultStage 1 (MapPartitionsRDD[9] at show at TEST.scala:34), which has no missing parents
22/09/02 16:36:52 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 15.5 KB, free 1979.5 MB)
22/09/02 16:36:52 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 5.6 KB, free 1979.5 MB)
22/09/02 16:36:52 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on a01a01b02.alico.corp:63526 (size: 5.6 KB, free: 1979.9 MB)
22/09/02 16:36:52 INFO SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:1039
22/09/02 16:36:52 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 1 (MapPartitionsRDD[9] at show at TEST.scala:34) (first 15 tasks are for partitions Vector(0))
22/09/02 16:36:52 INFO TaskSchedulerImpl: Adding task set 1.0 with 1 tasks
22/09/02 16:36:52 INFO TaskSetManager: Starting task 0.0 in stage 1.0 (TID 2, localhost, executor driver, partition 0, ANY, 7754 bytes)
22/09/02 16:36:52 INFO Executor: Running task 0.0 in stage 1.0 (TID 2)
22/09/02 16:36:52 INFO ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
22/09/02 16:36:52 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 7 ms
22/09/02 16:36:52 INFO Executor: Finished task 0.0 in stage 1.0 (TID 2). 1674 bytes result sent to driver
22/09/02 16:36:52 INFO TaskSetManager: Finished task 0.0 in stage 1.0 (TID 2) in 43 ms on localhost (executor driver) (1/1)
22/09/02 16:36:52 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool
22/09/02 16:36:52 INFO DAGScheduler: ResultStage 1 (show at TEST.scala:34) finished in 0.066 s
22/09/02 16:36:52 INFO DAGScheduler: Job 0 finished: show at TEST.scala:34, took 1.473738 s

|HIST_ID|CLNTPFX|CLNTCOY| CLNTNUM|SEQNUM| CRTDATE|VALIDFLAG|BNFIDENT|USER_PROFILE| JOB_NAME| DATIME|ETL_DATATYPE| ETL_DATATIME| DT_DATATIME|BATCHID|CRTDATE_tr| log_id| test_etl_datatime|

| 12| CN| 9|00989323| 10|20180802| 2| 1| ZHANGS|QPADEV001G|2018-08-02 15:47:...| U|2022-06-16 16:14:00|2022-06-15 20:33:00| 0|2018-08-02|rdz_la_zbnfpf1655...|2022-06-20 15:59:13|

22/09/02 16:36:52 INFO CodeGenerator: Code generated in 14.412 ms
22/09/02 16:36:52 INFO CodeGenerator: Code generated in 28.6352 ms
22/09/02 16:36:52 INFO CodeGenerator: Code generated in 17.2243 ms
22/09/02 16:36:53 INFO MemoryStore: Block broadcast_3 stored as values in memory (estimated size 387.7 KB, free 1979.2 MB)
22/09/02 16:36:53 INFO MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 37.3 KB, free 1979.1 MB)
22/09/02 16:36:53 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on a01a01b02.alico.corp:63526 (size: 37.3 KB, free: 1979.9 MB)
22/09/02 16:36:53 INFO SparkContext: Created broadcast 3 from
22/09/02 16:36:53 INFO DFSClient: Created HDFS_DELEGATION_TOKEN token 1374659 for test on 10.164.29.130:8020
22/09/02 16:36:53 INFO TokenCache: Got dt for hdfs://a01.alico.corp:8020; Kind: HDFS_DELEGATION_TOKEN, Service: 10.164.29.130:8020, Ident: (HDFS_DELEGATION_TOKEN token 1374659 for test)
22/09/02 16:36:53 INFO FileInputFormat: Total input paths to process : 1
22/09/02 16:36:53 INFO SparkContext: Starting job: show at TEST.scala:37
22/09/02 16:36:53 INFO DAGScheduler: Registering RDD 15 (show at TEST.scala:37)
22/09/02 16:36:53 INFO DAGScheduler: Got job 1 (show at TEST.scala:37) with 1 output partitions
22/09/02 16:36:53 INFO DAGScheduler: Final stage: ResultStage 3 (show at TEST.scala:37)
22/09/02 16:36:53 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 2)
22/09/02 16:36:53 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 2)
22/09/02 16:36:53 INFO DAGScheduler: Submitting ShuffleMapStage 2 (MapPartitionsRDD[15] at show at TEST.scala:37), which has no missing parents
22/09/02 16:36:53 INFO MemoryStore: Block broadcast_4 stored as values in memory (estimated size 19.6 KB, free 1979.1 MB)
22/09/02 16:36:53 INFO MemoryStore: Block broadcast_4_piece0 stored as bytes in memory (estimated size 8.2 KB, free 1979.1 MB)
22/09/02 16:36:53 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on a01a01b02.alico.corp:63526 (size: 8.2 KB, free: 1979.9 MB)
22/09/02 16:36:53 INFO SparkContext: Created broadcast 4 from broadcast at DAGScheduler.scala:1039
22/09/02 16:36:53 INFO DAGScheduler: Submitting 1 missing tasks from ShuffleMapStage 2 (MapPartitionsRDD[15] at show at TEST.scala:37) (first 15 tasks are for partitions Vector(0))
22/09/02 16:36:53 INFO TaskSchedulerImpl: Adding task set 2.0 with 1 tasks
22/09/02 16:36:53 INFO TaskSetManager: Starting task 0.0 in stage 2.0 (TID 3, localhost, executor driver, partition 0, ANY, 7950 bytes)
22/09/02 16:36:53 INFO Executor: Running task 0.0 in stage 2.0 (TID 3)
22/09/02 16:36:53 INFO HadoopRDD: Input split: hdfs://a01.alico.corp:8020/warehouse/tablespace/external/hive/test_rdz_source.db/rdz_la_zbnfpf/zbnfpf.txt:0+2880
22/09/02 16:36:53 INFO CodeGenerator: Code generated in 15.6463 ms
22/09/02 16:36:53 ERROR Executor: Exception in task 0.0 in stage 2.0 (TID 3)
java.lang.NoSuchMethodError: org.apache.hadoop.hive.serde2.lazy.LazyStruct.parseMultiDelimit([B[BI)V
at org.apache.hadoop.hive.contrib.serde2.MultiDelimitSerDe.doDeserialize(MultiDelimitSerDe.java:164)
at org.apache.hadoop.hive.serde2.AbstractEncodingAwareSerDe.deserialize(AbstractEncodingAwareSerDe.java:71)
at org.apache.spark.sql.hive.HadoopTableReader$$anonfun$fillObject$2.apply(TableReader.scala:435)
at org.apache.spark.sql.hive.HadoopTableReader$$anonfun$fillObject$2.apply(TableReader.scala:434)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:410)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:410)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$10$$anon$1.hasNext(WholeStageCodegenExec.scala:614)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)
at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:125)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
at org.apache.spark.scheduler.Task.run(Task.scala:109)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
22/09/02 16:36:53 WARN TaskSetManager: Lost task 0.0 in stage 2.0 (TID 3, localhost, executor driver): java.lang.NoSuchMethodError: org.apache.hadoop.hive.serde2.lazy.LazyStruct.parseMultiDelimit([B[BI)V
at org.apache.hadoop.hive.contrib.serde2.MultiDelimitSerDe.doDeserialize(MultiDelimitSerDe.java:164)
at org.apache.hadoop.hive.serde2.AbstractEncodingAwareSerDe.deserialize(AbstractEncodingAwareSerDe.java:71)
at org.apache.spark.sql.hive.HadoopTableReader$$anonfun$fillObject$2.apply(TableReader.scala:435)
at org.apache.spark.sql.hive.HadoopTableReader$$anonfun$fillObject$2.apply(TableReader.scala:434)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:410)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:410)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$10$$anon$1.hasNext(WholeStageCodegenExec.scala:614)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)
at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:125)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
at org.apache.spark.scheduler.Task.run(Task.scala:109)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

22/09/02 16:36:53 ERROR TaskSetManager: Task 0 in stage 2.0 failed 1 times; aborting job
22/09/02 16:36:53 INFO TaskSchedulerImpl: Removed TaskSet 2.0, whose tasks have all completed, from pool
22/09/02 16:36:53 INFO TaskSchedulerImpl: Cancelling stage 2
22/09/02 16:36:53 INFO DAGScheduler: ShuffleMapStage 2 (show at TEST.scala:37) failed in 0.127 s due to Job aborted due to stage failure: Task 0 in stage 2.0 failed 1 times, most recent failure: Lost task 0.0 in stage 2.0 (TID 3, localhost, executor driver): java.lang.NoSuchMethodError: org.apache.hadoop.hive.serde2.lazy.LazyStruct.parseMultiDelimit([B[BI)V
at org.apache.hadoop.hive.contrib.serde2.MultiDelimitSerDe.doDeserialize(MultiDelimitSerDe.java:164)
at org.apache.hadoop.hive.serde2.AbstractEncodingAwareSerDe.deserialize(AbstractEncodingAwareSerDe.java:71)
at org.apache.spark.sql.hive.HadoopTableReader$$anonfun$fillObject$2.apply(TableReader.scala:435)
at org.apache.spark.sql.hive.HadoopTableReader$$anonfun$fillObject$2.apply(TableReader.scala:434)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:410)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:410)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$10$$anon$1.hasNext(WholeStageCodegenExec.scala:614)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)
at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:125)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
at org.apache.spark.scheduler.Task.run(Task.scala:109)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

Driver stacktrace:
22/09/02 16:36:53 INFO DAGScheduler: Job 1 failed: show at TEST.scala:37, took 0.132460 s
Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 2.0 failed 1 times, most recent failure: Lost task 0.0 in stage 2.0 (TID 3, localhost, executor driver): java.lang.NoSuchMethodError: org.apache.hadoop.hive.serde2.lazy.LazyStruct.parseMultiDelimit([B[BI)V
at org.apache.hadoop.hive.contrib.serde2.MultiDelimitSerDe.doDeserialize(MultiDelimitSerDe.java:164)
at org.apache.hadoop.hive.serde2.AbstractEncodingAwareSerDe.deserialize(AbstractEncodingAwareSerDe.java:71)
at org.apache.spark.sql.hive.HadoopTableReader$$anonfun$fillObject$2.apply(TableReader.scala:435)
at org.apache.spark.sql.hive.HadoopTableReader$$anonfun$fillObject$2.apply(TableReader.scala:434)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:410)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:410)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$10$$anon$1.hasNext(WholeStageCodegenExec.scala:614)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)
at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:125)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
at org.apache.spark.scheduler.Task.run(Task.scala:109)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

Driver stacktrace:
at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1651)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1639)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1638)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1638)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:831)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:831)
at scala.Option.foreach(Option.scala:257)
at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:831)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1872)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1821)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1810)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:642)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2034)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2055)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2074)
at org.apache.spark.sql.execution.SparkPlan.executeTake(SparkPlan.scala:363)
at org.apache.spark.sql.execution.CollectLimitExec.executeCollect(limit.scala:38)
at org.apache.spark.sql.Dataset.org$apache$spark$sql$Dataset$$collectFromPlan(Dataset.scala:3278)
at org.apache.spark.sql.Dataset$$anonfun$head$1.apply(Dataset.scala:2489)
at org.apache.spark.sql.Dataset$$anonfun$head$1.apply(Dataset.scala:2489)
at org.apache.spark.sql.Dataset$$anonfun$52.apply(Dataset.scala:3259)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:77)
at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3258)
at org.apache.spark.sql.Dataset.head(Dataset.scala:2489)
at org.apache.spark.sql.Dataset.take(Dataset.scala:2703)
at org.apache.spark.sql.Dataset.showString(Dataset.scala:254)
at org.apache.spark.sql.Dataset.show(Dataset.scala:723)
at org.apache.spark.sql.Dataset.show(Dataset.scala:682)
at org.apache.spark.sql.Dataset.show(Dataset.scala:691)
at TEST$.main(TEST.scala:37)
at TEST.main(TEST.scala)
Caused by: java.lang.NoSuchMethodError: org.apache.hadoop.hive.serde2.lazy.LazyStruct.parseMultiDelimit([B[BI)V
at org.apache.hadoop.hive.contrib.serde2.MultiDelimitSerDe.doDeserialize(MultiDelimitSerDe.java:164)
at org.apache.hadoop.hive.serde2.AbstractEncodingAwareSerDe.deserialize(AbstractEncodingAwareSerDe.java:71)
at org.apache.spark.sql.hive.HadoopTableReader$$anonfun$fillObject$2.apply(TableReader.scala:435)
at org.apache.spark.sql.hive.HadoopTableReader$$anonfun$fillObject$2.apply(TableReader.scala:434)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:410)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:410)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$10$$anon$1.hasNext(WholeStageCodegenExec.scala:614)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)
at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:125)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
at org.apache.spark.scheduler.Task.run(Task.scala:109)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
22/09/02 16:36:53 INFO SparkContext: Invoking stop() from shutdown hook
22/09/02 16:36:53 INFO SparkUI: Stopped Spark web UI at http://a01a01b02.alico.corp:4040
22/09/02 16:36:53 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
22/09/02 16:36:53 INFO MemoryStore: MemoryStore cleared
22/09/02 16:36:53 INFO BlockManager: BlockManager stopped
22/09/02 16:36:53 INFO BlockManagerMaster: BlockManagerMaster stopped
22/09/02 16:36:53 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
22/09/02 16:36:53 WARN SparkEnv: Exception while deleting Spark temp dir: C:\Users\zilong\AppData\Local\Temp\1\spark-7917d05d-461f-4f96-8b42-03d7a2ce30d5\userFiles-8b4db58f-bd2b-42c8-b6a3-8f81ef07a2e6
java.io.IOException: Failed to delete: C:\Users\zilong\AppData\Local\Temp\1\spark-7917d05d-461f-4f96-8b42-03d7a2ce30d5\userFiles-8b4db58f-bd2b-42c8-b6a3-8f81ef07a2e6
at org.apache.spark.util.Utils$.deleteRecursively(Utils.scala:1074)
at org.apache.spark.SparkEnv.stop(SparkEnv.scala:103)
at org.apache.spark.SparkContext$$anonfun$stop$11.apply$mcV$sp(SparkContext.scala:1947)
at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1361)
at org.apache.spark.SparkContext.stop(SparkContext.scala:1946)
at org.apache.spark.SparkContext$$anonfun$2.apply$mcV$sp(SparkContext.scala:573)
at org.apache.spark.util.SparkShutdownHook.run(ShutdownHookManager.scala:216)
at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ShutdownHookManager.scala:188)
at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(ShutdownHookManager.scala:188)
at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(ShutdownHookManager.scala:188)
at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1992)
at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply$mcV$sp(ShutdownHookManager.scala:188)
at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(ShutdownHookManager.scala:188)
at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(ShutdownHookManager.scala:188)
at scala.util.Try$.apply(Try.scala:192)
at org.apache.spark.util.SparkShutdownHookManager.runAll(ShutdownHookManager.scala:188)
at org.apache.spark.util.SparkShutdownHookManager$$anon$2.run(ShutdownHookManager.scala:178)
at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54)
22/09/02 16:36:53 INFO SparkContext: Successfully stopped SparkContext
22/09/02 16:36:53 INFO ShutdownHookManager: Shutdown hook called
22/09/02 16:36:53 INFO ShutdownHookManager: Deleting directory C:\Users\zilong\AppData\Local\Temp\1\spark-7917d05d-461f-4f96-8b42-03d7a2ce30d5\userFiles-8b4db58f-bd2b-42c8-b6a3-8f81ef07a2e6
22/09/02 16:36:53 ERROR ShutdownHookManager: Exception while deleting Spark temp dir: C:\Users\zilong\AppData\Local\Temp\1\spark-7917d05d-461f-4f96-8b42-03d7a2ce30d5\userFiles-8b4db58f-bd2b-42c8-b6a3-8f81ef07a2e6
java.io.IOException: Failed to delete: C:\Users\zilong\AppData\Local\Temp\1\spark-7917d05d-461f-4f96-8b42-03d7a2ce30d5\userFiles-8b4db58f-bd2b-42c8-b6a3-8f81ef07a2e6
at org.apache.spark.util.Utils$.deleteRecursively(Utils.scala:1074)
at org.apache.spark.util.ShutdownHookManager$$anonfun$1$$anonfun$apply$mcV$sp$3.apply(ShutdownHookManager.scala:65)
at org.apache.spark.util.ShutdownHookManager$$anonfun$1$$anonfun$apply$mcV$sp$3.apply(ShutdownHookManager.scala:62)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
at org.apache.spark.util.ShutdownHookManager$$anonfun$1.apply$mcV$sp(ShutdownHookManager.scala:62)
at org.apache.spark.util.SparkShutdownHook.run(ShutdownHookManager.scala:216)
at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ShutdownHookManager.scala:188)
at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(ShutdownHookManager.scala:188)
at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(ShutdownHookManager.scala:188)
at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1992)
at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply$mcV$sp(ShutdownHookManager.scala:188)
at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(ShutdownHookManager.scala:188)
at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(ShutdownHookManager.scala:188)
at scala.util.Try$.apply(Try.scala:192)
at org.apache.spark.util.SparkShutdownHookManager.runAll(ShutdownHookManager.scala:188)
at org.apache.spark.util.SparkShutdownHookManager$$anon$2.run(ShutdownHookManager.scala:178)
at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54)
22/09/02 16:36:53 INFO ShutdownHookManager: Deleting directory C:\Users\zilong\AppData\Local\Temp\1\spark-7917d05d-461f-4f96-8b42-03d7a2ce30d5
22/09/02 16:36:53 ERROR ShutdownHookManager: Exception while deleting Spark temp dir: C:\Users\zilong\AppData\Local\Temp\1\spark-7917d05d-461f-4f96-8b42-03d7a2ce30d5
java.io.IOException: Failed to delete: C:\Users\zilong\AppData\Local\Temp\1\spark-7917d05d-461f-4f96-8b42-03d7a2ce30d5
at org.apache.spark.util.Utils$.deleteRecursively(Utils.scala:1074)
at org.apache.spark.util.ShutdownHookManager$$anonfun$1$$anonfun$apply$mcV$sp$3.apply(ShutdownHookManager.scala:65)
at org.apache.spark.util.ShutdownHookManager$$anonfun$1$$anonfun$apply$mcV$sp$3.apply(ShutdownHookManager.scala:62)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
at org.apache.spark.util.ShutdownHookManager$$anonfun$1.apply$mcV$sp(ShutdownHookManager.scala:62)
at org.apache.spark.util.SparkShutdownHook.run(ShutdownHookManager.scala:216)
at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ShutdownHookManager.scala:188)
at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(ShutdownHookManager.scala:188)
at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(ShutdownHookManager.scala:188)
at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1992)
at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply$mcV$sp(ShutdownHookManager.scala:188)
at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(ShutdownHookManager.scala:188)
at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(ShutdownHookManager.scala:188)
at scala.util.Try$.apply(Try.scala:192)
at org.apache.spark.util.SparkShutdownHookManager.runAll(ShutdownHookManager.scala:188)
at org.apache.spark.util.SparkShutdownHookManager$$anon$2.run(ShutdownHookManager.scala:178)
at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54)

进程已结束,退出代码为 1

我的解答思路和尝试过的方法

增加了可以解析多分隔符的包hive-contrib.jar
资源包里面的hive-site.xml也尝试过修改hive.aux.jars.path对应的hive-contrib.jar路径
也尝试过把hive-contrib.jar打入maven库

我想要达到的结果

以上我写了两个查询语句,一个是单分隔符的orc外部表(从日志上看可以正常读取数据)
一个是多分隔符的txt外部表,无法访问
有什么办法可以正常访问这个多分隔符外部表

  • 写回答

1条回答

      报告相同问题?

      相关推荐 更多相似问题

      问题事件

      • 已结题 9月16日
      • 修改了问题 9月5日
      • 创建了问题 9月2日

      悬赏问题

      • ¥15 请问Unity使用Magica Cloth插件怎么调整散兵的帘子和振袖比较好?
      • ¥15 请问为什么原神模型导入Unity做动画默认姿势会站不直?
      • ¥15 ld: cannot open C:\Program: No such file or directory
      • ¥15 请问汏佬要怎么访问这种过滤了post和get的php内容?
      • ¥15 C语言调用python脚本出现Debug Assertion Failed错误
      • ¥15 C语言设计学生成绩管理系统
      • ¥15 关于服务器传输和unity的一些问题
      • ¥15 python qt 实现数据表 表头筛选
      • ¥15 关于电子电气架构的问题,如何解决?
      • ¥75 数据总结公式(克服遗忘问题)