WJC_SX 2022-12-15 15:54 采纳率: 95%
浏览 56
已结题

Spark 读取 Hive 数据报错 NoSuchMethodError : org.apache.spark.sql.catalyst.catalog.SessionCatalog

Spark 读取 Hive 数据报错 NoSuchMethodError : org.apache.spark.sql.catalyst.catalog.SessionCatalog.

代码如下:

val spark: SparkSession = SparkSession.builder
  .appName("distinct") // optional and will be autogenerated if not specified
  //.master("yarn") // avoid hardcoding the deployment environment
  .master("local[*]")
  .enableHiveSupport() // self-explanatory, isn't it?
  .getOrCreate

spark.sql("use ods_hive_test")

报错代码信息:

22/12/15 15:32:43 INFO BlockManagerMasterEndpoint: Registering block manager LAPTOP-62E6B54A:9900 with 1986.6 MB RAM, BlockManagerId(driver, LAPTOP-62E6B54A, 9900, None)
22/12/15 15:32:43 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, LAPTOP-62E6B54A, 9900, None)
22/12/15 15:32:43 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, LAPTOP-62E6B54A, 9900, None)
22/12/15 15:32:43 INFO SharedState: Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir ('file:/D:/JavaProject/SparkTest/Spark-restapi/springboot-spark-master/sparkjob/spark-warehouse/').
22/12/15 15:32:43 INFO SharedState: Warehouse path is 'file:/D:/JavaProject/SparkTest/Spark-restapi/springboot-spark-master/sparkjob/spark-warehouse/'.
22/12/15 15:32:43 INFO StateStoreCoordinatorRef: Registered StateStoreCoordinator endpoint
Exception in thread "main" java.lang.NoSuchMethodError: org.apache.spark.sql.catalyst.catalog.SessionCatalog.<init>(Lscala/Function0;Lscala/Function0;Lorg/apache/spark/sql/catalyst/analysis/FunctionRegistry;Lorg/apache/spark/sql/internal/SQLConf;Lorg/apache/hadoop/conf/Configuration;Lorg/apache/spark/sql/catalyst/parser/ParserInterface;Lorg/apache/spark/sql/catalyst/catalog/FunctionResourceLoader;)V
    at org.apache.spark.sql.hive.HiveSessionCatalog.<init>(HiveSessionCatalog.scala:50)
    at org.apache.spark.sql.hive.HiveSessionStateBuilder.catalog$lzycompute(HiveSessionStateBuilder.scala:53)
    at org.apache.spark.sql.hive.HiveSessionStateBuilder.catalog(HiveSessionStateBuilder.scala:52)
    at org.apache.spark.sql.hive.HiveSessionStateBuilder$$anon$1.<init>(HiveSessionStateBuilder.scala:69)
    at org.apache.spark.sql.hive.HiveSessionStateBuilder.analyzer(HiveSessionStateBuilder.scala:69)
    at org.apache.spark.sql.internal.BaseSessionStateBuilder$$anonfun$build$2.apply(BaseSessionStateBuilder.scala:293)
    at org.apache.spark.sql.internal.BaseSessionStateBuilder$$anonfun$build$2.apply(BaseSessionStateBuilder.scala:293)
    at org.apache.spark.sql.internal.SessionState.analyzer$lzycompute(SessionState.scala:79)
    at org.apache.spark.sql.internal.SessionState.analyzer(SessionState.scala:79)
    at org.apache.spark.sql.execution.QueryExecution.analyzed$lzycompute(QueryExecution.scala:57)
    at org.apache.spark.sql.execution.QueryExecution.analyzed(QueryExecution.scala:55)
    at org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:47)
    at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:74)
    at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:642)
    at SparkTest$.main(SparkTest.scala:18)
    at SparkTest.main(SparkTest.scala)
22/12/15 15:32:44 INFO SparkContext: Invoking stop() from shutdown hook

  • 写回答

1条回答 默认 最新

  • AllenGd 大数据领域优质创作者 2022-12-15 17:33
    关注

    22/12/15 15:32:44 INFO SparkContext: Invoking stop() from shutdown hook
    集群资源不足,且动态资源分配申请的executors、内存、核数过多
    关闭动态资源分配:set spark.dynamicAllocation.enabled = false

    本回答被题主选为最佳回答 , 对您是否有帮助呢?
    评论

报告相同问题?

问题事件

  • 系统已结题 1月7日
  • 已采纳回答 12月30日
  • 创建了问题 12月15日

悬赏问题

  • ¥15 帮我解决一下膳食平衡的线性规划模型的数据实例
  • ¥40 万年历缺少农历,需要和阳历同时显示
  • ¥250 雷电模拟器内存穿透、寻基址和特征码的教学
  • ¥200 比特币ord程序wallet_constructor.rs文件支持一次性铸造1000个代币,并将它们分配到40个UTXO上(每个UTXO上分配25个代币),并设置找零地址
  • ¥15 关于Java的学习问题
  • ¥15 如何使用chatgpt完成文本分类任务?
  • ¥15 已知速度v关于位置s的等式,怎么转化为已知位置求速度v的等式
  • ¥15 我有个餐饮系统,用wampserver把环境配置好了,但是后端的网页却进去,是为什么,能不能帮远程一下?
  • ¥15 R运行没有名称为"species"的插槽对于此对象类"SDMmodelCV"
  • ¥20 基于决策树的数字信号处理,2ask 2psk 2fsk的代码,检查下报错的原因