weixin_48484941 2021-10-26 14:21 采纳率: 62.5%
浏览 21
已结题

spark在yarn模式下报UnsatisfiedLinkError错误,请问知道是什么问题么或应该怎么排查呢?

spark yarn 模式下报错。之前本地可以,Stand alone模式下报错。在hadoop的配置mapred-site.xml增加内容,修改之后standalone模式可以。但是yarn模式下又不行。错误如下: 请问有谁遇到过,或者应该怎么排查呢?

2021-10-26 14:11:12,544 INFO scheduler.DAGScheduler: ShuffleMapStage 2 (distinct at SparkStreamTest.java:136) failed in 1.367 s due to Job aborted due to stage failure: Task 0 in stage 2.0 failed 4 times, most recent failure: Lost task 0.3 in stage 2.0 (TID 13) (master executor 1): java.lang.UnsatisfiedLinkError: org.gdal.ogr.ogrJNI.CreateGeometryFromWkt__SWIG_1(Ljava/lang/String;)J
    at org.gdal.ogr.ogrJNI.CreateGeometryFromWkt__SWIG_1(Native Method)
    at org.gdal.ogr.ogr.CreateGeometryFromWkt(ogr.java:43)
    at org.gdal.ogr.Geometry.CreateFromWkt(Geometry.java:93)
    at com.test.SparkStreamTest.lambda$null$39b6b33d$1(SparkStreamTest.java:112)
    at org.apache.spark.api.java.JavaPairRDD$.$anonfun$toScalaFunction$1(JavaPairRDD.scala:1070)
    at scala.collection.Iterator$$anon$10.next(Iterator.scala:459)
    at scala.collection.Iterator$$anon$11.nextCur(Iterator.scala:484)
    at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:490)
    at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
    at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:192)
    at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:62)
    at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)
    at org.apache.spark.scheduler.Task.run(Task.scala:131)
    at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:497)
    at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1439)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:500)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
  • 写回答

1条回答 默认 最新

  • 明夜再说 2021-10-29 16:20
    关注

    SparkStreamTest.java:112这不都告诉你哪行出错了,去找找呗。看着像是jar包冲突了,这行代码的依赖是不是有问题?

    评论

报告相同问题?

问题事件

  • 已结题 (查看结题原因) 11月16日
  • 修改了问题 10月26日
  • 创建了问题 10月26日

悬赏问题

  • ¥30 dspbuilder中使用signalcompiler时报错Error during compilation: Fitter failed,求解决办法
  • ¥15 gwas 分析-数据质控之过滤稀有突变中出现的问题
  • ¥15 没有注册类 (异常来自 HRESULT: 0x80040154 (REGDB_E_CLASSNOTREG))
  • ¥15 知识蒸馏实战博客问题
  • ¥15 用PLC设计纸袋糊底机送料系统
  • ¥15 simulink仿真中dtc控制永磁同步电机如何控制开关频率
  • ¥15 用C语言输入方程怎么
  • ¥15 网站显示不安全连接问题
  • ¥15 51单片机显示器问题
  • ¥20 关于#qt#的问题:Qt代码的移植问题