weixin_48484941
2021-10-26 14:21
采纳率: 70%
浏览 18
已结题

spark在yarn模式下报UnsatisfiedLinkError错误,请问知道是什么问题么或应该怎么排查呢?

spark yarn 模式下报错。之前本地可以,Stand alone模式下报错。在hadoop的配置mapred-site.xml增加内容,修改之后standalone模式可以。但是yarn模式下又不行。错误如下: 请问有谁遇到过,或者应该怎么排查呢?

2021-10-26 14:11:12,544 INFO scheduler.DAGScheduler: ShuffleMapStage 2 (distinct at SparkStreamTest.java:136) failed in 1.367 s due to Job aborted due to stage failure: Task 0 in stage 2.0 failed 4 times, most recent failure: Lost task 0.3 in stage 2.0 (TID 13) (master executor 1): java.lang.UnsatisfiedLinkError: org.gdal.ogr.ogrJNI.CreateGeometryFromWkt__SWIG_1(Ljava/lang/String;)J
    at org.gdal.ogr.ogrJNI.CreateGeometryFromWkt__SWIG_1(Native Method)
    at org.gdal.ogr.ogr.CreateGeometryFromWkt(ogr.java:43)
    at org.gdal.ogr.Geometry.CreateFromWkt(Geometry.java:93)
    at com.test.SparkStreamTest.lambda$null$39b6b33d$1(SparkStreamTest.java:112)
    at org.apache.spark.api.java.JavaPairRDD$.$anonfun$toScalaFunction$1(JavaPairRDD.scala:1070)
    at scala.collection.Iterator$$anon$10.next(Iterator.scala:459)
    at scala.collection.Iterator$$anon$11.nextCur(Iterator.scala:484)
    at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:490)
    at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
    at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:192)
    at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:62)
    at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)
    at org.apache.spark.scheduler.Task.run(Task.scala:131)
    at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:497)
    at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1439)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:500)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

1条回答 默认 最新

相关推荐 更多相似问题