spark yarn 模式下报错。之前本地可以,Stand alone模式下报错。在hadoop的配置mapred-site.xml增加内容,修改之后standalone模式可以。但是yarn模式下又不行。错误如下: 请问有谁遇到过,或者应该怎么排查呢?
2021-10-26 14:11:12,544 INFO scheduler.DAGScheduler: ShuffleMapStage 2 (distinct at SparkStreamTest.java:136) failed in 1.367 s due to Job aborted due to stage failure: Task 0 in stage 2.0 failed 4 times, most recent failure: Lost task 0.3 in stage 2.0 (TID 13) (master executor 1): java.lang.UnsatisfiedLinkError: org.gdal.ogr.ogrJNI.CreateGeometryFromWkt__SWIG_1(Ljava/lang/String;)J
at org.gdal.ogr.ogrJNI.CreateGeometryFromWkt__SWIG_1(Native Method)
at org.gdal.ogr.ogr.CreateGeometryFromWkt(ogr.java:43)
at org.gdal.ogr.Geometry.CreateFromWkt(Geometry.java:93)
at com.test.SparkStreamTest.lambda$null$39b6b33d$1(SparkStreamTest.java:112)
at org.apache.spark.api.java.JavaPairRDD$.$anonfun$toScalaFunction$1(JavaPairRDD.scala:1070)
at scala.collection.Iterator$$anon$10.next(Iterator.scala:459)
at scala.collection.Iterator$$anon$11.nextCur(Iterator.scala:484)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:490)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:192)
at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:62)
at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)
at org.apache.spark.scheduler.Task.run(Task.scala:131)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:497)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1439)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:500)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)