请教:我的程序是只能在8088界面显示一个AppId 是running状态,其他都是ACCEPTED状态。尝试修改了spark-env以及yarn-site.xml,spark-defaults.conf,以及capacity-scheduler.xml都没有什么作用。
1. 1.1 vim yarn-site.xml
scp -r /usr/local/hadoop-2.7.1/etc/hadoop/yarn-site.xml root@xiuba112:/usr/local/hadoop-2.7.1/etc/hadoop/
yarn.nodemanager.aux-services.spark_shuffle.class
org.apache.spark.network.yarn.YarnShuffleService
spark.shuffle.service.port
7337
yarn.nodemanager.aux-services.mapreduce_shuffle.class
org.apache.hadoop.mapred.ShuffleHandler
1.2 添加依赖的jar包 cp /usr/local/spark-2.2.1-bin-hadoop2.7/yarn/spark-2.2.1-yarn-shuffle.jar /usr/local/hadoop-2.7.1/share/hadoop/yarn/lib/
拷贝“${SPARK_HOME}/lib/spark-1.3.0-yarn-shuffle.jar”到“${HADOOP_HOME}/share/hadoop/yarn/lib/”目录下。
note:高版本没有lib目录,有jars目录,比如说spark-2.0.2-yarn-shuffle.jar就在${SPARK_HOME}/yarn目录下,将其复制到${HADOOP_HOME}/share/hadoop/yarn/lib目录下。
1.3 重启NodeManager进程
- scp -r /usr/local/spark-2.2.1-bin-hadoop2.7/conf/spark-defaults.conf root@xiuba112:/usr/local/spark-2.2.1-bin-hadoop2.7/conf/ 在“spark-defaults.conf”中必须添加如下配置项: spark.shuffle.service.enabled=true spark.shuffle.service.port=7337
1和2不能解决问题
- vim /usr/local/spark-2.2.1-bin-hadoop2.7/conf/spark-env.sh conf/spark-env.sh中,同时在节点 /etc/profile中也添加一行
export YARN_CONF_DIR=$HADOOP_HOME/etc/hadoop
3不能解决问题。