cloudera中yarn重启不了
yarn的运行状态测试如图
我是从cloudera manager的yarn实例重启，重启成功后又会有问题
日志一直在重复打印
2023-01-12 02:27:45,443 INFO org.apache.hadoop.yarn.server.nodemanager.NodeManager: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NodeManager
STARTUP_MSG: host = cdh4/192.168.0.104
STARTUP_MSG: args = []
STARTUP_MSG: version = 3.0.0-cdh6.2.0
STARTUP_MSG: classpath = /var/run/cloudera-sc.....
STARTUP_MSG: build = http://github.com/cloudera/hadoop -r d1dff3d3a126da44e3458bbf148c3bc16ff55bd8; compiled by 'jenkins' on 2019-03-14T06:39Z
STARTUP_MSG: java = 1.8.0_181
************************************************************/
2023-01-12 02:27:45,471 INFO org.apache.hadoop.yarn.server.nodemanager.NodeManager: registered UNIX signal handlers for [TERM, HUP, INT]
2023-01-12 02:27:45,945 INFO org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService: Using state database at /var/lib/hadoop-yarn/yarn-nm-recovery/yarn-nm-state for recovery
2023-01-12 02:27:45,971 INFO org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService$LeveldbLogger: Recovering log #394474
2023-01-12 02:27:45,971 INFO org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService$LeveldbLogger: Level-0 table #394476: started
2023-01-12 02:27:45,971 INFO org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService$LeveldbLogger: Level-0 table #394476: 0 bytes OK
2023-01-12 02:27:46,031 INFO org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService$LeveldbLogger: Delete type=0 #394474
2023-01-12 02:27:46,031 INFO org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService$LeveldbLogger: Delete type=3 #394472
2023-01-12 02:27:46,045 INFO org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService: Loaded NM state version info 1.2
2023-01-12 02:27:46,283 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.resourceplugin.ResourcePluginManager: No Resource plugins found from configuration!
2023-01-12 02:27:46,283 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.resourceplugin.ResourcePluginManager: Found Resource plugins from configuration: null
2023-01-12 02:27:46,316 INFO org.apache.hadoop.yarn.server.nodemanager.NodeManager: Node Manager health check script is not available or doesn't have execute permission, so not starting the node health script runner.
2023-01-12 02:27:46,354 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerEventType for class org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ContainerEventDispatcher
2023-01-12 02:27:46,355 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationEventType for class org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher
2023-01-12 02:27:46,355 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.event.LocalizationEventType for class org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$LocalizationEventHandlerWrapper
2023-01-12 02:27:46,356 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServicesEventType for class org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices
2023-01-12 02:27:46,356 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorEventType for class org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl
2023-01-12 02:27:46,357 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainersLauncherEventType for class org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainersLauncher
2023-01-12 02:27:46,357 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.nodemanager.containermanager.scheduler.ContainerSchedulerEventType for class org.apache.hadoop.yarn.server.nodemanager.containermanager.scheduler.ContainerScheduler
2023-01-12 02:27:46,373 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.nodemanager.ContainerManagerEventType for class org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl
2023-01-12 02:27:46,374 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.nodemanager.NodeManagerEventType for class org.apache.hadoop.yarn.server.nodemanager.NodeManager
2023-01-12 02:27:46,416 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: Loaded properties from hadoop-metrics2.properties
2023-01-12 02:27:46,474 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s).
2023-01-12 02:27:46,474 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NodeManager metrics system started
2023-01-12 02:27:46,497 INFO org.apache.hadoop.yarn.server.nodemanager.DirectoryCollection: Disk Validator: yarn.nodemanager.disk-validator is loaded.
2023-01-12 02:27:46,507 INFO org.apache.hadoop.yarn.server.nodemanager.DirectoryCollection: Disk Validator: yarn.nodemanager.disk-validator is loaded.
2023-01-12 02:27:46,530 INFO org.apache.hadoop.yarn.server.nodemanager.NodeResourceMonitorImpl: Using ResourceCalculatorPlugin : org.apache.hadoop.yarn.util.ResourceCalculatorPlugin@1a45193b
2023-01-12 02:27:46,532 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.event.LogHandlerEventType for class org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService
2023-01-12 02:27:46,534 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.sharedcache.SharedCacheUploadEventType for class org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.sharedcache.SharedCacheUploadService
2023-01-12 02:27:46,534 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: AMRMProxyService is disabled
2023-01-12 02:27:46,534 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: per directory file limit = 8192
2023-01-12 02:27:46,537 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Disk Validator: yarn.nodemanager.disk-validator is loaded.
2023-01-12 02:27:46,542 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.event.LocalizerEventType for class org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerTracker
2023-01-12 02:27:46,571 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Adding auxiliary service mapreduce_shuffle, "mapreduce_shuffle"
2023-01-12 02:27:46,781 INFO org.apache.spark.network.yarn.YarnShuffleService: Initializing YARN shuffle service for Spark
2023-01-12 02:27:46,781 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Adding auxiliary service spark_shuffle, "spark_shuffle"
2023-01-12 02:27:46,836 INFO org.apache.spark.network.util.LevelDBProvider$LevelDBLogger: Recovering log #34007
2023-01-12 02:27:46,836 INFO org.apache.spark.network.util.LevelDBProvider$LevelDBLogger: Level-0 table #34009: started
2023-01-12 02:27:46,864 INFO org.apache.spark.network.util.LevelDBProvider$LevelDBLogger: Level-0 table #34009: 145 bytes OK
2023-01-12 02:27:46,898 INFO org.apache.spark.network.util.LevelDBProvider$LevelDBLogger: Delete type=0 #34007
2023-01-12 02:27:46,898 INFO org.apache.spark.network.util.LevelDBProvider$LevelDBLogger: Delete type=3 #34005
2023-01-12 02:27:46,898 INFO org.apache.spark.network.util.LevelDBProvider$LevelDBLogger: Compacting 4@0 + 1@1 files
2023-01-12 02:27:46,917 INFO org.apache.spark.network.util.LevelDBProvider$LevelDBLogger: Generated table #34011: 1 keys, 145 bytes
2023-01-12 02:27:46,917 INFO org.apache.spark.network.util.LevelDBProvider$LevelDBLogger: Compacted 4@0 + 1@1 files => 145 bytes
2023-01-12 02:27:46,923 INFO org.apache.spark.network.util.LevelDBProvider$LevelDBLogger: compacted to: files[ 0 1 0 0 0 0 0 ]
2023-01-12 02:27:46,923 INFO org.apache.spark.network.util.LevelDBProvider$LevelDBLogger: Delete type=2 #33998
2023-01-12 02:27:46,923 INFO org.apache.spark.network.util.LevelDBProvider$LevelDBLogger: Delete type=2 #34000
2023-01-12 02:27:46,923 INFO org.apache.spark.network.util.LevelDBProvider$LevelDBLogger: Delete type=2 #34003
2023-01-12 02:27:46,923 INFO org.apache.spark.network.util.LevelDBProvider$LevelDBLogger: Delete type=2 #34006
2023-01-12 02:27:46,924 INFO org.apache.spark.network.util.LevelDBProvider$LevelDBLogger: Delete type=2 #34009
2023-01-12 02:27:47,202 INFO org.apache.spark.network.yarn.YarnShuffleService: Started YARN shuffle service for Spark on port 7337. Authentication is not enabled. Registered executor file is /var/lib/hadoop-yarn/yarn-nm-recovery/nm-aux-services/spark_shuffle/registeredExecutors.ldb
2023-01-12 02:27:47,202 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Using ResourceCalculatorPlugin : org.apache.hadoop.yarn.util.ResourceCalculatorPlugin@127d7908
2023-01-12 02:27:47,202 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Using ResourceCalculatorProcessTree : null
2023-01-12 02:27:47,217 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Physical memory check enabled: true
2023-01-12 02:27:47,217 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Virtual memory check enabled: false
2023-01-12 02:27:47,217 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: ContainersMonitor enabled: true
2023-01-12 02:27:47,219 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService: rollingMonitorInterval is set as -1. The log rolling monitoring interval is disabled. The logs will be aggregated after this application is finished.
目前yarn集群有6个，但是只有3个节点，所有很多任务一直跑不动
