DolphinScheduler 任务流一直运行,但单个任务实例未运行,显示invalid date

Question

海豚日常任务偶尔出现单个任务流一直保持运行状态,但是任务流中的某个任务实例未提交成功,提交时间显示invalid date

Answer 1

附上master 报错日志

```bash
[INFO] 2022-02-16 10:38:01.380 org.apache.dolphinscheduler.server.master.runner.MasterSchedulerService:[145] - find one command: id: 56419, type: REPEAT_RUNNING
[INFO] 2022-02-16 10:38:01.410 org.apache.dolphinscheduler.server.master.runner.MasterSchedulerService:[153] - start master exec thread , split DAG ...
[INFO] 2022-02-16 10:38:01.418 org.apache.dolphinscheduler.server.master.runner.MasterExecThread:[315] - prepare process :62521 end
[INFO] 2022-02-16 10:38:01.426 org.apache.dolphinscheduler.server.master.runner.MasterExecThread:[792] - add task to stand by list: 测试优化降噪
[INFO] 2022-02-16 10:38:01.426 org.apache.dolphinscheduler.server.master.runner.MasterExecThread:[805] - remove task from stand by list: 测试优化降噪
[INFO] 2022-02-16 10:38:01.440 org.apache.dolphinscheduler.service.process.ProcessService:[845] - start submit task : 测试优化降噪, instance id:62521, state: RUNNING_EXECUTION
[INFO] 2022-02-16 10:38:01.448 org.apache.dolphinscheduler.service.process.ProcessService:[858] - end submit task to db successfully:测试优化降噪 state:SUBMITTED_SUCCESS complete, instance id:62521 state: RUNNING_EXECUTION  
[INFO] 2022-02-16 10:38:01.455 org.apache.dolphinscheduler.server.master.runner.MasterTaskExecThread:[216] - task ready to submit: TaskInstance{id=133185, name='测试优化降噪', taskType='SPARK', processDefinitionId=283, processInstanceId=62521, processInstanceName='null', taskJson='{"conditionResult":"{\"successNode\":[\"\"],\"failedNode\":[\"\"]}","conditionsTask":false,"depList":[],"dependence":"{}","forbidden":false,"id":"tasks-86743","maxRetryTimes":0,"name":"测试优化降噪","params":"{\"mainArgs\":\"--oss ${oss} --month ${month}\",\"driverMemory\":\"4G\",\"executorMemory\":\"8G\",\"programType\":\"SCALA\",\"mainClass\":\"tv.gz.bj.complement.NewSample\",\"driverCores\":1,\"deployMode\":\"cluster\",\"executorCores\":\"4\",\"appName\":\"BjMobileData\",\"mainJar\":{\"id\":301},\"sparkVersion\":\"SPARK2\",\"numExecutors\":\"40\",\"localParams\":[],\"others\":\"--queue pg \\\\\\n--principal pg \\\\\\n--keytab /home/pg/pg.keytab\",\"resourceList\":[]}","preTasks":"[]","retryInterval":1,"runFlag":"NORMAL","taskInstancePriority":"MEDIUM","taskTimeoutParameter":{"enable":false,"interval":0},"timeout":"{\"enable\":false,\"strategy\":\"\"}","type":"SPARK","workerGroup":"task"}', state=SUBMITTED_SUCCESS, submitTime=Wed Feb 16 10:38:01 CST 2022, startTime=null, endTime=null, host='null', executePath='null', logPath='null', retryTimes=0, alertFlag=NO, processInstance=null, processDefine=null, pid=0, appLink='null', flag=YES, dependency='null', duration=null, maxRetryTimes=0, retryInterval=1, taskInstancePriority=MEDIUM, processInstancePriority=MEDIUM, dependentResult='null', workerGroup='task', executorId=11, executorName='null'}
[INFO] 2022-02-16 10:38:01.455 org.apache.dolphinscheduler.server.master.runner.MasterTaskExecThread:[227] - master submit success, task : 测试优化降噪
[INFO] 2022-02-16 10:38:01.463 org.apache.dolphinscheduler.server.master.runner.MasterTaskExecThread:[116] - wait task: process id: 62521, task id:133185, task name:测试优化降噪 complete
[INFO] 2022-02-16 10:38:02.336 org.apache.dolphinscheduler.server.master.processor.TaskAckProcessor:[69] - taskAckCommand : TaskExecuteAckCommand{taskInstanceId=133185, startTime=Wed Feb 16 10:38:02 CST 2022, host='172.16.0.248:1234', status=1, logPath='/mnt/disk1/dolphinscheduler/logs/283/62521/133185.log', executePath='/mnt/disk1/dolphinscheduler/exec/process/9/283/62521/133185'}
[INFO] 2022-02-16 10:48:12.997 org.apache.dolphinscheduler.server.master.runner.MasterSchedulerService:[145] - find one command: id: 56421, type: REPEAT_RUNNING
[INFO] 2022-02-16 10:48:13.032 org.apache.dolphinscheduler.server.master.runner.MasterSchedulerService:[153] - start master exec thread , split DAG ...
[INFO] 2022-02-16 10:48:13.039 org.apache.dolphinscheduler.server.master.runner.MasterExecThread:[315] - prepare process :62748 end
[INFO] 2022-02-16 10:48:13.046 org.apache.dolphinscheduler.server.master.runner.MasterExecThread:[792] - add task to stand by list: hmp_fc_query_hit_rate_di
[INFO] 2022-02-16 10:48:13.046 org.apache.dolphinscheduler.server.master.runner.MasterExecThread:[805] - remove task from stand by list: hmp_fc_query_hit_rate_di
[INFO] 2022-02-16 10:48:13.059 org.apache.dolphinscheduler.service.process.ProcessService:[845] - start submit task : hmp_fc_query_hit_rate_di, instance id:62748, state: RUNNING_EXECUTION
[INFO] 2022-02-16 10:48:13.064 org.apache.dolphinscheduler.service.process.ProcessService:[858] - end submit task to db successfully:hmp_fc_query_hit_rate_di state:SUBMITTED_SUCCESS complete, instance id:62748 state: RUNNING_EXECUTION  
[INFO] 2022-02-16 10:48:13.070 org.apache.dolphinscheduler.server.master.runner.MasterTaskExecThread:[216] - task ready to submit: TaskInstance{id=133190, name='hmp_fc_query_hit_rate_di', taskType='SPARK', processDefinitionId=375, processInstanceId=62748, processInstanceName='null', taskJson='{"conditionResult":"{\"successNode\":[\"\"],\"failedNode\":[\"\"]}","conditionsTask":false,"depList":[],"dependence":"{}","forbidden":false,"id":"tasks-1-5oh3ve","maxRetryTimes":2,"name":"hmp_fc_query_hit_rate_di","params":"{\"mainArgs\":\"${dt}\",\"driverMemory\":\"1G\",\"executorMemory\":\"2G\",\"programType\":\"SCALA\",\"mainClass\":\"com.gz.HmpFrequencyControlLog.queryResponseHitRate\",\"driverCores\":1,\"deployMode\":\"cluster\",\"executorCores\":\"4\",\"appName\":\"\",\"mainJar\":{\"id\":364},\"sparkVersion\":\"SPARK2\",\"numExecutors\":\"24\",\"localParams\":[{\"prop\":\"dt\",\"direct\":\"IN\",\"type\":\"VARCHAR\",\"value\":\"$[yyyy-MM-dd,HH]\"}],\"others\":\"--principal cleandatamanager \\\\\\n--keytab /home/cleandatamanager/cleandatamanager.keytab\",\"resourceList\":[]}","preTasks":"[]","retryInterval":5,"runFlag":"NORMAL","taskInstancePriority":"MEDIUM","taskTimeoutParameter":{"enable":true,"interval":30,"strategy":"FAILED"},"timeout":"{\"enable\":true,\"interval\":30,\"strategy\":\"FAILED\"}","type":"SPARK","workerGroup":"task"}', state=SUBMITTED_SUCCESS, submitTime=Wed Feb 16 10:48:13 CST 2022, startTime=null, endTime=null, host='null', executePath='null', logPath='null', retryTimes=0, alertFlag=NO, processInstance=null, processDefine=null, pid=0, appLink='null', flag=YES, dependency='null', duration=null, maxRetryTimes=2, retryInterval=5, taskInstancePriority=MEDIUM, processInstancePriority=MEDIUM, dependentResult='null', workerGroup='task', executorId=17, executorName='null'}
[INFO] 2022-02-16 10:48:13.070 org.apache.dolphinscheduler.server.master.runner.MasterTaskExecThread:[227] - master submit success, task : hmp_fc_query_hit_rate_di
[INFO] 2022-02-16 10:48:13.077 org.apache.dolphinscheduler.server.master.runner.MasterTaskExecThread:[116] - wait task: process id: 62748, task id:133190, task name:hmp_fc_query_hit_rate_di complete
[ERROR] 2022-02-16 10:48:13.449 org.apache.dolphinscheduler.remote.handler.NettyClientHandler:[184] - exceptionCaught : {}
io.netty.channel.unix.Errors$NativeIoException: readAddress(..) failed: Connection reset by peer
[INFO] 2022-02-16 10:48:33.530 org.apache.dolphinscheduler.server.master.runner.MasterExecThread:[764] - work flow process instance [id: 62521, name:降噪优化测试-0-1644831891724], state change from RUNNING_EXECUTION to READY_STOP, cmd type: REPEAT_RUNNING
[INFO] 2022-02-16 10:48:34.832 org.apache.dolphinscheduler.server.master.runner.MasterTaskExecThread:[199] - master kill taskInstance name :测试优化降噪 taskInstance id:133185
[INFO] 2022-02-16 10:48:34.866 org.apache.dolphinscheduler.server.master.processor.TaskResponseProcessor:[71] - received command : TaskExecuteResponseCommand{taskInstanceId=133185, status=9, endTime=Wed Feb 16 10:48:34 CST 2022, processId=13691, appIds='application_1642180738154_11248'}
[INFO] 2022-02-16 10:48:36.810 org.apache.dolphinscheduler.server.master.processor.TaskKillResponseProcessor:[49] - received task kill response command : TaskKillResponseCommand{taskInstanceId=0, host='null', status=6, processId=0, appIds=[application_1642180738154_11248]}
[INFO] 2022-02-16 10:48:36.871 org.apache.dolphinscheduler.server.master.runner.MasterTaskExecThread:[102] - task :测试优化降噪 id:133185, process id:62521, exec thread completed 
[INFO] 2022-02-16 10:48:37.590 org.apache.dolphinscheduler.server.master.runner.MasterExecThread:[864] - task :测试优化降噪, id:133185 complete, state is KILL 
[INFO] 2022-02-16 10:48:38.601 org.apache.dolphinscheduler.server.master.runner.MasterExecThread:[764] - work flow process instance [id: 62521, name:降噪优化测试-0-1644831891724], state change from READY_STOP to STOP, cmd type: STOP
[INFO] 2022-02-16 10:48:38.621 org.apache.dolphinscheduler.server.master.runner.MasterExecThread:[925] - process:62521 end, state :STOP
[INFO] 2022-02-16 10:48:38.646 org.apache.dolphinscheduler.server.utils.AlertManager:[255] - add alert to db , alert: Alert{id=4845, title='stop failed', showType=TABLE, content='', alertType=EMAIL, alertStatus=null, log='null', alertGroupId=0, receivers='[e-mail]', receiversCc='', createTime=Wed Feb 16 10:48:38 CST 2022, updateTime=null, info={}}
[INFO] 2022-02-16 10:49:32.756 org.apache.dolphinscheduler.server.master.runner.MasterSchedulerService:[145] - find one command: id: 56422, type: RECOVER_SUSPENDED_PROCESS
[INFO] 2022-02-16 10:49:32.786 org.apache.dolphinscheduler.server.master.runner.MasterSchedulerService:[153] - start master exec thread , split DAG ...
[INFO] 2022-02-16 10:49:32.801 org.apache.dolphinscheduler.server.master.runner.MasterExecThread:[315] - prepare process :62521 end
[INFO] 2022-02-16 10:49:32.808 org.apache.dolphinscheduler.server.master.runner.MasterExecThread:[792] - add task to stand by list: 测试优化降噪
[INFO] 2022-02-16 10:49:32.809 org.apache.dolphinscheduler.server.master.runner.MasterExecThread:[805] - remove task from stand by list: 测试优化降噪
[INFO] 2022-02-16 10:49:32.822 org.apache.dolphinscheduler.service.process.ProcessService:[845] - start submit task : 测试优化降噪, instance id:62521, state: RUNNING_EXECUTION
[INFO] 2022-02-16 10:49:32.829 org.apache.dolphinscheduler.service.process.ProcessService:[858] - end submit task to db successfully:测试优化降噪 state:SUBMITTED_SUCCESS complete, instance id:62521 state: RUNNING_EXECUTION  
[INFO] 2022-02-16 10:49:32.836 org.apache.dolphinscheduler.server.master.runner.MasterTaskExecThread:[216] - task ready to submit: TaskInstance{id=133191, name='测试优化降噪', taskType='SPARK', processDefinitionId=283, processInstanceId=62521, processInstanceName='null', taskJson='{"conditionResult":"{\"successNode\":[\"\"],\"failedNode\":[\"\"]}","conditionsTask":false,"depList":[],"dependence":"{}","forbidden":false,"id":"tasks-86743","maxRetryTimes":0,"name":"测试优化降噪","params":"{\"mainArgs\":\"--oss ${oss} --month ${month}\",\"driverMemory\":\"4G\",\"executorMemory\":\"8G\",\"programType\":\"SCALA\",\"mainClass\":\"tv.gz.bj.complement.NewSample\",\"driverCores\":1,\"deployMode\":\"cluster\",\"executorCores\":\"4\",\"appName\":\"BjMobileData\",\"mainJar\":{\"id\":301},\"sparkVersion\":\"SPARK2\",\"numExecutors\":\"40\",\"localParams\":[],\"others\":\"--queue pg \\\\\\n--principal pg \\\\\\n--keytab /home/pg/pg.keytab\",\"resourceList\":[]}","preTasks":"[]","retryInterval":1,"runFlag":"NORMAL","taskInstancePriority":"MEDIUM","taskTimeoutParameter":{"enable":false,"interval":0},"timeout":"{\"enable\":false,\"strategy\":\"\"}","type":"SPARK","workerGroup":"task"}', state=SUBMITTED_SUCCESS, submitTime=Wed Feb 16 10:49:32 CST 2022, startTime=null, endTime=null, host='null', executePath='null', logPath='null', retryTimes=0, alertFlag=NO, processInstance=null, processDefine=null, pid=0, appLink='null', flag=YES, dependency='null', duration=null, maxRetryTimes=0, retryInterval=1, taskInstancePriority=MEDIUM, processInstancePriority=MEDIUM, dependentResult='null', workerGroup='task', executorId=11, executorName='null'}
[INFO] 2022-02-16 10:49:32.836 org.apache.dolphinscheduler.server.master.runner.MasterTaskExecThread:[227] - master submit success, task : 测试优化降噪
[INFO] 2022-02-16 10:49:32.843 org.apache.dolphinscheduler.server.master.runner.MasterTaskExecThread:[116] - wait task: process id: 62521, task id:133191, task name:测试优化降噪 complete
[INFO] 2022-02-16 10:49:33.530 org.apache.dolphinscheduler.server.master.processor.TaskAckProcessor:[69] - taskAckCommand : TaskExecuteAckCommand{taskInstanceId=133191, startTime=Wed Feb 16 10:49:33 CST 2022, host='172.16.0.248:1234', status=1, logPath='/mnt/disk1/dolphinscheduler/logs/283/62521/133191.log', executePath='/mnt/disk1/dolphinscheduler/exec/process/9/283/62521/133191'}
[INFO] 2022-02-16 10:50:31.894 org.apache.dolphinscheduler.server.master.processor.TaskResponseProcessor:[71] - received command : TaskExecuteResponseCommand{taskInstanceId=133191, status=7, endTime=Wed Feb 16 10:50:31 CST 2022, processId=20837, appIds='application_1642180738154_11254'}
[INFO] 2022-02-16 10:50:33.763 org.apache.dolphinscheduler.server.master.runner.MasterTaskExecThread:[102] - task :测试优化降噪 id:133191, process id:62521, exec thread completed 
[INFO] 2022-02-16 10:50:34.305 org.apache.dolphinscheduler.server.master.runner.MasterExecThread:[864] - task :测试优化降噪, id:133191 complete, state is SUCCESS 
[INFO] 2022-02-16 10:50:35.313 org.apache.dolphinscheduler.server.master.runner.MasterExecThread:[764] - work flow process instance [id: 62521, name:降噪优化测试-0-1644831891724], state change from RUNNING_EXECUTION to SUCCESS, cmd type: RECOVER_SUSPENDED_PROCESS
[INFO] 2022-02-16 10:50:35.331 org.apache.dolphinscheduler.server.master.runner.MasterExecThread:[925] - process:62521 end, state :SUCCESS
[INFO] 2022-02-16 10:50:35.357 org.apache.dolphinscheduler.server.utils.AlertManager:[255] - add alert to db , alert: Alert{id=4846, title='recover suspended process success', showType=TEXT, content='["id:62521","name:降噪优化测试-0-1644831891724","job type: recover suspended process","state: SUCCESS","recovery:NO","run time: 28","start time: 2022-02-16 10:38:01","end time: 2022-02-16 10:50:35","host: 172.16.0.242:5678"]', alertType=EMAIL, alertStatus=null, log='null', alertGroupId=0, receivers='[e-mail]', receiversCc='', createTime=Wed Feb 16 10:50:35 CST 2022, updateTime=null, info={}}
[ERROR] 2022-02-16 10:51:55.136 org.apache.dolphinscheduler.remote.handler.NettyServerHandler:[153] - exceptionCaught : java.lang.IllegalArgumentException: illegal packet [magic]3
io.netty.handler.codec.DecoderException: java.lang.IllegalArgumentException: illegal packet [magic]3
    at io.netty.handler.codec.ReplayingDecoder.callDecode(ReplayingDecoder.java:421)
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
    at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795)
    at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475)
    at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException: illegal packet [magic]3
    at org.apache.dolphinscheduler.remote.codec.NettyDecoder.checkMagic(NettyDecoder.java:118)
    at org.apache.dolphinscheduler.remote.codec.NettyDecoder.decode(NettyDecoder.java:54)
    at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501)
    at io.netty.handler.codec.ReplayingDecoder.callDecode(ReplayingDecoder.java:366)
    ... 14 common frames omitted
[ERROR] 2022-02-16 10:51:55.136 org.apache.dolphinscheduler.remote.handler.NettyServerHandler:[153] - exceptionCaught : java.lang.IllegalArgumentException: illegal packet [magic]0
io.netty.handler.codec.DecoderException: java.lang.IllegalArgumentException: illegal packet [magic]0
    at io.netty.handler.codec.ReplayingDecoder.callDecode(ReplayingDecoder.java:421)
    at io.netty.handler.codec.ReplayingDecoder.channelInputClosed(ReplayingDecoder.java:329)
    at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:371)
    at io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:354)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241)
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1405)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248)
    at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:901)
    at io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:819)
    at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
    at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
    at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384)
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException: illegal packet [magic]0
    at org.apache.dolphinscheduler.remote.codec.NettyDecoder.checkMagic(NettyDecoder.java:118)
    at org.apache.dolphinscheduler.remote.codec.NettyDecoder.decode(NettyDecoder.java:54)
    at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501)
    at io.netty.handler.codec.ReplayingDecoder.callDecode(ReplayingDecoder.java:366)
    ... 17 common frames omitted

```