23-09-2021 16:37:49 CST jobA INFO - Starting job jobA at 1632386269130
23-09-2021 16:37:49 CST jobA INFO - job JVM args: '-Dazkaban.flowid=dwd' '-Dazkaban.execid=66' '-Dazkaban.jobid=jobA'
23-09-2021 16:37:49 CST jobA INFO - user.to.proxy property was not set, defaulting to submit user MBY
23-09-2021 16:37:49 CST jobA INFO - Attached Ramp Props : [{}]
23-09-2021 16:37:49 CST jobA INFO - Building command job executor.
23-09-2021 16:37:49 CST jobA INFO - Failed with 5 inputs with exception e = null
23-09-2021 16:37:49 CST jobA INFO - Memory granted for job jobA
23-09-2021 16:37:49 CST jobA INFO - 1 commands to execute.
23-09-2021 16:37:49 CST jobA INFO - cwd=/root/executions/66
23-09-2021 16:37:49 CST jobA INFO - effective user is: MBY
23-09-2021 16:37:49 CST jobA INFO - Command: sh dwd.sh
23-09-2021 16:37:49 CST jobA INFO - Environment variables: {JOB_OUTPUT_PROP_FILE=/root/executions/66/jobA_output_4222314727082316537_tmp, JOB_PROP_FILE=/root/executions/66/jobA_job_props_3495378608077107121_tmp, KRB5CCNAME=/tmp/krb5cc__dwd__dwd__jobA__66__MBY, JOB_NAME=jobA}
23-09-2021 16:37:49 CST jobA INFO - Working directory: /root/executions/66
23-09-2021 16:37:49 CST jobA INFO - Spawned process with id 4423
23-09-2021 16:37:49 CST jobA INFO - which: no hbase in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/module/jdk/bin:/opt/module/hadoop/bin:/opt/module/hadoop/sbin)
23-09-2021 16:37:51 CST jobA INFO - SLF4J: Class path contains multiple SLF4J bindings.
23-09-2021 16:37:51 CST jobA INFO - SLF4J: Found binding in [jar:file:/opt/module/hive/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
23-09-2021 16:37:51 CST jobA INFO - SLF4J: Found binding in [jar:file:/opt/module/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
23-09-2021 16:37:51 CST jobA INFO - SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
23-09-2021 16:37:51 CST jobA INFO - SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
23-09-2021 16:37:51 CST jobA INFO -
23-09-2021 16:37:51 CST jobA INFO - Logging initialized using configuration in file:/opt/module/hive/conf/hive-log4j2.properties Async: true
23-09-2021 16:37:54 CST jobA INFO - Exception in thread "main" java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
23-09-2021 16:37:54 CST jobA INFO - at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:591)
23-09-2021 16:37:54 CST jobA INFO - at org.apache.hadoop.hive.ql.session.SessionState.beginStart(SessionState.java:531)
23-09-2021 16:37:54 CST jobA INFO - at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:705)
23-09-2021 16:37:54 CST jobA INFO - at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:641)
23-09-2021 16:37:54 CST jobA INFO - at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
23-09-2021 16:37:54 CST jobA INFO - at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
23-09-2021 16:37:54 CST jobA INFO - at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
23-09-2021 16:37:54 CST jobA INFO - at java.lang.reflect.Method.invoke(Method.java:498)
23-09-2021 16:37:54 CST jobA INFO - at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
23-09-2021 16:37:54 CST jobA INFO - at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
23-09-2021 16:37:54 CST jobA INFO - Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
23-09-2021 16:37:54 CST jobA INFO - at org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:226)
23-09-2021 16:37:54 CST jobA INFO - at org.apache.hadoop.hive.ql.metadata.Hive.<init>(Hive.java:366)
23-09-2021 16:37:54 CST jobA INFO - at org.apache.hadoop.hive.ql.metadata.Hive.create(Hive.java:310)
23-09-2021 16:37:54 CST jobA INFO - at org.apache.hadoop.hive.ql.metadata.Hive.getInternal(Hive.java:290)
23-09-2021 16:37:54 CST jobA INFO - at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:266)
23-09-2021 16:37:54 CST jobA INFO - at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:558)
23-09-2021 16:37:54 CST jobA INFO - ... 9 more
23-09-2021 16:37:54 CST jobA INFO - Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
23-09-2021 16:37:54 CST jobA INFO - at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1654)
23-09-2021 16:37:54 CST jobA INFO - at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:80)
23-09-2021 16:37:54 CST jobA INFO - at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:130)
23-09-2021 16:37:54 CST jobA INFO - at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:101)
23-09-2021 16:37:54 CST jobA INFO - at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3367)
23-09-2021 16:37:54 CST jobA INFO - at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3406)
23-09-2021 16:37:54 CST jobA INFO - at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3386)
23-09-2021 16:37:54 CST jobA INFO - at org.apache.hadoop.hive.ql.metadata.Hive.getAllFunctions(Hive.java:3640)
23-09-2021 16:37:54 CST jobA INFO - at org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:236)
23-09-2021 16:37:54 CST jobA INFO - at org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:221)
23-09-2021 16:37:54 CST jobA INFO - ... 14 more
23-09-2021 16:37:54 CST jobA INFO - Caused by: java.lang.reflect.InvocationTargetException
23-09-2021 16:37:54 CST jobA INFO - at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
23-09-2021 16:37:54 CST jobA INFO - at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
23-09-2021 16:37:54 CST jobA INFO - at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
23-09-2021 16:37:54 CST jobA INFO - at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
23-09-2021 16:37:54 CST jobA INFO - at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1652)
23-09-2021 16:37:54 CST jobA INFO - ... 23 more
23-09-2021 16:37:54 CST jobA INFO - Caused by: MetaException(message:Version information not found in metastore. )
23-09-2021 16:37:54 CST jobA INFO - at org.apache.hadoop.hive.metastore.ObjectStore.checkSchema(ObjectStore.java:7753)
23-09-2021 16:37:54 CST jobA INFO - at org.apache.hadoop.hive.metastore.ObjectStore.verifySchema(ObjectStore.java:7731)
23-09-2021 16:37:54 CST jobA INFO - at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
23-09-2021 16:37:54 CST jobA INFO - at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
23-09-2021 16:37:54 CST jobA INFO - at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
23-09-2021 16:37:54 CST jobA INFO - at java.lang.reflect.Method.invoke(Method.java:498)
23-09-2021 16:37:54 CST jobA INFO - at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:101)
23-09-2021 16:37:54 CST jobA INFO - at com.sun.proxy.$Proxy21.verifySchema(Unknown Source)
23-09-2021 16:37:54 CST jobA INFO - at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:565)
23-09-2021 16:37:54 CST jobA INFO - at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:626)
23-09-2021 16:37:54 CST jobA INFO - at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:416)
23-09-2021 16:37:54 CST jobA INFO - at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:78)
23-09-2021 16:37:54 CST jobA INFO - at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:84)
23-09-2021 16:37:54 CST jobA INFO - at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:6490)
23-09-2021 16:37:54 CST jobA INFO - at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:238)
23-09-2021 16:37:54 CST jobA INFO - at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:70)
23-09-2021 16:37:54 CST jobA INFO - ... 28 more
23-09-2021 16:37:54 CST jobA INFO - Process with id 4423 completed unsuccessfully in 4 seconds.
23-09-2021 16:37:54 CST jobA ERROR - Job run failed!
java.lang.RuntimeException: azkaban.jobExecutor.utils.process.ProcessFailureException: Process exited with code 1
at azkaban.jobExecutor.ProcessJob.run(ProcessJob.java:312)
at azkaban.execapp.JobRunner.runJob(JobRunner.java:823)
at azkaban.execapp.JobRunner.doRun(JobRunner.java:602)
at azkaban.execapp.JobRunner.run(JobRunner.java:563)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: azkaban.jobExecutor.utils.process.ProcessFailureException: Process exited with code 1
at azkaban.jobExecutor.utils.process.AzkabanProcess.run(AzkabanProcess.java:125)
at azkaban.jobExecutor.ProcessJob.run(ProcessJob.java:304)
... 8 more
23-09-2021 16:37:54 CST jobA ERROR - azkaban.jobExecutor.utils.process.ProcessFailureException: Process exited with code 1 cause: azkaban.jobExecutor.utils.process.ProcessFailureException: Process exited with code 1
23-09-2021 16:37:54 CST jobA INFO - Finishing job jobA at 1632386274019 with status FAILED
将上面的文件压缩成zip包在azkaban中进行调度执行失败
以下是对应zip包里的文件
dwd.sh
#!/bin/bash
yesterday=20210913
action=02
/opt/module/hive/bin/hive \
-hiveconf action=$action \
-hiveconf yesterday=$yesterday \
-f dwd.hql
dwd.project
azkaban-flow-version: 2.0
dwd.hql
set hive.exec.mode.local.auto=true;
set hive.exec.dynamic.partition=true;
set hive.exec.dynamic.partition.mode=nonstrict;
set hive.exec.max.dynamic.partitions=1000;
set hive.exec.max.dynamic.partitions.pernode=100;
insert overwrite table dwd_nshop.dwd_nshop_actlog_launch partition(bdp_day)
select
customer_id,
device_num,
device_type,
os,
os_version,
manufacturer,
carrier,
network_type,
area_code,
from_unixtime(cast(ct/1000 as int),'HH'),
ct,
bdp_day
from ods_nshop.ods_nshop_01_useractlog
where
bdp_day='${hiveconf:bdp_day}'
and
action='${hiveconf:action}';
dwd.flow
nodes:
- name: jobA
type: command
config:
command: sh dwd.sh
但是直接在linux中执行sh文件是可以成功的，文件对应的也改了uinx格式，这个怎么破？
