weixin_43113933 2021-12-14 16:58 采纳率: 58.3%
浏览 34
已结题

cdh6 hive udf一直报错

在本地虚拟机跑没有问题,这个udf是写的复杂些,但本地虚拟机跑是正常的,并且后来按照网上教程改了cdh对应的udf编写,可是一直报错,我都服了,是对json脱敏的hive udf,下面是任务日志,我以前绝望了,百度谷歌都不管用,没有一个有效方案,udf写的很简单的cdh上的hue是可以用,比如切割字符串,我写的这个就不行看能不能有大niu看看情况

img

2021-12-14 11:54:53,846 ERROR [IPC Server handler 7 on 45968] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task: attempt_1639027134846_0042_m_000000_0 - exited : java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"line":"{\"_id\":{\"$oid\":\"5fbe1bb08f35080030ea58d4\"},\"hasGeneratedId\":true,\"organisation\":{\"$oid\":\"5ef1d02f924838008888a688\"},\"lrs_id\":{\"$oid\":\"5ef1d12314e1340047ee4774\"},\"client\":{\"$oid\":\"5ef1d12314e1340047ee4775\"},\"person\":null,\"active\":true,\"voided\":false,\"timestamp\":{\"$date\":\"2020-11-25T08:54:08.922Z\"},\"stored\":{\"$date\":\"2020-11-25T08:54:08.922Z\"},\"hash\":\"e41e1684c538a83376913681cabaaf125f63708d\",\"agents\":[\"http://www.ai-augmented.com|19900000630\"],\"relatedAgents\":[\"http://www.ai-augmented.com|19900000630\",\"http://www-staging.ai-augmented.com/|xiaoya-test\"],\"registrations\":[],\"verbs\":[\"https://w3id.org/xapi/adl/verbs/logged-in\"],\"activities\":[\"http://www.api-staging.xyfolder.com/auth/user/signin\"],\"relatedActivities\":[\"http://www.api-staging.xyfolder.com/auth/user/signin\"],\"statement\":{\"version\":\"1.0.3\",\"actor\":{\"account\":{\"homePage\":\"http://www.ai-augmented.com\",\"name\":\"19900000630\"},\"objectType\":\"Agent\",\"name\":\"19900000630\"},\"verb\":{\"display\":{\"en\":\"logged-in\",\"zh-CN\":\"??\"},\"id\":\"https://w3id.org/xapi/adl/verbs/logged-in\"},\"object\":{\"id\":\"http://www.api-staging.xyfolder.com/auth/user/signin\",\"objectType\":\"Activity\",\"definition\":{\"type\":\"http://xapi.nelebd.ccnu.edu.cn/activitytypes/application\",\"name\":{\"zh-CN\":\"??????-???www.ai-augmented.com?\"},\"description\":{\"zh-CN\":\"??????-???www.ai-augmented.com?\"}}},\"authority\":{\"objectType\":\"Agent\",\"name\":\"xiaoya-test\",\"account\":{\"homePage\":\"http://www-staging.ai-augmented.com/\",\"name\":\"xiaoya-test\"}},\"id\":\"8c4d2096-4534-4335-a2e6-edd3d9301772\",\"timestamp\":\"2020-11-25T08:54:08.922Z\",\"stored\":\"2020-11-25T08:54:08.922Z\"}}"}
    at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:169)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:465)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:349)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"line":"{\"_id\":{\"$oid\":\"5fbe1bb08f35080030ea58d4\"},\"hasGeneratedId\":true,\"organisation\":{\"$oid\":\"5ef1d02f924838008888a688\"},\"lrs_id\":{\"$oid\":\"5ef1d12314e1340047ee4774\"},\"client\":{\"$oid\":\"5ef1d12314e1340047ee4775\"},\"person\":null,\"active\":true,\"voided\":false,\"timestamp\":{\"$date\":\"2020-11-25T08:54:08.922Z\"},\"stored\":{\"$date\":\"2020-11-25T08:54:08.922Z\"},\"hash\":\"e41e1684c538a83376913681cabaaf125f63708d\",\"agents\":[\"http://www.ai-augmented.com|19900000630\"],\"relatedAgents\":[\"http://www.ai-augmented.com|19900000630\",\"http://www-staging.ai-augmented.com/|xiaoya-test\"],\"registrations\":[],\"verbs\":[\"https://w3id.org/xapi/adl/verbs/logged-in\"],\"activities\":[\"http://www.api-staging.xyfolder.com/auth/user/signin\"],\"relatedActivities\":[\"http://www.api-staging.xyfolder.com/auth/user/signin\"],\"statement\":{\"version\":\"1.0.3\",\"actor\":{\"account\":{\"homePage\":\"http://www.ai-augmented.com\",\"name\":\"19900000630\"},\"objectType\":\"Agent\",\"name\":\"19900000630\"},\"verb\":{\"display\":{\"en\":\"logged-in\",\"zh-CN\":\"??\"},\"id\":\"https://w3id.org/xapi/adl/verbs/logged-in\"},\"object\":{\"id\":\"http://www.api-staging.xyfolder.com/auth/user/signin\",\"objectType\":\"Activity\",\"definition\":{\"type\":\"http://xapi.nelebd.ccnu.edu.cn/activitytypes/application\",\"name\":{\"zh-CN\":\"??????-???www.ai-augmented.com?\"},\"description\":{\"zh-CN\":\"??????-???www.ai-augmented.com?\"}}},\"authority\":{\"objectType\":\"Agent\",\"name\":\"xiaoya-test\",\"account\":{\"homePage\":\"http://www-staging.ai-augmented.com/\",\"name\":\"xiaoya-test\"}},\"id\":\"8c4d2096-4534-4335-a2e6-edd3d9301772\",\"timestamp\":\"2020-11-25T08:54:08.922Z\",\"stored\":\"2020-11-25T08:54:08.922Z\"}}"}
    at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:494)
    at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:160)
    ... 8 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to execute method public java.lang.String test.hive.udf.testudf_bak.evaluate(java.lang.String)  on object test.hive.udf.testudf_bak@6bda1d19 of class test.hive.udf.testudf_bak with arguments {{"_id":{"$oid":"5fbe1bb08f35080030ea58d4"},"hasGeneratedId":true,"organisation":{"$oid":"5ef1d02f924838008888a688"},"lrs_id":{"$oid":"5ef1d12314e1340047ee4774"},"client":{"$oid":"5ef1d12314e1340047ee4775"},"person":null,"active":true,"voided":false,"timestamp":{"$date":"2020-11-25T08:54:08.922Z"},"stored":{"$date":"2020-11-25T08:54:08.922Z"},"hash":"e41e1684c538a83376913681cabaaf125f63708d","agents":["http://www.ai-augmented.com|19900000630"],"relatedAgents":["http://www.ai-augmented.com|19900000630","http://www-staging.ai-augmented.com/|xiaoya-test"],"registrations":[],"verbs":["https://w3id.org/xapi/adl/verbs/logged-in"],"activities":["http://www.api-staging.xyfolder.com/auth/user/signin"],"relatedActivities":["http://www.api-staging.xyfolder.com/auth/user/signin"],"statement":{"version":"1.0.3","actor":{"account":{"homePage":"http://www.ai-augmented.com","name":"19900000630"},"objectType":"Agent","name":"19900000630"},"verb":{"display":{"en":"logged-in","zh-CN":"??"},"id":"https://w3id.org/xapi/adl/verbs/logged-in"},"object":{"id":"http://www.api-staging.xyfolder.com/auth/user/signin","objectType":"Activity","definition":{"type":"http://xapi.nelebd.ccnu.edu.cn/activitytypes/application","name":{"zh-CN":"??????-???www.ai-augmented.com?"},"description":{"zh-CN":"??????-???www.ai-augmented.com?"}}},"authority":{"objectType":"Agent","name":"xiaoya-test","account":{"homePage":"http://www-staging.ai-augmented.com/","name":"xiaoya-test"}},"id":"8c4d2096-4534-4335-a2e6-edd3d9301772","timestamp":"2020-11-25T08:54:08.922Z","stored":"2020-11-25T08:54:08.922Z"}}:java.lang.String} of size 1
    at org.apache.hadoop.hive.ql.exec.FunctionRegistry.invoke(FunctionRegistry.java:1024)
    at org.apache.hadoop.hive.ql.udf.generic.GenericUDFBridge.evaluate(GenericUDFBridge.java:194)
    at org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator._evaluate(ExprNodeGenericFuncEvaluator.java:186)
    at org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:77)
    at org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:65)
    at org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:88)
    at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:882)
    at org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:130)
    at org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:146)
    at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:484)
    ... 9 more
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.hadoop.hive.ql.exec.FunctionRegistry.invoke(FunctionRegistry.java:998)
    ... 18 more
Caused by: java.lang.ArrayIndexOutOfBoundsException: 1
    at test.hive.udf.testudf_bak.evaluate(testudf_bak.java:94)
    ... 23 more
  • 写回答

1条回答 默认 最新

  • 超级强大的大阳 2021-12-14 17:04
    关注

    94行数组越界,本地虚拟机跑没问题,服务器上就有问题了是吧,可以检查下jdk版本,对应的小版本也检查一下,指不定就是哪个的问题,最好换成一样的。

    本回答被题主选为最佳回答 , 对您是否有帮助呢?
    评论

报告相同问题?

问题事件

  • 系统已结题 10月15日
  • 已采纳回答 10月7日
  • 创建了问题 12月14日

悬赏问题

  • ¥15 有人能看一下我宿舍管理系统的报修功能该怎么改啊?链表那里总是越界
  • ¥15 cs loadimage运行不了,easyx也下了,没有用
  • ¥15 r包runway详细安装教程
  • ¥15 Html中读取Json文件中数据并制作表格
  • ¥15 谁有RH342练习环境
  • ¥15 STM32F407 DMA中断问题
  • ¥15 uniapp连接阿里云无法发布消息和订阅
  • ¥25 麦当劳点餐系统代码纠错
  • ¥15 轮班监督委员会问题。
  • ¥20 关于变压器的具体案例分析