用的Hadoop3.3.0 jdk8
记录一下，真的很奇怪，网上根本找不到解决方法
随便写一段sql，
hive> select a.moviename ,(select b.movieid from t_rating b) as movie_id from t_movie a;
运行过程：
Warning: Map Join MAPJOIN[32][bigTable=?] in task 'Stage-5:MAPRED' is a cross product
Warning: Shuffle Join JOIN[17][tables = [$hdt$_0, $hdt$_1, $hdt$_2]] in Stage 'Stage-2:MAPRED' is a cross product
Warning: Map Join MAPJOIN[33][bigTable=?] in task 'Stage-7:MAPRED' is a cross product
Query ID = hadoop_20200902124158_ac107116-4a4e-41a0-a773-0da5e06f711b
Total jobs = 4
Launching Job 1 out of 4
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapreduce.job.reduces=<number>
Starting Job = job_1599017699586_0003, Tracking URL = http://master:8088/proxy/application_1599017699586_0003/
Kill Command = /home/hadoop/app/hadoop/bin/mapred job -kill job_1599017699586_0003
Hadoop job information for Stage-3: number of mappers: 1; number of reducers: 1
2020-09-02 12:42:11,737 Stage-3 map = 0%, reduce = 0%
2020-09-02 12:42:40,880 Stage-3 map = 100%, reduce = 100%
Ended Job = job_1599017699586_0003 with errors
Error during job, obtaining debugging information...
Examining task ID: task_1599017699586_0003_m_000000 (and more) from job job_1599017699586_0003
Task with the most failures(4):
-----
Task ID:
task_1599017699586_0003_m_000000
URL:
http://0.0.0.0:8088/taskdetails.jsp?jobid=job_1599017699586_0003&tipid=task_1599017699586_0003_m_000000
-----
Diagnostic Messages for this Task:
Error: java.lang.RuntimeException: java.lang.NoSuchFieldException: parentOffset
at org.apache.hadoop.hive.ql.exec.SerializationUtilities$ArrayListSubListSerializer.<init>(SerializationUtilities.java:388)
at org.apache.hadoop.hive.ql.exec.SerializationUtilities$1.create(SerializationUtilities.java:234)
at org.apache.hive.com.esotericsoftware.kryo.pool.KryoPoolQueueImpl.borrow(KryoPoolQueueImpl.java:51)
at org.apache.hadoop.hive.ql.exec.SerializationUtilities.borrowKryo(SerializationUtilities.java:278)
at org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:413)
at org.apache.hadoop.hive.ql.exec.Utilities.getMapWork(Utilities.java:335)
at org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:435)
at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:881)
at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:874)
at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:716)
at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.<init>(MapTask.java:175)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:444)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:349)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:178)
at java.base/java.security.AccessController.doPrivileged(AccessController.java:691)
at java.base/javax.security.auth.Subject.doAs(Subject.java:425)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1845)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:172)
Caused by: java.lang.NoSuchFieldException: parentOffset
at java.base/java.lang.Class.getDeclaredField(Class.java:2489)
at org.apache.hadoop.hive.ql.exec.SerializationUtilities$ArrayListSubListSerializer.<init>(SerializationUtilities.java:382)
... 17 more
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
MapReduce Jobs Launched:
Stage-Stage-3: Map: 1 Reduce: 1 HDFS Read: 0 HDFS Write: 0 FAIL
Total MapReduce CPU Time Spent: 0 msec
反编译找到对应的代码，就是这段：
class SerializationUtilities$ArrayListSubListSerializer
extends Serializer<List<?>>
{
private Field _parentField;
private Field _parentOffsetField;
private Field _sizeField;
public SerializationUtilities$ArrayListSubListSerializer()
{
try
{
Class<?> clazz = Class.forName("java.util.ArrayList$SubList");
this._parentField = clazz.getDeclaredField("parent");
this._parentOffsetField = clazz.getDeclaredField("parentOffset");
this._sizeField = clazz.getDeclaredField("size");
this._parentField.setAccessible(true);
this._parentOffsetField.setAccessible(true);
this._sizeField.setAccessible(true);
}
catch (Exception e)
{
throw new RuntimeException(e);
}
}
...
报反射错误，说这个找不到。
java.lang.NoSuchFieldException: parentOffset
我寻思怎么找不到呢，这个东西干嘛使得？？
