Sqoop failed:java.util.LinkedList cannot be cast to org.apache.hive.hcatalog.mapreduce.InputJobInfo

Question

在使用sqoop命令将MySQL数据库的数据表数据迁移到Hive数据仓库时，发生了错误。

使用了如下命令（命令1）：

sqoop import -connect jdbc:mysql://master:3306/*** --username **** --password **** --query "select id,referrer,from_url,landing_page_url,url_title,platform_description,other_params,history from web_chat_text_ems_2019_07 where 1=1 and $CONDITIONS" --hcatalog-database itcast_ods --hcatalog-table web_chat_text_ems_ods -m 10 --split-by id

错误信息如下（部分）：

2023-01-09 07:06:31,280 INFO metastore.HMSHandler: Created RawStore: org.apache.hadoop.hive.metastore.ObjectStore@2ba31591
2023-01-09 07:07:13,181 INFO mapreduce.Job: Task Id : attempt_1673201919048_0006_m_000000_0, Status : FAILED
Error: java.lang.ClassCastException: java.util.LinkedList cannot be cast to org.apache.hive.hcatalog.mapreduce.InputJobInfo
    at org.apache.sqoop.mapreduce.hcat.SqoopHCatImportHelper.<init>(SqoopHCatImportHelper.java:88)
    at org.apache.sqoop.mapreduce.hcat.SqoopHCatImportMapper.setup(SqoopHCatImportMapper.java:45)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:800)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:348)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:178)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:172)

2023-01-09 07:07:13,301 INFO mapreduce.Job: Task Id : attempt_1673201919048_0006_m_000009_0, Status : FAILED
Error: java.lang.ClassCastException: java.util.LinkedList cannot be cast to org.apache.hive.hcatalog.mapreduce.InputJobInfo
    at org.apache.sqoop.mapreduce.hcat.SqoopHCatImportHelper.<init>(SqoopHCatImportHelper.java:88)
    at org.apache.sqoop.mapreduce.hcat.SqoopHCatImportMapper.setup(SqoopHCatImportMapper.java:45)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:800)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:348)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:178)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:172)

2023-01-09 07:07:13,311 INFO mapreduce.Job: Task Id : attempt_1673201919048_0006_m_000003_0, Status : FAILED
Error: java.lang.ClassCastException: java.util.LinkedList cannot be cast to org.apache.hive.hcatalog.mapreduce.InputJobInfo
    at org.apache.sqoop.mapreduce.hcat.SqoopHCatImportHelper.<init>(SqoopHCatImportHelper.java:88)
    at org.apache.sqoop.mapreduce.hcat.SqoopHCatImportMapper.setup(SqoopHCatImportMapper.java:45)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:800)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:348)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:178)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:172)

2023-01-09 07:07:13,343 INFO mapreduce.Job: Task Id : attempt_1673201919048_0006_m_000006_0, Status : FAILED
Error: java.lang.ClassCastException: java.util.LinkedList cannot be cast to org.apache.hive.hcatalog.mapreduce.InputJobInfo
    at org.apache.sqoop.mapreduce.hcat.SqoopHCatImportHelper.<init>(SqoopHCatImportHelper.java:88)
    at org.apache.sqoop.mapreduce.hcat.SqoopHCatImportMapper.setup(SqoopHCatImportMapper.java:45)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:800)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:348)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:178)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:172)

2023-01-09 07:07:21,192 INFO mapreduce.Job: Task Id : attempt_1673201919048_0006_m_000005_0, Status : FAILED
Error: java.lang.ClassCastException: java.util.LinkedList cannot be cast to org.apache.hive.hcatalog.mapreduce.InputJobInfo
    at org.apache.sqoop.mapreduce.hcat.SqoopHCatImportHelper.<init>(SqoopHCatImportHelper.java:88)
    at org.apache.sqoop.mapreduce.hcat.SqoopHCatImportMapper.setup(SqoopHCatImportMapper.java:45)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:800)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:348)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:178)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:172)

2023-01-09 07:07:21,211 INFO mapreduce.Job: Task Id : attempt_1673201919048_0006_m_000002_0, Status : FAILED
Error: java.lang.ClassCastException: java.util.LinkedList cannot be cast to org.apache.hive.hcatalog.mapreduce.InputJobInfo
    at org.apache.sqoop.mapreduce.hcat.SqoopHCatImportHelper.<init>(SqoopHCatImportHelper.java:88)
    at org.apache.sqoop.mapreduce.hcat.SqoopHCatImportMapper.setup(SqoopHCatImportMapper.java:45)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:800)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:348)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:178)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:172)

2023-01-09 07:07:21,218 INFO mapreduce.Job: Task Id : attempt_1673201919048_0006_m_000008_0, Status : FAILED
Error: java.lang.ClassCastException: java.util.LinkedList cannot be cast to org.apache.hive.hcatalog.mapreduce.InputJobInfo
    at org.apache.sqoop.mapreduce.hcat.SqoopHCatImportHelper.<init>(SqoopHCatImportHelper.java:88)
    at org.apache.sqoop.mapreduce.hcat.SqoopHCatImportMapper.setup(SqoopHCatImportMapper.java:45)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:800)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:348)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:178)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:172)

2023-01-09 07:07:27,629 INFO mapreduce.Job: Task Id : attempt_1673201919048_0006_m_000001_0, Status : FAILED
Error: java.lang.ClassCastException: java.util.LinkedList cannot be cast to org.apache.hive.hcatalog.mapreduce.InputJobInfo
    at org.apache.sqoop.mapreduce.hcat.SqoopHCatImportHelper.<init>(SqoopHCatImportHelper.java:88)
    at org.apache.sqoop.mapreduce.hcat.SqoopHCatImportMapper.setup(SqoopHCatImportMapper.java:45)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:800)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:348)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:178)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:172)

2023-01-09 07:07:27,644 INFO mapreduce.Job: Task Id : attempt_1673201919048_0006_m_000007_0, Status : FAILED
Error: java.lang.ClassCastException: java.util.LinkedList cannot be cast to org.apache.hive.hcatalog.mapreduce.InputJobInfo
    at org.apache.sqoop.mapreduce.hcat.SqoopHCatImportHelper.<init>(SqoopHCatImportHelper.java:88)
    at org.apache.sqoop.mapreduce.hcat.SqoopHCatImportMapper.setup(SqoopHCatImportMapper.java:45)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:800)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:348)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:178)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:172)

2023-01-09 07:07:27,683 INFO mapreduce.Job: Task Id : attempt_1673201919048_0006_m_000004_0, Status : FAILED
Error: java.lang.ClassCastException: java.util.LinkedList cannot be cast to org.apache.hive.hcatalog.mapreduce.InputJobInfo
    at org.apache.sqoop.mapreduce.hcat.SqoopHCatImportHelper.<init>(SqoopHCatImportHelper.java:88)
    at org.apache.sqoop.mapreduce.hcat.SqoopHCatImportMapper.setup(SqoopHCatImportMapper.java:45)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:800)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:348)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:178)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:172)

我所使用的软件的版本信息如下：

Hadoop 3.3.1 ，4.0.0-alpha-2，mysql8.0，sqoob 1.4.7，java 1.8

尝试增加了HCAT_HOME,将$HIVE_HOME/lib中的hive-common-4.0.0-alpha-2.jar 拷贝到$SQOOP_HOME/lib中，没有解决问题。

请问以上错误发生的原因， 及如何解决

Answer 1

引用 皆我百晓生 小程序回复内容作答：

根据错误信息，可以看到错误是由于java.util.LinkedList无法强制转换为org.apache.hive.hcatalog.mapreduce.InputJobInfo引起的。这可能是由于Sqoop版本与Hive版本不兼容导致的。

解决此错误的方法是确保使用的Sqoop版本与Hive版本兼容。您提供的信息中，Sqoop版本为1.4.7，而Hive版本为4.0.0-alpha-2。这两个版本可能不兼容，因此可以尝试降低Hive版本或升级Sqoop版本来解决问题。

另外，还可以尝试以下解决方法：

1. 确保在将hive-common-4.0.0-alpha-2.jar拷贝到$SQOOP_HOME/lib之前，将其从$HIVE_HOME/lib中删除。
2. 检查是否安装了正确版本的Hive，并确保环境变量HIVE_HOME设置正确。
3. 检查Sqoop的配置文件（sqoop-site.xml）中的相关配置项，确保其与Hive的配置相匹配。

如果尝试了上述解决方法后仍然无法解决问题，可以尝试以下进一步的解决方法：

1. 确保在运行Sqoop命令之前，已正确配置Hive Metastore。可以通过检查Hive Metastore是否正常运行，并且Sqoop可以连接到Hive Metastore来确认。

2. 检查Sqoop和Hive的日志文件，查看是否有其他错误或异常信息。根据日志中的详细错误信息，可以更准确地确定问题所在。

3. 尝试使用Sqoop的--driver选项指定正确的Hive驱动程序。例如，如果您使用的是MySQL数据库作为Hive Metastore的后端数据库，则应使用--driver com.mysql.jdbc.Driver选项。

4. 如果您使用的是Cloudera Distribution of Hadoop（CDH），请确保Sqoop和Hive的版本与CDH版本兼容。可以参考CDH文档或与CDH支持团队联系以获取更多帮助。

5. 如果可能的话，可以尝试使用其他工具或方法将MySQL数据迁移到Hive，例如使用Apache NiFi、Apache Kafka等。

总之，解决Sqoop与Hive版本不兼容导致的错误可能需要尝试不同的方法和工具。通过仔细检查日志、确认配置和版本兼容性，以及尝试不同的解决方法，您应该能够解决这个问题。