Hive sqoop 数据迁移

Question

hive实现数据迁移报错

[root@node1 ~]# sqoop import \
> --connect jdbc:mysql://localhost:3306/nev \
> --username root \
> --password hadoop \
> --query "select user_id,age_range,gender from userinfo where 1=1 and \$CONDITIONS" \
> --hcatalog-database itcast_ods \
> --hcatalog-table userinfo \
> -m 10 \
> --split-by user_id
Warning: /export/server/sqoop-1.4.7/../hbase does not exist! HBase imports will fail.
Please set $HBASE_HOME to the root of your HBase installation.
Warning: /export/server/sqoop-1.4.7/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /export/server/sqoop-1.4.7/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
2024-06-05 07:50:21,841 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
2024-06-05 07:50:21,949 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
2024-06-05 07:50:22,120 INFO conf.HiveConf: Found configuration file file:/export/server/sqoop-1.4.7/conf/hive-site.xml
2024-06-05 07:50:22,540 INFO common.HCatUtil: mapreduce.lib.hcatoutput.hive.conf not set. Generating configuration differences.
2024-06-05 07:50:22,541 INFO common.HCatUtil: Configuration differences={hive.metastore.event.db.notification.api.auth=false, hive.server2.thrift.bind.host=node1, javax.jdo.option.ConnectionPassword=hadoop, javax.jdo.option.ConnectionURL=jdbc:mysql://node1:3306/hive3?createDatabaseIfNotExist=true&useSSL=false&useUnicode=true&characterEncoding=UTF-8, javax.jdo.option.ConnectionDriverName=com.mysql.jdbc.Driver, javax.jdo.option.ConnectionUserName=root, hive.metastore.uris=thrift://node1:9083}
2024-06-05 07:50:22,568 WARN common.HCatUtil: HCatUtil.getHiveClient is unsafe and can be a resource leak depending on HMSC implementation and caching mechanism. Use HCatUtil.getHiveMetastoreClient instead.
Exception in thread "main" java.lang.NoClassDefFoundError: com/facebook/fb303/FacebookService$Iface
    at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:756)
    at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
    at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
    at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
    at org.apache.hive.hcatalog.common.HCatUtil.getHiveClient(HCatUtil.java:590)
    at org.apache.sqoop.mapreduce.hcat.SqoopHCatUtilities.isHCatView(SqoopHCatUtilities.java:178)
    at org.apache.sqoop.tool.BaseSqoopTool.validateHCatalogOptions(BaseSqoopTool.java:1655)
    at org.apache.sqoop.tool.ImportTool.validateOptions(ImportTool.java:1179)
    at org.apache.sqoop.Sqoop.run(Sqoop.java:137)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
    at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
    at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)
    at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243)
    at org.apache.sqoop.Sqoop.main(Sqoop.java:252)
Caused by: java.lang.ClassNotFoundException: com.facebook.fb303.FacebookService$Iface
    at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
    ... 22 more
[root@node1 ~]# 

Answer 1

引自免费微信小程序：皆我百晓生

看起来你在尝试使用sqoop将MySQL中的数据迁移到Hive的HCatalog表中，但是遇到了一个类找不到的错误。这个错误是由于缺少FacebookService$Iface类导致的。这个类通常与Apache Thrift库相关，而Thrift是HCatalog的一个依赖。

为了解决这个问题，你需要确保你的系统中安装了正确的版本的Thrift，并且在运行sqoop时可以访问到它。这里有一个可能的解决步骤：

1. 检查Thrift的安装：确认你的系统上已经安装了Thrift，并且是正确的版本。可以通过pip list | grep thrift（如果你使用Python包管理器）或者在你的系统路径中查找相关的jar文件来检查。

2. 添加Thrift到classpath：如果Thrift已经安装，但仍然有这个错误，可能是环境变量设置的问题。在运行sqoop命令前，你需要确保Thrift的jar文件被包含在Java classpath中。这可以通过修改系统的CLASSPATH环境变量或者在执行sqoop命令时显式指定jar文件路径来完成。例如：

   CLASSPATH=$CLASSPATH:/path/to/thrift.jar
   export CLASSPATH
   

3. 修复依赖：有时候，是因为Hadoop或HCatalog的相关配置文件没有正确地引用Thrift的jar。检查你的HIVE_HOME/lib目录，确保里面有Thrift的jar文件，如果没有，从正确的源复制过去。

4. 重新启动服务：在调整完配置后，重启Hive Metastore服务，以便新的配置生效。

5. 再次尝试运行sqoop：在上述步骤完成后，再运行你的sqoop命令看是否还存在同样的错误。

如果以上步骤都无法解决问题，可能还需要检查其他依赖项或者查看更详细的日志输出以确定问题的具体原因。如果你正在使用的sqoop版本过旧，可能需要更新到最新版，以确保兼容性和稳定性。