sqoop导出到mysql出错

Question

2024-05-12 23:16:02,434 INFO mapreduce.Job: Job job_1715526694917_0002 failed with state FAILED due to: Task failed task_1715526694917_0002_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0 killedMaps:0 killedReduces: 0

2024-05-12 23:16:02,483 INFO mapreduce.Job: Counters: 8
    Job Counters 
        Failed map tasks=4
        Launched map tasks=4
        Data-local map tasks=4
        Total time spent by all maps in occupied slots (ms)=68328
        Total time spent by all reduces in occupied slots (ms)=0
        Total time spent by all map tasks (ms)=34164
        Total vcore-milliseconds taken by all map tasks=34164
        Total megabyte-milliseconds taken by all map tasks=102492000
2024-05-12 23:16:02,491 WARN mapreduce.Counters: Group FileSystemCounters is deprecated. Use org.apache.hadoop.mapreduce.FileSystemCounter instead
2024-05-12 23:16:02,493 INFO mapreduce.ExportJobBase: Transferred 0 bytes in 32.3546 seconds (0 bytes/sec)
2024-05-12 23:16:02,499 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
2024-05-12 23:16:02,499 INFO mapreduce.ExportJobBase: Exported 0 records.
2024-05-12 23:16:02,499 ERROR mapreduce.ExportJobBase: Export job failed!
2024-05-12 23:16:02,501 ERROR tool.ExportTool: Error during export: 
Export job failed!
    at org.apache.sqoop.mapreduce.ExportJobBase.runExport(ExportJobBase.java:445)
    at org.apache.sqoop.manager.SqlManager.exportTable(SqlManager.java:931)
    at org.apache.sqoop.tool.ExportTool.exportTable(ExportTool.java:80)
    at org.apache.sqoop.tool.ExportTool.run(ExportTool.java:99)
    at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
    at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
    at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)
    at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243)
    at org.apache.sqoop.Sqoop.main(Sqoop.java:252)

主要显示是这样的

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/hadoop/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/hadoop/hbase/lib/client-facing-thirdparty/slf4j-reload4j-1.7.33.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/hadoop/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
2024-05-12 23:15:27,128 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
2024-05-12 23:15:27,150 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
2024-05-12 23:15:27,245 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
2024-05-12 23:15:27,248 INFO tool.CodeGenTool: Beginning code generation
Sun May 12 23:15:27 HKT 2024 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2024-05-12 23:15:27,841 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `job_data` AS t LIMIT 1
2024-05-12 23:15:27,867 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `job_data` AS t LIMIT 1
2024-05-12 23:15:27,883 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /home/hadoop/hadoop
Note: /tmp/sqoop-hadoop/compile/3c15eb70be2dfc1c4ec16ac14b2f2b86/job_data.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
2024-05-12 23:15:29,424 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-hadoop/compile/3c15eb70be2dfc1c4ec16ac14b2f2b86/job_data.jar
2024-05-12 23:15:29,437 INFO mapreduce.ExportJobBase: Beginning export of job_data
2024-05-12 23:15:29,437 INFO Configuration.deprecation: mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2024-05-12 23:15:29,521 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
2024-05-12 23:15:30,041 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2024-05-12 23:15:30,122 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative
2024-05-12 23:15:30,125 INFO Configuration.deprecation: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative
2024-05-12 23:15:30,126 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
2024-05-12 23:15:30,180 INFO client.RMProxy: Connecting to ResourceManager at master/192.168.56.101:8032
2024-05-12 23:15:30,571 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/hadoop/.staging/job_1715526694917_0002
2024-05-12 23:15:30,656 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2024-05-12 23:15:31,117 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2024-05-12 23:15:31,546 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2024-05-12 23:15:31,978 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2024-05-12 23:15:32,005 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2024-05-12 23:15:32,046 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2024-05-12 23:15:32,487 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2024-05-12 23:15:32,932 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2024-05-12 23:15:32,957 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2024-05-12 23:15:33,381 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2024-05-12 23:15:33,809 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2024-05-12 23:15:34,226 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2024-05-12 23:15:34,244 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2024-05-12 23:15:34,664 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2024-05-12 23:15:35,088 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2024-05-12 23:15:35,109 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2024-05-12 23:15:35,531 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2024-05-12 23:15:35,963 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2024-05-12 23:15:35,986 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2024-05-12 23:15:36,416 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2024-05-12 23:15:36,845 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2024-05-12 23:15:37,264 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2024-05-12 23:15:37,689 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2024-05-12 23:15:38,111 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2024-05-12 23:15:38,534 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2024-05-12 23:15:38,565 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2024-05-12 23:15:39,079 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2024-05-12 23:15:39,500 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2024-05-12 23:15:39,929 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2024-05-12 23:15:40,348 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2024-05-12 23:15:40,364 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2024-05-12 23:15:40,386 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2024-05-12 23:15:40,821 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2024-05-12 23:15:40,840 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2024-05-12 23:15:41,263 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2024-05-12 23:15:41,681 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2024-05-12 23:15:42,105 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2024-05-12 23:15:42,124 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2024-05-12 23:15:42,544 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2024-05-12 23:15:42,568 INFO input.FileInputFormat: Total input files to process : 1
2024-05-12 23:15:42,571 INFO input.FileInputFormat: Total input files to process : 1
2024-05-12 23:15:42,612 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2024-05-12 23:15:43,028 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2024-05-12 23:15:43,036 INFO mapreduce.JobSubmitter: number of splits:4
2024-05-12 23:15:43,069 INFO Configuration.deprecation: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative
2024-05-12 23:15:43,171 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2024-05-12 23:15:43,595 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1715526694917_0002
2024-05-12 23:15:43,595 INFO mapreduce.JobSubmitter: Executing with tokens: []
2024-05-12 23:15:43,751 INFO conf.Configuration: resource-types.xml not found
2024-05-12 23:15:43,751 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
2024-05-12 23:15:43,917 INFO impl.YarnClientImpl: Submitted application application_1715526694917_0002
2024-05-12 23:15:43,944 INFO mapreduce.Job: The url to track the job: http://master:8088/proxy/application_1715526694917_0002/
2024-05-12 23:15:43,944 INFO mapreduce.Job: Running job: job_1715526694917_0002
2024-05-12 23:15:50,079 INFO mapreduce.Job: Job job_1715526694917_0002 running in uber mode : false
2024-05-12 23:15:50,080 INFO mapreduce.Job:  map 0% reduce 0%
2024-05-12 23:16:01,422 INFO mapreduce.Job:  map 100% reduce 0

具体的日志文件
怎么解决这个问题

Answer 1

以下内容由CHATGPT及笑面虎共同生成、有用望采纳：

---

您想要解决的问题：

您遇到的问题是在使用Sqoop将数据从Hadoop导出到MySQL时遇到了错误，导致导出作业失败。

问题出现原因：

1. 任务失败：日志显示有4个map任务失败，没有reduce任务失败。
2. SSL连接警告：日志中提到了SSL连接没有服务器身份验证的警告，这可能与网络配置有关。
3. API弃用警告：日志中有多条API弃用警告，这表明使用的Sqoop版本可能较旧，或者与Hadoop集群的版本不兼容。
4. 编译警告：日志中提到了编译时使用了弃用的API，这可能表明代码生成或编译过程中存在问题。
5. 多SLF4J绑定：日志中提到了多个SLF4J绑定，这通常意味着类路径中存在多个版本的SLF4J，这可能导致运行时问题。

问题解决方案：

1. 检查任务失败的具体原因：查看更详细的任务日志，确定失败的原因。可能是数据问题、网络问题或者配置问题。
2. 解决SSL连接警告：根据警告信息，检查MySQL连接配置，确保SSL连接正确设置。
3. 升级Sqoop和Hadoop：如果可能，升级到最新版本的Sqoop和Hadoop，以解决API弃用问题。
4. 解决编译警告：根据编译警告，检查和更新代码，以使用最新的API。
5. 解决多SLF4J绑定问题：清理类路径，确保只包含一个版本的SLF4J。

提供代码：

由于这个问题涉及到Sqoop的内部错误和配置问题，没有具体的代码可以提供。但是，您可以使用以下步骤来检查和解决问题：

# 查看Sqoop任务的详细日志
sqoop job --status job_name

# 检查MySQL连接配置
sqoop list-databases --connect jdbc:mysql://hostname:port/database --username username --password password

# 检查Sqoop的类路径，移除多余的SLF4J绑定
export CLASSPATH=`echo $CLASSPATH | sed 's/:/ /g' | sed 's/slf4j-log4j12.jar//g' | sed 's/slf4j-reload4j.jar//g' | tr ' ' ':'`

# 重新运行Sqoop导出命令
sqoop export ...

代码运行方式：

您需要在Hadoop集群的命令行环境中运行上述命令，确保您有足够的权限来访问相关资源。

代码预期运行结果：

如果问题解决，Sqoop导出命令应该能够成功执行，没有错误日志。

推荐相关链接：

• Sqoop官方文档
• MySQL与Sqoop集成
• SLF4J多绑定问题