使用sqoop从mariadb里面导数据到hive报错

Question

RT
执行代码如下

sqoop import --connect jdbc:mysql://localhost:3306/test --username root --password 1 --table exit_tran --hive-import --hive-table exit_tran -m 1 --hive-overwrite

导出数据总是报错

20/03/03 17:35:40 INFO mapreduce.Job: Task Id : attempt_1583223426401_0007_m_000000_2, Status : FAILED
Error: java.io.IOException: SQLException in nextKeyValue
    at org.apache.sqoop.mapreduce.db.DBRecordReader.nextKeyValue(DBRecordReader.java:277)
    at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:556)
    at org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)
    at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
    at org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.sql.SQLException: HOUR_OF_DAY: 2 -> 3
    at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:129)
    at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97)
    at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:89)
    at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:63)
    at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:73)
    at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:85)
    at com.mysql.cj.jdbc.result.ResultSetImpl.getTimestamp(ResultSetImpl.java:903)
    at org.apache.sqoop.lib.JdbcWritableBridge.readTimestamp(JdbcWritableBridge.java:111)
    at com.cloudera.sqoop.lib.JdbcWritableBridge.readTimestamp(JdbcWritableBridge.java:83)
    at exit_tran.readFields(exit_tran.java:229)
    at org.apache.sqoop.mapreduce.db.DBRecordReader.nextKeyValue(DBRecordReader.java:244)
    ... 12 more
Caused by: com.mysql.cj.exceptions.WrongArgumentException: HOUR_OF_DAY: 2 -> 3
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:61)
    at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:105)
    at com.mysql.cj.result.SqlTimestampValueFactory.localCreateFromTimestamp(SqlTimestampValueFactory.java:112)
    at com.mysql.cj.result.SqlTimestampValueFactory.localCreateFromTimestamp(SqlTimestampValueFactory.java:50)
    at com.mysql.cj.result.AbstractDateTimeValueFactory.createFromTimestamp(AbstractDateTimeValueFactory.java:87)
    at com.mysql.cj.protocol.a.MysqlTextValueDecoder.decodeTimestamp(MysqlTextValueDecoder.java:79)
    at com.mysql.cj.protocol.result.AbstractResultsetRow.decodeAndCreateReturnValue(AbstractResultsetRow.java:87)
    at com.mysql.cj.protocol.result.AbstractResultsetRow.getValueFromBytes(AbstractResultsetRow.java:241)
    at com.mysql.cj.protocol.a.result.TextBufferRow.getValue(TextBufferRow.java:132)
    ... 17 more
Caused by: java.lang.IllegalArgumentException: HOUR_OF_DAY: 2 -> 3
    at java.util.GregorianCalendar.computeTime(GregorianCalendar.java:2829)
    at java.util.Calendar.updateTime(Calendar.java:3393)
    at java.util.Calendar.getTimeInMillis(Calendar.java:1782)
    at com.mysql.cj.result.SqlTimestampValueFactory.localCreateFromTimestamp(SqlTimestampValueFactory.java:108)
    ... 23 more

Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143

求指导~

Answer 1

根据错误信息，这个问题是由于MySQL时间戳的格式不符合Hadoop序列化协议而引起的。这个问题可以通过在sqoop命令后添加--map-column-hive标志来解决。这个标志可以使用特定的时间戳格式将时间戳转换为Hadoop能够序列化的格式。例如，如果时间戳使用"yyyy-MM-dd HH:mm:ss"格式，那么可以使用以下命令来导入数据：

sqoop import --connect jdbc:mysql://localhost:3306/test --username root --password 1 --table exit_tran --hive-import --hive-table exit_tran -m 1 --hive-overwrite --map-column-hive create_time=timestamp

其中，"create_time"是数据表中的时间戳列。

另外，也可以尝试升级MySQL驱动程序来解决这个问题。可以下载最新版本的MySQL驱动程序，并将它放在sqoop的lib目录中。