代码目标：将informix数据库中的数据通过sqoop导入到hive中。
求助：
1.是不是我的开发机也要安装sqoop？
2.java代码 eclipse所在的机器和sqoop服务器是否一定要在同一台服务器？
3.sqoop和hadoop（hdfs）是否一定要在同一台服务器？
java代码：
package myMultiProject;
import org.apache.hadoop.conf.Configuration;
import org.apache.sqoop.Sqoop;
import org.apache.sqoop.hive.HiveConfig;
import org.apache.sqoop.tool.ImportTool;
import org.apache.sqoop.tool.SqoopTool;
import org.apache.sqoop.util.OptionsFileUtil;
import java.io.IOException;
public class mySqoopToHive {
public void SqoopToHive() {
System.out.println(" begin test sqoop");
String[] argument = new String[] {
"--connect","jdbc:informix-sqli://源头informix连接字符串",
"-driver","com.informix.jdbc.IfxDriver",
"--table","tbl_file",
"--hive-import","--hive-database","default",
"--hive-overwrite","--create-hive-table",
"--hive-table","java_tbl_file",
"-m", "1",
"--target-dir","/tmp/",
};
String[] expandArguments=null;
try {
expandArguments = OptionsFileUtil.expandArguments(argument);
} catch (Exception e1) {
// TODO Auto-generated catch block
e1.printStackTrace();
}
com.cloudera.sqoop.tool.SqoopTool sqoopTool=(com.cloudera.sqoop.tool.SqoopTool)SqoopTool.getTool("import");
Configuration conf= new Configuration();
//这里是我的hdfs url路径ip
conf.set("fs.defaultFS","hdfs://hdfs url路径ip:9000");
Configuration hive=null;
try {
hive=HiveConfig.getHiveConf(conf);
} catch (IOException e) {
//TODO Auto-generated catch block
e.printStackTrace();
}
Sqoop sqoop = new Sqoop(sqoopTool,SqoopTool.loadPlugins(conf) );
//int res = Sqoop.runSqoop(sqoop,argument);
int res = Sqoop.runSqoop(sqoop,expandArguments);
System.out.println(res);
System.out.println("執行sqoop結束");
}
}
报错的信息
begin test sqoop
2020-07-22 10:22:03,726 INFO [main] conf.HiveConf (HiveConf.java:findConfigFile(181)) - Found configuration file null
2020-07-22 10:22:04,050 WARN [main] util.Shell (Shell.java:(679)) - Did not find winutils.exe: java.io.FileNotFoundException: java.io.FileNotFoundException: HADOOP_HOME and hadoop.home.dir are unset. -see https://wiki.apache.org/hadoop/WindowsProblems
2020-07-22 10:22:04,064 WARN [main] tool.SqoopTool (SqoopTool.java:loadPluginsFromConfDir(177)) - $SQOOP_CONF_DIR has not been set in the environment. Cannot check for additional configuration.
2020-07-22 10:22:04,140 INFO [main] sqoop.Sqoop (Sqoop.java:(96)) - Running Sqoop version: 1.4.7
2020-07-22 10:22:04,158 INFO [main] tool.BaseSqoopTool (BaseSqoopTool.java:validateOutputFormatOptions(1465)) - Using Hive-specific delimiters for output. You can override
2020-07-22 10:22:04,159 INFO [main] tool.BaseSqoopTool (BaseSqoopTool.java:validateOutputFormatOptions(1466)) - delimiters with --fields-terminated-by, etc.
2020-07-22 10:22:04,165 WARN [main] sqoop.ConnFactory (ConnFactory.java:loadManagersFromConfDir(273)) - $SQOOP_CONF_DIR has not been set in the environment. Cannot check for additional configuration.
2020-07-22 10:22:04,174 WARN [main] sqoop.ConnFactory (ConnFactory.java:getManager(132)) - Parameter --driver is set to an explicit driver however appropriate connection manager is not being set (via --connection-manager). Sqoop is going to fall back to org.apache.sqoop.manager.GenericJdbcManager. Please specify explicitly which connection manager should be used next time.
2020-07-22 10:22:04,181 INFO [main] manager.SqlManager (SqlManager.java:initOptionDefaults(98)) - Using default fetchSize of 1000
2020-07-22 10:22:04,183 INFO [main] tool.CodeGenTool (CodeGenTool.java:generateORM(91)) - Beginning code generation
2020-07-22 10:22:04,494 INFO [main] manager.SqlManager (SqlManager.java:execute(776)) - Executing SQL statement: SELECT t.* FROM tbl_file AS t WHERE 1=0
2020-07-22 10:22:04,511 INFO [main] manager.SqlManager (SqlManager.java:execute(776)) - Executing SQL statement: SELECT t.* FROM tbl_file AS t WHERE 1=0
2020-07-22 10:22:04,531 INFO [main] orm.CompilationManager (CompilationManager.java:findHadoopJars(91)) - $HADOOP_MAPRED_HOME is not set
2020-07-22 10:22:04,534 ERROR [main] orm.CompilationManager (CompilationManager.java:compile(193)) - It seems as though you are running sqoop with a JRE.
2020-07-22 10:22:04,534 ERROR [main] orm.CompilationManager (CompilationManager.java:compile(194)) - Sqoop requires a JDK that can compile Java code.
2020-07-22 10:22:04,534 ERROR [main] orm.CompilationManager (CompilationManager.java:compile(195)) - Please install a JDK and set $JAVA_HOME to use it.
2020-07-22 10:22:04,536 ERROR [main] tool.ImportTool (ImportTool.java:run(634)) - Import failed: java.io.IOException: Could not start Java compiler.
at org.apache.sqoop.orm.CompilationManager.compile(CompilationManager.java:196)
at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:107)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:501)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:628)
at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
at myMultiProject.mySqoopToHive.SqoopToHive(mySqoopToHive.java:47)
at myMultiProject.myMain.main(myMain.java:15)
1
執行sqoop結束
