@Configuration
public class SparkContextBean{
private String sparkHome = ".";
private String appName = "sparkTest";
private String master = "local";
@Autowired
KerborseUtil kerborseUtil;
@Bean
@ConditionalOnMissingBean(SparkConf.class)
public SparkConf sparkConf() throws Exception {
kerborseUtil.zkAuthentication();
kerborseUtil.hiveAuthentication();
SparkConf conf = new SparkConf().setAppName(appName).setMaster(master);
return conf;
}
@Bean
@ConditionalOnMissingBean(JavaSparkContext.class)
public JavaSparkContext javaSparkContext() throws Exception {
return new JavaSparkContext(sparkConf());
}
@Bean
@ConditionalOnMissingBean(HiveContext.class)
public HiveContext hiveContext() throws Exception {
return new HiveContext(javaSparkContext().sc());
}
}
然后项目启动的时候报错
Caused by: java.lang.reflect.InvocationTargetException: null
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at javax.jdo.JDOHelper$16.run(JDOHelper.java:1965)
at java.security.AccessController.doPrivileged(Native Method)
at javax.jdo.JDOHelper.invoke(JDOHelper.java:1960)
at javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1166)
... 75 common frames omitted
Caused by: org.datanucleus.exceptions.NucleusUserException: Persistence process has been specified to use a ClassLoaderResolver of name "datanucleus" yet this has not been found by the DataNucleus plugin mechanism. Please check your CLASSPATH and plugin specification.
at org.datanucleus.NucleusContext.(NucleusContext.java:283)
at org.datanucleus.NucleusContext.(NucleusContext.java:247)
at org.datanucleus.NucleusContext.(NucleusContext.java:225)
at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.(JDOPersistenceManagerFactory.java:416)
at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.createPersistenceManagerFactory(JDOPersistenceManagerFactory.java:301)
at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.getPersistenceManagerFactory(JDOPersistenceManagerFactory.java:202)
... 83 common frames omitted
环境:hadoop2.7.2+hive1.3.0+spark1.5.1+scala2.10.4
网上看了
https://blog.csdn.net/qq_38426934/article/details/81902830
但是没有搞定,不知道怎么解决,哎惭愧
猜想应该是和datanucleus相关的三个jar包有关系,jar包依赖也贴上
求大神们帮忙,看看
c币没了,不好意思,还是想得到大神的帮助,不胜感激
<dependency>
<groupId>org.datanucleus</groupId>
<artifactId>datanucleus-core</artifactId>
<version>3.2.10</version>
</dependency>
<dependency>
<groupId>org.datanucleus</groupId>
<artifactId>datanucleus-api-jdo</artifactId>
<version>3.2.6</version>
</dependency>
<dependency>
<groupId>org.datanucleus</groupId>
<artifactId>datanucleus-rdbms</artifactId>
<version>3.2.9</version>
</dependency>