小虾米有大梦想 2019-04-22 17:23
浏览 405
已结题

使用collect方法将JavaRDD<DependencyLink>转成List<DependencyLink>时出错

DependencyLink是一个类,字段包括parent、child、callCount、errorCount、appName 和dev 。其中,前四个字段是zipkin-dependencies源码自带的,后面两个字段是我新增的。我获取到了JavaRDD数据后,将里面的字段全部打印输出,每个字段都有数据,但在JavaRDD的基础上再使用collect()方法后,自己添加的字段值就没有了

JavaPairRDD<Tuple2<String,String>, DependencyLink> objects3 = new SQLContext(sc).read()
            .format("org.apache.spark.sql.execution.datasources.jdbc.JdbcRelationProvider")
            .options(options)
            .load()
            .toJavaRDD()
            .groupBy(rowTraceId)
            .flatMapValues(new RowsToDependencyLinks(logInitializer, hasTraceIdHigh))
            .values()
            .mapToPair(l -> Tuple2.apply(Tuple2.apply(l.parent(), l.child()), l));
    JavaRDD<DependencyLink> objects5 = objects3.values();
    class ForeachFunTest5 implements VoidFunction<DependencyLink>, Serializable {       
        private static final long serialVersionUID = -4892194648703458594L;
        @Override
        public void call(DependencyLink t) throws Exception {
            System.out.println("MySQLDependenciesJob------ values after reduceByKey parent = " + t.parent());
            System.out.println("MySQLDependenciesJob------ values after reduceByKey child = " + t.child());
            System.out.println("MySQLDependenciesJob------ values after reduceByKey appName = " + t.appName());
            System.out.println("MySQLDependenciesJob------ values after reduceByKey dev = " + t.dev()); 
            System.out.println("MySQLDependenciesJob------ values after reduceByKey callCount = " + t.callCount());
            System.out.println("MySQLDependenciesJob------ values after reduceByKey errorCount = " + t.errorCount());
        }
    }
    objects5.foreach(new ForeachFunTest5());
    List<DependencyLink> objects6 = objects5.collect();
    for(DependencyLink dependencyLink: objects6)  {
        System.out.println("MySQLDependenciesJob------ objects6 parent = " + dependencyLink.parent());
        System.out.println("MySQLDependenciesJob------ objects6 child = " + dependencyLink.child());
        System.out.println("MySQLDependenciesJob------ objects6 appName = " + dependencyLink.appName());
        System.out.println("MySQLDependenciesJob------ objects6 dev = " + dependencyLink.dev());
        System.out.println("MySQLDependenciesJob------ objects6 callCount = " + dependencyLink.callCount());
        System.out.println("MySQLDependenciesJob------ objects6 errorCount = " + dependencyLink.errorCount());
    }
  • 写回答

0条回答 默认 最新

    报告相同问题?

    悬赏问题

    • ¥15 HFSS 中的 H 场图与 MATLAB 中绘制的 B1 场 部分对应不上
    • ¥15 如何在scanpy上做差异基因和通路富集?
    • ¥20 关于#硬件工程#的问题,请各位专家解答!
    • ¥15 关于#matlab#的问题:期望的系统闭环传递函数为G(s)=wn^2/s^2+2¢wn+wn^2阻尼系数¢=0.707,使系统具有较小的超调量
    • ¥15 FLUENT如何实现在堆积颗粒的上表面加载高斯热源
    • ¥30 截图中的mathematics程序转换成matlab
    • ¥15 动力学代码报错,维度不匹配
    • ¥15 Power query添加列问题
    • ¥50 Kubernetes&Fission&Eleasticsearch
    • ¥15 報錯:Person is not mapped,如何解決?