ontwthe 2021-12-15 13:33 采纳率: 0%
浏览 90
已结题

spark写入mysql特别慢

最近写了个程序,大致是通过flume收集数据通过kafka传递给sparkstreaming,中间将数据转换成datadrame最后用.write.jdbc传入mysql。前面转换数据都没问题,单独运行挺快的,就是后面存入mysql时非常的慢,一分钟只能存2,3条左右。
查看运行记录发现每次存入mysql时会出现200个 (executor driver),请问这该怎么解决。 具体记录如下:

5 13:27:02 INFO SparkContext: Starting job: jdbc at wodebishe.scala:173
21/12/15 13:27:02 INFO DAGScheduler: Registering RDD 223 (jdbc at wodebishe.scala:173)
21/12/15 13:27:02 INFO DAGScheduler: Got job 34 (jdbc at wodebishe.scala:173) with 200 output partitions
21/12/15 13:27:02 INFO DAGScheduler: Final stage: ResultStage 46 (jdbc at wodebishe.scala:173)
21/12/15 13:27:02 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 45)
21/12/15 13:27:02 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 45)
21/12/15 13:27:02 INFO DAGScheduler: Submitting ShuffleMapStage 45 (MapPartitionsRDD[223] at jdbc at wodebishe.scala:173), which has no missing parents
21/12/15 13:27:02 INFO MemoryStore: Block broadcast_49 stored as values in memory (estimated size 58.9 KB, free 1988.3 MB)
21/12/15 13:27:02 INFO MemoryStore: Block broadcast_49_piece0 stored as bytes in memory (estimated size 22.9 KB, free 1988.3 MB)
21/12/15 13:27:02 INFO BlockManagerInfo: Added broadcast_49_piece0 in memory on windows10.microdone.cn:51542 (size: 22.9 KB, free: 1988.6 MB)
21/12/15 13:27:02 INFO SparkContext: Created broadcast 49 from broadcast at DAGScheduler.scala:1039
21/12/15 13:27:02 INFO DAGScheduler: Submitting 3 missing tasks from ShuffleMapStage 45 (MapPartitionsRDD[223] at jdbc at wodebishe.scala:173) (first 15 tasks are for partitions Vector(0, 1, 2))
21/12/15 13:27:02 INFO TaskSchedulerImpl: Adding task set 45.0 with 3 tasks
21/12/15 13:27:02 INFO TaskSetManager: Starting task 0.0 in stage 45.0 (TID 639, localhost, executor driver, partition 0, PROCESS_LOCAL, 7724 bytes)
21/12/15 13:27:02 INFO TaskSetManager: Starting task 1.0 in stage 45.0 (TID 640, localhost, executor driver, partition 1, PROCESS_LOCAL, 7724 bytes)
21/12/15 13:27:02 INFO TaskSetManager: Starting task 2.0 in stage 45.0 (TID 641, localhost, executor driver, partition 2, PROCESS_LOCAL, 7724 bytes)
21/12/15 13:27:02 INFO Executor: Running task 1.0 in stage 45.0 (TID 640)
21/12/15 13:27:02 INFO Executor: Running task 0.0 in stage 45.0 (TID 639)
21/12/15 13:27:02 INFO Executor: Running task 2.0 in stage 45.0 (TID 641)
21/12/15 13:27:02 INFO BlockManager: Found block rdd_58_1 locally
21/12/15 13:27:02 INFO BlockManager: Found block rdd_58_2 locally
21/12/15 13:27:02 INFO BlockManager: Found block rdd_58_0 locally
21/12/15 13:27:02 INFO Executor: Finished task 1.0 in stage 45.0 (TID 640). 2256 bytes result sent to driver
21/12/15 13:27:02 INFO TaskSetManager: Finished task 1.0 in stage 45.0 (TID 640) in 8 ms on localhost (executor driver) (1/3)
21/12/15 13:27:02 INFO Executor: Finished task 0.0 in stage 45.0 (TID 639). 2256 bytes result sent to driver
21/12/15 13:27:02 INFO TaskSetManager: Finished task 0.0 in stage 45.0 (TID 639) in 10 ms on localhost (executor driver) (2/3)
21/12/15 13:27:02 INFO Executor: Finished task 2.0 in stage 45.0 (TID 641). 2299 bytes result sent to driver
21/12/15 13:27:02 INFO TaskSetManager: Finished task 2.0 in stage 45.0 (TID 641) in 13 ms on localhost (executor driver) (3/3)
21/12/15 13:27:02 INFO TaskSchedulerImpl: Removed TaskSet 45.0, whose tasks have all completed, from pool 
21/12/15 13:27:02 INFO DAGScheduler: ShuffleMapStage 45 (jdbc at wodebishe.scala:173) finished in 0.017 s
21/12/15 13:27:02 INFO DAGScheduler: looking for newly runnable stages
21/12/15 13:27:02 INFO DAGScheduler: running: Set()
21/12/15 13:27:02 INFO DAGScheduler: waiting: Set(ResultStage 46)
21/12/15 13:27:02 INFO DAGScheduler: failed: Set()
21/12/15 13:27:02 INFO DAGScheduler: Submitting ResultStage 46 (MapPartitionsRDD[227] at jdbc at wodebishe.scala:173), which has no missing parents
21/12/15 13:27:02 INFO MemoryStore: Block broadcast_50 stored as values in memory (estimated size 61.1 KB, free 1988.2 MB)
21/12/15 13:27:02 INFO MemoryStore: Block broadcast_50_piece0 stored as bytes in memory (estimated size 24.5 KB, free 1988.2 MB)
21/12/15 13:27:02 INFO BlockManagerInfo: Added broadcast_50_piece0 in memory on windows10.microdone.cn:51542 (size: 24.5 KB, free: 1988.6 MB)
21/12/15 13:27:02 INFO SparkContext: Created broadcast 50 from broadcast at DAGScheduler.scala:1039
21/12/15 13:27:02 INFO DAGScheduler: Submitting 200 missing tasks from ResultStage 46 (MapPartitionsRDD[227] at jdbc at wodebishe.scala:173) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14))
21/12/15 13:27:02 INFO TaskSchedulerImpl: Adding task set 46.0 with 200 tasks
21/12/15 13:27:02 INFO TaskSetManager: Starting task 0.0 in stage 46.0 (TID 642, localhost, executor driver, partition 0, PROCESS_LOCAL, 7754 bytes)
21/12/15 13:27:02 INFO TaskSetManager: Starting task 1.0 in stage 46.0 (TID 643, localhost, executor driver, partition 1, PROCESS_LOCAL, 7754 bytes)
21/12/15 13:27:02 INFO TaskSetManager: Starting task 2.0 in stage 46.0 (TID 644, localhost, executor driver, partition 2, PROCESS_LOCAL, 7754 bytes)
21/12/15 13:27:02 INFO TaskSetManager: Starting task 3.0 in stage 46.0 (TID 645, localhost, executor driver, partition 3, PROCESS_LOCAL, 7754 bytes)
21/12/15 13:27:02 INFO Executor: Running task 0.0 in stage 46.0 (TID 642)
21/12/15 13:27:02 INFO Executor: Running task 1.0 in stage 46.0 (TID 643)
21/12/15 13:27:02 INFO Executor: Running task 2.0 in stage 46.0 (TID 644)
21/12/15 13:27:02 INFO Executor: Running task 3.0 in stage 46.0 (TID 645)
21/12/15 13:27:02 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 3 blocks
21/12/15 13:27:02 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 3 blocks
21/12/15 13:27:02 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
21/12/15 13:27:02 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 3 blocks
21/12/15 13:27:02 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
21/12/15 13:27:02 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
21/12/15 13:27:02 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 3 blocks
21/12/15 13:27:02 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
21/12/15 13:27:02 INFO Executor: Finished task 1.0 in stage 46.0 (TID 643). 2724 bytes result sent to driver
21/12/15 13:27:02 INFO TaskSetManager: Starting task 4.0 in stage 46.0 (TID 646, localhost, executor driver, partition 4, PROCESS_LOCAL, 7754 bytes)
21/12/15 13:27:02 INFO Executor: Running task 4.0 in stage 46.0 (TID 646)
21/12/15 13:27:02 INFO TaskSetManager: Finished task 1.0 in stage 46.0 (TID 643) in 93 ms on localhost (executor driver) (1/200)
21/12/15 13:27:02 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 3 blocks
21/12/15 13:27:02 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
21/12/15 13:27:02 INFO Executor: Finished task 3.0 in stage 46.0 (TID 645). 2724 bytes result sent to driver
21/12/15 13:27:02 INFO TaskSetManager: Starting task 5.0 in stage 46.0 (TID 647, localhost, executor driver, partition 5, PROCESS_LOCAL, 7754 bytes)
21/12/15 13:27:02 INFO Executor: Running task 5.0 in stage 46.0 (TID 647)
21/12/15 13:27:02 INFO TaskSetManager: Finished task 3.0 in stage 46.0 (TID 645) in 176 ms on localhost (executor driver) (2/200)
21/12/15 13:27:02 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 3 blocks
21/12/15 13:27:02 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
21/12/15 13:27:03 INFO Executor: Finished task 2.0 in stage 46.0 (TID 644). 2724 bytes result sent to driver
21/12/15 13:27:03 INFO TaskSetManager: Starting task 6.0 in stage 46.0 (TID 648, localhost, executor driver, partition 6, PROCESS_LOCAL, 7754 bytes)
21/12/15 13:27:03 INFO Executor: Running task 6.0 in stage 46.0 (TID 648)
21/12/15 13:27:03 INFO TaskSetManager: Finished task 2.0 in stage 46.0 (TID 644) in 269 ms on localhost (executor driver) (3/200)
21/12/15 13:27:03 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 3 blocks
21/12/15 13:27:03 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
21/12/15 13:27:03 INFO Executor: Finished task 0.0 in stage 46.0 (TID 642). 2767 bytes result sent to driver
21/12/15 13:27:03 INFO TaskSetManager: Starting task 7.0 in stage 46.0 (TID 649, localhost, executor driver, partition 7, PROCESS_LOCAL, 7754 bytes)
21/12/15 13:27:03 INFO Executor: Running task 7.0 in stage 46.0 (TID 649)
21/12/15 13:27:03 INFO TaskSetManager: Finished task 0.0 in stage 46.0 (TID 642) in 342 ms on localhost (executor driver) (4/200)
21/12/15 13:27:03 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 3 blocks
21/12/15 13:27:03 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
21/12/15 13:27:03 INFO Executor: Finished task 4.0 in stage 46.0 (TID 646). 2767 bytes result sent to driver
21/12/15 13:27:03 INFO TaskSetManager: Starting task 8.0 in stage 46.0 (TID 650, localhost, executor driver, partition 8, PROCESS_LOCAL, 7754 bytes)
21/12/15 13:27:03 INFO Executor: Running task 8.0 in stage 46.0 (TID 650)
21/12/15 13:27:03 INFO TaskSetManager: Finished task 4.0 in stage 46.0 (TID 646) in 342 ms on localhost (executor driver) (5/200)
21/12/15 13:27:03 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 3 blocks
21/12/15 13:27:03 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
21/12/15 13:27:03 INFO Executor: Finished task 5.0 in stage 46.0 (TID 647). 2724 bytes result sent to driver
21/12/15 13:27:03 INFO TaskSetManager: Starting task 9.0 in stage 46.0 (TID 651, localhost, executor driver, partition 9, PROCESS_LOCAL, 7754 bytes)
21/12/15 13:27:03 INFO Executor: Running task 9.0 in stage 46.0 (TID 651)
21/12/15 13:27:03 INFO TaskSetManager: Finished task 5.0 in stage 46.0 (TID 647) in 344 ms on localhost (executor driver) (6/200)

  • 写回答

0条回答 默认 最新

    报告相同问题?

    问题事件

    • 系统已结题 12月23日
    • 创建了问题 12月15日

    悬赏问题

    • ¥15 vscode下编写第三方库opencv与pcl代码时没有代码提示
    • ¥15 能够跑通不报错,如何解决?(标签-matlab)
    • ¥15 MOS在RDS较大,频率高时开关波形异常
    • ¥15 SCENIC分析报错求解答
    • ¥15 ceph初始化mon不成功 下图不报错 这个是什么元
    • ¥15 数学建模数学建模少ai
    • ¥15 如何把LibreOffice添加到自定义层中
    • ¥35 这个等效电路图是什么意思?(答疑)(有没有不是chat的,我就是在chat搜不到才来问的,chat还开了VIP)(可以追加酬金)
    • ¥200 基于同花顺supermind的量化策略脚本编辑
    • ¥15 黄永刚的晶体塑性子程序中输入的材料参数里的晶体取向参数是什么形式的?