子集DataFrames（Gota）时的Goroutine开销和性能分析

I've been working since the beginning of 2016 on implementing a Pandas/R DataFrame implementation for Go: https://github.com/kniren/gota.

Recently, I've been focusing on improving the performance of the library to try to match that of Pandas/Dplyr. You can follow the progress so far here: https://github.com/kniren/gota/issues/16

Since one of the more frequently used operations is the DataFrame subsetting, I thought it could be a good idea to introduce concurrency to try to increase the performance of the system.

Before:

columns := make([]series.Series, df.ncols)
for i, column := range df.columns {
    s := column.Subset(indexes)
    columns[i] = s
}

After:

columns := make([]series.Series, df.ncols)
var wg sync.WaitGroup
wg.Add(df.ncols)
for i := range df.columns {
    go func(i int) {
        columns[i] = df.columns[i].Subset(indexes)
        wg.Done()
    }(i)
}
wg.Wait()

As far as I understand, creating a goroutine for each of the columns of a DataFrame should not introduce much overhead, so I was expecting to achieve at least a x2 speedup with respect to the serial version (At least for large datasets). However, when benchmarking this change with datasets and indexes of different sizes the results are very disappointing (NROWSxNCOLS_INDEXSIZE-CPUCORES):

benchmark                                          old ns/op      new ns/op      delta
BenchmarkDataFrame_Subset/1000000x20_100           55230          109349         +97.99%
BenchmarkDataFrame_Subset/1000000x20_100-2         51457          67714          +31.59%
BenchmarkDataFrame_Subset/1000000x20_100-4         49845          70141          +40.72%
BenchmarkDataFrame_Subset/1000000x20_1000          518506         518085         -0.08%
BenchmarkDataFrame_Subset/1000000x20_1000-2        476661         311379         -34.67%
BenchmarkDataFrame_Subset/1000000x20_1000-4        505023         316583         -37.31%
BenchmarkDataFrame_Subset/1000000x20_10000         6621116        6314112        -4.64%
BenchmarkDataFrame_Subset/1000000x20_10000-2       7316062        4509601        -38.36%
BenchmarkDataFrame_Subset/1000000x20_10000-4       6483812        8394113        +29.46%
BenchmarkDataFrame_Subset/1000000x20_100000        105341711      106427967      +1.03%
BenchmarkDataFrame_Subset/1000000x20_100000-2      94567729       56778647       -39.96%
BenchmarkDataFrame_Subset/1000000x20_100000-4      91896690       60971444       -33.65%
BenchmarkDataFrame_Subset/1000000x20_1000000       1538680081     1632044752     +6.07%
BenchmarkDataFrame_Subset/1000000x20_1000000-2     1292113119     1100075806     -14.86%
BenchmarkDataFrame_Subset/1000000x20_1000000-4     1282367864     949615298      -25.95%
BenchmarkDataFrame_Subset/100000x20_100            50286          106850         +112.48%
BenchmarkDataFrame_Subset/100000x20_100-2          54537          70492          +29.26%
BenchmarkDataFrame_Subset/100000x20_100-4          58024          76617          +32.04%
BenchmarkDataFrame_Subset/100000x20_1000           541600         625967         +15.58%
BenchmarkDataFrame_Subset/100000x20_1000-2         493894         362894         -26.52%
BenchmarkDataFrame_Subset/100000x20_1000-4         535373         349211         -34.77%
BenchmarkDataFrame_Subset/100000x20_10000          6298063        7678499        +21.92%
BenchmarkDataFrame_Subset/100000x20_10000-2        5827185        4832560        -17.07%
BenchmarkDataFrame_Subset/100000x20_10000-4        8195048        3660077        -55.34%
BenchmarkDataFrame_Subset/100000x20_100000         105108807      82976477       -21.06%
BenchmarkDataFrame_Subset/100000x20_100000-2       92112736       58317114       -36.69%
BenchmarkDataFrame_Subset/100000x20_100000-4       92044966       63469935       -31.04%
BenchmarkDataFrame_Subset/1000x20_10               9741           53365          +447.84%
BenchmarkDataFrame_Subset/1000x20_10-2             9366           36457          +289.25%
BenchmarkDataFrame_Subset/1000x20_10-4             9463           46682          +393.31%
BenchmarkDataFrame_Subset/1000x20_100              50841          103523         +103.62%
BenchmarkDataFrame_Subset/1000x20_100-2            49972          62344          +24.76%
BenchmarkDataFrame_Subset/1000x20_100-4            72014          81808          +13.60%
BenchmarkDataFrame_Subset/1000x20_1000             457799         571292         +24.79%
BenchmarkDataFrame_Subset/1000x20_1000-2           460551         405116         -12.04%
BenchmarkDataFrame_Subset/1000x20_1000-4           462928         416522         -10.02%
BenchmarkDataFrame_Subset/1000x200_10              90125          688443         +663.88%
BenchmarkDataFrame_Subset/1000x200_10-2            85259          392705         +360.60%
BenchmarkDataFrame_Subset/1000x200_10-4            87412          387509         +343.31%
BenchmarkDataFrame_Subset/1000x200_100             486600         1082901        +122.54%
BenchmarkDataFrame_Subset/1000x200_100-2           471154         732304         +55.43%
BenchmarkDataFrame_Subset/1000x200_100-4           542846         659571         +21.50%
BenchmarkDataFrame_Subset/1000x200_1000            5926086        6686480        +12.83%
BenchmarkDataFrame_Subset/1000x200_1000-2          5364091        3986970        -25.67%
BenchmarkDataFrame_Subset/1000x200_1000-4          5904977        4504084        -23.72%
BenchmarkDataFrame_Subset/1000x2000_10             1187297        7800052        +556.96%
BenchmarkDataFrame_Subset/1000x2000_10-2           1217022        3930742        +222.98%
BenchmarkDataFrame_Subset/1000x2000_10-4           1301666        3617871        +177.94%
BenchmarkDataFrame_Subset/1000x2000_100            6942015        10790196       +55.43%
BenchmarkDataFrame_Subset/1000x2000_100-2          6588351        7592847        +15.25%
BenchmarkDataFrame_Subset/1000x2000_100-4          7067226        14391327       +103.63%
BenchmarkDataFrame_Subset/1000x2000_1000           62392457       69560711       +11.49%
BenchmarkDataFrame_Subset/1000x2000_1000-2         57793006       37416703       -35.26%
BenchmarkDataFrame_Subset/1000x2000_1000-4         59572261       58398203       -1.97%

benchmark                                          old allocs     new allocs     delta
BenchmarkDataFrame_Subset/1000000x20_100           41             42             +2.44%
BenchmarkDataFrame_Subset/1000000x20_100-2         41             42             +2.44%
BenchmarkDataFrame_Subset/1000000x20_100-4         41             42             +2.44%
BenchmarkDataFrame_Subset/1000000x20_1000          41             42             +2.44%
BenchmarkDataFrame_Subset/1000000x20_1000-2        41             42             +2.44%
BenchmarkDataFrame_Subset/1000000x20_1000-4        41             42             +2.44%
BenchmarkDataFrame_Subset/1000000x20_10000         41             42             +2.44%
BenchmarkDataFrame_Subset/1000000x20_10000-2       41             42             +2.44%
BenchmarkDataFrame_Subset/1000000x20_10000-4       41             42             +2.44%
BenchmarkDataFrame_Subset/1000000x20_100000        41             42             +2.44%
BenchmarkDataFrame_Subset/1000000x20_100000-2      41             42             +2.44%
BenchmarkDataFrame_Subset/1000000x20_100000-4      41             42             +2.44%
BenchmarkDataFrame_Subset/1000000x20_1000000       41             42             +2.44%
BenchmarkDataFrame_Subset/1000000x20_1000000-2     41             43             +4.88%
BenchmarkDataFrame_Subset/1000000x20_1000000-4     41             46             +12.20%
BenchmarkDataFrame_Subset/100000x20_100            41             42             +2.44%
BenchmarkDataFrame_Subset/100000x20_100-2          41             42             +2.44%
BenchmarkDataFrame_Subset/100000x20_100-4          41             42             +2.44%
BenchmarkDataFrame_Subset/100000x20_1000           41             42             +2.44%
BenchmarkDataFrame_Subset/100000x20_1000-2         41             42             +2.44%
BenchmarkDataFrame_Subset/100000x20_1000-4         41             42             +2.44%
BenchmarkDataFrame_Subset/100000x20_10000          41             42             +2.44%
BenchmarkDataFrame_Subset/100000x20_10000-2        41             42             +2.44%
BenchmarkDataFrame_Subset/100000x20_10000-4        41             42             +2.44%
BenchmarkDataFrame_Subset/100000x20_100000         41             42             +2.44%
BenchmarkDataFrame_Subset/100000x20_100000-2       41             42             +2.44%
BenchmarkDataFrame_Subset/100000x20_100000-4       41             42             +2.44%
BenchmarkDataFrame_Subset/1000x20_10               41             42             +2.44%
BenchmarkDataFrame_Subset/1000x20_10-2             41             42             +2.44%
BenchmarkDataFrame_Subset/1000x20_10-4             41             42             +2.44%
BenchmarkDataFrame_Subset/1000x20_100              41             42             +2.44%
BenchmarkDataFrame_Subset/1000x20_100-2            41             42             +2.44%
BenchmarkDataFrame_Subset/1000x20_100-4            41             42             +2.44%
BenchmarkDataFrame_Subset/1000x20_1000             41             42             +2.44%
BenchmarkDataFrame_Subset/1000x20_1000-2           41             42             +2.44%
BenchmarkDataFrame_Subset/1000x20_1000-4           41             42             +2.44%
BenchmarkDataFrame_Subset/1000x200_10              401            402            +0.25%
BenchmarkDataFrame_Subset/1000x200_10-2            401            402            +0.25%
BenchmarkDataFrame_Subset/1000x200_10-4            401            402            +0.25%
BenchmarkDataFrame_Subset/1000x200_100             401            402            +0.25%
BenchmarkDataFrame_Subset/1000x200_100-2           401            402            +0.25%
BenchmarkDataFrame_Subset/1000x200_100-4           401            402            +0.25%
BenchmarkDataFrame_Subset/1000x200_1000            401            402            +0.25%
BenchmarkDataFrame_Subset/1000x200_1000-2          401            402            +0.25%
BenchmarkDataFrame_Subset/1000x200_1000-4          401            402            +0.25%
BenchmarkDataFrame_Subset/1000x2000_10             4001           4002           +0.02%
BenchmarkDataFrame_Subset/1000x2000_10-2           4001           4002           +0.02%
BenchmarkDataFrame_Subset/1000x2000_10-4           4001           4002           +0.02%
BenchmarkDataFrame_Subset/1000x2000_100            4001           4002           +0.02%
BenchmarkDataFrame_Subset/1000x2000_100-2          4001           4002           +0.02%
BenchmarkDataFrame_Subset/1000x2000_100-4          4001           4002           +0.02%
BenchmarkDataFrame_Subset/1000x2000_1000           4001           4002           +0.02%
BenchmarkDataFrame_Subset/1000x2000_1000-2         4001           4010           +0.22%
BenchmarkDataFrame_Subset/1000x2000_1000-4         4001           4003           +0.05%

benchmark                                          old bytes     new bytes     delta
BenchmarkDataFrame_Subset/1000000x20_100           32400         32416         +0.05%
BenchmarkDataFrame_Subset/1000000x20_100-2         32400         32416         +0.05%
BenchmarkDataFrame_Subset/1000000x20_100-4         32400         32416         +0.05%
BenchmarkDataFrame_Subset/1000000x20_1000          298880        298896        +0.01%
BenchmarkDataFrame_Subset/1000000x20_1000-2        298880        298896        +0.01%
BenchmarkDataFrame_Subset/1000000x20_1000-4        298880        298896        +0.01%
BenchmarkDataFrame_Subset/1000000x20_10000         2971520       2971536       +0.00%
BenchmarkDataFrame_Subset/1000000x20_10000-2       2971520       2971536       +0.00%
BenchmarkDataFrame_Subset/1000000x20_10000-4       2971520       2971536       +0.00%
BenchmarkDataFrame_Subset/1000000x20_100000        29083520      29083536      +0.00%
BenchmarkDataFrame_Subset/1000000x20_100000-2      29083520      29083547      +0.00%
BenchmarkDataFrame_Subset/1000000x20_100000-4      29083542      29083563      +0.00%
BenchmarkDataFrame_Subset/1000000x20_1000000       290121600     290121616     +0.00%
BenchmarkDataFrame_Subset/1000000x20_1000000-2     290121600     290121696     +0.00%
BenchmarkDataFrame_Subset/1000000x20_1000000-4     290121600     290121840     +0.00%
BenchmarkDataFrame_Subset/100000x20_100            32400         32416         +0.05%
BenchmarkDataFrame_Subset/100000x20_100-2          32400         32416         +0.05%
BenchmarkDataFrame_Subset/100000x20_100-4          32400         32416         +0.05%
BenchmarkDataFrame_Subset/100000x20_1000           298880        298896        +0.01%
BenchmarkDataFrame_Subset/100000x20_1000-2         298880        298896        +0.01%
BenchmarkDataFrame_Subset/100000x20_1000-4         298880        298896        +0.01%
BenchmarkDataFrame_Subset/100000x20_10000          2971520       2971536       +0.00%
BenchmarkDataFrame_Subset/100000x20_10000-2        2971520       2971536       +0.00%
BenchmarkDataFrame_Subset/100000x20_10000-4        2971520       2971536       +0.00%
BenchmarkDataFrame_Subset/100000x20_100000         29083520      29083536      +0.00%
BenchmarkDataFrame_Subset/100000x20_100000-2       29083520      29083536      +0.00%
BenchmarkDataFrame_Subset/100000x20_100000-4       29083542      29083553      +0.00%
BenchmarkDataFrame_Subset/1000x20_10               4880          4896          +0.33%
BenchmarkDataFrame_Subset/1000x20_10-2             4880          4896          +0.33%
BenchmarkDataFrame_Subset/1000x20_10-4             4880          4896          +0.33%
BenchmarkDataFrame_Subset/1000x20_100              32400         32416         +0.05%
BenchmarkDataFrame_Subset/1000x20_100-2            32400         32416         +0.05%
BenchmarkDataFrame_Subset/1000x20_100-4            32400         32416         +0.05%
BenchmarkDataFrame_Subset/1000x20_1000             298880        298896        +0.01%
BenchmarkDataFrame_Subset/1000x20_1000-2           298880        298896        +0.01%
BenchmarkDataFrame_Subset/1000x20_1000-4           298880        298896        +0.01%
BenchmarkDataFrame_Subset/1000x200_10              49568         49584         +0.03%
BenchmarkDataFrame_Subset/1000x200_10-2            49568         49584         +0.03%
BenchmarkDataFrame_Subset/1000x200_10-4            49568         49585         +0.03%
BenchmarkDataFrame_Subset/1000x200_100             324768        324784        +0.00%
BenchmarkDataFrame_Subset/1000x200_100-2           324768        324784        +0.00%
BenchmarkDataFrame_Subset/1000x200_100-4           324768        324784        +0.00%
BenchmarkDataFrame_Subset/1000x200_1000            2989568       2989584       +0.00%
BenchmarkDataFrame_Subset/1000x200_1000-2          2989568       2989584       +0.00%
BenchmarkDataFrame_Subset/1000x200_1000-4          2989569       2989588       +0.00%
BenchmarkDataFrame_Subset/1000x2000_10             491072        491088        +0.00%
BenchmarkDataFrame_Subset/1000x2000_10-2           491072        491133        +0.01%
BenchmarkDataFrame_Subset/1000x2000_10-4           491072        491088        +0.00%
BenchmarkDataFrame_Subset/1000x2000_100            3243072       3243088       +0.00%
BenchmarkDataFrame_Subset/1000x2000_100-2          3243074       3243102       +0.00%
BenchmarkDataFrame_Subset/1000x2000_100-4          3243076       3243100       +0.00%
BenchmarkDataFrame_Subset/1000x2000_1000           29891072      29891088      +0.00%
BenchmarkDataFrame_Subset/1000x2000_1000-2         29891086      29891797      +0.00%
BenchmarkDataFrame_Subset/1000x2000_1000-4         29891115      29891167      +0.00%

Running the profiler (cpu/mem) over this benchmark didn't seem to reveal nothing significant. The concurrent version seem to spend some time on rumtime.match_semaphore_signal but I guess this is to be expected when waiting for the goroutines to finish.

I've tried limiting the number of goroutines launched to the maximum number of cores as reported by runtime.GOMAXPROCS(0) but the results are somewhat even worse. Am I doing something horribly wrong here or is the goroutines overhead so big that it has such a significant effect on the performance?

写回答
好问题 0 提建议
追加酬金
关注问题
分享
邀请回答
编辑收藏删除结题
收藏举报

1条回答默认最新

关注

码龄粉丝数原力等级 --

被采纳

被点赞

采纳率
dongyou6795 2017-08-03 14:17
关注
Goroutines are cheap, but not free.

I didn't read your code, but if you are spawning NCOLS_INDEXSIZE goroutines for each row you process, then it's a very bad practice.

This can be seen in your benchmark where you have 2k columns and only 1k rows - you get very big improvement. But in all other cases, when number of columns << number of rows, goroutine spawning becomes a bottleneck.

Instead you should spawn a pool of goroutines (close to your CPU count) and distribute work between them through channels - it's the canonical way. You may want to read https://blog.golang.org/pipelines

解决无用
评论打赏
分享
举报

评论

按下Enter换行，Ctrl+Enter发表内容

报告相同问题？

关注问题

C++语言编程子集和问题（回溯法解）注意用C++啊 c++
2018-05-01 15:55

回答 2 已采纳 ``` #include #include #include #include using namespace std; int n,c,a[10000],b
java集合父集和子集操作异常 java
2016-01-21 11:48

回答 2 已采纳出现这个异常存在的原因是在遍历集合的同时，又对集合元素进行修改导致的。比如这段代码： ``` public static void main(String[] args) { Lis
词法分析——NFA 转换成 DFA：子集构造算法
2018-10-30 12:59

数学家是我理想的博客因为 NFA 的状态转移不确定，不适合直接做词法分析器的识别，在写算法时往往需要使用回溯。所以我们一般使用子集构造算法，将 NFA 转换成 DFA，得到确定的状态转移，再转化成一个词法分析器的代码。下面给出一个...
用python语言实现“子集和数”问题的分支限界算法 python
2022-05-28 00:54

回答 2 已采纳 def subset_sum(lst, target): for i in range(1, 2**len(lst)): pick = list(mask(lst, bin(
求“最大公共偏序子集” leetcode 排序算法算法
2022-02-09 22:41

回答 2 已采纳 https://leetcode-cn.com/circle/discuss/7HUB28/
用Python将数据集分为两个子集 big data python sqlite 有问必答
2021-09-28 04:17

回答 2 已采纳 df_eedi = df[df["Technical Efficiency Index"] == "EEDI"] df_evi = df[df["Technical Efficiency Index
元素和子集，属于与包含
2021-01-16 04:21

左直拳的博客子集一般地，对于两个集合A、B，如果集合A中任意一个元素都是集合B中的元素，我们就说这两个集合有包含关系，称集合A为集合B的子集（subset）。记作A⊆B（或B⊇A），读作“A包含于B”（或“B包含A”）。即，对于...
求大神解答子集生成问题（十分感谢） c++ c语言
2020-09-27 19:24

回答 1 已采纳 ``` #include #include using namespace std; int ex(int n,int m) { long pfs=n; long r
python爬虫如何取出定位标签下的所有子集文本 python
2021-03-27 16:46

回答 1 已采纳将原代码中这段内容： for j in page_spec_data: for k in j.a: # print(k.string) value_word
用Python进行一系列的数据分析问题 big data python 有问必答
2021-09-28 04:45

回答 1 已采纳数据点中的异常值处理，可参考如下方法：这个首先获取两个列数据中的四分位差IQR=Q3-Q1,a,b=df['device'].quantile([0.25,0.75])IQR=b-a获取到IQR,然后
子集和问题(C语言)--回溯法
2019-06-19 20:40

LFuser.的博客子集和问题题目描述给定一个正整数集合X={x1,x2,…,xn}和一个正整数c，设计回溯算法，求集合X的一个子集Y，使得Y中元素之和等于c。解题思路类似于全排列的思想，尝试所有结果，如果不符合则回溯具体代码实现 #...
如何读取数组或切片的子集
2018-07-17 16:16

回答 1 已采纳 The key here I think is to stop thinking of it as rows and colums; in your code, you're operating
java实现子集和问题
2019-07-22 21:49

南墙的博客求n个正整数构成的一个给定集合A = {a1,a2,a3,…,an}的子集，子集的和要等于一个给定的正整数d。请输出所有符合条件的子集。 2 解决方案 2.1 全排列思想求解方法1：首先，将子集保存在一个数组链表中，每次往链表中...
Java实现 LeetCode 416 分割等和子集
2020-03-15 10:28

南墙的博客 416. 分割等和子集给定一个只包含正整数的非空数组。是否可以将这个数组分割成两个子集，使得两个子集的元素和相等。注意: 每个数组中的元素不会超过 100 数组的大小不会超过 200 示例 1: 输入: [1, 5, 11, 5] ...
没有解决我的问题, 去提问

悬赏问题

¥15 Mac系统vs code使用phpstudy如何配置debug来调试php
¥15 目前主流的音乐软件，像网易云音乐，QQ音乐他们的前端和后台部分是用的什么技术实现的?求解！
¥60 pb数据库修改与连接
¥15 spss统计中二分类变量和有序变量的相关性分析可以用kendall相关分析吗？
¥15 拟通过pc下指令到安卓系统，如果追求响应速度，尽可能无延迟，是不是用安卓模拟器会优于实体的安卓手机？如果是，可以快多少毫秒？
¥20 神经网络Sequential name=sequential, built=False
¥16 Qphython 用xlrd读取excel报错
¥15 单片机学习顺序问题！！
¥15 ikuai客户端多拨vpn，重启总是有个别重拨不上
¥20 关于#anlogic#sdram#的问题，如何解决？(关键词-performance)

子集DataFrames（Gota）时的Goroutine开销和性能分析

1条回答 默认 最新

悬赏问题

1条回答默认最新