各位大佬,请问一下,我有一个spark的dataframe,1000多列,想求出每一列数据的25%分位数,50分位数和75分为数,
temp_result = original_data.agg(*[F.max(original_data.columns[i]).alias('max')for i in range(1000)],
*[F.min(original_data.columns[i]).alias('min')for i in range(1000)],
*[F.mean(original_data.columns[i]).alias('mean')for i in range(1000)],
*[F.expr('percentile_approx('+ original_data.columns[i] + ', array(0.25))').alias('25%')for i in range(1000)],
*[F.expr('percentile_approx('+ original_data.columns[i] + ', array(0.50))').alias('50%')for i in range(1000)],
*[F.expr('percentile_approx('+ original_data.columns[i] + ', array(0.75))').alias('75%')for i in range(1000)]).collect()
我这样求的话很耗时,有没有什么好方法?跪求!!!