0

What's difference between 'groupby.apply'and'groupby.agg'? , Why processed data are displayed as 'NoneType'？

  1. This is for Spark 2.1.0，I want to turn a column into multiple rows，same column can be connected with commas，when I use groupby.apply, there are no output, when I use groupby.agg, procedures can be followed as what I think.
  2. I use .count() of procedures, but it shows 'NoneType' object has no attribute 'count'. 
data1=df.groupBy('_c0').agg(collect_list('_c1')).show()
print(data1.count())

data1=df.groupBy('_c0').apply(collect_list('_c1')).show()

I want to know how many number of rows and cols of procedures.

shizhen
  • 12,251
  • 9
  • 52
  • 88
mandy
  • 5
  • 3

1 Answers1

0

data1=df.groupBy('_c0').apply(collect_list('_c1')).show()

returns NoneType because show does not return anything. That is why you cannot apply count on it. Get rid of show. apply returns an sql.DataFrame. For what you are trying to do refer to this SO Question

Refer to source code of show.

Sıddık Açıl
  • 957
  • 8
  • 18