Details
Major Description
The current freqItems API is really awkward to use. It returns a DataFrame with a single row, in which each value is an array of frequent items.
This design doesn't work well for exploratory data analysis (running show – when there are more than 2 or 3 frequent values, the values get cut off):
In [74]: df.stat.freqItems(["a", "b", "c"], 0.4).show() +------------------+------------------+-----------------+ | a_freqItems| b_freqItems| c_freqItems| +------------------+------------------+-----------------+ |ArrayBuffer(11, 1)|ArrayBuffer(2, 22)|ArrayBuffer(1, 3)| +------------------+------------------+-----------------+
It also doesn't work well for serious engineering, since it is hard to get the value out.
We should create a new function (so we maintain source/binary compatibility) that returns a list of list of values.
Attachments
Issue Links
- links to
