[SPARK-8999] Support non-temporal sequence in PrefixSpan

• Type: Improvement
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.5.0
• Fix Version/s: 1.5.0
• Component/s: MLlib
• Target Version/s: 1.5.0
• Assignee: Zhang JiaJin
• Reporter: Xiangrui Meng
• Shepherd: Xiangrui Meng
• Created: 11/Jul/15 04:18
• Updated: 17/Aug/15 23:42
• Resolved: 01/Aug/15 08:56

Description

In SPARK-6487, we assume that all items are ordered. However, we should support non-temporal sequences in PrefixSpan. This should be done before 1.5 because it changes PrefixSpan APIs.

We can use `Array[Array[Int]]` or follow SPMF to use `Array[Int]` and use -1 to mark itemset boundaries. The latter is more efficient for storage. If we support generic item type, we can use null.

Issue Links

• blocks: SPARK-9000 Support generic item type in PrefixSpan
• depends upon: SPARK-6487 Add sequential pattern mining algorithm PrefixSpan to Spark MLlib
• is blocked by: SPARK-8997 Improve LocalPrefixSpan performance
• is blocked by: SPARK-8998 Collect enough frequent prefixes before projection in PrefixSpan
• relates to: SPARK-10028 Add Python API for PrefixSpan
• links to: [Github] Pull Request #7594 (zhangjiajin)
• links to: [Github] Pull Request #7646 (zhangjiajin)
• links to: [Github] Pull Request #7818 (feynmanliang)