[SPARK-8449] HDF5 read/write support for Spark MLlib

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 1.4.0
• Component/s: MLlib
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Alexander Ulanov
• Created: 18/Jun/15 19:09
• Updated: 21/May/19 04:37
• Resolved: 21/May/19 04:37

Description

Add support for reading and writing HDF5 file format to/from LabeledPoint. HDFS and local file system have to be supported. Other Spark formats to be discussed.

Interface proposal:
/* path - directory path in any Hadoop-supported file system URI */
MLUtils.saveAsHDF5(sc: SparkContext, path: String, RDD[LabeledPoint]): Unit
/* path - file or directory path in any Hadoop-supported file system URI */
MLUtils.loadHDF5(sc: SparkContext, path: String): RDD[LabeledPoint]

Issue Links

• is related to: SPARK-5575 Artificial neural networks for MLlib deep learning