[SPARK-3573] Dataset

• Type: Sub-task
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 1.2.0
• Component/s: MLlib
• Target Version/s: 1.2.0
• Assignee: Xiangrui Meng
• Reporter: Xiangrui Meng
• Shepherd: Michael Armbrust
• Created: 17/Sep/14 20:14
• Updated: 04/Nov/14 06:32
• Resolved: 04/Nov/14 06:32

Description

This JIRA is for discussion of ML dataset, essentially a SchemaRDD with extra ML-specific metadata embedded in its schema.

.Sample code

Suppose we have training events stored on HDFS and user/ad features in Hive, we want to assemble features for training and then apply decision tree.
The proposed pipeline with dataset looks like the following (need more refinements):

sqlContext.jsonFile("/path/to/training/events", 0.01).registerTempTable("event")
val training = sqlContext.sql("""
  SELECT event.id AS eventId, event.userId AS userId, event.adId AS adId, event.action AS label,
         user.gender AS userGender, user.country AS userCountry, user.features AS userFeatures,
         ad.targetGender AS targetGender
    FROM event JOIN user ON event.userId = user.id JOIN ad ON event.adId = ad.id;""").cache()

val indexer = new Indexer()
val interactor = new Interactor()
val fvAssembler = new FeatureVectorAssembler()
val treeClassifer = new DecisionTreeClassifer()

val paramMap = new ParamMap()
  .put(indexer.features, Map("userCountryIndex" -> "userCountry"))
  .put(indexer.sortByFrequency, true)
  .put(interactor.features, Map("genderMatch" -> Array("userGender", "targetGender")))
  .put(fvAssembler.features, Map("features" -> Array("genderMatch", "userCountryIndex", "userFeatures")))
  .put(fvAssembler.dense, true)
  .put(treeClassifer.maxDepth, 4) // By default, classifier recognizes "features" and "label" columns.

val pipeline = Pipeline.create(indexer, interactor, fvAssembler, treeClassifier)
val model = pipeline.fit(training, paramMap)

sqlContext.jsonFile("/path/to/events", 0.01).registerTempTable("event")
val test = sqlContext.sql("""
  SELECT event.id AS eventId, event.userId AS userId, event.adId AS adId,
         user.gender AS userGender, user.country AS userCountry, user.features AS userFeatures,
         ad.targetGender AS targetGender
    FROM event JOIN user ON event.userId = user.id JOIN ad ON event.adId = ad.id;""")

val prediction = model.transform(test).select('eventId, 'prediction)

Issue Links

• depends upon: SPARK-3569 Add metadata field to StructField
• depends upon: SPARK-3572 Internal API for User-Defined Types
• is related to: SPARK-3530 Pipeline and Parameters
• links to: [Github] Pull Request #3070 (mengxr)