[GOBBLIN-587] Implement partition level lineage for fs based destination

• Type: Task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.14.0
• Assignee: Zhixiong Chen
• Reporter: Zhixiong Chen
• Created: 12/Sep/18 22:28
• Updated: 30/Nov/18 06:34
• Resolved: 14/Sep/18 19:33

Description

Currently, gobblin lineage is sent at dataset level. The task is to send partition level lineage for fs sink. An example kafka-hdfs partition lineage is

{
  "timestamp": 1536785248451,
  "namespace": {
    "string": "gobblin.event.lineage"
  },
  "name": "LoginEvent",
  "metadata": {
    "destination": "{\"object-type\":\"org.apache.gobblin.dataset.PartitionDescriptor\",\"object-data\":{\"dataset\":{\"object-type\":\"org.apache.gobblin.dataset.DatasetDescriptor\",\"object-data\":{\"platform\":\"hdfs\",\"metadata\":{\"branch\":\"0\"},\"name\":\"/data/tracking/LoginEvent\"}},\"name\":\"hourly/2018/09/12/12\"}}",
    "eventType": "LineageEvent",
    "source": "{\"object-type\":\"org.apache.gobblin.dataset.DatasetDescriptor\",\"object-data\":{\"platform\":\"kafka\",\"metadata\":{},\"name\":\"LoginEvent\"}}",
    "metricContextName": "org.apache.gobblin.runtime.SafeDatasetCommit.1693032310",
    "metricContextID": "1a7895b0-9e93-414e-ac0b-038f9375c82e",
    "class": "org.apache.gobblin.runtime.SafeDatasetCommit",
  }
}

Note: Lineage is not available automatically. You might have to implement the support in your source-destination pair.