[GOBBLIN-1972] Fix Iceberg-Distcp, which now erroneously shows the job finishing as "SUCCESSFUL", despite the commit step failing with an exception.

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Component/s: gobblin-core
• Assignee: Abhishek Tiwari
• Reporter: Kip Kohn
• Created: 06/Dec/23 20:36
• Updated: 14/Dec/23 20:13
• Resolved: 14/Dec/23 20:12

Description

Find and fix what appears to be a regression in `CopyDataPublisher` or `SafeDatasetCommit`...

Here, the same Iceberg-Distcp `CommitStep` fails (`org.apache.gobblin.data.management.copy.iceberg.IcebergRegisterStep`), but with apache gobblin build `0.18.0-dev-493` it correctly shows "FAILED" yet with `0.18.0-dev-515` it claims "SUCCEEDED".

BOTH display the same error:

ERROR [CopyDataPublisher] [Azkaban] Failed to publish tracking_db.MyEvent
  [then the exception stacktrace for the commit step, followed by...]

ERROR [SafeDatasetCommit] [Azkaban] Failed to commit dataset
 - java.io.IOException: Not all datasets published successfully
 -     at org.apache.gobblin.data.management.copy.publisher.CopyDataPublisher.publishData(CopyDataPublisher.java:160)
 -     at org.apache.gobblin.publisher.DataPublisher.publish(DataPublisher.java:79)
 -     at org.apache.gobblin.runtime.SafeDatasetCommit.commitDataset(SafeDatasetCommit.java:260)
 -     at org.apache.gobblin.runtime.SafeDatasetCommit.call(SafeDatasetCommit.java:168)
  [...]

but only "FAILED" continues with:

WARN [SafeDatasetCommit] [Azkaban] At least one task in CopyEntity.DatasetAndPartition(dataset=CopyableDatasetMetadata(datasetURN=tracking_db.MyEvent), partition=tracking_db.MyEvent) did not get committed successfully. Setting dataset state to FAILED. java.io.IOException: Not all datasets published successfully
 -     at org.apache.gobblin.data.management.copy.publisher.CopyDataPublisher.publishData(CopyDataPublisher.java:160)
 -     at org.apache.gobblin.publisher.DataPublisher.publish(DataPublisher.java:79)
 -     at org.apache.gobblin.runtime.SafeDatasetCommit.commitDataset(SafeDatasetCommit.java:260)
 -     at org.apache.gobblin.runtime.SafeDatasetCommit.call(SafeDatasetCommit.java:168)
[...]

INFO [AbstractJobLauncher] [Azkaban] Due to task failure, will report that no records or bytes were written for CopyEntity.DatasetAndPartition(dataset=CopyableDatasetMetadata(datasetURN=tracking_db.MyEvent), partition=tracking_db.MyEvent)

by contrast, "SUCCEEDED" announces:

INFO [SafeDatasetCommit] [Azkaban] Persisting dataset state for dataset CopyEntity.DatasetAndPartition(dataset=CopyableDatasetMetadata(datasetURN=tracking_db.MyEvent), partition=tracking_db.MyEvent)
[...]
INFO [AbstractJobLauncher] [Azkaban] DatasetMetrics for 'CopyEntity.DatasetAndPartition(dataset=CopyableDatasetMetadata(datasetURN=tracking_db.MyEvent), partition=tracking_db.MyEvent)' - (records: 0; bytes: 0)