Gobblin 0.9.0 persists Task and WorkUnit state to state.store.fs.uri when running under yarn. The issue here is that the state store might be switched to something like s3a which is eventually consistent. In 0.6.0, gobblin persisted the Task and WorkUnit state to HDFS and reserved the state.store.fs.uri for the final job state. This worked much better as the Task and WorkUnit state is only used for communication between the worker nodes and the master node.
Joel Baranick wrote on 2017-06-22T21:36:42Z : @htran1 Are you able to look into this?