Uploaded image for project: 'Apache Gobblin'
  1. Apache Gobblin
  2. GOBBLIN-33

StateStores persists Task and WorkUnit state to state.store.fs.uri

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.12.0
    • Component/s: None
    • Labels:
      None

      Description

      Gobblin 0.9.0 persists Task and WorkUnit state to state.store.fs.uri when running under yarn. The issue here is that the state store might be switched to something like s3a which is eventually consistent. In 0.6.0, gobblin persisted the Task and WorkUnit state to HDFS and reserved the state.store.fs.uri for the final job state. This worked much better as the Task and WorkUnit state is only used for communication between the worker nodes and the master node.

      Github Url : https://github.com/linkedin/gobblin/issues/1847
      Github Reporter : Joel Baranick
      Github Created At : 2017-05-09T17:26:58Z
      Github Updated At : 2017-06-22T21:36:43Z

      Comments


      Joel Baranick wrote on 2017-06-22T21:36:42Z : @htran1 Are you able to look into this?

      Github Url : https://github.com/linkedin/gobblin/issues/1847#issuecomment-310509687

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                jbaranick Joel Baranick
                Reporter:
                jbaranick Joel Baranick
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: