Uploaded image for project: 'Apache Gobblin'
  1. Apache Gobblin
  2. GOBBLIN-227

JobLauncherUtils.cleanTaskStagingData fails for jobs with forks

    Details

    • Type: Bug
    • Status: Open
    • Priority: Critical
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Precondition:
      Using Hocon configuration and have two forks configured.

      Summary:
      When AbstractJobLauncher calls JobLauncherUtils.cleanTaskStagingData it tries to lookup writer.staging.dir in the configuration and fails.

      Details:
      Hocon configuration doesn't allow the following config:

      writer.staging.dir=/foo
      writer.staging.dir.0=/foo
      writer.staging.dir.1=/foo
      

      Initially writer.staging.dir is of type String, but when the Hocon parser encounters writer.staging.dir.0, it decides that writer.staging.dir is now of type Object, overwriting the prior value with {"0": "/foo"}.
      The effective Hocon configuration is:

      {
        "writer": {
          "staging": {
            "dir": {
              "0": "/foo",
              "1": "/foo"
            }
          }
        }
      }
      

      Fork specific configuration uses the same config keys as regular configuration except the fork number is appended like: .1. The code that looks up fork specific configuration doesn't automatically fallback to regular configuration. For example, if the code is trying to find writer.staging.dir.0 and it isn't configured, the job will fail. Then means that all forks must configure fork specific versions of writer.staging.dir.

      When AbstractJobLauncher calls JobLauncherUtils.cleanTaskStagingData it cleans up the based on the current job's configuration. Because of this, fork.branches is always set to 1. The call to WriterUtils.getWriterStagingDir(state, numBranches, branchId) is made with numBranches=1 and branchId=0. This results in the method looking for writer.staging.dir. Unfortunately, when using Hocon configuration the value writer.staging.dir doesn't exist and the job fails.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              jbaranick Joel Baranick
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated: