Uploaded image for project: 'Apache Tez'
  1. Apache Tez
  2. TEZ-2368

Make a dag identifier available in Context classes

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.7.0
    • Component/s: None
    • Labels:
      None
    • Target Version/s:

      Description

      Provide the dag number, which is a unique number, for each dag running within an application in the TezInputContext, TezOutputContext, TezProcessorContext.
      When containers are re-used, or for external services, this can be used to generate intermediate data to a dag specific directory instead of an application specific directory, where it becomes difficult to differentiate between different dags.
      The DAG name does provide this - but is not suitable for use in a directory name. Hashing the name is an option, but can lead to collisions.
      Generating data into a dag specific directory will eventually only be usable when we move away from the default MR handler, or enhance it to support an additional parameter.

        Attachments

        1. TEZ-2368.3.txt
          6 kB
          Siddharth Seth
        2. TEZ-2368.2.txt
          6 kB
          Siddharth Seth
        3. TEZ-2368.1.txt
          6 kB
          Siddharth Seth

          Activity

            People

            • Assignee:
              sseth Siddharth Seth
              Reporter:
              sseth Siddharth Seth
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: