Uploaded image for project: 'Apache Tez'
  1. Apache Tez
  2. TEZ-2368

Make a dag identifier available in Context classes

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.7.0
    • None
    • None

    Description

      Provide the dag number, which is a unique number, for each dag running within an application in the TezInputContext, TezOutputContext, TezProcessorContext.
      When containers are re-used, or for external services, this can be used to generate intermediate data to a dag specific directory instead of an application specific directory, where it becomes difficult to differentiate between different dags.
      The DAG name does provide this - but is not suitable for use in a directory name. Hashing the name is an option, but can lead to collisions.
      Generating data into a dag specific directory will eventually only be usable when we move away from the default MR handler, or enhance it to support an additional parameter.

      Attachments

        1. TEZ-2368.1.txt
          6 kB
          Siddharth Seth
        2. TEZ-2368.2.txt
          6 kB
          Siddharth Seth
        3. TEZ-2368.3.txt
          6 kB
          Siddharth Seth

        Activity

          People

            sseth Siddharth Seth
            sseth Siddharth Seth
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: