Uploaded image for project: 'Apache Tez'
  1. Apache Tez
  2. TEZ-690 Tez API Ease of Use
  3. TEZ-778

Simplify Local Resource specification while building a DAG

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Follow up from TEZ-752.
      It would be useful to allow users to specify LocalResources as just a simple path - rather than the current requirement of a YARN local resource.

      Something along the lines of
      Vertex.addResource(String name, Path hdfsPath)
      Vertex.addResource(String name, Path hdfsPath, AdditionalOptions)

      Restricting the path to HDFS only could be an option - if we want to avoid having to manage movement of files, cleanup etc.

      After this TezClient would take care of obtaining Tokens, converting this over to what is required by YARN etc.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              sseth Siddharth Seth
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated: