Uploaded image for project: 'Apache Tez'
  1. Apache Tez
  2. TEZ-1876 [Umbrella] Polish up support for local mode in Tez
  3. TEZ-1348

Allow Tez local mode to run against filesystems other than local FS

Attach filesAttach ScreenshotVotersWatch issueWatchersLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • None
    • 0.9.2, 0.10.0
    • None
    • None
    • Committed to branch-0.9.

    Description

      In TEZ-717, I incorrect thought setting fs.defaultFS programmatically in tez-site would work for local mode.
      Currently the requirement is that tez-site.xml must have fs.defaultFS set to file:///.

      While that works, it doesn't allow for seamless execution in either local-mode or on a cluster.

      The main issue here is that when Inputs / Outputs are configured - they use a version of configuration which reads tez-site, and do not use the configuration from the client itself (which is correct behaviour).

      Not sure what a good way to fix this is
      1) It may be possible to override this value each time an instance of Configuration/TezConfiguration is created. One possible way would be to statically add a default resource to Configuration the moment a local client is created.
      2) Provide information in the contexts on whether this is local or not. This is fairly ugly, and would get in the way of running mixed mode tasks.

      Anyone have other suggestions ?

      Attachments

        1. tez-1348.patch
          16 kB
          Todd Lipcon
        2. tez-1348.patch
          17 kB
          Todd Lipcon
        3. tez-1348.txt
          17 kB
          Todd Lipcon

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            tlipcon Todd Lipcon
            sseth Siddharth Seth
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 40m
                40m

                Slack

                  Issue deployment