Uploaded image for project: 'Apache Tez'
  1. Apache Tez
  2. TEZ-3848

Tez Local mode doesn't localize distributed cache files

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.9.1
    • Component/s: None
    • Labels:
      None

      Description

      Tez doesn't symlink LocalResources into place in LocalContainerLauncher.

      In YARN mode, Yarn takes care of this when it launches the container. But in local mode, if you're depending on a file existing in the distributed cache, it's never symlinked into place (so you're out of luck).

      We test our pig scripts in local mode and have some tools to set up the distributed cache the same way it would work in production. This works fine in Mapreduce mode but are unable to use Pig + Tez local mode for testing due to this problem.

      I have a fix working and will submit a PR once I rebase it.

      Jonathan Turner Eagles Will Lauer

        Attachments

        1. TEZ-3848.1.patch
          11 kB
          Jacob Tolar

          Issue Links

            Activity

              People

              • Assignee:
                jtolar Jacob Tolar
                Reporter:
                jtolar Jacob Tolar
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: