Uploaded image for project: 'Apache Tez'
  1. Apache Tez
  2. TEZ-2397

Translation of LocalResources via Tez plan serialization can be lossy

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 0.4.0
    • Fix Version/s: 0.7.0, 0.5.4, 0.6.1
    • Component/s: None
    • Labels:
      None
    • Target Version/s:

      Description

      Happens when there's no port information. The way we serialize a YarnURL into a string causes the reconstructed path to include the port as -1, which is an invalid URL. Path/URL reconstruction from this causes the hostname to be lost.
      This is problematic on clusters running HDFA HA - since there's no host:port information, only a service name. I'd imaging it'll be a problem for viewfs as well.

      1. TEZ-2397.1.txt
        4 kB
        Siddharth Seth

        Activity

        Hide
        hitesh Hitesh Shah added a comment -

        Closing issue as 0.5.4, 0.6.1 and 0.7.0 have been released.

        Show
        hitesh Hitesh Shah added a comment - Closing issue as 0.5.4, 0.6.1 and 0.7.0 have been released.
        Hide
        hitesh Hitesh Shah added a comment -

        Thanks Siddharth Seth, Bikas Saha. Committed to 0.5 , 0.6 and master branches.

        Show
        hitesh Hitesh Shah added a comment - Thanks Siddharth Seth , Bikas Saha . Committed to 0.5 , 0.6 and master branches.
        Hide
        hitesh Hitesh Shah added a comment -

        Committing shortly.

        Show
        hitesh Hitesh Shah added a comment - Committing shortly.
        Hide
        tezqa TezQA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12729707/TEZ-2397.1.txt
        against master revision a02a5ea.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in .

        Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/595//testReport/
        Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/595//console

        This message is automatically generated.

        Show
        tezqa TezQA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12729707/TEZ-2397.1.txt against master revision a02a5ea. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/595//testReport/ Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/595//console This message is automatically generated.
        Hide
        jeagles Jonathan Eagles added a comment -

        I'll make sure this is in 0.6.1 before the rc.

        Show
        jeagles Jonathan Eagles added a comment - I'll make sure this is in 0.6.1 before the rc.
        Hide
        bikassaha Bikas Saha added a comment -

        Good catch! +1 for the patch.

        0.5.4 should wait for this. /cc Hitesh Shah 0.6.1 should wait for this /cc Jonathan Eagles

        Show
        bikassaha Bikas Saha added a comment - Good catch! +1 for the patch. 0.5.4 should wait for this. /cc Hitesh Shah 0.6.1 should wait for this /cc Jonathan Eagles
        Hide
        sseth Siddharth Seth added a comment -

        Patch to use translation via ConverterUtils instead of trying to append strings. ConverterUtils handles the port being set to -1 in case it's absent properly.

        Hitesh Shah - please review.

        Show
        sseth Siddharth Seth added a comment - Patch to use translation via ConverterUtils instead of trying to append strings. ConverterUtils handles the port being set to -1 in case it's absent properly. Hitesh Shah - please review.

          People

          • Assignee:
            sseth Siddharth Seth
            Reporter:
            sseth Siddharth Seth
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development