Hadoop Common
  1. Hadoop Common
  2. HADOOP-8327

distcpv2 and distcpv1 jars should not coexist

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.2
    • Fix Version/s: 0.23.3, 2.0.2-alpha
    • Component/s: None
    • Labels:
      None
    • Release Note:
      Resolve sporadic distcp issue due to having two DistCp classes (v1 & v2) in the classpath.
    • Tags:
      distcp distcpv1 distcpv2 classpath
    • Target Version/s:

      Description

      Distcp v2 (hadoop-tools/hadoop-distcp/...) and Distcp v1 (hadoop-tools/hadoop-extras/...) are currently both built, and the resulting hadoop-distcp-x.jar and hadoop-extras-x.jar end up in the same class path directory. This causes some undeterministic problems, where v1 is launched when v2 is intended, or even v2 is launched, but may later fail on various nodes because of mismatch with v1.

      According to
      http://docs.oracle.com/javase/6/docs/technotes/tools/windows/classpath.html ("Understanding class path wildcards")

      "The order in which the JAR files in a directory are enumerated in the expanded class path is not specified and may vary from platform to platform and even from moment to moment on the same machine."

      Suggest distcpv1 be deprecated at this point, possibly by discontinuing build of distcpv1.

      1. HADOOP-8327.patch
        142 kB
        Dave Thompson
      2. HADOOP-8327.patch
        142 kB
        Dave Thompson
      3. HADOOP-8327-branch-0.23.2.patch
        142 kB
        Dave Thompson

        Activity

        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1073 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1073/)
        HADOOP-8327. distcpv2 and distcpv1 jars should not coexist (Dave Thompson via bobby) (Revision 1335075)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1335075
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-tools/hadoop-extras/src/main/java/org/apache/hadoop/tools/DistCp.java
        • /hadoop/common/trunk/hadoop-tools/hadoop-extras/src/main/java/org/apache/hadoop/tools/DistCpV1.java
        • /hadoop/common/trunk/hadoop-tools/hadoop-extras/src/main/java/org/apache/hadoop/tools/Logalyzer.java
        • /hadoop/common/trunk/hadoop-tools/hadoop-extras/src/test/java/org/apache/hadoop/tools/TestCopyFiles.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1073 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1073/ ) HADOOP-8327 . distcpv2 and distcpv1 jars should not coexist (Dave Thompson via bobby) (Revision 1335075) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1335075 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-tools/hadoop-extras/src/main/java/org/apache/hadoop/tools/DistCp.java /hadoop/common/trunk/hadoop-tools/hadoop-extras/src/main/java/org/apache/hadoop/tools/DistCpV1.java /hadoop/common/trunk/hadoop-tools/hadoop-extras/src/main/java/org/apache/hadoop/tools/Logalyzer.java /hadoop/common/trunk/hadoop-tools/hadoop-extras/src/test/java/org/apache/hadoop/tools/TestCopyFiles.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #1038 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1038/)
        HADOOP-8327. distcpv2 and distcpv1 jars should not coexist (Dave Thompson via bobby) (Revision 1335075)

        Result = FAILURE
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1335075
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-tools/hadoop-extras/src/main/java/org/apache/hadoop/tools/DistCp.java
        • /hadoop/common/trunk/hadoop-tools/hadoop-extras/src/main/java/org/apache/hadoop/tools/DistCpV1.java
        • /hadoop/common/trunk/hadoop-tools/hadoop-extras/src/main/java/org/apache/hadoop/tools/Logalyzer.java
        • /hadoop/common/trunk/hadoop-tools/hadoop-extras/src/test/java/org/apache/hadoop/tools/TestCopyFiles.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1038 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1038/ ) HADOOP-8327 . distcpv2 and distcpv1 jars should not coexist (Dave Thompson via bobby) (Revision 1335075) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1335075 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-tools/hadoop-extras/src/main/java/org/apache/hadoop/tools/DistCp.java /hadoop/common/trunk/hadoop-tools/hadoop-extras/src/main/java/org/apache/hadoop/tools/DistCpV1.java /hadoop/common/trunk/hadoop-tools/hadoop-extras/src/main/java/org/apache/hadoop/tools/Logalyzer.java /hadoop/common/trunk/hadoop-tools/hadoop-extras/src/test/java/org/apache/hadoop/tools/TestCopyFiles.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Build #251 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/251/)
        svn merge -c 1335075 FIXES: HADOOP-8327. distcpv2 and distcpv1 jars should not coexist (Dave Thompson via bobby) (Revision 1335079)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1335079
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-extras/src/main/java/org/apache/hadoop/tools/DistCp.java
        • /hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-extras/src/main/java/org/apache/hadoop/tools/DistCpV1.java
        • /hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-extras/src/main/java/org/apache/hadoop/tools/Logalyzer.java
        • /hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-extras/src/test/java/org/apache/hadoop/tools/TestCopyFiles.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #251 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/251/ ) svn merge -c 1335075 FIXES: HADOOP-8327 . distcpv2 and distcpv1 jars should not coexist (Dave Thompson via bobby) (Revision 1335079) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1335079 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-extras/src/main/java/org/apache/hadoop/tools/DistCp.java /hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-extras/src/main/java/org/apache/hadoop/tools/DistCpV1.java /hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-extras/src/main/java/org/apache/hadoop/tools/Logalyzer.java /hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-extras/src/test/java/org/apache/hadoop/tools/TestCopyFiles.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #2212 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2212/)
        HADOOP-8327. distcpv2 and distcpv1 jars should not coexist (Dave Thompson via bobby) (Revision 1335075)

        Result = ABORTED
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1335075
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-tools/hadoop-extras/src/main/java/org/apache/hadoop/tools/DistCp.java
        • /hadoop/common/trunk/hadoop-tools/hadoop-extras/src/main/java/org/apache/hadoop/tools/DistCpV1.java
        • /hadoop/common/trunk/hadoop-tools/hadoop-extras/src/main/java/org/apache/hadoop/tools/Logalyzer.java
        • /hadoop/common/trunk/hadoop-tools/hadoop-extras/src/test/java/org/apache/hadoop/tools/TestCopyFiles.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2212 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2212/ ) HADOOP-8327 . distcpv2 and distcpv1 jars should not coexist (Dave Thompson via bobby) (Revision 1335075) Result = ABORTED bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1335075 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-tools/hadoop-extras/src/main/java/org/apache/hadoop/tools/DistCp.java /hadoop/common/trunk/hadoop-tools/hadoop-extras/src/main/java/org/apache/hadoop/tools/DistCpV1.java /hadoop/common/trunk/hadoop-tools/hadoop-extras/src/main/java/org/apache/hadoop/tools/Logalyzer.java /hadoop/common/trunk/hadoop-tools/hadoop-extras/src/test/java/org/apache/hadoop/tools/TestCopyFiles.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #2194 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2194/)
        HADOOP-8327. distcpv2 and distcpv1 jars should not coexist (Dave Thompson via bobby) (Revision 1335075)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1335075
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-tools/hadoop-extras/src/main/java/org/apache/hadoop/tools/DistCp.java
        • /hadoop/common/trunk/hadoop-tools/hadoop-extras/src/main/java/org/apache/hadoop/tools/DistCpV1.java
        • /hadoop/common/trunk/hadoop-tools/hadoop-extras/src/main/java/org/apache/hadoop/tools/Logalyzer.java
        • /hadoop/common/trunk/hadoop-tools/hadoop-extras/src/test/java/org/apache/hadoop/tools/TestCopyFiles.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2194 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2194/ ) HADOOP-8327 . distcpv2 and distcpv1 jars should not coexist (Dave Thompson via bobby) (Revision 1335075) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1335075 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-tools/hadoop-extras/src/main/java/org/apache/hadoop/tools/DistCp.java /hadoop/common/trunk/hadoop-tools/hadoop-extras/src/main/java/org/apache/hadoop/tools/DistCpV1.java /hadoop/common/trunk/hadoop-tools/hadoop-extras/src/main/java/org/apache/hadoop/tools/Logalyzer.java /hadoop/common/trunk/hadoop-tools/hadoop-extras/src/test/java/org/apache/hadoop/tools/TestCopyFiles.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #2269 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2269/)
        HADOOP-8327. distcpv2 and distcpv1 jars should not coexist (Dave Thompson via bobby) (Revision 1335075)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1335075
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-tools/hadoop-extras/src/main/java/org/apache/hadoop/tools/DistCp.java
        • /hadoop/common/trunk/hadoop-tools/hadoop-extras/src/main/java/org/apache/hadoop/tools/DistCpV1.java
        • /hadoop/common/trunk/hadoop-tools/hadoop-extras/src/main/java/org/apache/hadoop/tools/Logalyzer.java
        • /hadoop/common/trunk/hadoop-tools/hadoop-extras/src/test/java/org/apache/hadoop/tools/TestCopyFiles.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2269 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2269/ ) HADOOP-8327 . distcpv2 and distcpv1 jars should not coexist (Dave Thompson via bobby) (Revision 1335075) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1335075 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-tools/hadoop-extras/src/main/java/org/apache/hadoop/tools/DistCp.java /hadoop/common/trunk/hadoop-tools/hadoop-extras/src/main/java/org/apache/hadoop/tools/DistCpV1.java /hadoop/common/trunk/hadoop-tools/hadoop-extras/src/main/java/org/apache/hadoop/tools/Logalyzer.java /hadoop/common/trunk/hadoop-tools/hadoop-extras/src/test/java/org/apache/hadoop/tools/TestCopyFiles.java
        Hide
        Robert Joseph Evans added a comment -

        Thanks Dave,

        I have put this into trunk, branch-2, and branch-0.23

        Show
        Robert Joseph Evans added a comment - Thanks Dave, I have put this into trunk, branch-2, and branch-0.23
        Hide
        Robert Joseph Evans added a comment -

        The patch looks good to me +1.

        Show
        Robert Joseph Evans added a comment - The patch looks good to me +1.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12525853/HADOOP-8327.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        -1 javadoc. The javadoc tool appears to have generated 2 warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-tools/hadoop-extras.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/948//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/948//artifact/trunk/trunk/patchprocess/newPatchFindbugsWarningshadoop-extras.html
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/948//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12525853/HADOOP-8327.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. -1 javadoc. The javadoc tool appears to have generated 2 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-tools/hadoop-extras. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/948//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/948//artifact/trunk/trunk/patchprocess/newPatchFindbugsWarningshadoop-extras.html Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/948//console This message is automatically generated.
        Hide
        Dave Thompson added a comment -

        Same trunk patch, just uploaded with grant license toggle flipped.

        Show
        Dave Thompson added a comment - Same trunk patch, just uploaded with grant license toggle flipped.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12525302/HADOOP-8327.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        -1 javadoc. The javadoc tool appears to have generated 2 warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-tools/hadoop-extras.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/920//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/920//artifact/trunk/trunk/patchprocess/newPatchFindbugsWarningshadoop-extras.html
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/920//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12525302/HADOOP-8327.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. -1 javadoc. The javadoc tool appears to have generated 2 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-tools/hadoop-extras. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/920//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/920//artifact/trunk/trunk/patchprocess/newPatchFindbugsWarningshadoop-extras.html Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/920//console This message is automatically generated.
        Hide
        Dave Thompson added a comment -

        Though the intent is for the fix is branch 0.23.2, I'm attaching a trunk patch now as it has been speculated that the auto patch is testing to trunk rather than 0.23.2 as the attached patch name specifies.

        Show
        Dave Thompson added a comment - Though the intent is for the fix is branch 0.23.2, I'm attaching a trunk patch now as it has been speculated that the auto patch is testing to trunk rather than 0.23.2 as the attached patch name specifies.
        Hide
        Dave Thompson added a comment -

        I don't seem to have any visibility as to why the above auto patching isn't succeeding. Patch looks good to me.

        Show
        Dave Thompson added a comment - I don't seem to have any visibility as to why the above auto patching isn't succeeding. Patch looks good to me.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12525221/HADOOP-8327-branch-0.23.2.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/917//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12525221/HADOOP-8327-branch-0.23.2.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/917//console This message is automatically generated.
        Hide
        Dave Thompson added a comment -

        The attached patch renames the v1 DistCp class to DistCpV1. Further, the utility dependencies that use it Logalyzer and TestCopyFiles are adjusted to use the renamed DistCpV1.

        Show
        Dave Thompson added a comment - The attached patch renames the v1 DistCp class to DistCpV1. Further, the utility dependencies that use it Logalyzer and TestCopyFiles are adjusted to use the renamed DistCpV1.
        Hide
        Dave Thompson added a comment -

        Logalyzer and TestCopyFiles are two utilities that depend on distcp V1, and further use an incompatible constructor. I suggest renaming the DistCp (v1) class to DistCPV1 for now, which will prevent random distcp failures from the above problem, and not affect those utilities that still depend on DistCpV1. Further any external utilities that use this class will be flushed out, but the class will still be accessible (though now called DistCpV1).

        DistCp (v2) will still remain (untouched) as DistCp.

        Show
        Dave Thompson added a comment - Logalyzer and TestCopyFiles are two utilities that depend on distcp V1, and further use an incompatible constructor. I suggest renaming the DistCp (v1) class to DistCPV1 for now, which will prevent random distcp failures from the above problem, and not affect those utilities that still depend on DistCpV1. Further any external utilities that use this class will be flushed out, but the class will still be accessible (though now called DistCpV1). DistCp (v2) will still remain (untouched) as DistCp.

          People

          • Assignee:
            Dave Thompson
            Reporter:
            Dave Thompson
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development