Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-3549

dist tar build fails in hadoop-hdfs-raid project

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 3.0.0-alpha1
    • Fix Version/s: 3.0.0-alpha1
    • Component/s: build
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      Trying to build the distribution tarball in a clean tree via mvn install -Pdist -Dtar -DskipTests -Dmaven.javadoc.skip fails with this error:

      main:
           [exec] tar: hadoop-hdfs-raid-3.0.0-SNAPSHOT: Cannot stat: No such file or directory
           [exec] tar: Exiting with failure status due to previous errors
      
      1. HDFS-3549.patch
        2 kB
        Jason Lowe
      2. HDFS-3549.patch
        2 kB
        Jason Lowe
      3. HDFS-3549.patch
        2 kB
        Jason Lowe
      4. HDFS-3549.patch
        0.5 kB
        Jason Lowe

        Activity

        Hide
        hudson Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1121 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1121/)
        HDFS-3549. Fix dist tar build fails in hadoop-hdfs-raid project. (Jason Lowe via daryn) (Revision 1353695)

        Result = FAILURE
        daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1353695
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-raid/pom.xml
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1121 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1121/ ) HDFS-3549 . Fix dist tar build fails in hadoop-hdfs-raid project. (Jason Lowe via daryn) (Revision 1353695) Result = FAILURE daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1353695 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-raid/pom.xml /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #1088 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1088/)
        HDFS-3549. Fix dist tar build fails in hadoop-hdfs-raid project. (Jason Lowe via daryn) (Revision 1353695)

        Result = FAILURE
        daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1353695
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-raid/pom.xml
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1088 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1088/ ) HDFS-3549 . Fix dist tar build fails in hadoop-hdfs-raid project. (Jason Lowe via daryn) (Revision 1353695) Result = FAILURE daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1353695 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-raid/pom.xml /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #2403 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2403/)
        HDFS-3549. Fix dist tar build fails in hadoop-hdfs-raid project. (Jason Lowe via daryn) (Revision 1353695)

        Result = FAILURE
        daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1353695
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-raid/pom.xml
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2403 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2403/ ) HDFS-3549 . Fix dist tar build fails in hadoop-hdfs-raid project. (Jason Lowe via daryn) (Revision 1353695) Result = FAILURE daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1353695 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-raid/pom.xml /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #2455 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2455/)
        HDFS-3549. Fix dist tar build fails in hadoop-hdfs-raid project. (Jason Lowe via daryn) (Revision 1353695)

        Result = SUCCESS
        daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1353695
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-raid/pom.xml
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2455 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2455/ ) HDFS-3549 . Fix dist tar build fails in hadoop-hdfs-raid project. (Jason Lowe via daryn) (Revision 1353695) Result = SUCCESS daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1353695 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-raid/pom.xml /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #2385 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2385/)
        HDFS-3549. Fix dist tar build fails in hadoop-hdfs-raid project. (Jason Lowe via daryn) (Revision 1353695)

        Result = SUCCESS
        daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1353695
        Files :

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-raid/pom.xml
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2385 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2385/ ) HDFS-3549 . Fix dist tar build fails in hadoop-hdfs-raid project. (Jason Lowe via daryn) (Revision 1353695) Result = SUCCESS daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1353695 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-raid/pom.xml /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        jlowe Jason Lowe added a comment -

        Thanks Daryn! Filed HDFS-3563 to track fixing the 31 findbugs warnings.

        Show
        jlowe Jason Lowe added a comment - Thanks Daryn! Filed HDFS-3563 to track fixing the 31 findbugs warnings.
        Hide
        daryn Daryn Sharp added a comment -

        Committed to trunk, thanks Jason.

        Show
        daryn Daryn Sharp added a comment - Committed to trunk, thanks Jason.
        Hide
        daryn Daryn Sharp added a comment -

        +1 TestRaidNode appears to fail due to race condition with querying the job status. Findbugs warnings are of course due to making findbugs work again.

        Show
        daryn Daryn Sharp added a comment - +1 TestRaidNode appears to fail due to race condition with querying the job status. Findbugs warnings are of course due to making findbugs work again.
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12532901/HDFS-3549.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 javadoc. The javadoc tool appears to have generated 13 warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        -1 findbugs. The patch appears to introduce 31 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs-raid:

        org.apache.hadoop.raid.TestRaidNode

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2679//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/2679//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs-raid.html
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2679//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12532901/HDFS-3549.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 javadoc. The javadoc tool appears to have generated 13 warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 31 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs-raid: org.apache.hadoop.raid.TestRaidNode +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2679//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/2679//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs-raid.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2679//console This message is automatically generated.
        Hide
        jlowe Jason Lowe added a comment -

        Looks like Jenkins was having some issues, uploading patch again to kick it.

        Show
        jlowe Jason Lowe added a comment - Looks like Jenkins was having some issues, uploading patch again to kick it.
        Hide
        jlowe Jason Lowe added a comment -

        Updated patch to remove need for stub findbugsExcludeFile.xml. Also removed the dist tar config since we can inherit it from the hadoop-project-dist parent. That simplifies the pom a bit and fixes a malformed XML issue due to the use of '2> /dev/null' which was already fixed in the parent pom.

        Show
        jlowe Jason Lowe added a comment - Updated patch to remove need for stub findbugsExcludeFile.xml. Also removed the dist tar config since we can inherit it from the hadoop-project-dist parent. That simplifies the pom a bit and fixes a malformed XML issue due to the use of '2> /dev/null' which was already fixed in the parent pom.
        Hide
        jlowe Jason Lowe added a comment -

        The javadoc warnings and findbug failure were introduced in MAPREDUCE-3868, which also introduced the dist tar build failure. Attaching an updated patch which adds a stub findbugsExcludeFile.xml to fix the findbugs build failure.

        Given the nature of the change, I don't believe the TestRaidNode failure is related.

        Show
        jlowe Jason Lowe added a comment - The javadoc warnings and findbug failure were introduced in MAPREDUCE-3868 , which also introduced the dist tar build failure. Attaching an updated patch which adds a stub findbugsExcludeFile.xml to fix the findbugs build failure. Given the nature of the change, I don't believe the TestRaidNode failure is related.
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12532747/HDFS-3549.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 javadoc. The javadoc tool appears to have generated 13 warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        -1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs-raid:

        org.apache.hadoop.raid.TestRaidNode

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2673//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2673//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12532747/HDFS-3549.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 javadoc. The javadoc tool appears to have generated 13 warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs-raid: org.apache.hadoop.raid.TestRaidNode +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2673//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2673//console This message is automatically generated.
        Hide
        revans2 Robert Joseph Evans added a comment -

        +1 the change looks good to me, but I am not an HDFS committer so you are going to need someone else to +1 and commit it.

        Show
        revans2 Robert Joseph Evans added a comment - +1 the change looks good to me, but I am not an HDFS committer so you are going to need someone else to +1 and commit it.
        Hide
        jlowe Jason Lowe added a comment -

        Patch to work around the issue by moving the assembly to the prepare-package phase which ensures the assembly is completed before tar tries to pick it up.

        Show
        jlowe Jason Lowe added a comment - Patch to work around the issue by moving the assembly to the prepare-package phase which ensures the assembly is completed before tar tries to pick it up.
        Hide
        jlowe Jason Lowe added a comment -

        The problem only occurs on a build from a clean tree. If one builds without -Dtar it succeeds, and then subsequent builds with -Dtar will succeed since hadoop-hdfs-raid-3.0.0-SNAPSHOT will exist from a prior build.

        It appears to be an issue of ordering during the package phase. I thought Maven was supposed to preserve the relative order of plugins within a phase as they appear in the pom, but this comment on MNG-2258 implies the ordering can still be wrong sometimes even after the fix in Maven 3.0.3.

        Show
        jlowe Jason Lowe added a comment - The problem only occurs on a build from a clean tree. If one builds without -Dtar it succeeds, and then subsequent builds with -Dtar will succeed since hadoop-hdfs-raid-3.0.0-SNAPSHOT will exist from a prior build. It appears to be an issue of ordering during the package phase. I thought Maven was supposed to preserve the relative order of plugins within a phase as they appear in the pom, but this comment on MNG-2258 implies the ordering can still be wrong sometimes even after the fix in Maven 3.0.3.

          People

          • Assignee:
            jlowe Jason Lowe
            Reporter:
            jlowe Jason Lowe
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development