Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.0.0-alpha
    • Component/s: contrib/raid
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

          [javac] Compiling 33 source files to /Users/szetszwo/hadoop/t2/hadoop-mapreduce-project/build/contrib/raid/classes
          [javac] /Users/szetszwo/hadoop/t2/hadoop-mapreduce-project/src/contrib/raid/src/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRaidUtil.java:42:
       unreported exception org.apache.hadoop.ipc.StandbyException; must be caught or declared to be thrown
          [javac]     return namesystem.getFileInfo(src, resolveLink);
          [javac]                                  ^
      

        Issue Links

          Activity

          Tsz Wo Nicholas Sze created issue -
          Tsz Wo Nicholas Sze made changes -
          Field Original Value New Value
          Link This issue is related to HDFS-3089 [ HDFS-3089 ]
          Hide
          Tsz Wo Nicholas Sze added a comment -

          HDFS-3089 is going to break RAID since it is changing FSDatasetInterface.

          -- ./src/java/org/apache/hadoop/hdfs/server/datanode/RaidBlockSender.java ----
          444:     private final FSDatasetInterface data;
          446:     private BlockInputStreamFactory(ExtendedBlock block, FSDatasetInt ...
             2 occurrence(s)
          

          However, RAID is already broken before HDFS-3089. It looks like that it was broken by the HA changes.

          Show
          Tsz Wo Nicholas Sze added a comment - HDFS-3089 is going to break RAID since it is changing FSDatasetInterface. -- ./src/java/org/apache/hadoop/hdfs/server/datanode/RaidBlockSender.java ---- 444: private final FSDatasetInterface data; 446: private BlockInputStreamFactory(ExtendedBlock block, FSDatasetInt ... 2 occurrence(s) However, RAID is already broken before HDFS-3089 . It looks like that it was broken by the HA changes.
          Tsz Wo Nicholas Sze made changes -
          Link This issue is broken by HDFS-1623 [ HDFS-1623 ]
          Devaraj K made changes -
          Attachment MAPREDUCE-4057.patch [ 12521892 ]
          Hide
          Devaraj K added a comment -

          I have attached patch to fix these compilation errors.

          Show
          Devaraj K added a comment - I have attached patch to fix these compilation errors.
          Devaraj K made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12521892/MAPREDUCE-4057.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService
          org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry
          org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization
          org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2172//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2172//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12521892/MAPREDUCE-4057.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2172//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2172//console This message is automatically generated.
          Hide
          Devaraj K added a comment -

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService
          org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry
          org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization
          org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs

          These test failures are not related to this patch. Already MAPREDUCE-4094
          jira exist to address these failures.

          Show
          Devaraj K added a comment - -1 core tests. The patch failed these unit tests: org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs These test failures are not related to this patch. Already MAPREDUCE-4094 jira exist to address these failures.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Raid still cannot be compiled after the patch:

          compile-test:
               [echo] contrib: raid
              [javac] /Users/szetszwo/hadoop/t-commit/hadoop-mapreduce-project/src/contrib/build-contrib.xml:227: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds
              [javac] Compiling 20 source files to /Users/szetszwo/hadoop/t-commit/hadoop-mapreduce-project/build/contrib/raid/test
              [javac] /Users/szetszwo/hadoop/t-commit/hadoop-mapreduce-project/src/contrib/raid/src/test/org/apache/hadoop/raid/TestBlockFixer.java:42: cannot find symbol
              [javac] symbol  : class MiniMRCluster
              [javac] location: package org.apache.hadoop.mapred
              [javac] import org.apache.hadoop.mapred.MiniMRCluster;
              [javac]                                ^
              ...
              [javac] Note: Some input files use or override a deprecated API.
              [javac] Note: Recompile with -Xlint:deprecation for details.
              [javac] 12 errors
          
          Show
          Tsz Wo Nicholas Sze added a comment - Raid still cannot be compiled after the patch: compile-test: [echo] contrib: raid [javac] /Users/szetszwo/hadoop/t-commit/hadoop-mapreduce-project/src/contrib/build-contrib.xml:227: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds [javac] Compiling 20 source files to /Users/szetszwo/hadoop/t-commit/hadoop-mapreduce-project/build/contrib/raid/test [javac] /Users/szetszwo/hadoop/t-commit/hadoop-mapreduce-project/src/contrib/raid/src/test/org/apache/hadoop/raid/TestBlockFixer.java:42: cannot find symbol [javac] symbol : class MiniMRCluster [javac] location: package org.apache.hadoop.mapred [javac] import org.apache.hadoop.mapred.MiniMRCluster; [javac] ^ ... [javac] Note: Some input files use or override a deprecated API. [javac] Note: Recompile with -Xlint:deprecation for details. [javac] 12 errors
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Devaraj, thanks for posting a patch. Could you also fix the errors in the tests?

          Show
          Tsz Wo Nicholas Sze added a comment - Devaraj, thanks for posting a patch. Could you also fix the errors in the tests?
          Tsz Wo Nicholas Sze made changes -
          Assignee Devaraj K [ devaraj.k ]
          Hide
          Devaraj K added a comment -

          Thanks Nicholas for looking into the patch.

          I am not getting any errors after applying the patch in my env.

          compile-test:
               [echo] contrib: raid
              [javac] D:\opensource\hadoop-trunk\hadoop-mapreduce-project\src\contrib\build-contrib.xml:227: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds
          
          init:
          
          ivy-download:
          

          As per the error, it is giving MiniMRCluster cannot find.

          Can you please check any jar is missing or am I doing anything wrong?

          Show
          Devaraj K added a comment - Thanks Nicholas for looking into the patch. I am not getting any errors after applying the patch in my env. compile-test: [echo] contrib: raid [javac] D:\opensource\hadoop-trunk\hadoop-mapreduce-project\src\contrib\build-contrib.xml:227: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds init: ivy-download: As per the error, it is giving MiniMRCluster cannot find. Can you please check any jar is missing or am I doing anything wrong?
          Hide
          Tsz Wo Nicholas Sze added a comment -

          The ant command did not work when running in hadoop-mapreduce-project/src/contrib/raid/ but it worked when running it in hadoop-mapreduce-project/ . Thanks for the hint, Devaraj.

          +1 patch looks good.

          Show
          Tsz Wo Nicholas Sze added a comment - The ant command did not work when running in hadoop-mapreduce-project/src/contrib/raid/ but it worked when running it in hadoop-mapreduce-project/ . Thanks for the hint, Devaraj. +1 patch looks good.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          I have committed this. Thanks, Devaraj!

          Show
          Tsz Wo Nicholas Sze added a comment - I have committed this. Thanks, Devaraj!
          Tsz Wo Nicholas Sze made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Fix Version/s 2.0.0 [ 12320354 ]
          Fix Version/s 3.0.0 [ 12320355 ]
          Resolution Fixed [ 1 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #2116 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2116/)
          MAPREDUCE-4057. Update RAID for the HA and fsdataset changes. Contributed by Devaraj K (Revision 1311959)

          Result = SUCCESS
          szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1311959
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/raid/src/java/org/apache/hadoop/hdfs/server/datanode/RaidBlockSender.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/raid/src/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRaidUtil.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/raid/src/java/org/apache/hadoop/raid/BlockFixer.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2116 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2116/ ) MAPREDUCE-4057 . Update RAID for the HA and fsdataset changes. Contributed by Devaraj K (Revision 1311959) Result = SUCCESS szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1311959 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/raid/src/java/org/apache/hadoop/hdfs/server/datanode/RaidBlockSender.java /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/raid/src/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRaidUtil.java /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/raid/src/java/org/apache/hadoop/raid/BlockFixer.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #2042 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2042/)
          MAPREDUCE-4057. Update RAID for the HA and fsdataset changes. Contributed by Devaraj K (Revision 1311959)

          Result = SUCCESS
          szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1311959
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/raid/src/java/org/apache/hadoop/hdfs/server/datanode/RaidBlockSender.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/raid/src/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRaidUtil.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/raid/src/java/org/apache/hadoop/raid/BlockFixer.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2042 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2042/ ) MAPREDUCE-4057 . Update RAID for the HA and fsdataset changes. Contributed by Devaraj K (Revision 1311959) Result = SUCCESS szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1311959 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/raid/src/java/org/apache/hadoop/hdfs/server/datanode/RaidBlockSender.java /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/raid/src/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRaidUtil.java /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/raid/src/java/org/apache/hadoop/raid/BlockFixer.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #2055 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2055/)
          MAPREDUCE-4057. Update RAID for the HA and fsdataset changes. Contributed by Devaraj K (Revision 1311959)

          Result = ABORTED
          szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1311959
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/raid/src/java/org/apache/hadoop/hdfs/server/datanode/RaidBlockSender.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/raid/src/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRaidUtil.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/raid/src/java/org/apache/hadoop/raid/BlockFixer.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2055 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2055/ ) MAPREDUCE-4057 . Update RAID for the HA and fsdataset changes. Contributed by Devaraj K (Revision 1311959) Result = ABORTED szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1311959 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/raid/src/java/org/apache/hadoop/hdfs/server/datanode/RaidBlockSender.java /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/raid/src/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRaidUtil.java /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/raid/src/java/org/apache/hadoop/raid/BlockFixer.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1011 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1011/)
          MAPREDUCE-4057. Update RAID for the HA and fsdataset changes. Contributed by Devaraj K (Revision 1311959)

          Result = FAILURE
          szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1311959
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/raid/src/java/org/apache/hadoop/hdfs/server/datanode/RaidBlockSender.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/raid/src/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRaidUtil.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/raid/src/java/org/apache/hadoop/raid/BlockFixer.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1011 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1011/ ) MAPREDUCE-4057 . Update RAID for the HA and fsdataset changes. Contributed by Devaraj K (Revision 1311959) Result = FAILURE szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1311959 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/raid/src/java/org/apache/hadoop/hdfs/server/datanode/RaidBlockSender.java /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/raid/src/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRaidUtil.java /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/raid/src/java/org/apache/hadoop/raid/BlockFixer.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1046 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1046/)
          MAPREDUCE-4057. Update RAID for the HA and fsdataset changes. Contributed by Devaraj K (Revision 1311959)

          Result = SUCCESS
          szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1311959
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/raid/src/java/org/apache/hadoop/hdfs/server/datanode/RaidBlockSender.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/raid/src/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRaidUtil.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/raid/src/java/org/apache/hadoop/raid/BlockFixer.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1046 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1046/ ) MAPREDUCE-4057 . Update RAID for the HA and fsdataset changes. Contributed by Devaraj K (Revision 1311959) Result = SUCCESS szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1311959 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/raid/src/java/org/apache/hadoop/hdfs/server/datanode/RaidBlockSender.java /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/raid/src/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRaidUtil.java /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/raid/src/java/org/apache/hadoop/raid/BlockFixer.java
          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Allen Wittenauer made changes -
          Fix Version/s 3.0.0 [ 12320355 ]
          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Patch Available Patch Available
          16d 18h 45m 1 Devaraj K 08/Apr/12 17:49
          Patch Available Patch Available Resolved Resolved
          2d 3h 1m 1 Tsz Wo Nicholas Sze 10/Apr/12 20:50
          Resolved Resolved Closed Closed
          43d 37m 1 Arun C Murthy 23/May/12 21:28

            People

            • Assignee:
              Devaraj K
              Reporter:
              Tsz Wo Nicholas Sze
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development