Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.22.0
    • Fix Version/s: 0.22.0
    • Component/s: contrib/raid
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/490/consoleFull

      Mapreduce trunk compilation is broken with

      compile:
      [echo] contrib: raid
      [javac] Compiling 27 source files to /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk/trunk/build/contrib/raid/classes
      [javac] /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk/trunk/src/contrib/raid/src/java/org/apache/hadoop/hdfs/server/datanode/RaidBlockSender.java:71: cannot find symbol
      [javac] symbol : class BlockTransferThrottler
      [javac] location: class org.apache.hadoop.hdfs.server.datanode.RaidBlockSender
      [javac] private BlockTransferThrottler throttler;
      [javac] ^
      [javac] /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk/trunk/src/contrib/raid/src/java/org/apache/hadoop/hdfs/server/datanode/RaidBlockSender.java:377: cannot find symbol
      [javac] symbol : class BlockTransferThrottler
      [javac] location: class org.apache.hadoop.hdfs.server.datanode.RaidBlockSender
      [javac] BlockTransferThrottler throttler) throws IOException {
      [javac] ^
      [javac] Note: Some input files use or override a deprecated API.
      [javac] Note: Recompile with -Xlint:deprecation for details.
      [javac] Note: Some input files use unchecked or unsafe operations.
      [javac] Note: Recompile with -Xlint:unchecked for details.
      [javac] 2 errors

      1. MAPREDUCE-2179.patch
        2 kB
        Ramkumar Vadali

        Activity

        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Build #29 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/29/)
        MAPREDUCE-3137. Fix broken merge of MAPREDUCE-2179. Contributed by Hitesh Shah.

        acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1178524
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop/yarn/applications/distributedshell/TestDistributedShell.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #29 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/29/ ) MAPREDUCE-3137 . Fix broken merge of MAPREDUCE-2179 . Contributed by Hitesh Shah. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1178524 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop/yarn/applications/distributedshell/TestDistributedShell.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-0.23-Build #36 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/36/)
        MAPREDUCE-3137. Fix broken merge of MAPREDUCE-2179. Contributed by Hitesh Shah.

        acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1178524
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop/yarn/applications/distributedshell/TestDistributedShell.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Build #36 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/36/ ) MAPREDUCE-3137 . Fix broken merge of MAPREDUCE-2179 . Contributed by Hitesh Shah. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1178524 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop/yarn/applications/distributedshell/TestDistributedShell.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #643 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/643/)

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #643 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/643/ )
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #533 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/533/)
        MAPREDUCE-2179. Fix RaidBlockSender compilation failure. (Ramkumar Vadali via
        schen)

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #533 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/533/ ) MAPREDUCE-2179 . Fix RaidBlockSender compilation failure. (Ramkumar Vadali via schen)
        Hide
        Scott Chen added a comment -

        Nigel: Should we fix the OOM issue in another patch?

        Show
        Scott Chen added a comment - Nigel: Should we fix the OOM issue in another patch?
        Hide
        Scott Chen added a comment -

        Nigel: Sorry. I missed your comments.

        Show
        Scott Chen added a comment - Nigel: Sorry. I missed your comments.
        Hide
        Scott Chen added a comment -

        I just committed this. Thanks Ram.

        Show
        Scott Chen added a comment - I just committed this. Thanks Ram.
        Hide
        Nigel Daley added a comment -

        FWIW, I got OOM errors running the tests. Adding this fixed it:

        ===================================================================
        --- src/contrib/raid/build.xml  (revision 1032497)
        +++ src/contrib/raid/build.xml  (working copy)
        @@ -35,7 +35,7 @@
         
           <target name="test-junit" depends="compile,compile-test" if="test.available">
             <junit showoutput="${test.output}" fork="yes" printsummary="yes" errorProperty="tests.failed" 
        -           haltonfailure="no" failureProperty="tests.failed">
        +           haltonfailure="no" failureProperty="tests.failed" maxmemory="1024m" >
         
                 <classpath refid="test.classpath"/>
                 <sysproperty key="test.build.data" value="${build.test}/data"/>
        
        Show
        Nigel Daley added a comment - FWIW, I got OOM errors running the tests. Adding this fixed it: =================================================================== --- src/contrib/raid/build.xml (revision 1032497) +++ src/contrib/raid/build.xml (working copy) @@ -35,7 +35,7 @@ <target name= "test-junit" depends= "compile,compile-test" if = "test.available" > <junit showoutput= "${test.output}" fork= "yes" printsummary= "yes" errorProperty= "tests.failed" - haltonfailure= "no" failureProperty= "tests.failed" > + haltonfailure= "no" failureProperty= "tests.failed" maxmemory= "1024m" > <classpath refid= "test.classpath" /> <sysproperty key= "test.build.data" value= "${build.test}/data" />
        Hide
        Ramkumar Vadali added a comment -

        ant test-patch will not run since trunk compilation is broken.

        I have run raid unit-tests:

        
        test-junit:
            [junit] WARNING: multiple versions of ant detected in path for junit
            [junit]          jar:file:/home/rvadali/local/external/ant/lib/ant.jar!/org/apache/tools/ant/Project.class
            [junit]      and jar:file:/home/rvadali/.ivy2/cache/ant/ant/jars/ant-1.6.5.jar!/org/apache/tools/ant/Project.class
            [junit] Running org.apache.hadoop.hdfs.TestRaidDfs
            [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 41.64 sec
            [junit] Running org.apache.hadoop.raid.TestBlockFixer
            [junit] Tests run: 5, Failures: 0, Errors: 0, Time elapsed: 139.487 sec
            [junit] Running org.apache.hadoop.raid.TestDirectoryTraversal
            [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 9.169 sec
            [junit] Running org.apache.hadoop.raid.TestErasureCodes
            [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 26.334 sec
            [junit] Running org.apache.hadoop.raid.TestGaloisField
            [junit] Tests run: 7, Failures: 0, Errors: 0, Time elapsed: 0.399 sec
            [junit] Running org.apache.hadoop.raid.TestHarIndexParser
            [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.051 sec
            [junit] Running org.apache.hadoop.raid.TestRaidFilter
            [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 4.473 sec
            [junit] Running org.apache.hadoop.raid.TestRaidHar
            [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 70.16 sec
            [junit] Running org.apache.hadoop.raid.TestRaidNode
            [junit] Tests run: 4, Failures: 0, Errors: 0, Time elapsed: 406.19 sec
            [junit] Running org.apache.hadoop.raid.TestRaidPurge
            [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 107.977 sec
            [junit] Running org.apache.hadoop.raid.TestRaidShell
            [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 26.168 sec
        
        test:
        
        BUILD SUCCESSFUL
        Total time: 14 minutes 12 seconds
        
        
        Show
        Ramkumar Vadali added a comment - ant test-patch will not run since trunk compilation is broken. I have run raid unit-tests: test-junit: [junit] WARNING: multiple versions of ant detected in path for junit [junit] jar:file:/home/rvadali/local/external/ant/lib/ant.jar!/org/apache/tools/ant/Project.class [junit] and jar:file:/home/rvadali/.ivy2/cache/ant/ant/jars/ant-1.6.5.jar!/org/apache/tools/ant/Project.class [junit] Running org.apache.hadoop.hdfs.TestRaidDfs [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 41.64 sec [junit] Running org.apache.hadoop.raid.TestBlockFixer [junit] Tests run: 5, Failures: 0, Errors: 0, Time elapsed: 139.487 sec [junit] Running org.apache.hadoop.raid.TestDirectoryTraversal [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 9.169 sec [junit] Running org.apache.hadoop.raid.TestErasureCodes [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 26.334 sec [junit] Running org.apache.hadoop.raid.TestGaloisField [junit] Tests run: 7, Failures: 0, Errors: 0, Time elapsed: 0.399 sec [junit] Running org.apache.hadoop.raid.TestHarIndexParser [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.051 sec [junit] Running org.apache.hadoop.raid.TestRaidFilter [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 4.473 sec [junit] Running org.apache.hadoop.raid.TestRaidHar [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 70.16 sec [junit] Running org.apache.hadoop.raid.TestRaidNode [junit] Tests run: 4, Failures: 0, Errors: 0, Time elapsed: 406.19 sec [junit] Running org.apache.hadoop.raid.TestRaidPurge [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 107.977 sec [junit] Running org.apache.hadoop.raid.TestRaidShell [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 26.168 sec test: BUILD SUCCESSFUL Total time: 14 minutes 12 seconds
        Hide
        Scott Chen added a comment -

        +1
        The patch looks good to me.
        Ram: Can you post the test results here?

        Show
        Scott Chen added a comment - +1 The patch looks good to me. Ram: Can you post the test results here?
        Hide
        Scott Chen added a comment -

        Right now raid can be broken by HDFS because it is not using the public API of HDFS.
        We need to have HDFS-1461 and refactor RaidBlockSender to get rid of this dependency.

        Show
        Scott Chen added a comment - Right now raid can be broken by HDFS because it is not using the public API of HDFS. We need to have HDFS-1461 and refactor RaidBlockSender to get rid of this dependency.
        Hide
        Ramkumar Vadali added a comment -

        r1032836 (HDFS-1457) removed the class BlockTransferThrottler. The RAID code does not need that functionality, so this patch just removes the dependence on BlockTransferThrottler.

        Show
        Ramkumar Vadali added a comment - r1032836 ( HDFS-1457 ) removed the class BlockTransferThrottler. The RAID code does not need that functionality, so this patch just removes the dependence on BlockTransferThrottler.
        Hide
        Nigel Daley added a comment -

        Looks like HDFS-1457 removed BlockTransferThrottler.java: http://svn.apache.org/viewvc?view=revision&revision=1032836

        Show
        Nigel Daley added a comment - Looks like HDFS-1457 removed BlockTransferThrottler.java: http://svn.apache.org/viewvc?view=revision&revision=1032836

          People

          • Assignee:
            Ramkumar Vadali
            Reporter:
            Giridharan Kesavan
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development