Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.22.0
    • Fix Version/s: 0.23.0
    • Component/s: contrib/raid
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      It would be good to have a way to centralize all the recovery logs, since recovery can be executed by any hdfs client. The best place to store this information is HDFS itself.

      1. MAPREDUCE-1706.txt
        8 kB
        Scott Chen
      2. test.result
        50 kB
        Scott Chen
      3. MAPREDUCE-1706.rebased.txt
        8 kB
        Scott Chen
      4. test.result
        50 kB
        Scott Chen

        Activity

        Hide
        Ramkumar Vadali added a comment -

        This looks good to me.
        +1

        Show
        Ramkumar Vadali added a comment - This looks good to me. +1
        Hide
        Scott Chen added a comment -

        [exec]
        [exec] -1 overall.
        [exec]
        [exec] +1 @author. The patch does not contain any @author tags.
        [exec]
        [exec] +1 tests included. The patch appears to include 4 new or modified tests.
        [exec]
        [exec] +1 javadoc. The javadoc tool did not generate any warning messages.
        [exec]
        [exec] -1 javac. The patch appears to cause tar ant target to fail.
        [exec]
        [exec] -1 findbugs. The patch appears to introduce 13 new Findbugs warnings.
        [exec]
        [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
        [exec]
        [exec] -1 system test framework. The patch failed system test framework compile.
        [exec]
        [exec]
        [exec]
        [exec]
        [exec] ======================================================================
        [exec] ======================================================================
        [exec] Finished build.
        [exec] ======================================================================
        [exec] ======================================================================
        [exec]
        [exec]

        BUILD FAILED
        /data/users/schen/hadoop-mapreduce-trunk/build.xml:1749: exec returned: 3

        Show
        Scott Chen added a comment - [exec] [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 4 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] -1 javac. The patch appears to cause tar ant target to fail. [exec] [exec] -1 findbugs. The patch appears to introduce 13 new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. [exec] [exec] -1 system test framework. The patch failed system test framework compile. [exec] [exec] [exec] [exec] [exec] ====================================================================== [exec] ====================================================================== [exec] Finished build. [exec] ====================================================================== [exec] ====================================================================== [exec] [exec] BUILD FAILED /data/users/schen/hadoop-mapreduce-trunk/build.xml:1749: exec returned: 3
        Hide
        Scott Chen added a comment -

        Attached the test results in file.

              [junit] Running org.apache.hadoop.mapred.TestControlledMapReduceJob
              [junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 0 sec
        

        The failed test is expected (see MAPREDUCE-2121).

        Show
        Scott Chen added a comment - Attached the test results in file. [junit] Running org.apache.hadoop.mapred.TestControlledMapReduceJob [junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 0 sec The failed test is expected (see MAPREDUCE-2121 ).
        Hide
        dhruba borthakur added a comment -

        The current patch does not merge with latest trunk

        Show
        dhruba borthakur added a comment - The current patch does not merge with latest trunk
        Hide
        Scott Chen added a comment -

        Rebased the patch and resolve the conflict.

        Show
        Scott Chen added a comment - Rebased the patch and resolve the conflict.
        Hide
        dhruba borthakur added a comment -

        Compilation errors

        
            [javac] /mnt/vol/devrs004.snc1/dhruba/commitmap/src/contrib/raid/src/test/org/apache/hadoop/hdfs/TestRaidDfs.java:469: cannot find symbol
            [javac] symbol  : class BufferedReader
            [javac] location: class org.apache.hadoop.hdfs.TestRaidDfs
            [javac]       BufferedReader reader = new BufferedReader(new InputStreamReader(stm));
            [javac]       ^
            [javac] /mnt/vol/devrs004.snc1/dhruba/commitmap/src/contrib/raid/src/test/org/apache/hadoop/hdfs/TestRaidDfs.java:469: cannot find symbol
            [javac] symbol  : class BufferedReader
            [javac] location: class org.apache.hadoop.hdfs.TestRaidDfs
            [javac]       BufferedReader reader = new BufferedReader(new InputStreamReader(stm));
            [javac]                                   ^
            [javac] /mnt/vol/devrs004.snc1/dhruba/commitmap/src/contrib/raid/src/test/org/apache/hadoop/hdfs/TestRaidDfs.java:469: cannot find symbol
            [javac] symbol  : class InputStreamReader
            [javac] location: class org.apache.hadoop.hdfs.TestRaidDfs
            [javac]       BufferedReader reader = new BufferedReader(new InputStreamReader(stm));
            [javac]                                                      ^
            [javac] /mnt/vol/devrs004.snc1/dhruba/commitmap/src/contrib/raid/src/test/org/apache/hadoop/hdfs/TestRaidDfs.java:473: cannot find symbol
            [javac] symbol  : variable Pattern
            [javac] location: class org.apache.hadoop.hdfs.TestRaidDfs
            [javac]       assertTrue(Pattern.matches("Alternate path : .*/destraid",
        
        
        Show
        dhruba borthakur added a comment - Compilation errors [javac] /mnt/vol/devrs004.snc1/dhruba/commitmap/src/contrib/raid/src/test/org/apache/hadoop/hdfs/TestRaidDfs.java:469: cannot find symbol [javac] symbol : class BufferedReader [javac] location: class org.apache.hadoop.hdfs.TestRaidDfs [javac] BufferedReader reader = new BufferedReader( new InputStreamReader(stm)); [javac] ^ [javac] /mnt/vol/devrs004.snc1/dhruba/commitmap/src/contrib/raid/src/test/org/apache/hadoop/hdfs/TestRaidDfs.java:469: cannot find symbol [javac] symbol : class BufferedReader [javac] location: class org.apache.hadoop.hdfs.TestRaidDfs [javac] BufferedReader reader = new BufferedReader( new InputStreamReader(stm)); [javac] ^ [javac] /mnt/vol/devrs004.snc1/dhruba/commitmap/src/contrib/raid/src/test/org/apache/hadoop/hdfs/TestRaidDfs.java:469: cannot find symbol [javac] symbol : class InputStreamReader [javac] location: class org.apache.hadoop.hdfs.TestRaidDfs [javac] BufferedReader reader = new BufferedReader( new InputStreamReader(stm)); [javac] ^ [javac] /mnt/vol/devrs004.snc1/dhruba/commitmap/src/contrib/raid/src/test/org/apache/hadoop/hdfs/TestRaidDfs.java:473: cannot find symbol [javac] symbol : variable Pattern [javac] location: class org.apache.hadoop.hdfs.TestRaidDfs [javac] assertTrue(Pattern.matches( "Alternate path : .*/destraid" ,
        Hide
        Scott Chen added a comment -

        Sorry for the mistake. I will update the patch soon.

        Show
        Scott Chen added a comment - Sorry for the mistake. I will update the patch soon.
        Hide
        Scott Chen added a comment -

        I updated the patch. Will post the test-patch and test result again.

        Show
        Scott Chen added a comment - I updated the patch. Will post the test-patch and test result again.
        Hide
        Scott Chen added a comment -

        [exec]
        [exec] -1 overall.
        [exec]
        [exec] +1 @author. The patch does not contain any @author tags.
        [exec]
        [exec] +1 tests included. The patch appears to i
        [exec] BUILD FAILED
        [exec] /data/users/schen/hadoop-mapreduce-trunk/src/test/aop/build/aop.xml:233: The following error occurred while executnclude 4 new or modified tests.
        [exec]
        [exec] +1 javadoc. The javadoc tool did not generate any warning messages.
        [exec]
        [exec] +1 javac. The aping this line:
        [exec] /data/users/schen/hadoop-mapreduce-trunk/src/test/aop/build/aop.xml:214: The following error occurred while execuplied patch does not increase the total number of javac compiler warnings.
        [exec]
        [exec] +1 findbugs. The patch does not introduce any nting this line:
        [exec] /data/users/schen/hadoop-mapreduce-trunk/src/test/aop/build/aop.xml:90: compile errors: 6
        [exec]
        [exec] Total time: 1 minute ew Findbugs (version 1.3.9) warnings.
        [exec]
        [exec] +1 release audit. The applied patch does not increase the total number of release au54 seconds
        [exec] dit warnings.
        [exec]
        [exec] -1 system test framework. The patch failed system test framework compile.
        [exec]
        [exec]
        [exec]
        [exec]
        [exec] ======================================================================
        [exec] ======================================================================
        [exec] Finished build.
        [exec] ======================================================================
        [exec] ======================================================================
        [exec]
        [exec]

        Show
        Scott Chen added a comment - [exec] [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to i [exec] BUILD FAILED [exec] /data/users/schen/hadoop-mapreduce-trunk/src/test/aop/build/aop.xml:233: The following error occurred while executnclude 4 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The aping this line: [exec] /data/users/schen/hadoop-mapreduce-trunk/src/test/aop/build/aop.xml:214: The following error occurred while execuplied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any nting this line: [exec] /data/users/schen/hadoop-mapreduce-trunk/src/test/aop/build/aop.xml:90: compile errors: 6 [exec] [exec] Total time: 1 minute ew Findbugs (version 1.3.9) warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release au54 seconds [exec] dit warnings. [exec] [exec] -1 system test framework. The patch failed system test framework compile. [exec] [exec] [exec] [exec] [exec] ====================================================================== [exec] ====================================================================== [exec] Finished build. [exec] ====================================================================== [exec] ====================================================================== [exec] [exec]
        Hide
        Scott Chen added a comment -

        Attached the test result again.
        The failed test is expected (see MAPREDUCE-2121).

            [junit] Running org.apache.hadoop.mapred.TestControlledMapReduceJob
            [junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 0 sec 
        
        Show
        Scott Chen added a comment - Attached the test result again. The failed test is expected (see MAPREDUCE-2121 ). [junit] Running org.apache.hadoop.mapred.TestControlledMapReduceJob [junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 0 sec
        Hide
        Scott Chen added a comment -

        I have committed this. Thanks Ram for the review.

        Show
        Scott Chen added a comment - I have committed this. Thanks Ram for the review.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #599 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/599/)
        MAPREDUCE-1706. Log RAID recoveries on HDFS. (schen)

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #599 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/599/ ) MAPREDUCE-1706 . Log RAID recoveries on HDFS. (schen)
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #643 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/643/)

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #643 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/643/ )

          People

          • Assignee:
            Scott Chen
            Reporter:
            Rodrigo Schmidt
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development