Details

    • Type: Test
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 0.22.0
    • Fix Version/s: 3.0.0-alpha1
    • Component/s: examples
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Tags:
      unit tests examples

      Description

      Looking to add some more examples such as Mean, Median, and Standard Deviation to the examples.
      I have some generic JUnit testcases as well.

      1. MAPREDUCE-2669.patch
        31 kB
        Konstantin Shvachko
      2. MAPREDUCE-2669.patch
        31 kB
        Plamen Jeliazkov
      3. MAPREDUCE-2669.patch
        149 kB
        Plamen Jeliazkov
      4. MAPREDUCE-2669.patch
        147 kB
        Plamen Jeliazkov
      5. MAPREDUCE-2669.patch
        5.48 MB
        Plamen Jeliazkov
      6. MAPREDUCE-2669.patch
        39 kB
        Plamen Jeliazkov
      7. MAPREDUCE-2669.patch
        40 kB
        Plamen Jeliazkov
      8. MAPREDUCE-2669.patch
        41 kB
        Plamen Jeliazkov
      9. MAPREDUCE-2669.patch
        39 kB
        Plamen Jeliazkov
      10. mapreduce-new-examples-0.22.patch
        37 kB
        Plamen Jeliazkov

        Issue Links

          Activity

          Hide
          zero45 Plamen Jeliazkov added a comment -

          Posting them up for review for now.

          Show
          zero45 Plamen Jeliazkov added a comment - Posting them up for review for now.
          Hide
          qwertymaniac Harsh J added a comment -

          Hey Plamen,

          Could you post a review on http://reviews.apache.org? Just to note, a single patch that incorporates all changes is more preferable.

          Thanks!

          Show
          qwertymaniac Harsh J added a comment - Hey Plamen, Could you post a review on http://reviews.apache.org? Just to note, a single patch that incorporates all changes is more preferable. Thanks!
          Hide
          zero45 Plamen Jeliazkov added a comment -

          Went ahead and did just that – thanks Harsh. I will post a diff here of a slight revision and then continue to update via reviews.apache.org.

          Show
          zero45 Plamen Jeliazkov added a comment - Went ahead and did just that – thanks Harsh. I will post a diff here of a slight revision and then continue to update via reviews.apache.org.
          Hide
          zero45 Plamen Jeliazkov added a comment -

          The patch for review.

          Show
          zero45 Plamen Jeliazkov added a comment - The patch for review.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12486343/mapreduce-new-examples-0.22.patch
          against trunk revision 1146517.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 9 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          -1 javac. The applied patch generated 2254 javac compiler warnings (more than the trunk's current 2244 warnings).

          -1 findbugs. The patch appears to introduce 15 new Findbugs (version 1.3.9) warnings.

          -1 release audit. The applied patch generated 5 release audit warnings (more than the trunk's current 2 warnings).

          -1 core tests. The patch failed these core unit tests:
          org.apache.hadoop.cli.TestMRCLI
          org.apache.hadoop.fs.TestFileSystem

          -1 contrib tests. The patch failed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/468//testReport/
          Release audit warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/468//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/468//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/468//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12486343/mapreduce-new-examples-0.22.patch against trunk revision 1146517. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 9 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 2254 javac compiler warnings (more than the trunk's current 2244 warnings). -1 findbugs. The patch appears to introduce 15 new Findbugs (version 1.3.9) warnings. -1 release audit. The applied patch generated 5 release audit warnings (more than the trunk's current 2 warnings). -1 core tests. The patch failed these core unit tests: org.apache.hadoop.cli.TestMRCLI org.apache.hadoop.fs.TestFileSystem -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/468//testReport/ Release audit warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/468//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/468//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/468//console This message is automatically generated.
          Hide
          zero45 Plamen Jeliazkov added a comment -

          Making some changes for the QA bot.

          Show
          zero45 Plamen Jeliazkov added a comment - Making some changes for the QA bot.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12486531/MAPREDUCE-2669.patch
          against trunk revision 1146517.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 9 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          -1 javac. The applied patch generated 2254 javac compiler warnings (more than the trunk's current 2244 warnings).

          -1 findbugs. The patch appears to introduce 9 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these core unit tests:
          org.apache.hadoop.cli.TestMRCLI
          org.apache.hadoop.fs.TestFileSystem

          -1 contrib tests. The patch failed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/470//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/470//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/470//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12486531/MAPREDUCE-2669.patch against trunk revision 1146517. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 9 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 2254 javac compiler warnings (more than the trunk's current 2244 warnings). -1 findbugs. The patch appears to introduce 9 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.cli.TestMRCLI org.apache.hadoop.fs.TestFileSystem -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/470//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/470//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/470//console This message is automatically generated.
          Hide
          zero45 Plamen Jeliazkov added a comment -

          Another patch for review. I couldn't access those logs from the QA bot. :S. Hopefully this will remove some of those FindBug issues that it is reporting though.

          Show
          zero45 Plamen Jeliazkov added a comment - Another patch for review. I couldn't access those logs from the QA bot. :S. Hopefully this will remove some of those FindBug issues that it is reporting though.
          Hide
          zero45 Plamen Jeliazkov added a comment -

          Fixed some more FindBug issues.

          Show
          zero45 Plamen Jeliazkov added a comment - Fixed some more FindBug issues.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12486662/MAPREDUCE-2669.patch
          against trunk revision 1146517.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 9 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 9 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these core unit tests:
          org.apache.hadoop.cli.TestMRCLI
          org.apache.hadoop.fs.TestFileSystem

          -1 contrib tests. The patch failed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/471//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/471//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/471//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12486662/MAPREDUCE-2669.patch against trunk revision 1146517. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 9 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 9 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.cli.TestMRCLI org.apache.hadoop.fs.TestFileSystem -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/471//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/471//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/471//console This message is automatically generated.
          Hide
          zero45 Plamen Jeliazkov added a comment -

          More fixes.

          Show
          zero45 Plamen Jeliazkov added a comment - More fixes.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12486701/MAPREDUCE-2669.patch
          against trunk revision 1146517.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 9 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these core unit tests:
          org.apache.hadoop.cli.TestMRCLI
          org.apache.hadoop.fs.TestFileSystem

          -1 contrib tests. The patch failed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/473//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/473//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/473//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12486701/MAPREDUCE-2669.patch against trunk revision 1146517. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 9 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.cli.TestMRCLI org.apache.hadoop.fs.TestFileSystem -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/473//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/473//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/473//console This message is automatically generated.
          Hide
          devaraj.k Devaraj K added a comment -

          It is good to see these examples.

          There are few comments with this patch.

          1. Use indentation two spaces per level, not four.
          2. Before submitting the patch you can test the patch using the test-patch target to avoid these -1's from the Hadoop QA like find bug issues.

          You can refer here for more details: http://wiki.apache.org/hadoop/HowToContribute

          3. Streams can be closed properly by considering the exceptional cases also. Refer discussion in the defect MAPREDUCE-2243.
          4. Use some meaningful name for the test names instead of test.
          5. Move these test classes from source code to the src/test/mapred.

          You can file this in the review board because it is having good number of changes and also can be reviewed thoroughly by reviewers.

          Show
          devaraj.k Devaraj K added a comment - It is good to see these examples. There are few comments with this patch. 1. Use indentation two spaces per level, not four. 2. Before submitting the patch you can test the patch using the test-patch target to avoid these -1's from the Hadoop QA like find bug issues. You can refer here for more details: http://wiki.apache.org/hadoop/HowToContribute 3. Streams can be closed properly by considering the exceptional cases also. Refer discussion in the defect MAPREDUCE-2243 . 4. Use some meaningful name for the test names instead of test. 5. Move these test classes from source code to the src/test/mapred. You can file this in the review board because it is having good number of changes and also can be reviewed thoroughly by reviewers.
          Hide
          zero45 Plamen Jeliazkov added a comment -

          Thank you, Devaraj! Yes I have been filing it on the review board; I have been uploading the .patchs here as well as on the review board. I will add your comments to the patch and upload again soon.

          Show
          zero45 Plamen Jeliazkov added a comment - Thank you, Devaraj! Yes I have been filing it on the review board; I have been uploading the .patchs here as well as on the review board. I will add your comments to the patch and upload again soon.
          Hide
          zero45 Plamen Jeliazkov added a comment -

          The reason for the 5MB patch is that it includes a sample text file for the JUnit tests to use.

          I have done applied the patch myself and it appears to be working correctly.

          I don't know why the core tests are failing, or the contrib tests, but after looking them over twice now I am pretty sure I can conclude that they were present prior to my patch.

          In any case, here is the latest patch!

          Show
          zero45 Plamen Jeliazkov added a comment - The reason for the 5MB patch is that it includes a sample text file for the JUnit tests to use. I have done applied the patch myself and it appears to be working correctly. I don't know why the core tests are failing, or the contrib tests, but after looking them over twice now I am pretty sure I can conclude that they were present prior to my patch. In any case, here is the latest patch!
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12486940/MAPREDUCE-2669.patch
          against trunk revision 1147981.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 12 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

          -1 release audit. The applied patch generated 3 release audit warnings (more than the trunk's current 2 warnings).

          -1 core tests. The patch failed these core unit tests:
          org.apache.hadoop.cli.TestMRCLI
          org.apache.hadoop.fs.TestFileSystem

          -1 contrib tests. The patch failed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/479//testReport/
          Release audit warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/479//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/479//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/479//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12486940/MAPREDUCE-2669.patch against trunk revision 1147981. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 12 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. -1 release audit. The applied patch generated 3 release audit warnings (more than the trunk's current 2 warnings). -1 core tests. The patch failed these core unit tests: org.apache.hadoop.cli.TestMRCLI org.apache.hadoop.fs.TestFileSystem -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/479//testReport/ Release audit warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/479//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/479//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/479//console This message is automatically generated.
          Hide
          zero45 Plamen Jeliazkov added a comment -

          Smaller test input file. Many reviewer changes added in. Much more QA bot worthy.

          Show
          zero45 Plamen Jeliazkov added a comment - Smaller test input file. Many reviewer changes added in. Much more QA bot worthy.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12487250/MAPREDUCE-2669.patch
          against trunk revision 1148421.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 12 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          -1 javac. The applied patch generated 2249 javac compiler warnings (more than the trunk's current 2244 warnings).

          -1 findbugs. The patch appears to introduce 4 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these core unit tests:
          org.apache.hadoop.cli.TestMRCLI
          org.apache.hadoop.fs.TestFileSystem

          -1 contrib tests. The patch failed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/486//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/486//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/486//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12487250/MAPREDUCE-2669.patch against trunk revision 1148421. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 12 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 2249 javac compiler warnings (more than the trunk's current 2244 warnings). -1 findbugs. The patch appears to introduce 4 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.cli.TestMRCLI org.apache.hadoop.fs.TestFileSystem -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/486//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/486//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/486//console This message is automatically generated.
          Hide
          zero45 Plamen Jeliazkov added a comment -

          Should fix the javac warnings and 3 of the FindBugs; I am not sure where the last one is coming from as it doesn't appear to mention any of the files I am updating.

          Show
          zero45 Plamen Jeliazkov added a comment - Should fix the javac warnings and 3 of the FindBugs; I am not sure where the last one is coming from as it doesn't appear to mention any of the files I am updating.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12487324/MAPREDUCE-2669.patch
          against trunk revision 1149004.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 12 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          -1 javac. The applied patch generated 2245 javac compiler warnings (more than the trunk's current 2244 warnings).

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these core unit tests:
          org.apache.hadoop.cli.TestMRCLI
          org.apache.hadoop.fs.TestFileSystem

          -1 contrib tests. The patch failed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/488//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/488//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/488//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12487324/MAPREDUCE-2669.patch against trunk revision 1149004. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 12 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 2245 javac compiler warnings (more than the trunk's current 2244 warnings). -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.cli.TestMRCLI org.apache.hadoop.fs.TestFileSystem -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/488//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/488//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/488//console This message is automatically generated.
          Hide
          shv Konstantin Shvachko added a comment -

          Good examples! Some comments:

          1. You need to address findbugs and javac warning if they are related to the patch.
          2. Test failures do not seem to be relates to the new examples.
          3. I recommend using forward slashes in INPUT and OUTPUT constants
            +  private final static String INPUT = "src/test/mapred/org/apache/hadoop/examples/input";
            +  private final static String OUTPUT = "src/test/mapred/org/apache/hadoop/examples/robust_stddev_output";
            
          4. For INPUT directory I'd propose to use just INPUT = "src/test/mapred/org/apache/hadoop/examples/.
            As much as I love Shakespeare, committing "A Midsummer Night's Dream" under Apache License doesn't seem to be required to exemplify MR statistics.
            Having said that, it was interesting to learn that Shakespeare used pretty short words - around 4.42 on average - compared to modern texts from the Web.
          5. OUTPUT directory should point to a directory in build/data, so that it is cleaned up with the clean target.
          6. The tests should be converted to JUnit 4 format.
          7. Also it is better to combine all three test under common roof TestWordStats or similar. It will be less code and will save test execution time.
          Show
          shv Konstantin Shvachko added a comment - Good examples! Some comments: You need to address findbugs and javac warning if they are related to the patch. Test failures do not seem to be relates to the new examples. I recommend using forward slashes in INPUT and OUTPUT constants + private final static String INPUT = "src/test/mapred/org/apache/hadoop/examples/input" ; + private final static String OUTPUT = "src/test/mapred/org/apache/hadoop/examples/robust_stddev_output" ; For INPUT directory I'd propose to use just INPUT = "src/test/mapred/org/apache/hadoop/examples/ . As much as I love Shakespeare, committing "A Midsummer Night's Dream" under Apache License doesn't seem to be required to exemplify MR statistics. Having said that, it was interesting to learn that Shakespeare used pretty short words - around 4.42 on average - compared to modern texts from the Web. OUTPUT directory should point to a directory in build/data , so that it is cleaned up with the clean target. The tests should be converted to JUnit 4 format. Also it is better to combine all three test under common roof TestWordStats or similar. It will be less code and will save test execution time.
          Hide
          zero45 Plamen Jeliazkov added a comment -

          Patch based on Konstantin's suggestions. Removed input file; cannot find any FindBugs, managed to find a deprecation javac warning though.

          Also reduced the testcases to 1 file; JUnit4.

          Show
          zero45 Plamen Jeliazkov added a comment - Patch based on Konstantin's suggestions. Removed input file; cannot find any FindBugs, managed to find a deprecation javac warning though. Also reduced the testcases to 1 file; JUnit4.
          Hide
          shv Konstantin Shvachko added a comment -

          Updated patch to the new project structure.

          Show
          shv Konstantin Shvachko added a comment - Updated patch to the new project structure.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12492659/MAPREDUCE-2669.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 8 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in ..

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/586//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/586//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/586//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-api.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/586//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/586//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/586//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-nodemanager.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/586//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/586//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/586//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/586//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-hs.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/586//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-shuffle.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/586//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-jobclient.html
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/586//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12492659/MAPREDUCE-2669.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 4 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 8 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in .. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/586//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/586//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/586//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-api.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/586//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/586//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/586//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-nodemanager.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/586//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/586//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/586//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/586//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-hs.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/586//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-shuffle.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/586//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-jobclient.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/586//console This message is automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12492659/MAPREDUCE-2669.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/708//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/708//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-nodemanager.html
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/708//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12492659/MAPREDUCE-2669.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 4 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/708//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/708//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-nodemanager.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/708//console This message is automatically generated.
          Hide
          shv Konstantin Shvachko added a comment -

          Findbug warnings are related to yarn rather than the patch.

          Show
          shv Konstantin Shvachko added a comment - Findbug warnings are related to yarn rather than the patch.
          Hide
          shv Konstantin Shvachko added a comment -

          I just committed this. Thank you Plamen.

          Show
          shv Konstantin Shvachko added a comment - I just committed this. Thank you Plamen.
          Hide
          hudson Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #873 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/873/)
          MAPREDUCE-2669. Add new examples for Mean, Median, and Standard Deviation. Contributed by Plamen Jeliazkov.

          shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1169874
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/src/examples/org/apache/hadoop/examples/WordMean.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/src/examples/org/apache/hadoop/examples/WordMedian.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/src/examples/org/apache/hadoop/examples/WordStandardDeviation.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/examples/TestWordStats.java
          Show
          hudson Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #873 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/873/ ) MAPREDUCE-2669 . Add new examples for Mean, Median, and Standard Deviation. Contributed by Plamen Jeliazkov. shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1169874 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/src/examples/org/apache/hadoop/examples/WordMean.java /hadoop/common/trunk/hadoop-mapreduce-project/src/examples/org/apache/hadoop/examples/WordMedian.java /hadoop/common/trunk/hadoop-mapreduce-project/src/examples/org/apache/hadoop/examples/WordStandardDeviation.java /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/examples/TestWordStats.java
          Hide
          hudson Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #950 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/950/)
          MAPREDUCE-2669. Add new examples for Mean, Median, and Standard Deviation. Contributed by Plamen Jeliazkov.

          shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1169874
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/src/examples/org/apache/hadoop/examples/WordMean.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/src/examples/org/apache/hadoop/examples/WordMedian.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/src/examples/org/apache/hadoop/examples/WordStandardDeviation.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/examples/TestWordStats.java
          Show
          hudson Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #950 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/950/ ) MAPREDUCE-2669 . Add new examples for Mean, Median, and Standard Deviation. Contributed by Plamen Jeliazkov. shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1169874 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/src/examples/org/apache/hadoop/examples/WordMean.java /hadoop/common/trunk/hadoop-mapreduce-project/src/examples/org/apache/hadoop/examples/WordMedian.java /hadoop/common/trunk/hadoop-mapreduce-project/src/examples/org/apache/hadoop/examples/WordStandardDeviation.java /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/examples/TestWordStats.java
          Hide
          hudson Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #885 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/885/)
          MAPREDUCE-2669. Add new examples for Mean, Median, and Standard Deviation. Contributed by Plamen Jeliazkov.

          shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1169874
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/src/examples/org/apache/hadoop/examples/WordMean.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/src/examples/org/apache/hadoop/examples/WordMedian.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/src/examples/org/apache/hadoop/examples/WordStandardDeviation.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/examples/TestWordStats.java
          Show
          hudson Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #885 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/885/ ) MAPREDUCE-2669 . Add new examples for Mean, Median, and Standard Deviation. Contributed by Plamen Jeliazkov. shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1169874 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/src/examples/org/apache/hadoop/examples/WordMean.java /hadoop/common/trunk/hadoop-mapreduce-project/src/examples/org/apache/hadoop/examples/WordMedian.java /hadoop/common/trunk/hadoop-mapreduce-project/src/examples/org/apache/hadoop/examples/WordStandardDeviation.java /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/examples/TestWordStats.java
          Hide
          hudson Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #816 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/816/)
          MAPREDUCE-2669. Add new examples for Mean, Median, and Standard Deviation. Contributed by Plamen Jeliazkov.

          shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1169874
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/src/examples/org/apache/hadoop/examples/WordMean.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/src/examples/org/apache/hadoop/examples/WordMedian.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/src/examples/org/apache/hadoop/examples/WordStandardDeviation.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/examples/TestWordStats.java
          Show
          hudson Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #816 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/816/ ) MAPREDUCE-2669 . Add new examples for Mean, Median, and Standard Deviation. Contributed by Plamen Jeliazkov. shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1169874 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/src/examples/org/apache/hadoop/examples/WordMean.java /hadoop/common/trunk/hadoop-mapreduce-project/src/examples/org/apache/hadoop/examples/WordMedian.java /hadoop/common/trunk/hadoop-mapreduce-project/src/examples/org/apache/hadoop/examples/WordStandardDeviation.java /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/examples/TestWordStats.java
          Hide
          hudson Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #792 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/792/)
          MAPREDUCE-2669. Add new examples for Mean, Median, and Standard Deviation. Contributed by Plamen Jeliazkov.

          shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1169874
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/src/examples/org/apache/hadoop/examples/WordMean.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/src/examples/org/apache/hadoop/examples/WordMedian.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/src/examples/org/apache/hadoop/examples/WordStandardDeviation.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/examples/TestWordStats.java
          Show
          hudson Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #792 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/792/ ) MAPREDUCE-2669 . Add new examples for Mean, Median, and Standard Deviation. Contributed by Plamen Jeliazkov. shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1169874 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/src/examples/org/apache/hadoop/examples/WordMean.java /hadoop/common/trunk/hadoop-mapreduce-project/src/examples/org/apache/hadoop/examples/WordMedian.java /hadoop/common/trunk/hadoop-mapreduce-project/src/examples/org/apache/hadoop/examples/WordStandardDeviation.java /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/examples/TestWordStats.java

            People

            • Assignee:
              zero45 Plamen Jeliazkov
              Reporter:
              zero45 Plamen Jeliazkov
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - 48h
                48h
                Remaining:
                Remaining Estimate - 48h
                48h
                Logged:
                Time Spent - Not Specified
                Not Specified

                  Development