Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.21.0
    • Fix Version/s: 0.21.0
    • Component/s: test
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Tags:
      herriot

      Description

      Herriot has remote cluster control API. The functionality works fairly well, however, JavaDocs are missed here and there. This has to be fixed.

      1. hadoop-6572.0.22.patch
        8 kB
        Konstantin Boudnik
      2. hadoop-6572.patch
        12 kB
        Konstantin Boudnik

        Issue Links

          Activity

          Konstantin Boudnik created issue -
          Konstantin Boudnik made changes -
          Field Original Value New Value
          Assignee Balaji Rajagopalan [ balajirg ]
          Hide
          Konstantin Boudnik added a comment -

          Initial patch sent in by Balaji

          Show
          Konstantin Boudnik added a comment - Initial patch sent in by Balaji
          Konstantin Boudnik made changes -
          Attachment hadoop-6572.patch [ 12443885 ]
          Hide
          Vinay Kumar Thota added a comment -

          Exception exclusion functionality is not working correctly because of that tests are failing by not matching the error count.
          I debugged the issue and found that the problem with shell command which is generating in the getNumberOfMatchesInLogFile function.
          Currently building the shell command in the following way.

          if(list != null){
            for(int i =0; i < list.length; ++i)
            {
              filePattern.append(" | grep -v " + list[i] );
            }
          }
              String[] cmd =
                  new String[] {
                      "bash",
                      "-c",
                      "grep -c "
                          + pattern + " " + filePattern
                          + " | awk -F: '{s+=$2} END {print s}'" };    
          

          The above commnad won't work correctly because you are counting the exceptions in the file before excluding the known exceptions.
          In this case it gives the mismatch error counts everytime.The shell command should be in the following way to work correctly.

          if (list != null) {
            int index = 0;
            for (String excludeExp : list) {
              filePattern.append((++index < list.length)? "| grep -v " : 
                      "| grep -vc " + list[i] );  
            }
          }
          String[] cmd =
             new String[] {
                 "bash",
                 "-c",
                 "grep "
                     + pattern + " " + filePattern
                     + " | awk -F: '{s+=$2} END {print s}'" };    
          
          Show
          Vinay Kumar Thota added a comment - Exception exclusion functionality is not working correctly because of that tests are failing by not matching the error count. I debugged the issue and found that the problem with shell command which is generating in the getNumberOfMatchesInLogFile function. Currently building the shell command in the following way. if(list != null){ for(int i =0; i < list.length; ++i) { filePattern.append(" | grep -v " + list[i] ); } } String[] cmd = new String[] { "bash", "-c", "grep -c " + pattern + " " + filePattern + " | awk -F: '{s+=$2} END {print s}'" }; The above commnad won't work correctly because you are counting the exceptions in the file before excluding the known exceptions. In this case it gives the mismatch error counts everytime.The shell command should be in the following way to work correctly. if (list != null) { int index = 0; for (String excludeExp : list) { filePattern.append((++index < list.length)? "| grep -v " : "| grep -vc " + list[i] ); } } String[] cmd = new String[] { "bash", "-c", "grep " + pattern + " " + filePattern + " | awk -F: '{s+=$2} END {print s}'" };
          Hide
          Konstantin Boudnik added a comment -

          I am changing the description of the JIRA because an issue in the core functionality was found.

          Show
          Konstantin Boudnik added a comment - I am changing the description of the JIRA because an issue in the core functionality was found.
          Konstantin Boudnik made changes -
          Summary Remote cluster control functionality needs some JavaDocs improvement Remote cluster control functionality needs JavaDocs improvement; exceptionList doesn't work property
          Hide
          Konstantin Boudnik added a comment -

          Reverting: the bug is unrelated to this JIRA

          Show
          Konstantin Boudnik added a comment - Reverting: the bug is unrelated to this JIRA
          Konstantin Boudnik made changes -
          Summary Remote cluster control functionality needs JavaDocs improvement; exceptionList doesn't work property Remote cluster control functionality needs JavaDocs improvement
          Hide
          Konstantin Boudnik added a comment -

          Vinay, I think this comment belongs to a different JIRA where the exception filtering has been done. If such a JIRA doesn't exist (somehow I can find it right now; was it committed anywhere yet?) Balaji should know where this functionality has been implemented initially and the comment (along with patch modification) clearly belongs there.

          Also, in your comment above you're altering filePattern which I believe contains the list of files to be grepped.

          Show
          Konstantin Boudnik added a comment - Vinay, I think this comment belongs to a different JIRA where the exception filtering has been done. If such a JIRA doesn't exist (somehow I can find it right now; was it committed anywhere yet?) Balaji should know where this functionality has been implemented initially and the comment (along with patch modification) clearly belongs there. Also, in your comment above you're altering filePattern which I believe contains the list of files to be grepped.
          Hide
          Konstantin Boudnik added a comment -

          Ok, I think I am confused. This JIRA is about adding JavaDocs to the public APIs of the code which is part of HADOOP-6332's patches. The issue Vinay has found is in that code. And needed to be tracked separately from this particular JIRA: we can't mix different issues in the same patch.

          Show
          Konstantin Boudnik added a comment - Ok, I think I am confused. This JIRA is about adding JavaDocs to the public APIs of the code which is part of HADOOP-6332 's patches. The issue Vinay has found is in that code. And needed to be tracked separately from this particular JIRA: we can't mix different issues in the same patch.
          Hide
          Balaji Rajagopalan added a comment -

          Please create a new JIRA to fix any issues with exception exclustion pattern. The original work was submitted as part of pushConfig against a yahoo internal bugzilla ticket, can't publish internal details here. I will use this ticket to address cos java docs comment.

          balaji

          Show
          Balaji Rajagopalan added a comment - Please create a new JIRA to fix any issues with exception exclustion pattern. The original work was submitted as part of pushConfig against a yahoo internal bugzilla ticket, can't publish internal details here. I will use this ticket to address cos java docs comment. balaji
          Hide
          Konstantin Boudnik added a comment -

          Balaji, I will be committing HADOOP-6332 to the Common trunk today. Your patch here should be applicable to that work without hassle because it is pretty much the same code.

          In other words, it's gonna be committed soon too

          Show
          Konstantin Boudnik added a comment - Balaji, I will be committing HADOOP-6332 to the Common trunk today. Your patch here should be applicable to that work without hassle because it is pretty much the same code. In other words, it's gonna be committed soon too
          Konstantin Boudnik made changes -
          Link This issue is blocked by HADOOP-6332 [ HADOOP-6332 ]
          Hide
          Konstantin Boudnik added a comment -

          Correct patch for the trunk and 0.21

          Show
          Konstantin Boudnik added a comment - Correct patch for the trunk and 0.21
          Konstantin Boudnik made changes -
          Attachment hadoop-6572.0.22.patch [ 12444551 ]
          Hide
          Konstantin Boudnik added a comment -

          To verify...

          Show
          Konstantin Boudnik added a comment - To verify...
          Konstantin Boudnik made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12444551/hadoop-6572.0.22.patch
          against trunk revision 944521.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 12 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/521/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/521/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/521/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/521/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12444551/hadoop-6572.0.22.patch against trunk revision 944521. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 12 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/521/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/521/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/521/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/521/console This message is automatically generated.
          Hide
          Konstantin Boudnik added a comment -

          +1 patch looks good.

          Show
          Konstantin Boudnik added a comment - +1 patch looks good.
          Hide
          Konstantin Boudnik added a comment -

          I have just committed this. Thanks Balaji.

          Show
          Konstantin Boudnik added a comment - I have just committed this. Thanks Balaji.
          Konstantin Boudnik made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags [Reviewed]
          Fix Version/s 0.21.0 [ 12313563 ]
          Resolution Fixed [ 1 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #257 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/257/)
          HADOOP-6752. CHANGES.txt should be updated in 0.21 section instead of 0.22
          HADOOP-6752. Remote cluster control functionality needs JavaDocs improvement. Contributed by Balaji Rajagopalan.

          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #257 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/257/ ) HADOOP-6752 . CHANGES.txt should be updated in 0.21 section instead of 0.22 HADOOP-6752 . Remote cluster control functionality needs JavaDocs improvement. Contributed by Balaji Rajagopalan.
          Konstantin Boudnik made changes -
          Affects Version/s 0.21.0 [ 12313563 ]
          Affects Version/s 0.22.0 [ 12314296 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk #346 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/346/)

          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk #346 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/346/ )
          Tom White made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Konstantin Boudnik made changes -
          Tags herriot

            People

            • Assignee:
              Balaji Rajagopalan
              Reporter:
              Konstantin Boudnik
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development