Hadoop Common
  1. Hadoop Common
  2. HADOOP-6788

[Herriot] Exception exclusion functionality is not working correctly.

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.21.0
    • Fix Version/s: 0.21.0
    • Component/s: test
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Tags:
      herriot

      Description

      Exception exclusion functionality is not working correctly because of that tests are failing by not matching the error count.
      I debugged the issue and found that the problem with shell command which is generating in the getNumberOfMatchesInLogFile function.

      Currently building the shell command in the following way.

      if(list != null){
      for(int i =0; i < list.length; ++i)

      { filePattern.append(" | grep -v " + list[i] ); }

      }
      String[] cmd =
      new String[] {
      "bash",
      "-c",
      "grep -c "
      + pattern + " " + filePattern
      + " | awk -F: '

      {s+=$2} END {print s}'" };

      However, The above commnad won't work correctly because you are counting the exceptions in the file before excluding the known exceptions.
      In this case it gives the mismatch error counts everytime.The shell command should be in the following way to work correctly.

      if (list != null) {
      int index = 0;
      for (String excludeExp : list) { filePattern.append((++index < list.length)? "| grep -v " : "| grep -vc " + list[i] ); }
      }
      String[] cmd =
      new String[] {
      "bash",
      "-c",
      "grep "
      + pattern + " " + filePattern
      + " | awk -F: '{s+=$2}

      END

      {print s}

      '" };

      1. 6788-ydist-security.patch
        2 kB
        Vinay Kumar Thota
      2. 6788-ydist-security.patch
        2 kB
        Vinay Kumar Thota
      3. HADOOP-6788.patch
        2 kB
        Vinay Kumar Thota
      4. HADOOP-6788.patch
        2 kB
        Vinay Kumar Thota

        Issue Links

          Activity

          Gavin made changes -
          Link This issue depends upon HADOOP-6332 [ HADOOP-6332 ]
          Gavin made changes -
          Link This issue depends on HADOOP-6332 [ HADOOP-6332 ]
          Konstantin Boudnik made changes -
          Tags herriot
          Tom White made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Konstantin Boudnik made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Fix Version/s 0.21.0 [ 12313563 ]
          Resolution Fixed [ 1 ]
          Konstantin Boudnik made changes -
          Assignee Balaji Rajagopalan [ balajirg ] Vinay Kumar Thota [ vinaythota ]
          Vinod Kumar Vavilapalli made changes -
          Summary Exception exclusion functionality is not working correctly. [Herriot] Exception exclusion functionality is not working correctly.
          Konstantin Boudnik made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hadoop Flags [Reviewed]
          Affects Version/s 0.21.0 [ 12313563 ]
          Vinay Kumar Thota made changes -
          Attachment HADOOP-6788.patch [ 12446103 ]
          Vinay Kumar Thota made changes -
          Attachment 6788-ydist-security.patch [ 12446102 ]
          Vinay Kumar Thota made changes -
          Summary Exception exculsion functionality is not working correctly. Exception exclusion functionality is not working correctly.
          Vinay Kumar Thota made changes -
          Attachment HADOOP-6788.patch [ 12445620 ]
          Vinay Kumar Thota made changes -
          Attachment 6788-ydist-security.patch [ 12445619 ]
          Vinay Kumar Thota made changes -
          Link This issue depends on HADOOP-6332 [ HADOOP-6332 ]
          Vinay Kumar Thota made changes -
          Link This issue requires MAPREDUCE-1774 [ MAPREDUCE-1774 ]
          Vinay Kumar Thota made changes -
          Project Hadoop Map/Reduce [ 12310941 ] Hadoop Common [ 12310240 ]
          Key MAPREDUCE-1793 HADOOP-6788
          Component/s test [ 12311440 ]
          Component/s test [ 12312904 ]
          Konstantin Boudnik made changes -
          Field Original Value New Value
          Link This issue requires MAPREDUCE-1774 [ MAPREDUCE-1774 ]
          Vinay Kumar Thota created issue -

            People

            • Assignee:
              Vinay Kumar Thota
              Reporter:
              Vinay Kumar Thota
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development