Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-4433

Improve data loader for collecting metrics and log files from hadoop and system

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.19.0
    • Component/s: None
    • Labels:
      None
    • Environment:

      Red Hat Enterprise Linux 4.5, 5, Java 6

    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      - Added startup and shutdown script
      - Added torque metrics data loader
      - Improve handling of Exec Plugin
      - Added Test cases for File Tailing Adaptors
      - Added Test cases for Start streaming at specific offset
      Show
      - Added startup and shutdown script - Added torque metrics data loader - Improve handling of Exec Plugin - Added Test cases for File Tailing Adaptors - Added Test cases for Start streaming at specific offset

      Description

      Enhancement should include:

      • Ability to stream /var/log/messages
      • Better handling of startup and shutdown chukwa processes
      • Ability to monitor torque/hod allocated clusters
      • Add serialization and deserialization of hadoop metrics to JSON log files.
      1. HADOOP-4433.patch
        131 kB
        Eric Yang
      2. HADOOP-4433-2.patch
        127 kB
        Eric Yang

        Issue Links

          Activity

          Hide
          eyang Eric Yang added a comment -

          Patch for this issue depends on patch for HADOOP-4431

          Show
          eyang Eric Yang added a comment - Patch for this issue depends on patch for HADOOP-4431
          Hide
          eyang Eric Yang added a comment -
          • Added startup and shutdown script
          • Added torque metrics data loader
          • Improve handling of Exec Plugin
          • Added Test cases for File Tailing Adaptors
          • Added Test cases for Start streaming at specific offset
          Show
          eyang Eric Yang added a comment - Added startup and shutdown script Added torque metrics data loader Improve handling of Exec Plugin Added Test cases for File Tailing Adaptors Added Test cases for Start streaming at specific offset
          Hide
          eyang Eric Yang added a comment -
          • Added startup and shutdown script
          • Added torque metrics data loader
          • Improve handling of Exec Plugin
          • Added Test cases for File Tailing Adaptors
          • Added Test cases for Start streaming at specific offset
          Show
          eyang Eric Yang added a comment - Added startup and shutdown script Added torque metrics data loader Improve handling of Exec Plugin Added Test cases for File Tailing Adaptors Added Test cases for Start streaming at specific offset
          Hide
          eyang Eric Yang added a comment -

          This patch depends on the patch for HADOOP-4431.

          Show
          eyang Eric Yang added a comment - This patch depends on the patch for HADOOP-4431 .
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12392397/HADOOP-4433.patch
          against trunk revision 705942.

          -1 @author. The patch appears to contain 1 @author tags which the Hadoop community has agreed to not allow in code contributions.

          +1 tests included. The patch appears to include 8 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3495/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3495/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3495/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3495/console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12392397/HADOOP-4433.patch against trunk revision 705942. -1 @author. The patch appears to contain 1 @author tags which the Hadoop community has agreed to not allow in code contributions. +1 tests included. The patch appears to include 8 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 Eclipse classpath. The patch retains Eclipse classpath integrity. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3495/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3495/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3495/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3495/console This message is automatically generated.
          Hide
          eyang Eric Yang added a comment -

          False positive on the author tag. The author tag was existed in Log4j DailyRollingFileAppender.

          Show
          eyang Eric Yang added a comment - False positive on the author tag. The author tag was existed in Log4j DailyRollingFileAppender.
          Hide
          jboulon Jerome Boulon added a comment -

          Important patch for Chukwa since it's fixing some blocking bugs at the adaptor level and the ExecPlugin
          +1

          Show
          jboulon Jerome Boulon added a comment - Important patch for Chukwa since it's fixing some blocking bugs at the adaptor level and the ExecPlugin +1
          Hide
          owen.omalley Owen O'Malley added a comment -

          The patch doesn't apply against trunk, please regenerate it. It fails on build.xml. In the future, please make smaller patches.

          Show
          owen.omalley Owen O'Malley added a comment - The patch doesn't apply against trunk, please regenerate it. It fails on build.xml. In the future, please make smaller patches.
          Hide
          eyang Eric Yang added a comment -
          • Merge the changes from trunk build.xml for ChukImp Test cases.
          • Remove author tag from ChukwaFileAppender class.

          Patches will become smaller and smaller, I promise.

          Show
          eyang Eric Yang added a comment - Merge the changes from trunk build.xml for ChukImp Test cases. Remove author tag from ChukwaFileAppender class. Patches will become smaller and smaller, I promise.
          Hide
          eyang Eric Yang added a comment -

          Changes from previous patch:

          -Resolved trunk build.xml file conflict.

          Show
          eyang Eric Yang added a comment - Changes from previous patch: -Resolved trunk build.xml file conflict.
          Hide
          asrabkin Ari Rabkin added a comment -

          Looks okay to me. +1

          Show
          asrabkin Ari Rabkin added a comment - Looks okay to me. +1
          Hide
          owen.omalley Owen O'Malley added a comment -

          I just committed this. Please make much smaller patches.

          Show
          owen.omalley Owen O'Malley added a comment - I just committed this. Please make much smaller patches.
          Hide
          hudson Hudson added a comment -
          Show
          hudson Hudson added a comment - Integrated in Hadoop-trunk #652 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/652/ )

            People

            • Assignee:
              eyang Eric Yang
              Reporter:
              eyang Eric Yang
            • Votes:
              1 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - 96h
                96h
                Remaining:
                Remaining Estimate - 96h
                96h
                Logged:
                Time Spent - Not Specified
                Not Specified

                  Development