Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-4433

Improve data loader for collecting metrics and log files from hadoop and system

        Details

        • Type: Improvement
        • Status: Closed
        • Priority: Major
        • Resolution: Fixed
        • Affects Version/s: None
        • Fix Version/s: 0.19.0
        • Component/s: None
        • Labels:
          None
        • Environment:

          Red Hat Enterprise Linux 4.5, 5, Java 6

        • Hadoop Flags:
          Reviewed
        • Release Note:
          Hide
          - Added startup and shutdown script
          - Added torque metrics data loader
          - Improve handling of Exec Plugin
          - Added Test cases for File Tailing Adaptors
          - Added Test cases for Start streaming at specific offset
          Show
          - Added startup and shutdown script - Added torque metrics data loader - Improve handling of Exec Plugin - Added Test cases for File Tailing Adaptors - Added Test cases for Start streaming at specific offset

          Description

          Enhancement should include:

          • Ability to stream /var/log/messages
          • Better handling of startup and shutdown chukwa processes
          • Ability to monitor torque/hod allocated clusters
          • Add serialization and deserialization of hadoop metrics to JSON log files.
          1. Text File
            HADOOP-4433.patch
            131 kB
            Eric Yang
          2. Text File
            HADOOP-4433-2.patch
            127 kB
            Eric Yang

            Issue Links

            depends upon

            Improvement - An improvement or enhancement to an existing feature or task. HADOOP-4431 Add versionning/tags to Chukwa Chunk

            • Major - Major loss of function.
            • Closed
            is depended upon by

            Improvement - An improvement or enhancement to an existing feature or task. HADOOP-4459 - Improve Chukwa Collector

            • Major - Major loss of function.
            • Closed

              Activity

              Ascending order - Click to sort in descending order
              Hide
              Permalink
              eyang Eric Yang added a comment -

              Patch for this issue depends on patch for HADOOP-4431

              Show
              eyang Eric Yang added a comment - Patch for this issue depends on patch for HADOOP-4431
              Hide
              Permalink
              eyang Eric Yang added a comment -
              • Added startup and shutdown script
              • Added torque metrics data loader
              • Improve handling of Exec Plugin
              • Added Test cases for File Tailing Adaptors
              • Added Test cases for Start streaming at specific offset
              Show
              eyang Eric Yang added a comment - Added startup and shutdown script Added torque metrics data loader Improve handling of Exec Plugin Added Test cases for File Tailing Adaptors Added Test cases for Start streaming at specific offset
              Hide
              Permalink
              eyang Eric Yang added a comment -
              • Added startup and shutdown script
              • Added torque metrics data loader
              • Improve handling of Exec Plugin
              • Added Test cases for File Tailing Adaptors
              • Added Test cases for Start streaming at specific offset
              Show
              eyang Eric Yang added a comment - Added startup and shutdown script Added torque metrics data loader Improve handling of Exec Plugin Added Test cases for File Tailing Adaptors Added Test cases for Start streaming at specific offset
              Hide
              Permalink
              eyang Eric Yang added a comment -

              This patch depends on the patch for HADOOP-4431.

              Show
              eyang Eric Yang added a comment - This patch depends on the patch for HADOOP-4431 .
              Hide
              Permalink
              hadoopqa Hadoop QA added a comment -

              -1 overall. Here are the results of testing the latest attachment
              http://issues.apache.org/jira/secure/attachment/12392397/HADOOP-4433.patch
              against trunk revision 705942.

              -1 @author. The patch appears to contain 1 @author tags which the Hadoop community has agreed to not allow in code contributions.

              +1 tests included. The patch appears to include 8 new or modified tests.

              +1 javadoc. The javadoc tool did not generate any warning messages.

              +1 javac. The applied patch does not increase the total number of javac compiler warnings.

              +1 findbugs. The patch does not introduce any new Findbugs warnings.

              +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

              +1 core tests. The patch passed core unit tests.

              +1 contrib tests. The patch passed contrib unit tests.

              Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3495/testReport/
              Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3495/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
              Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3495/artifact/trunk/build/test/checkstyle-errors.html
              Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3495/console

              This message is automatically generated.

              Show
              hadoopqa Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12392397/HADOOP-4433.patch against trunk revision 705942. -1 @author. The patch appears to contain 1 @author tags which the Hadoop community has agreed to not allow in code contributions. +1 tests included. The patch appears to include 8 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 Eclipse classpath. The patch retains Eclipse classpath integrity. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3495/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3495/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3495/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3495/console This message is automatically generated.
              Hide
              Permalink
              eyang Eric Yang added a comment -

              False positive on the author tag. The author tag was existed in Log4j DailyRollingFileAppender.

              Show
              eyang Eric Yang added a comment - False positive on the author tag. The author tag was existed in Log4j DailyRollingFileAppender.
              Hide
              Permalink
              jboulon Jerome Boulon added a comment -

              Important patch for Chukwa since it's fixing some blocking bugs at the adaptor level and the ExecPlugin
              +1

              Show
              jboulon Jerome Boulon added a comment - Important patch for Chukwa since it's fixing some blocking bugs at the adaptor level and the ExecPlugin +1
              Hide
              Permalink
              owen.omalley Owen O'Malley added a comment -

              The patch doesn't apply against trunk, please regenerate it. It fails on build.xml. In the future, please make smaller patches.

              Show
              owen.omalley Owen O'Malley added a comment - The patch doesn't apply against trunk, please regenerate it. It fails on build.xml. In the future, please make smaller patches.
              Hide
              Permalink
              eyang Eric Yang added a comment -
              • Merge the changes from trunk build.xml for ChukImp Test cases.
              • Remove author tag from ChukwaFileAppender class.

              Patches will become smaller and smaller, I promise.

              Show
              eyang Eric Yang added a comment - Merge the changes from trunk build.xml for ChukImp Test cases. Remove author tag from ChukwaFileAppender class. Patches will become smaller and smaller, I promise.
              Hide
              Permalink
              eyang Eric Yang added a comment -

              Changes from previous patch:

              -Resolved trunk build.xml file conflict.

              Show
              eyang Eric Yang added a comment - Changes from previous patch: -Resolved trunk build.xml file conflict.
              Hide
              Permalink
              asrabkin Ari Rabkin added a comment -

              Looks okay to me. +1

              Show
              asrabkin Ari Rabkin added a comment - Looks okay to me. +1
              Hide
              Permalink
              owen.omalley Owen O'Malley added a comment -

              I just committed this. Please make much smaller patches.

              Show
              owen.omalley Owen O'Malley added a comment - I just committed this. Please make much smaller patches.
              Hide
              Permalink
              hudson Hudson added a comment -

              Integrated in Hadoop-trunk #652 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/652/)

              Show
              hudson Hudson added a comment - Integrated in Hadoop-trunk #652 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/652/ )

                People

                • Assignee:
                  eyang Eric Yang
                  Reporter:
                  eyang Eric Yang
                • Votes:
                  1 Vote for this issue
                  Watchers:
                  1 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved:

                    Time Tracking

                    Estimated:
                    Original Estimate - 96h
                    96h
                    Remaining:
                    Remaining Estimate - 96h
                    96h
                    Logged:
                    Time Spent - Not Specified
                    Not Specified

                      Development