Hadoop Common
  1. Hadoop Common
  2. HADOOP-4709

Update chukwa source code and develop chukwa from public repository

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Environment:

      Redhat EL 4.5, Java 1.6, Hadoop trunk

    • Release Note:
      Hide
      What is new in HADOOP-4709:

        Dashboard for visual Hadoop metrics and system metrics.
        Database loader for loading chukwa records to jdbc connected database.
        FileAdaptor for streaming small file as one record for data processing.
        Scripts for capturing tcpdump output and stream to Chukwa for archive.

      Improvement

        New load blanace algorithm for ChukwaAgent to distribute data to Chukwa Collectors more evenly.
        Unit test and validation for agent, collector, and demux map reduce job.

      Contributed by Jerome Boulon, Andy Konwinski, Ariel Rabkin, Eric Yang
      Show
      What is new in HADOOP-4709 :   Dashboard for visual Hadoop metrics and system metrics.   Database loader for loading chukwa records to jdbc connected database.   FileAdaptor for streaming small file as one record for data processing.   Scripts for capturing tcpdump output and stream to Chukwa for archive. Improvement   New load blanace algorithm for ChukwaAgent to distribute data to Chukwa Collectors more evenly.   Unit test and validation for agent, collector, and demux map reduce job. Contributed by Jerome Boulon, Andy Konwinski, Ariel Rabkin, Eric Yang

      Description

      New features that were added to Chukwa source code since last time chukwa was committed to SVN repository:

      • Added Hadoop Infrastructure Care Center (UI for visualize data collected by Chukwa)
      • Added FileAdaptor for streaming small file in one chunk
      • Added compression to archive and demux output
      • Added unit tests and validation for agent, collector, and demux map reduce job
      • Added database loader for loading demux output (sequence file) to jdbc connected database
      • Added algorithm to distribute collector load more evenly
      1. HADOOP-4709-binary.tar.gz
        60 kB
        Eric Yang
      2. HADOOP-4709-3.patch
        1.85 MB
        Eric Yang

        Issue Links

          Activity

          Hide
          Eric Yang added a comment -

          Giri,

          Please file this as a different bug. The original intention was to utilize the same library version when chukwa is built inside hadoop's contrib directory. In this case, jsp-api.jar should be loaded from hadoop/lib/jsp-2.1/jsp-api-2.1.jar The copy of jsp-api.jar was supplied for building chukwa in standalone mode. This means, the build.xml file need to be changed to pick up jsp-api-*.jar from the proper location.

          Thanks

          Show
          Eric Yang added a comment - Giri, Please file this as a different bug. The original intention was to utilize the same library version when chukwa is built inside hadoop's contrib directory. In this case, jsp-api.jar should be loaded from hadoop/lib/jsp-2.1/jsp-api-2.1.jar The copy of jsp-api.jar was supplied for building chukwa in standalone mode. This means, the build.xml file need to be changed to pick up jsp-api-*.jar from the proper location. Thanks
          Hide
          Giridharan Kesavan added a comment -

          Clarification:

          I was going through the chukwa/build.xml 's package-hadoop target and found that we are trying to copy jsp-api.jar from the hadoop/lib dir but actually the jsp-api. jar resides in chukwa/lib directory.

          Can someone please clarify on this?

          Also, let me know if this is a bug and if you want me to file a different bug?

          Thanks,
          Giri

          Show
          Giridharan Kesavan added a comment - Clarification: I was going through the chukwa/build.xml 's package-hadoop target and found that we are trying to copy jsp-api.jar from the hadoop/lib dir but actually the jsp-api. jar resides in chukwa/lib directory. Can someone please clarify on this? Also, let me know if this is a bug and if you want me to file a different bug? Thanks, Giri
          Hide
          Hudson added a comment -
          Show
          Hudson added a comment - Integrated in Hadoop-trunk #680 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/680/ )
          Hide
          Chris Douglas added a comment -

          I just committed this. Thanks Jerome, Eric, Andy, and Ari

          Show
          Chris Douglas added a comment - I just committed this. Thanks Jerome, Eric, Andy, and Ari
          Hide
          Owen O'Malley added a comment -

          Unfortunately, Yahoo has been using an internal subversion repository for Chukwa. This patch is trying to sync the two repositories. Keeping the hidden repository is destructive to the community and needs to stop. I propose we let this patch in, but put Chukwa on probation and point out (as you have done) that this is not acceptable open source development practice.

          Show
          Owen O'Malley added a comment - Unfortunately, Yahoo has been using an internal subversion repository for Chukwa. This patch is trying to sync the two repositories. Keeping the hidden repository is destructive to the community and needs to stop. I propose we let this patch in, but put Chukwa on probation and point out (as you have done) that this is not acceptable open source development practice.
          Hide
          Doug Cutting added a comment -

          Big code drops like this are a bad practice. Rather you should be adding patches issue by issue. Otherwise you're effectively not operating in public, and eliminate the opportunity for others to participate in the process. Before we can add Chukwa committers, we need its contributors to demonstrate that they understand how to collaborate openly. This patch is a step backwards.

          Show
          Doug Cutting added a comment - Big code drops like this are a bad practice. Rather you should be adding patches issue by issue. Otherwise you're effectively not operating in public, and eliminate the opportunity for others to participate in the process. Before we can add Chukwa committers, we need its contributors to demonstrate that they understand how to collaborate openly. This patch is a step backwards.
          Hide
          Eric Yang added a comment -

          Same patch with additional partition name and user name removed.

          Show
          Eric Yang added a comment - Same patch with additional partition name and user name removed.
          Hide
          Eric Yang added a comment -

          Same patch with user name and Yahoo specific partition names removed

          Show
          Eric Yang added a comment - Same patch with user name and Yahoo specific partition names removed
          Hide
          Eric Yang added a comment -

          The current patch contains user name which should not be part of the patch.

          Show
          Eric Yang added a comment - The current patch contains user name which should not be part of the patch.
          Hide
          Eric Yang added a comment -

          Yes this drop includes patches from HADOOP 4459, 4434, and 4529.

          Show
          Eric Yang added a comment - Yes this drop includes patches from HADOOP 4459, 4434, and 4529.
          Hide
          Chris Douglas added a comment -

          Does this drop include the other PA issues for Chukwa (HADOOP-4459, HADOOP-4434, and HADOOP-4529)?

          Show
          Chris Douglas added a comment - Does this drop include the other PA issues for Chukwa ( HADOOP-4459 , HADOOP-4434 , and HADOOP-4529 )?
          Hide
          Eric Yang added a comment -

          Same patch with author tag removed.

          Show
          Eric Yang added a comment - Same patch with author tag removed.
          Hide
          Eric Yang added a comment -

          Remove author tag

          Show
          Eric Yang added a comment - Remove author tag
          Hide
          Eric Yang added a comment -

          Same as above patch and removed author tag.

          Show
          Eric Yang added a comment - Same as above patch and removed author tag.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12394722/HADOOP-4709.patch
          against trunk revision 721415.

          -1 @author. The patch appears to contain 2 @author tags which the Hadoop community has agreed to not allow in code contributions.

          +1 tests included. The patch appears to include 80 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3666/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3666/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3666/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3666/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12394722/HADOOP-4709.patch against trunk revision 721415. -1 @author. The patch appears to contain 2 @author tags which the Hadoop community has agreed to not allow in code contributions. +1 tests included. The patch appears to include 80 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 Eclipse classpath. The patch retains Eclipse classpath integrity. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3666/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3666/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3666/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3666/console This message is automatically generated.
          Hide
          Eric Yang added a comment -

          Apply with patch -p0 < HADOOP-4709.patch in trunk.

          Show
          Eric Yang added a comment - Apply with patch -p0 < HADOOP-4709 .patch in trunk.
          Hide
          Eric Yang added a comment -

          Image files and Apache JSP Taglib 1.1

          Instruction:

          untar the tarball in trunk, and add files in src/contrib/chukwa/lib/jstl*, src/contrib/chukwa/src/web/hicc/images, src/contrib/chukwa/src/web/hicc/css/images, to SVN.

          Show
          Eric Yang added a comment - Image files and Apache JSP Taglib 1.1 Instruction: untar the tarball in trunk, and add files in src/contrib/chukwa/lib/jstl*, src/contrib/chukwa/src/web/hicc/images, src/contrib/chukwa/src/web/hicc/css/images, to SVN.

            People

            • Assignee:
              Eric Yang
              Reporter:
              Eric Yang
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development