Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-4709

Update chukwa source code and develop chukwa from public repository

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • None
    • None
    • None

    • Redhat EL 4.5, Java 1.6, Hadoop trunk

    • Hide
      What is new in HADOOP-4709:

        Dashboard for visual Hadoop metrics and system metrics.
        Database loader for loading chukwa records to jdbc connected database.
        FileAdaptor for streaming small file as one record for data processing.
        Scripts for capturing tcpdump output and stream to Chukwa for archive.

      Improvement

        New load blanace algorithm for ChukwaAgent to distribute data to Chukwa Collectors more evenly.
        Unit test and validation for agent, collector, and demux map reduce job.

      Contributed by Jerome Boulon, Andy Konwinski, Ariel Rabkin, Eric Yang
      Show
      What is new in HADOOP-4709 :   Dashboard for visual Hadoop metrics and system metrics.   Database loader for loading chukwa records to jdbc connected database.   FileAdaptor for streaming small file as one record for data processing.   Scripts for capturing tcpdump output and stream to Chukwa for archive. Improvement   New load blanace algorithm for ChukwaAgent to distribute data to Chukwa Collectors more evenly.   Unit test and validation for agent, collector, and demux map reduce job. Contributed by Jerome Boulon, Andy Konwinski, Ariel Rabkin, Eric Yang

    Description

      New features that were added to Chukwa source code since last time chukwa was committed to SVN repository:

      • Added Hadoop Infrastructure Care Center (UI for visualize data collected by Chukwa)
      • Added FileAdaptor for streaming small file in one chunk
      • Added compression to archive and demux output
      • Added unit tests and validation for agent, collector, and demux map reduce job
      • Added database loader for loading demux output (sequence file) to jdbc connected database
      • Added algorithm to distribute collector load more evenly

      Attachments

        1. HADOOP-4709-3.patch
          1.85 MB
          Eric Yang
        2. HADOOP-4709-binary.tar.gz
          60 kB
          Eric Yang

        Issue Links

          incorporates

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-4710 Chukwa - Add duplicate detection, and implement virtual offset of the log file to checkpoint file

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-4811 Import style sheet images for Chukwa

          • Trivial - Cosmetic problem like misspelt words or misaligned text.
          • Closed

          New Feature - A new feature of the product, which has yet to be developed. HADOOP-4529 Support CHUKWA_CONF_DIR environment property for configuration storage

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-4460 Update Chukwa Demux process

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-4461 Update Chukwa parsers

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-4462 Rolling mechanism for demux output

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-4463 Utility classes for Archive and ChukwaRecord files

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-4434 Improve log rotation and reduce communication between agent and logging process, make agent more fault tolerant

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-4459 - Improve Chukwa Collector

          • Major - Major loss of function.
          • Closed

          Activity

            People

              eyang Eric Yang
              eyang Eric Yang
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: