Uploaded image for project: 'Chukwa'
  1. Chukwa
  2. CHUKWA-276

Hourly and Daily rolling should use a single reducer + fix classpath issue

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.2.0, 0.3.0
    • Component/s: Data Processors
    • Labels:
      None

      Description

      Hourly and Daily rolling should not use more than one reducer and they are currently relying on the default parameter to be correct.

      <property>
      <name>mapred.reduce.tasks</name>
      <value>1</value>
      <description>The default number of reduce tasks per job. Typically set
      to a prime close to the number of available hosts. Ignored when
      mapred.job.tracker is "local".
      </description>
      </property>

      This should be forced within the Rolling class in order to avoid any issue.

      1. CHUKWA-276.patch
        5 kB
        Jerome Boulon

        Activity

        Hide
        hudson Hudson added a comment -
        Show
        hudson Hudson added a comment - Integrated in Chukwa-trunk #61 (See http://hudson.zones.apache.org/hudson/job/Chukwa-trunk/61/ )
        Hide
        eyang Eric Yang added a comment -

        I just committed this to trunk and 0.2, thanks Jerome.

        Show
        eyang Eric Yang added a comment - I just committed this to trunk and 0.2, thanks Jerome.
        Hide
        eyang Eric Yang added a comment -

        +1 looks good.

        Show
        eyang Eric Yang added a comment - +1 looks good.
        Hide
        jboulon Jerome Boulon added a comment -
        • Fix classpath: add HADOOP_CONF_DIR
        • Force reducer count to 1
        Show
        jboulon Jerome Boulon added a comment - Fix classpath: add HADOOP_CONF_DIR Force reducer count to 1

          People

          • Assignee:
            jboulon Jerome Boulon
            Reporter:
            jboulon Jerome Boulon
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development