Uploaded image for project: 'Chukwa (retired)'
  1. Chukwa (retired)
  2. CHUKWA-581

Demux do not support user custom reducer package name.

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • 0.4.0, 0.5.0, 0.6.0
    • 0.6.0
    • MR Data Processors
    • Ubuntu 10.0.4,Hadoop 0.20

    • CustomReducer

    Description

      In ReduceProcessorFactory.java,
      it only gets

      {reduceType}

      .class from this package,as following:

      String path = "org.apache.hadoop.chukwa.extraction.demux.processor.reducer." + reduceType;

      So our custom reducer could only be placed in that package.
      Thus, When we add our

      {demux}

      .jar to Hadoop classpath, since the Hadoop framework will not be able to merge the class files from multiple jar files with the same package name, custom reduce class can not be found.
      I get a way to overcome this defect.Put custom reducer into a package named with "org.apache.hadoop.chukwa.extraction.demux.processor.reducer."+

      {custom}
      Here,{custom}

      is a part of package name.Then in parser,we can write like this:

      Key.setReducerType("{custom}.YourReducer")

      In this way, Hadoop framework could be able to find our reduce class in this partial user defined new package.
      But I think it will be better to fix chukwa to remove the default package name and fully support custom package name for reducer.

      Thanks for Eric Yang's help.

      Attachments

        1. CHUKWA-581-update.patch
          3 kB
          Eric Yang
        2. custom_reducer_backwardcompatible.patch
          16 kB
          Ivy Tang
        3. custom_reducer_conf_backwardcompatible.patch
          4 kB
          Ivy Tang

        Activity

          People

            ivytang Ivy Tang
            ygnhzeus Yingzhong Xu
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - 24h
                24h
                Remaining:
                Remaining Estimate - 24h
                24h
                Logged:
                Time Spent - Not Specified
                Not Specified