Chukwa
  1. Chukwa
  2. CHUKWA-581

Demux do not support user custom reducer package name.

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 0.4.0, 0.5.0, 0.6.0
    • Fix Version/s: 0.6.0
    • Component/s: MR Data Processors
    • Labels:
    • Environment:

      Ubuntu 10.0.4,Hadoop 0.20

    • Tags:
      CustomReducer

      Description

      In ReduceProcessorFactory.java,
      it only gets

      {reduceType}

      .class from this package,as following:

      String path = "org.apache.hadoop.chukwa.extraction.demux.processor.reducer." + reduceType;

      So our custom reducer could only be placed in that package.
      Thus, When we add our

      {demux}

      .jar to Hadoop classpath, since the Hadoop framework will not be able to merge the class files from multiple jar files with the same package name, custom reduce class can not be found.
      I get a way to overcome this defect.Put custom reducer into a package named with "org.apache.hadoop.chukwa.extraction.demux.processor.reducer."+

      {custom}
      Here,{custom}

      is a part of package name.Then in parser,we can write like this:

      Key.setReducerType("{custom}.YourReducer")

      In this way, Hadoop framework could be able to find our reduce class in this partial user defined new package.
      But I think it will be better to fix chukwa to remove the default package name and fully support custom package name for reducer.

      Thanks for Eric Yang's help.

        Activity

        Yingzhong Xu created issue -
        Yingzhong Xu made changes -
        Field Original Value New Value
        Description In ReduceProcessorFactory.java,
        it only gets {reduceType}.class from this package,as following:
        bq. String path = "org.apache.hadoop.chukwa.extraction.demux.processor.reducer." + reduceType;

        So our custom reducer could only be placed in that package.
        Thus, When we add our {demux}.jar to Hadoop classpath, since the Hadoop framework will not be able to merge the class files from multiple jar files with the same package name, custom reduce class can not be found.
        I get a way to overcome this defect.Put custom reducer into a package named with "org.apache.hadoop.chukwa.extraction.demux.processor.reducer."+{custom}
        Here,{custom} is a part of package name.Then in parser,we can write like this:
        bq. Key.setReducerType("{custom}.YourReducer")
        In this way, Hadoop framework could be able to find our reduce class in this partial user defined new package.
        But I think it will be better to fix chukwa to fully support custom package name for reducer.

        Thanks for Eric Yang's help.
        In ReduceProcessorFactory.java,
        it only gets {reduceType}.class from this package,as following:
        bq. String path = "org.apache.hadoop.chukwa.extraction.demux.processor.reducer." + reduceType;

        So our custom reducer could only be placed in that package.
        Thus, When we add our {demux}.jar to Hadoop classpath, since the Hadoop framework will not be able to merge the class files from multiple jar files with the same package name, custom reduce class can not be found.
        I get a way to overcome this defect.Put custom reducer into a package named with "org.apache.hadoop.chukwa.extraction.demux.processor.reducer."+{custom}
        Here,{custom} is a part of package name.Then in parser,we can write like this:
        bq. Key.setReducerType("{custom}.YourReducer")
        In this way, Hadoop framework could be able to find our reduce class in this partial user defined new package.
        But I think it will be better to fix chukwa to remove the default package name and fully support custom package name for reducer.

        Thanks for Eric Yang's help.
        Eric Yang made changes -
        Fix Version/s 0.4.0 [ 12314359 ]
        IvyTang made changes -
        Assignee IvyTang [ ivytang ]
        IvyTang made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Affects Version/s 0.5.0 [ 12315030 ]
        Affects Version/s 0.6.0 [ 12319494 ]
        Labels patch
        Fix Version/s 0.6.0 [ 12319494 ]
        Fix Version/s 0.5.0 [ 12315030 ]
        Tags CustomReducer
        IvyTang made changes -
        Attachment custom_reducer_conf.patch [ 12592729 ]
        Attachment custom_reducer.patch [ 12592730 ]
        IvyTang made changes -
        Attachment custom_reducer_backwardcompatible.patch [ 12594881 ]
        Attachment custom_reducer_conf_backwardcompatible.patch [ 12594882 ]
        IvyTang made changes -
        Attachment custom_reducer_conf.patch [ 12592729 ]
        IvyTang made changes -
        Attachment custom_reducer.patch [ 12592730 ]
        IvyTang made changes -
        Attachment custom_reducer_backwardcompatible.patch [ 12594881 ]
        IvyTang made changes -
        Attachment custom_reducer_backwardcompatible.patch [ 12597423 ]
        Eric Yang made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Fix Version/s 0.5.0 [ 12315030 ]
        Resolution Fixed [ 1 ]
        Eric Yang made changes -
        Resolution Fixed [ 1 ]
        Status Resolved [ 5 ] Reopened [ 4 ]
        Eric Yang made changes -
        Attachment CHUKWA-581-update.patch [ 12598678 ]
        Eric Yang made changes -
        Status Reopened [ 4 ] Patch Available [ 10002 ]
        Eric Yang made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]

          People

          • Assignee:
            IvyTang
            Reporter:
            Yingzhong Xu
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Time Tracking

              Estimated:
              Original Estimate - 24h
              24h
              Remaining:
              Remaining Estimate - 24h
              24h
              Logged:
              Time Spent - Not Specified
              Not Specified

                Development