Chukwa
  1. Chukwa
  2. CHUKWA-581

Demux do not support user custom reducer package name.

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 0.4.0, 0.5.0, 0.6.0
    • Fix Version/s: 0.6.0
    • Component/s: MR Data Processors
    • Labels:
    • Environment:

      Ubuntu 10.0.4,Hadoop 0.20

    • Tags:
      CustomReducer

      Description

      In ReduceProcessorFactory.java,
      it only gets

      {reduceType}

      .class from this package,as following:

      String path = "org.apache.hadoop.chukwa.extraction.demux.processor.reducer." + reduceType;

      So our custom reducer could only be placed in that package.
      Thus, When we add our

      {demux}

      .jar to Hadoop classpath, since the Hadoop framework will not be able to merge the class files from multiple jar files with the same package name, custom reduce class can not be found.
      I get a way to overcome this defect.Put custom reducer into a package named with "org.apache.hadoop.chukwa.extraction.demux.processor.reducer."+

      {custom}
      Here,{custom}

      is a part of package name.Then in parser,we can write like this:

      Key.setReducerType("{custom}.YourReducer")

      In this way, Hadoop framework could be able to find our reduce class in this partial user defined new package.
      But I think it will be better to fix chukwa to remove the default package name and fully support custom package name for reducer.

      Thanks for Eric Yang's help.

        Activity

        Hide
        IvyTang added a comment -

        The ReduceProcessorFactory inits the reducer class using the classname "org.apache.hadoop.chukwa.extraction.demux.processor.reducer."+ reduceType.So the reducer class must be put in package org.apache.hadoop.chukwa.extraction.demux.processor.reducer and named reduceType.

        This is not that custom.
        So i chaneg the implementation of ReduceProcessorFactory inits the reducer class ,let the ReduceProcessorFactory read the reducer class name from chukwa-demux-conf.xml ,like the way how the MapperProcessorFactory handle the custom mapper class.
        So for the users, If you write your own mapper class ClassA and reducer class ClassB. And you want your DataType1 log be processed by ClassA,and you set the ChukwaRecordKey#ReduceType to be ReduceType1 in ClassA.process .All you need is setting
        <property>
        <name>DataType1_mapper</name>
        <value>ClassA's name</value>
        <description></description>
        </property>
        <property>
        <name>ReduceType1_reducer</name>
        <value>ClassB's name</value>
        <description></description>
        </property>
        .The the DataType1 log will be processed by MapperClass ClassA and ReducerClass ClassB.
        I

        Show
        IvyTang added a comment - The ReduceProcessorFactory inits the reducer class using the classname "org.apache.hadoop.chukwa.extraction.demux.processor.reducer."+ reduceType.So the reducer class must be put in package org.apache.hadoop.chukwa.extraction.demux.processor.reducer and named reduceType. This is not that custom. So i chaneg the implementation of ReduceProcessorFactory inits the reducer class ,let the ReduceProcessorFactory read the reducer class name from chukwa-demux-conf.xml ,like the way how the MapperProcessorFactory handle the custom mapper class. So for the users, If you write your own mapper class ClassA and reducer class ClassB. And you want your DataType1 log be processed by ClassA,and you set the ChukwaRecordKey#ReduceType to be ReduceType1 in ClassA.process .All you need is setting <property> <name>DataType1_mapper</name> <value>ClassA's name</value> <description></description> </property> <property> <name>ReduceType1_reducer</name> <value>ClassB's name</value> <description></description> </property> .The the DataType1 log will be processed by MapperClass ClassA and ReducerClass ClassB. I
        Hide
        Eric Yang added a comment -

        There seems to be a problem with custom_reducer_conf patch. It doesn't apply on trunk. Ivy, could you take a look?

        Show
        Eric Yang added a comment - There seems to be a problem with custom_reducer_conf patch. It doesn't apply on trunk. Ivy, could you take a look?
        Hide
        Eric Yang added a comment -

        How about make it more backward compatible with some tweaks to config?

        <property>
          <name>DataType1</name>
          <value>mapperClass,reducerClass</value>
        </property>
        
        <property>
          <name>mapperType1</name>
          <value>mapperClass</value>
        </property>
        
        <property>
          <name>ReduceType1</name>
          <value>,reducerClass</value>
        </property>
        
        Show
        Eric Yang added a comment - How about make it more backward compatible with some tweaks to config? <property> <name>DataType1</name> <value>mapperClass,reducerClass</value> </property> <property> <name>mapperType1</name> <value>mapperClass</value> </property> <property> <name>ReduceType1</name> <value>,reducerClass</value> </property>
        Hide
        IvyTang added a comment -

        The new patch for the backwardcompatible demux conf.

        Show
        IvyTang added a comment - The new patch for the backwardcompatible demux conf.
        Hide
        Eric Yang added a comment -

        +1 looks good.

        Show
        Eric Yang added a comment - +1 looks good.
        Hide
        Eric Yang added a comment -

        The test case is failing for TestDemuxMapperConfigs:

        Running org.apache.hadoop.chukwa.extraction.demux.TestDemuxMapperConfigs
        Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.343 sec <<< FAILURE!

        Results :

        Failed tests: testSeCustomeMapProcessor(org.apache.hadoop.chukwa.extraction.demux.TestDemuxMapperConfigs): MockMapProcessor never invoked

        Tests run: 2, Failures: 1, Errors: 0, Skipped: 0

        Stacktrace:

        -------------------------------------------------------------------------------
        Test set: org.apache.hadoop.chukwa.extraction.demux.TestDemuxMapperConfigs
        -------------------------------------------------------------------------------
        Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.343 sec <<< FAILURE!
        testSeCustomeMapProcessor(org.apache.hadoop.chukwa.extraction.demux.TestDemuxMapperConfigs) Time elapsed: 0.281 sec <<< FAILURE!
        junit.framework.AssertionFailedError: MockMapProcessor never invoked
        at junit.framework.Assert.fail(Assert.java:50)
        at junit.framework.Assert.assertTrue(Assert.java:20)
        at junit.framework.Assert.assertNotNull(Assert.java:218)
        at org.apache.hadoop.chukwa.extraction.demux.TestDemuxMapperConfigs.testSeCustomeMapProcessor(TestDemuxMapperConfigs.java:86)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at junit.framework.TestCase.runTest(TestCase.java:168)
        at junit.framework.TestCase.runBare(TestCase.java:134)
        at junit.framework.TestResult$1.protect(TestResult.java:110)
        at junit.framework.TestResult.runProtected(TestResult.java:128)
        at junit.framework.TestResult.run(TestResult.java:113)
        at junit.framework.TestCase.run(TestCase.java:124)
        at junit.framework.TestSuite.runTest(TestSuite.java:243)
        at junit.framework.TestSuite.run(TestSuite.java:238)
        at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:83)
        at org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:53)
        at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:123)
        at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:104)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:164)
        at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:110)
        at org.apache.maven.surefire.booter.SurefireStarter.invokeProvider(SurefireStarter.java:175)
        at org.apache.maven.surefire.booter.SurefireStarter.runSuitesInProcessWhenForked(SurefireStarter.java:81)
        at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:68)

        Show
        Eric Yang added a comment - The test case is failing for TestDemuxMapperConfigs: Running org.apache.hadoop.chukwa.extraction.demux.TestDemuxMapperConfigs Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.343 sec <<< FAILURE! Results : Failed tests: testSeCustomeMapProcessor(org.apache.hadoop.chukwa.extraction.demux.TestDemuxMapperConfigs): MockMapProcessor never invoked Tests run: 2, Failures: 1, Errors: 0, Skipped: 0 Stacktrace: ------------------------------------------------------------------------------- Test set: org.apache.hadoop.chukwa.extraction.demux.TestDemuxMapperConfigs ------------------------------------------------------------------------------- Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.343 sec <<< FAILURE! testSeCustomeMapProcessor(org.apache.hadoop.chukwa.extraction.demux.TestDemuxMapperConfigs) Time elapsed: 0.281 sec <<< FAILURE! junit.framework.AssertionFailedError: MockMapProcessor never invoked at junit.framework.Assert.fail(Assert.java:50) at junit.framework.Assert.assertTrue(Assert.java:20) at junit.framework.Assert.assertNotNull(Assert.java:218) at org.apache.hadoop.chukwa.extraction.demux.TestDemuxMapperConfigs.testSeCustomeMapProcessor(TestDemuxMapperConfigs.java:86) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at junit.framework.TestCase.runTest(TestCase.java:168) at junit.framework.TestCase.runBare(TestCase.java:134) at junit.framework.TestResult$1.protect(TestResult.java:110) at junit.framework.TestResult.runProtected(TestResult.java:128) at junit.framework.TestResult.run(TestResult.java:113) at junit.framework.TestCase.run(TestCase.java:124) at junit.framework.TestSuite.runTest(TestSuite.java:243) at junit.framework.TestSuite.run(TestSuite.java:238) at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:83) at org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:53) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:123) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:104) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:164) at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:110) at org.apache.maven.surefire.booter.SurefireStarter.invokeProvider(SurefireStarter.java:175) at org.apache.maven.surefire.booter.SurefireStarter.runSuitesInProcessWhenForked(SurefireStarter.java:81) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:68)
        Hide
        IvyTang added a comment -

        Fix bug

        Show
        IvyTang added a comment - Fix bug
        Hide
        Eric Yang added a comment -

        +1, I just committed this. Thanks Ivy.

        Show
        Eric Yang added a comment - +1, I just committed this. Thanks Ivy.
        Hide
        Hudson added a comment -

        FAILURE: Integrated in Chukwa-trunk #470 (See https://builds.apache.org/job/Chukwa-trunk/470/)
        CHUKWA-581. Added support for custom reducer package name. (Ivy Tang via Eric Yang) (eyang: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1515060)

        • /incubator/chukwa/trunk/CHANGES.txt
        • /incubator/chukwa/trunk/conf/chukwa-demux-conf.xml
        • /incubator/chukwa/trunk/src/main/java/org/apache/hadoop/chukwa/extraction/demux/Demux.java
        • /incubator/chukwa/trunk/src/main/java/org/apache/hadoop/chukwa/extraction/demux/processor/reducer/ReduceProcessorFactory.java
        • /incubator/chukwa/trunk/src/test/java/org/apache/hadoop/chukwa/extraction/demux/TestDemuxMapperConfigs.java
        • /incubator/chukwa/trunk/src/test/java/org/apache/hadoop/chukwa/extraction/demux/TestDemuxReducerConfigs.java
        Show
        Hudson added a comment - FAILURE: Integrated in Chukwa-trunk #470 (See https://builds.apache.org/job/Chukwa-trunk/470/ ) CHUKWA-581 . Added support for custom reducer package name. (Ivy Tang via Eric Yang) (eyang: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1515060 ) /incubator/chukwa/trunk/CHANGES.txt /incubator/chukwa/trunk/conf/chukwa-demux-conf.xml /incubator/chukwa/trunk/src/main/java/org/apache/hadoop/chukwa/extraction/demux/Demux.java /incubator/chukwa/trunk/src/main/java/org/apache/hadoop/chukwa/extraction/demux/processor/reducer/ReduceProcessorFactory.java /incubator/chukwa/trunk/src/test/java/org/apache/hadoop/chukwa/extraction/demux/TestDemuxMapperConfigs.java /incubator/chukwa/trunk/src/test/java/org/apache/hadoop/chukwa/extraction/demux/TestDemuxReducerConfigs.java
        Hide
        Eric Yang added a comment -

        Some demux configuration contains both mapper and reducer. It's missing in chukwa-demux-conf.xml for integration test.

        Show
        Eric Yang added a comment - Some demux configuration contains both mapper and reducer. It's missing in chukwa-demux-conf.xml for integration test.
        Hide
        Eric Yang added a comment -

        Updated integration test demux configuration.

        Show
        Eric Yang added a comment - Updated integration test demux configuration.
        Hide
        Eric Yang added a comment -

        I just committed the update.

        Show
        Eric Yang added a comment - I just committed the update.
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in Chukwa-trunk #472 (See https://builds.apache.org/job/Chukwa-trunk/472/)
        CHUKWA-581. Added support for custom reducer package name. (Ivy Tang via Eric Yang) (eyang: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1515186)

        • /incubator/chukwa/trunk/conf/chukwa-demux-conf.xml
        Show
        Hudson added a comment - SUCCESS: Integrated in Chukwa-trunk #472 (See https://builds.apache.org/job/Chukwa-trunk/472/ ) CHUKWA-581 . Added support for custom reducer package name. (Ivy Tang via Eric Yang) (eyang: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1515186 ) /incubator/chukwa/trunk/conf/chukwa-demux-conf.xml

          People

          • Assignee:
            IvyTang
            Reporter:
            Yingzhong Xu
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Time Tracking

              Estimated:
              Original Estimate - 24h
              24h
              Remaining:
              Remaining Estimate - 24h
              24h
              Logged:
              Time Spent - Not Specified
              Not Specified

                Development