Uploaded image for project: 'Chukwa'
  1. Chukwa
  2. CHUKWA-650

Re-configure Demux ReduceNumber without re-starting the DemuxManager service

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.6.0
    • Fix Version/s: None
    • Component/s: Data Processors
    • Labels:
      None

      Description

      Currently, DemuxManager controls the Demux's reduce number by reading "CHUKWA_DEMUX_REDUCER_COUNT_FIELD" at the initialization part. It is impossible to tune that reduce number before the next round of demux happens. The only way is to restart that DemuxManager service. I wonder if it is OK to parse that configuration item in runDemux() function.

        Activity

        Hide
        asrabkin Ari Rabkin added a comment -

        Hrm. Does config get re-loaded without restarting demux? If you tell me you have deployed this and it works, I'll commit it.

        Show
        asrabkin Ari Rabkin added a comment - Hrm. Does config get re-loaded without restarting demux? If you tell me you have deployed this and it works, I'll commit it.
        Hide
        grace.huang Jie Huang added a comment -

        Oops, I will upload the correct patch soon.

        Show
        grace.huang Jie Huang added a comment - Oops, I will upload the correct patch soon.
        Hide
        grace.huang Jie Huang added a comment -

        Sorry, I missed one line to reload the configuration before reading that item. Here attaches the updated solution. I will verify it by running over the real cluster ASAP.
        Besides, instead of reloading the existing conf object, we'd better to copy a new configuration instance before reloading, and pass this new instance to Demux. It not only makes the reduce number updated, but also reloads other necessary items as well for Demux process.

        Any idea about this proposal? Thanks

        Show
        grace.huang Jie Huang added a comment - Sorry, I missed one line to reload the configuration before reading that item. Here attaches the updated solution. I will verify it by running over the real cluster ASAP. Besides, instead of reloading the existing conf object, we'd better to copy a new configuration instance before reloading, and pass this new instance to Demux. It not only makes the reduce number updated, but also reloads other necessary items as well for Demux process. Any idea about this proposal? Thanks
        Hide
        grace.huang Jie Huang added a comment -

        @Ari, I have verified the patch on a single-node deployment. The following Demux log to demonstrate the final effect.

        2012-07-30 14:38:22,395 INFO main DemuxManager - old reduce number: 5
        2012-07-30 14:38:22,415 INFO main DemuxManager - new reduce number: 10
        2012-07-30 14:38:22,424 INFO main ChukwaConfiguration - chukwaConf is /home/Grace/work/chukwa-incubating-0.6.0/etc/chukwa
        2012-07-30 14:38:22,453 WARN main JobClient - Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
        2012-07-30 14:38:22,544 INFO main FileInputFormat - Total input paths to process : 1
        2012-07-30 14:38:22,678 INFO main JobClient - Running job: job_201207291428_0009
        

        Any comment, please let me know. thanks for your time.

        Show
        grace.huang Jie Huang added a comment - @Ari, I have verified the patch on a single-node deployment. The following Demux log to demonstrate the final effect. 2012-07-30 14:38:22,395 INFO main DemuxManager - old reduce number: 5 2012-07-30 14:38:22,415 INFO main DemuxManager - new reduce number: 10 2012-07-30 14:38:22,424 INFO main ChukwaConfiguration - chukwaConf is /home/Grace/work/chukwa-incubating-0.6.0/etc/chukwa 2012-07-30 14:38:22,453 WARN main JobClient - Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same. 2012-07-30 14:38:22,544 INFO main FileInputFormat - Total input paths to process : 1 2012-07-30 14:38:22,678 INFO main JobClient - Running job: job_201207291428_0009 Any comment, please let me know. thanks for your time.
        Hide
        asrabkin Ari Rabkin added a comment -

        I just committed this. Thanks.

        Show
        asrabkin Ari Rabkin added a comment - I just committed this. Thanks.
        Hide
        hudson Hudson added a comment -

        Integrated in Chukwa-trunk #455 (See https://builds.apache.org/job/Chukwa-trunk/455/)
        CHUKWA-650. Re-configure Demux ReduceNumber without re-starting the DemuxManager service. Contributed by Jie Huang (Revision 1367839)

        Result = FAILURE
        asrabkin : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1367839
        Files :

        • /incubator/chukwa/trunk/CHANGES.txt
        • /incubator/chukwa/trunk/src/main/java/org/apache/hadoop/chukwa/extraction/demux/DemuxManager.java
        Show
        hudson Hudson added a comment - Integrated in Chukwa-trunk #455 (See https://builds.apache.org/job/Chukwa-trunk/455/ ) CHUKWA-650 . Re-configure Demux ReduceNumber without re-starting the DemuxManager service. Contributed by Jie Huang (Revision 1367839) Result = FAILURE asrabkin : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1367839 Files : /incubator/chukwa/trunk/CHANGES.txt /incubator/chukwa/trunk/src/main/java/org/apache/hadoop/chukwa/extraction/demux/DemuxManager.java

          People

          • Assignee:
            grace.huang Jie Huang
            Reporter:
            grace.huang Jie Huang
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development