Uploaded image for project: 'Chukwa'
  1. Chukwa
  2. CHUKWA-239

demux settings should work correctly out-of-the-box

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.1.2, 0.2.0
    • Component/s: Data Processors
    • Labels:
      None

      Description

      The sort buffer sizes in chukwa-demux-conf are incompatible with the Hadoop defaults.
      As a result, the demux mapred jobs all fail almost immediately with an
      out-of-memory if run on an out-of-the-box Hadoop.

        Activity

        Hide
        asrabkin Ari Rabkin added a comment -

        These settings worked for me. Jerome, is this is a sensible default for small Chukwa deployments? (under 50 nodes)

        Show
        asrabkin Ari Rabkin added a comment - These settings worked for me. Jerome, is this is a sensible default for small Chukwa deployments? (under 50 nodes)
        Hide
        jboulon Jerome Boulon added a comment -

        Ari, yes for the one you changed, on my laptop I'm running with similar setting however you may also want to change the default value for io.sort.factor to be 10.
        But be careful, there's a bug in Demux.java, demux force a load of the config coming from CHUKWA_HOME/conf so if you have a separate config dir then you will only see the default value. I'm not sure if I have open a Jira for that but I will fix it early next week.

        Show
        jboulon Jerome Boulon added a comment - Ari, yes for the one you changed, on my laptop I'm running with similar setting however you may also want to change the default value for io.sort.factor to be 10. But be careful, there's a bug in Demux.java, demux force a load of the config coming from CHUKWA_HOME/conf so if you have a separate config dir then you will only see the default value. I'm not sure if I have open a Jira for that but I will fix it early next week.
        Hide
        asrabkin Ari Rabkin added a comment -

        Revised patch, also changing io.sort.factor

        Show
        asrabkin Ari Rabkin added a comment - Revised patch, also changing io.sort.factor
        Hide
        asrabkin Ari Rabkin added a comment -

        I just committed this.

        Show
        asrabkin Ari Rabkin added a comment - I just committed this.
        Hide
        hudson Hudson added a comment -

        Integrated in Chukwa-trunk #28 (See http://hudson.zones.apache.org/hudson/job/Chukwa-trunk/28/)
        . Demux settings now work out of the box.

        Show
        hudson Hudson added a comment - Integrated in Chukwa-trunk #28 (See http://hudson.zones.apache.org/hudson/job/Chukwa-trunk/28/ ) . Demux settings now work out of the box.
        Hide
        hudson Hudson added a comment -
        Show
        hudson Hudson added a comment - Integrated in Chukwa-trunk #45 (See http://hudson.zones.apache.org/hudson/job/Chukwa-trunk/45/ )

          People

          • Assignee:
            asrabkin Ari Rabkin
            Reporter:
            asrabkin Ari Rabkin
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development