Chukwa
  1. Chukwa
  2. CHUKWA-447

Load increased by 50% when using ExecAdaptor instead of org.apache.hadoop.chukwa.inputtools.plugin.metrics.Exec+FileTailingAdaptor

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.3.0, 0.4.0
    • Fix Version/s: 0.5.0
    • Component/s: Input Tools
    • Labels:
      None
    • Environment:

      Redhat EL 5.1, Java 6

      Description

      While I am coming up a fix for CHUKWA-446, I notice the load average increased by ExecAdaptor. There is probably something that could be improved to reduce the cost of ExecAdaptor.

      1. CHUKWA-447.patch
        0.7 kB
        Eric Yang
      2. Picture 4.png
        47 kB
        Eric Yang

        Activity

        Hide
        Eric Yang added a comment -

        The load one minute average graph for a week. The change was implemented at Jan 16 midnight, and notice the load jumped by 50%.

        Show
        Eric Yang added a comment - The load one minute average graph for a week. The change was implemented at Jan 16 midnight, and notice the load jumped by 50%.
        Hide
        Ari Rabkin added a comment -

        Interesting. And sort of surprising, since both are spawning processes in a loop. Increased 50% is how large an increase in absolute terms?

        Show
        Ari Rabkin added a comment - Interesting. And sort of surprising, since both are spawning processes in a loop. Increased 50% is how large an increase in absolute terms?
        Hide
        Ari Rabkin added a comment -

        NM, didn't see graph. Hrm. Let me do some profiling here. But this probably does add more impetus to CHUKWA-419 or equivalent.

        Show
        Ari Rabkin added a comment - NM, didn't see graph. Hrm. Let me do some profiling here. But this probably does add more impetus to CHUKWA-419 or equivalent.
        Hide
        Ari Rabkin added a comment -

        Can you say anything about the configuration settings in use? Which processes are you forking? Same as before, or different?

        Show
        Ari Rabkin added a comment - Can you say anything about the configuration settings in use? Which processes are you forking? Same as before, or different?
        Hide
        Eric Yang added a comment -

        The commands are identical before and after. Here is what is in my initial_adaptors:

        add org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Iostat 60 /usr/bin/iostat -x -k 55 2 0
        add org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Df 60 /bin/df -l 0
        add org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Sar 60 /usr/bin/sar -q -r -n ALL 55 0
        add org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Top 60 /usr/bin/top -b -n 1 -c 0
        
        Show
        Eric Yang added a comment - The commands are identical before and after. Here is what is in my initial_adaptors: add org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Iostat 60 /usr/bin/iostat -x -k 55 2 0 add org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Df 60 /bin/df -l 0 add org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Sar 60 /usr/bin/sar -q -r -n ALL 55 0 add org.apache.hadoop.chukwa.datacollection.adaptor.ExecAdaptor Top 60 /usr/bin/top -b -n 1 -c 0
        Hide
        Eric Yang added a comment -

        The expected period parameter is in seconds, but the code is running as:

        timer.schedule(execTimer, 0L, period);
        

        Hence, it's running 1000 times more frequent than expected.

        Show
        Eric Yang added a comment - The expected period parameter is in seconds, but the code is running as: timer.schedule(execTimer, 0L, period); Hence, it's running 1000 times more frequent than expected.
        Hide
        Eric Yang added a comment -

        Changed "period" unit from millisecond to second. Only converts to millisecond for scheduling.

        Show
        Eric Yang added a comment - Changed "period" unit from millisecond to second. Only converts to millisecond for scheduling.
        Hide
        Eric Yang added a comment -

        Retain change in code instead of interface.

        Show
        Eric Yang added a comment - Retain change in code instead of interface.
        Hide
        Ari Rabkin added a comment -

        Looks like a good catch. I'll commit this.

        Show
        Ari Rabkin added a comment - Looks like a good catch. I'll commit this.
        Hide
        Ari Rabkin added a comment -

        I just committed this.

        Not sure this entirely will resolve issue; I had thought most of the time the ExecAdaptor is blocked waiting on the underlying process, but that may depend which processes are being invoked. Certainly the fix is worthwhile regardless.

        Show
        Ari Rabkin added a comment - I just committed this. Not sure this entirely will resolve issue; I had thought most of the time the ExecAdaptor is blocked waiting on the underlying process, but that may depend which processes are being invoked. Certainly the fix is worthwhile regardless.

          People

          • Assignee:
            Eric Yang
            Reporter:
            Eric Yang
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development