Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-1013

Flume master CPU consumption is large

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Won't Fix
    • Affects Version/s: 0.9.4
    • Fix Version/s: 0.9.5
    • Component/s: Master
    • Labels:
      None

      Description

      We set up flume master in one EC2 m1.large machine with centos 6.2 os.
      We found CPU usage is too large on this server.
      [root@ip-10-114-242-72 flume]# top -H
      top - 03:52:45 up 7 days, 23:09, 3 users, load average: 2.18, 2.48, 2.42
      Tasks: 340 total, 3 running, 337 sleeping, 0 stopped, 0 zombie
      Cpu(s): 70.4%us, 0.3%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 29.2%st
      Mem: 7646264k total, 7458172k used, 188092k free, 85068k buffers
      Swap: 4194296k total, 0k used, 4194296k free, 6161968k cached

      PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
      8188 flume 20 0 3161m 413m 11m R 73.0 5.5 917:57.24 java
      8189 flume 20 0 3161m 413m 11m R 71.0 5.5 915:11.29 java
      8150 flume 20 0 3051m 42m 10m S 12.2 0.6 57:56.77 java
      8031 hdfs 20 0 1863m 91m 11m S 11.6 1.2 56:52.38 java

      We did jstack for it.
      /usr/java/jdk1.6.0_30/bin/jstack -F -l 8152

      Thread 8189: (state = IN_JAVA)
      Error occurred during stack walking:
      java.lang.NullPointerException
         at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:78)
         at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
         at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
         at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
         at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
         at java.lang.reflect.Method.invoke(Method.java:597)
         at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
         at sun.tools.jstack.JStack.main(JStack.java:84)

      Locked ownable synchronizers:
         - None

      Thread 8188: (state = IN_JAVA)
      Error occurred during stack walking:
      java.lang.NullPointerException
         at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:78)
         at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
         at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
         at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
         at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
         at java.lang.reflect.Method.invoke(Method.java:597)
         at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
         at sun.tools.jstack.JStack.main(JStack.java:84)

      Thread 8189: (state = IN_JAVA)

      • java.util.HashMap.getEntry(java.lang.Object) @bci=74, line=347 (Compiled frame; information may be imprecise)
      • java.util.HashMap.containsKey(java.lang.Object) @bci=2, line=335 (Compiled frame)
      • java.util.HashSet.contains(java.lang.Object) @bci=5, line=184 (Compiled frame)
      • com.cloudera.distributed.GossipMulticast.doServerGossip(java.io.DataInputStream, java.io.DataOutputStream) @bci=75, line=458 (Interpreted frame)
      • com.cloudera.distributed.GossipMulticast$GossipServer.run() @bci=153, line=386 (Interpreted frame)

      Locked ownable synchronizers:
         - None

      Thread 8188: (state = IN_JAVA)

      • java.util.HashMap.getEntry(java.lang.Object) @bci=74, line=347 (Compiled frame; information may be imprecise)
      • java.util.HashMap.containsKey(java.lang.Object) @bci=2, line=335 (Compiled frame)
      • java.util.HashSet.contains(java.lang.Object) @bci=5, line=184 (Compiled frame)
      • com.cloudera.distributed.GossipMulticast.doClientGossip(java.io.DataInputStream, java.io.DataOutputStream) @bci=122, line=485 (Interpreted frame)
      • com.cloudera.distributed.GossipMulticast$GossipThread.run() @bci=199, line=308 (Interpreted frame)

      Locked ownable synchronizers:
         - None

      I am not sure if it is bug or wrong configuration. We use default set up and no any tuning.

      Below is the version information:

      1. rpm -q flume-master
        flume-master-0.9.4+25.40-1.noarch

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              xwu Xiang Jun Wu
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: