Critical Description
We set up flume master in one EC2 m1.large machine with centos 6.2 os.
We found CPU usage is too large on this server.
[root@ip-10-114-242-72 flume]# top -H
top - 03:52:45 up 7 days, 23:09, 3 users, load average: 2.18, 2.48, 2.42
Tasks: 340 total, 3 running, 337 sleeping, 0 stopped, 0 zombie
Cpu(s): 70.4%us, 0.3%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 29.2%st
Mem: 7646264k total, 7458172k used, 188092k free, 85068k buffers
Swap: 4194296k total, 0k used, 4194296k free, 6161968k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
8188 flume 20 0 3161m 413m 11m R 73.0 5.5 917:57.24 java
8189 flume 20 0 3161m 413m 11m R 71.0 5.5 915:11.29 java
8150 flume 20 0 3051m 42m 10m S 12.2 0.6 57:56.77 java
8031 hdfs 20 0 1863m 91m 11m S 11.6 1.2 56:52.38 java
We did jstack for it.
/usr/java/jdk1.6.0_30/bin/jstack -F -l 8152
Thread 8189: (state = IN_JAVA)
Error occurred during stack walking:
java.lang.NullPointerException
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:78)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
at sun.tools.jstack.JStack.main(JStack.java:84)
Locked ownable synchronizers:
- None
Thread 8188: (state = IN_JAVA)
Error occurred during stack walking:
java.lang.NullPointerException
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:78)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
at sun.tools.jstack.JStack.main(JStack.java:84)
Thread 8189: (state = IN_JAVA)
- java.util.HashMap.getEntry(java.lang.Object) @bci=74, line=347 (Compiled frame; information may be imprecise)
- java.util.HashMap.containsKey(java.lang.Object) @bci=2, line=335 (Compiled frame)
- java.util.HashSet.contains(java.lang.Object) @bci=5, line=184 (Compiled frame)
- com.cloudera.distributed.GossipMulticast.doServerGossip(java.io.DataInputStream, java.io.DataOutputStream) @bci=75, line=458 (Interpreted frame)
- com.cloudera.distributed.GossipMulticast$GossipServer.run() @bci=153, line=386 (Interpreted frame)
Locked ownable synchronizers:
- None
Thread 8188: (state = IN_JAVA)
- java.util.HashMap.getEntry(java.lang.Object) @bci=74, line=347 (Compiled frame; information may be imprecise)
- java.util.HashMap.containsKey(java.lang.Object) @bci=2, line=335 (Compiled frame)
- java.util.HashSet.contains(java.lang.Object) @bci=5, line=184 (Compiled frame)
- com.cloudera.distributed.GossipMulticast.doClientGossip(java.io.DataInputStream, java.io.DataOutputStream) @bci=122, line=485 (Interpreted frame)
- com.cloudera.distributed.GossipMulticast$GossipThread.run() @bci=199, line=308 (Interpreted frame)
Locked ownable synchronizers:
- None
I am not sure if it is bug or wrong configuration. We use default set up and no any tuning.
Below is the version information:
- rpm -q flume-master
flume-master-0.9.4+25.40-1.noarch