Details
-
Bug
-
Status: Closed
-
Critical
-
Resolution: Fixed
-
0.7, 0.9.1
-
None
-
Thrift Server start up by Resin Server at Linux system.
Description
【1】The first version application Environment:
# cat /proc/version Linux version 3.5.0-23-generic (buildd@komainu) (gcc version 4.6.3 (Ubuntu/Linaro 4.6.3-1ubuntu5) ) #35~precise1-Ubuntu SMP Fri Jan 25 17:13:26 UTC 2013 # java -version java version "1.6.0_45" Java(TM) SE Runtime Environment (build 1.6.0_45-b06) Java HotSpot(TM) 64-Bit Server VM (build 20.45-b01, mixed mode) #thrift version is 0.7
Java Server code like following:use TNonblockingServer type server.
public void start() { try { logger.info("TNonblockingServer start ...."); TProcessor tprocessor = new IconOpenService.Processor<IconOpenService.Iface>(iconOpenServiceImpl); TNonblockingServerSocket tnbSocketTransport = new TNonblockingServerSocket(19875); TNonblockingServer.Args tnbArgs = new TNonblockingServer.Args(tnbSocketTransport); tnbArgs.processor(tprocessor); tnbArgs.transportFactory(new TFramedTransport.Factory()); tnbArgs.protocolFactory(new TBinaryProtocol.Factory()); TServer server = new TNonblockingServer(tnbArgs); logger.info("TNonblockingServer start OK1...."); server.serve(); logger.info("TNonblockingServer start OK2...."); } catch (Exception e) { logger.info("TNonblockingServer start ERROR!!!"); e.printStackTrace(); } }
Problem description:
the thrift server interface invoked per day is much higher.as following:
- wc -l thrift.log.20131209
55598276 thrift.log
In general, the thrift server will be crashed down about 16 pm.
I much restart thrift server again an again,
it is very terrible,my boss will kill me(oh.oh).
so i use tool to monitor the thrift server.
linux top command result is that the RES item is 9.2G
Why use so many memory?
so crazy.
So I use Jmap command to print the jvm info that as following:
Object Histogram:
num #instances #bytes Class description -------------------------------------------------------------------------- 1: 2493990 191419432 char[] 2: 1371792 43897344 java.lang.String 3: 525011 42306448 int[] 4: 356604 17116992 java.nio.HeapByteBuffer 5: 299290 14365920 java.nio.HeapCharBuffer 6: 70742 10203032 * ConstMethodKlass 7: 70742 9629584 * MethodKlass 8: 342788 8226912 java.lang.StringBuffer 9: 341661 8199864 java.lang.StringBuilder 10: 6644 7149072 * ConstantPoolKlass 11: 133484 6407232 org.springframework.aop.framework.ReflectiveMethodInvocation 12: 101052 6096736 * SymbolKlass 13: 144401 5776040 java.util.HashMap$KeyIterator 14: 6644 5039192 * InstanceKlassKlass 15: 160734 4881928 java.lang.Object[] 16: 149160 4773120 java.util.AbstractList$Itr 17: 59294 4269168 org.apache.log4j.spi.LoggingEvent 18: 5359 4171640 * ConstantPoolCacheKlass 19: 164812 3955488 java.util.LinkedList$Entry 20: 109117 3491744 java.util.HashMap$Entry 21: 59069 3307864 org.apache.thrift.protocol.TBinaryProtocol
Just a moment, thrift server throw an exception as following:
2013-12-09 16:01:32,181 ERROR [,Thread-44] - [org.apache.thrift.server.TNonblockingServer] - run() exiting due to uncaught error
java.lang.OutOfMemoryError: Java heap space
at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:39)
at java.nio.ByteBuffer.allocate(ByteBuffer.java:312)
at org.apache.thrift.server.TNonblockingServer$FrameBuffer.read(TNonblockingServer.java:491)
at org.apache.thrift.server.TNonblockingServer$SelectThread.handleRead(TNonblockingServer.java:359)
at org.apache.thrift.server.TNonblockingServer$SelectThread.select(TNonblockingServer.java:304)
at org.apache.thrift.server.TNonblockingServer$SelectThread.run(TNonblockingServer.java:243)
【2】The seconde version application Environment:
because The first version application throw an
java.lang.OutOfMemoryError: Java heap space that i don't know how to do.
so the first step i want to do is that upgrade my thrift version..
the version 0.7 upgrade to the version 0.9.1
of course, I have changed my java code as following:
use the TThreadedSelectorServer type server.
try { logger.info("TNonblockingServer start ...."); TProcessor tprocessor = new IconOpenService.Processor<IconOpenService.Iface>(iconOpenServiceImpl); TNonblockingServerSocket tnbSocketTransport = new TNonblockingServerSocket(9876); TThreadedSelectorServer.Args tnbArgs = new TThreadedSelectorServer.Args(tnbSocketTransport); tnbArgs.processor(tprocessor); tnbArgs.transportFactory(new TFramedTransport.Factory(256)); tnbArgs.protocolFactory(new TBinaryProtocol.Factory()); TServer server = new TThreadedSelectorServer(tnbArgs); logger.info("TNonblockingServer start OK1...."); server.serve(); logger.info("TNonblockingServer start OK2...."); } catch (Exception e) { logger.info("TNonblockingServer start ERROR!!!"); e.printStackTrace(); }
now it work well one day ,but i use linux top command find that the java process use much memory, i worry about that the thrift server will run out of memory again.what should i do ?
Cpu(s): 1.7%us, 1.0%sy, 0.0%ni, 97.1%id, 0.1%wa, 0.0%hi, 0.1%si, 0.0%st Mem: 16374956k total, 16205088k used, 169868k free, 70912k buffers Swap: 3999740k total, 4660k used, 3995080k free, 10265608k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 60104 www-data 20 0 12.4g 4.5g 11m S 71 29.0 154:50.91 java 52938 root 20 0 264m 37m 1168 S 2 0.2 43:10.60 rsyslogd 903 root 20 0 17468 1384 944 R 1 0.0 0:00.02 top 53674 root 20 0 191m 23m 4740 S 1 0.1 71:11.18 admin-agent-msg 487 root 20 0 0 0 0 S 0 0.0 1:05.70 kworker/13:1 547 root 20 0 0 0 0 S 0 0.0 34:44.21 kworker/0:2
jmap info as following:
Object Histogram:
num #instances #bytes Class description -------------------------------------------------------------------------- 1: 140522 1205495664 byte[] 2: 707542 58720520 char[] 3: 393972 12607104 java.lang.String 4: 143339 12535960 int[] 5: 71035 10236744 * ConstMethodKlass 6: 71035 9669432 * MethodKlass 7: 6674 7173400 * ConstantPoolKlass 8: 101411 6121416 * SymbolKlass
if you have any good idean,please share it to me,thanks.
address:zhuhai city of china
QQ:1010859834
mail:hiyoucai@126.com