Affects Version/s: 4.1.0-incubating
Fix Version/s: None
I found the following error in the client's log :
2017-08-10 09:06:57 ERROR [DubboServerHandler-10.28.109.45:20994-thread-475] c.c.d.a.c.b.r.RocketMQMsgProducer[RocketMQMsgProducer.java:42] -> Send ons msg failed, topic=TopicSubOrderDataSync, tag=TagActivityDataSync, key=activity-order-411320385584760066, msg=411320385584760066
org.apache.rocketmq.client.exception.MQBrokerException: CODE: 2 DESC: [TIMEOUT_CLEAN_QUEUE]broker busy, start flow control for a while, period in queue: 208ms, size of queue: 17
For more information, please visit the url, http://rocketmq.apache.org/docs/faq/
at org.apache.rocketmq.client.impl.MQClientAPIImpl.processSendResponse(MQClientAPIImpl.java:531) ~[rocketmq-client-4.0.0-incubating.jar!/:4.0.0-incubating]
at org.apache.rocketmq.client.impl.MQClientAPIImpl.sendMessageSync(MQClientAPIImpl.java:345) ~[rocketmq-client-4.0.0-incubating.jar!/:4.0.0-incubating]
at org.apache.rocketmq.client.impl.MQClientAPIImpl.sendMessage(MQClientAPIImpl.java:327) ~[rocketmq-client-4.0.0-incubating.jar!/:4.0.0-incubating]
at org.apache.rocketmq.client.impl.MQClientAPIImpl.sendMessage(MQClientAPIImpl.java:290) ~[rocketmq-client-4.0.0-incubating.jar!/:4.0.0-incubating]
at org.apache.rocketmq.client.impl.producer.DefaultMQProducerImpl.sendKernelImpl(DefaultMQProducerImpl.java:688) ~[rocketmq-client-4.0.0-incubating.jar!/:4.0.0-incubating]
at org.apache.rocketmq.client.impl.producer.DefaultMQProducerImpl.sendDefaultImpl(DefaultMQProducerImpl.java:458) ~[rocketmq-client-4.0.0-incubating.jar!/:4.0.0-incubating]
at org.apache.rocketmq.client.impl.producer.DefaultMQProducerImpl.send(DefaultMQProducerImpl.java:1049) ~[rocketmq-client-4.0.0-incubating.jar!/:4.0.0-incubating]
at org.apache.rocketmq.client.impl.producer.DefaultMQProducerImpl.send(DefaultMQProducerImpl.java:1008) ~[rocketmq-client-4.0.0-incubating.jar!/:4.0.0-incubating]
at org.apache.rocketmq.client.producer.DefaultMQProducer.send(DefaultMQProducer.java:204) ~[rocketmq-client-4.0.0-incubating.jar!/:4.0.0-incubating]
and I look into store.log in RocketMQ Broker, and found that at the same time, the following line is there:
2017-08-10 09:06:57 INFO FlushRealTimeService - Flush data to disk costs 1240 ms
I look into the source code, and found that rocketmq have some index files, which rocketmq will not write immediately（because it is wrote randomlly）, but when a index file write finished(about 500MB), rocketmq finally force it into disk, that means dirty pages will be about 500MB maximally， (I have executed the bin/os.sh under RocketMQ, which change the default behavior of linux pdflush）.because the [vm.dirty_ratio] is 50, and the available memory is about 1600MB at my linux machine, 500MB will not exceed 50% of 1600M, so pdflush will not executed in this way. So I guess writeback will impact the RT of write.
So I write a testcase and proved this, the code is:
before running this testcase, please make sure you are running this code in linux platform, and set the following linux configs:
vm.dirty_background_ratio = 50
vm.dirty_ratio = 50
vm.dirty_expire_centisecs = 3000
vm.dirty_writeback_centisecs = 500
The code will do the following things:
1.open a index file, and write some dirty data into it (do not flush manully,let linux pdflush to writeback it into disk)
2.open a commitLog file, and write some dirty data into it, every time the dirty data reach 20KB，it will notify another thread to force data into disk. look into the outputs，it's RT is healthy for now.
3.wait about 30 seconds，until the pdflush wakeup and found dirty data in index file exists for 30 seconds，then pdflush will writeback it into disk（you can execute [cat /proc/meminfo|grep Dirty] to show the size of all dirty data）. At the same time，look into the outputs，and you will see some output like this:
---write cost ms: 213
flushToDisk cost ms:502
the RT of write call is unacceptable, this will block the thread that handle messages, then client will fail to send message,until pdflush is done. And if you set the dirty data in index file bigger, the write RT will grow bigger too.
the bin/os.sh set vm.dirty_writeback_centisecs to 360000(an hour), and I look into store.log, found that the log [Flush data to disk costs * ms] appear hourly, this proved my guess much more realistic.
I attention that the index file have three sections, with the first is head, the second is index wrote randomly, and the third is another type of index wrote sequential. so maybe the second section can be move into another file and this will only use 20MB of memory，which will make writeback faster，at the same time，the third section can be wrote sequential and writeback into disk as soon as possible ，just like what commitlog does. but when recover from system crash, consistency of this two index files is also a problem.