[ROCKETMQ-152] lockForPutMessage CAS may cause system load very high and putMessage cost more than 500ms if the sendMessageThread num is very much - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Improvement
Status: Closed
Priority: Major
Resolution: Won't Fix
Affects Version/s: 4.0.0-incubating
Fix Version/s: None
Component/s: rocketmq-store
Labels:
None
Environment:
sendMessageThreadPoolNums=128 and
JDK jdk1.7.0_80 linux 2.6.32-504.el6.x86_64 and have more than 50 g free mem

Description

broker store sendMessageThreadPoolNums=128
16 thread 1000bytes benchmark test ， System load to 100
put message acquire the lock may spin，cause System load high and putMessage cost more than 500ms

/**

Spin util acquired the lock.
*/
private void lockForPutMessage() {
if (this.defaultMessageStore.getMessageStoreConfig().isUseReentrantLockWhenPutMessage()) { putMessageNormalLock.lock(); }
else

Unknown macro: { boolean flag; do { flag = this.putMessageSpinLock.compareAndSet(true, false); } while (!flag); }

}

2017-03-22 14:40:16 WARN SendMessageThread_101 - putMessage not in lock eclipse time(ms)=1030, bodyLength=10013
2017-03-22 14:40:16 WARN SendMessageThread_34 - putMessage not in lock eclipse time(ms)=1932, bodyLength=10013
2017-03-22 14:40:16 WARN SendMessageThread_3 - putMessage not in lock eclipse time(ms)=581, bodyLength=10013
2017-03-22 14:40:16 WARN SendMessageThread_57 - putMessage not in lock eclipse time(ms)=583, bodyLength=10013
2017-03-22 14:40:16 WARN SendMessageThread_123 - putMessage not in lock eclipse time(ms)=2225, bodyLength=10013
2017-03-22 14:40:16 WARN SendMessageThread_1 - putMessage not in lock eclipse time(ms)=1642, bodyLength=10013
2017-03-22 14:40:16 WARN SendMessageThread_58 - putMessage not in lock eclipse time(ms)=587, bodyLength=10013
2017-03-22 14:40:16 WARN SendMessageThread_19 - putMessage not in lock eclipse time(ms)=1369, bodyLength=10013
2017-03-22 14:40:16 WARN SendMessageThread_32 - putMessage not in lock eclipse time(ms)=1896, bodyLength=10013
2017-03-22 14:40:16 WARN SendMessageThread_2 - putMessage not in lock eclipse time(ms)=1018, bodyLength=10013
2017-03-22 14:40:16 WARN SendMessageThread_122 - putMessage not in lock eclipse time(ms)=657, bodyLength=10013
2017-03-22 14:40:16 WARN SendMessageThread_120 - putMessage not in lock eclipse time(ms)=592, bodyLength=10013
2017-03-22 14:40:16 WARN SendMessageThread_88 - putMessage not in lock eclipse time(ms)=559, bodyLength=10013
2017-03-22 14:40:16 WARN SendMessageThread_44 - putMessage not in lock eclipse time(ms)=892, bodyLength=10013
2017-03-22 14:40:16 WARN SendMessageThread_79 - putMessage not in lock eclipse time(ms)=699, bodyLength=10013
2017-03-22 14:40:16 WARN SendMessageThread_84 - putMessage not in lock eclipse time(ms)=616, bodyLength=10013
2017-03-22 14:40:16 WARN SendMessageThread_112 - putMessage not in lock eclipse time(ms)=515, bodyLength=10013
2017-03-22 14:40:16 WARN SendMessageThread_59 - putMessage not in lock eclipse time(ms)=1301, bodyLength=10013
2017-03-22 14:40:16 WARN SendMessageThread_96 - putMessage not in lock eclipse time(ms)=635, bodyLength=10013

2017-03-22 14:40:16 WARN SendMessageThread_90 - ********lockForPutMessage cost:317
2017-03-22 14:40:16 WARN SendMessageThread_10 - ********lockForPutMessage cost:447
2017-03-22 14:40:16 WARN SendMessageThread_62 - ********lockForPutMessage cost:450
2017-03-22 14:40:16 WARN SendMessageThread_93 - ********lockForPutMessage cost:131
2017-03-22 14:40:16 WARN SendMessageThread_8 - ********lockForPutMessage cost:171
2017-03-22 14:40:16 WARN SendMessageThread_93 - ********lockForPutMessage cost:4
2017-03-22 14:40:16 WARN SendMessageThread_105 - ********lockForPutMessage cost:666
2017-03-22 14:40:16 WARN SendMessageThread_127 - ********lockForPutMessage cost:689
2017-03-22 14:40:16 WARN SendMessageThread_30 - ********lockForPutMessage cost:341
2017-03-22 14:40:16 WARN SendMessageThread_70 - ********lockForPutMessage cost:317
2017-03-22 14:40:16 WARN SendMessageThread_92 - ********lockForPutMessage cost:58
2017-03-22 14:40:16 WARN SendMessageThread_63 - ********lockForPutMessage cost:261
2017-03-22 14:40:16 WARN SendMessageThread_98 - ********lockForPutMessage cost:261
2017-03-22 14:40:16 WARN SendMessageThread_27 - ********lockForPutMessage cost:546
2017-03-22 14:40:16 WARN SendMessageThread_53 - ********lockForPutMessage cost:257
2017-03-22 14:40:16 WARN SendMessageThread_115 - ********lockForPutMessage cost:1047
2017-03-22 14:40:16 WARN SendMessageThread_82 - ********lockForPutMessage cost:1891
2017-03-22 14:40:16 WARN SendMessageThread_57 - ********lockForPutMessage cost:559
2017-03-22 14:40:16 WARN SendMessageThread_40 - ********lockForPutMessage cost:235
2017-03-22 14:40:16 WARN SendMessageThread_77 - ********lockForPutMessage cost:265
2017-03-22 14:40:16 WARN SendMessageThread_35 - ********lockForPutMessage cost:909
2017-03-22 14:40:16 WARN SendMessageThread_113 - ********lockForPutMessage cost:244
2017-03-22 14:40:16 WARN SendMessageThread_97 - ********lockForPutMessage cost:922
2017-03-22 14:40:16 WARN SendMessageThread_114 - ********lockForPutMessage cost:40
2017-03-22 14:40:16 WARN SendMessageThread_43 - ********lockForPutMessage cost:351
2017-03-22 14:40:16 WARN SendMessageThread_126 - ********lockForPutMessage cost:346
2017-03-22 14:40:16 WARN SendMessageThread_110 - ********lockForPutMessage cost:887
2017-03-22 14:40:16 WARN SendMessageThread_15 - ********lockForPutMessage cost:320
2017-03-22 14:40:16 WARN SendMessageThread_3 - ********lockForPutMessage cost:566
2017-03-22 14:40:16 WARN SendMessageThread_68 - ********lockForPutMessage cost:748
2017-03-22 14:40:16 WARN SendMessageThread_22 - ********lockForPutMessage cost:308
2017-03-22 14:40:16 WARN SendMessageThread_34 - ********lockForPutMessage cost:1926
2017-03-22 14:40:16 WARN SendMessageThread_91 - ********lockForPutMessage cost:358
2017-03-22 14:40:16 WARN SendMessageThread_25 - ********lockForPutMessage cost:107
2017-03-22 14:40:16 WARN SendMessageThread_101 - ********lockForPutMessage cost:1024
2017-03-22 14:40:16 WARN SendMessageThread_89 - ********lockForPutMessage cost:326
2017-03-22 14:40:16 WARN SendMessageThread_43 - ********lockForPutMessage cost:5
2017-03-22 14:40:16 WARN SendMessageThread_30 - ********lockForPutMessage cost:1
2017-03-22 14:40:16 WARN SendMessageThread_42 - ********lockForPutMessage cost:326
2017-03-22 14:40:16 WARN SendMessageThread_95 - ********lockForPutMessage cost:58
2017-03-22 14:40:16 WARN SendMessageThread_19 - ********lockForPutMessage cost:1363
2017-03-22 14:40:16 WARN SendMessageThread_86 - ********lockForPutMessage cost:92
2017-03-22 14:40:16 WARN SendMessageThread_1 - ********lockForPutMessage cost:1640
2017-03-22 14:40:16 WARN SendMessageThread_82 - ********lockForPutMessage cost:30
2017-03-22 14:40:16 WARN SendMessageThread_115 - ********lockForPutMessage cost:33
2017-03-22 14:40:16 WARN SendMessageThread_58 - ********lockForPutMessage cost:582
2017-03-22 14:40:16 WARN SendMessageThread_53 - ********lockForPutMessage cost:34
2017-03-22 14:40:16 WARN SendMessageThread_123 - ********lockForPutMessage cost:2224
2017-03-22 14:40:16 WARN SendMessageThread_78 - ********lockForPutMessage cost:286
2017-03-22 14:40:16 WARN SendMessageThread_63 - ********lockForPutMessage cost:27
2017-03-22 14:40:16 WARN SendMessageThread_32 - ********lockForPutMessage cost:1874
2017-03-22 14:40:16 WARN SendMessageThread_87 - ********lockForPutMessage cost:312
2017-03-22 14:40:16 WARN SendMessageThread_53 - ********lockForPutMessage cost:14
2017-03-22 14:40:16 WARN SendMessageThread_123 - ********lockForPutMessage cost:21
2017-03-22 14:40:16 WARN SendMessageThread_45 - ********lockForPutMessage cost:382
2017-03-22 14:40:16 WARN SendMessageThread_89 - ********lockForPutMessage cost:37
2017-03-22 14:40:16 WARN SendMessageThread_15 - ********lockForPutMessage cost:51
2017-03-22 14:40:16 WARN SendMessageThread_2 - ********lockForPutMessage cost:1015
2017-03-22 14:40:16 WARN SendMessageThread_68 - ********lockForPutMessage cost:50
2017-03-22 14:40:16 WARN SendMessageThread_22 - ********lockForPutMessage cost:51
2017-03-22 14:40:16 WARN SendMessageThread_84 - ********lockForPutMessage cost:606
2017-03-22 14:40:16 WARN SendMessageThread_3 - ********lockForPutMessage cost:42
2017-03-22 14:40:16 WARN SendMessageThread_122 - ********lockForPutMessage cost:652
2017-03-22 14:40:16 WARN SendMessageThread_115 - ********lockForPutMessage cost:19
2017-03-22 14:40:16 WARN SendMessageThread_126 - ********lockForPutMessage cost:24
2017-03-22 14:40:16 WARN SendMessageThread_57 - ********lockForPutMessage cost:3
2017-03-22 14:40:16 WARN SendMessageThread_110 - ********lockForPutMessage cost:3
2017-03-22 14:40:16 WARN SendMessageThread_70 - ********lockForPutMessage cost:40
2017-03-22 14:40:16 WARN SendMessageThread_118 - ********lockForPutMessage cost:279
2017-03-22 14:40:16 WARN SendMessageThread_103 - ********lockForPutMessage cost:301
2017-03-22 14:40:16 WARN SendMessageThread_36 - ********lockForPutMessage cost:384
2017-03-22 14:40:16 WARN SendMessageThread_120 - ********lockForPutMessage cost:587
2017-03-22 14:40:16 WARN SendMessageThread_88 - ********lockForPutMessage cost:554
2017-03-22 14:40:16 WARN SendMessageThread_124 - ********lockForPutMessage cost:324
2017-03-22 14:40:16 WARN SendMessageThread_44 - ********lockForPutMessage cost:888
2017-03-22 14:40:16 WARN SendMessageThread_49 - ********lockForPutMessage cost:311
2017-03-22 14:40:16 WARN SendMessageThread_41 - ********lockForPutMessage cost:312
2017-03-22 14:40:16 WARN SendMessageThread_5 - ********lockForPutMessage cost:115
2017-03-22 14:40:16 WARN SendMessageThread_4 - ********lockForPutMessage cost:151
2017-03-22 14:40:16 WARN SendMessageThread_119 - ********lockForPutMessage cost:79
2017-03-22 14:40:16 WARN SendMessageThread_65 - ********lockForPutMessage cost:107
2017-03-22 14:40:16 WARN SendMessageThread_128 - ********lockForPutMessage cost:112
2017-03-22 14:40:16 WARN SendMessageThread_96 - ********lockForPutMessage cost:609
2017-03-22 14:40:16 WARN SendMessageThread_79 - ********lockForPutMessage cost:697
2017-03-22 14:40:16 WARN SendMessageThread_7 - ********lockForPutMessage cost:417
2017-03-22 14:40:16 WARN SendMessageThread_94 - ********lockForPutMessage cost:419
2017-03-22 14:40:16 WARN SendMessageThread_72 - ********lockForPutMessage cost:208
2017-03-22 14:40:16 WARN SendMessageThread_8 - ********lockForPutMessage cost:42
2017-03-22 14:40:16 WARN SendMessageThread_99 - ********lockForPutMessage cost:237
2017-03-22 14:40:16 WARN SendMessageThread_16 - ********lockForPutMessage cost:123
2017-03-22 14:40:16 WARN SendMessageThread_85 - ********lockForPutMessage cost:133
2017-03-22 14:40:16 WARN SendMessageThread_111 - ********lockForPutMessage cost:139
2017-03-22 14:40:16 WARN SendMessageThread_112 - ********lockForPutMessage cost:510
2017-03-22 14:40:16 WARN SendMessageThread_45 - ********lockForPutMessage cost:2
2017-03-22 14:40:16 WARN SendMessageThread_2 - ********lockForPutMessage cost:3
2017-03-22 14:40:16 WARN SendMessageThread_59 - ********lockForPutMessage cost:1291
2017-03-22 14:40:16 WARN SendMessageThread_64 - ********lockForPutMessage cost:105
2017-03-22 14:40:16 WARN SendMessageThread_59 - ********lockForPutMessage cost:48
2017-03-22 14:40:16 WARN SendMessageThread_46 - ********lockForPutMessage cost:255
2017-03-22 14:40:16 WARN SendMessageThread_49 - ********lockForPutMessage cost:95
2017-03-22 14:40:16 WARN SendMessageThread_44 - ********lockForPutMessage cost:7
2017-03-22 14:40:16 WARN SendMessageThread_112 - ********lockForPutMessage cost:37
2017-03-22 14:40:16 WARN SendMessageThread_7 - ********lockForPutMessage cost:75

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending

QQ截图20170322152327.png
22/Mar/17 08:01
16 kB
zhaoziyan

Activity

People

Assignee:: Xinyu Zhou

Reporter:: zhaoziyan

Votes:: 0 Vote for this issue

Watchers:: 2 Start watching this issue

Dates

Created:: 22/Mar/17 08:01

Updated:: 20/Apr/17 06:39

Resolved:: 20/Apr/17 06:39