Uploaded image for project: 'Apache RocketMQ'
  1. Apache RocketMQ
  2. ROCKETMQ-152

lockForPutMessage CAS may cause system load very high and putMessage cost more than 500ms if the sendMessageThread num is very much

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Won't Fix
    • Affects Version/s: 4.0.0-incubating
    • Fix Version/s: None
    • Component/s: rocketmq-store
    • Labels:
      None
    • Environment:
      sendMessageThreadPoolNums=128 and
      JDK jdk1.7.0_80 linux 2.6.32-504.el6.x86_64 and have more than 50 g free mem

      Description

      broker store sendMessageThreadPoolNums=128
      16 thread 1000bytes benchmark test , System load to 100
      put message acquire the lock may spin,cause System load high and putMessage cost more than 500ms

      /**

      • Spin util acquired the lock.
        */
        private void lockForPutMessage() {
        if (this.defaultMessageStore.getMessageStoreConfig().isUseReentrantLockWhenPutMessage()) { putMessageNormalLock.lock(); }

        else

        Unknown macro: { boolean flag; do { flag = this.putMessageSpinLock.compareAndSet(true, false); } while (!flag); }

        }

      2017-03-22 14:40:16 WARN SendMessageThread_101 - putMessage not in lock eclipse time(ms)=1030, bodyLength=10013
      2017-03-22 14:40:16 WARN SendMessageThread_34 - putMessage not in lock eclipse time(ms)=1932, bodyLength=10013
      2017-03-22 14:40:16 WARN SendMessageThread_3 - putMessage not in lock eclipse time(ms)=581, bodyLength=10013
      2017-03-22 14:40:16 WARN SendMessageThread_57 - putMessage not in lock eclipse time(ms)=583, bodyLength=10013
      2017-03-22 14:40:16 WARN SendMessageThread_123 - putMessage not in lock eclipse time(ms)=2225, bodyLength=10013
      2017-03-22 14:40:16 WARN SendMessageThread_1 - putMessage not in lock eclipse time(ms)=1642, bodyLength=10013
      2017-03-22 14:40:16 WARN SendMessageThread_58 - putMessage not in lock eclipse time(ms)=587, bodyLength=10013
      2017-03-22 14:40:16 WARN SendMessageThread_19 - putMessage not in lock eclipse time(ms)=1369, bodyLength=10013
      2017-03-22 14:40:16 WARN SendMessageThread_32 - putMessage not in lock eclipse time(ms)=1896, bodyLength=10013
      2017-03-22 14:40:16 WARN SendMessageThread_2 - putMessage not in lock eclipse time(ms)=1018, bodyLength=10013
      2017-03-22 14:40:16 WARN SendMessageThread_122 - putMessage not in lock eclipse time(ms)=657, bodyLength=10013
      2017-03-22 14:40:16 WARN SendMessageThread_120 - putMessage not in lock eclipse time(ms)=592, bodyLength=10013
      2017-03-22 14:40:16 WARN SendMessageThread_88 - putMessage not in lock eclipse time(ms)=559, bodyLength=10013
      2017-03-22 14:40:16 WARN SendMessageThread_44 - putMessage not in lock eclipse time(ms)=892, bodyLength=10013
      2017-03-22 14:40:16 WARN SendMessageThread_79 - putMessage not in lock eclipse time(ms)=699, bodyLength=10013
      2017-03-22 14:40:16 WARN SendMessageThread_84 - putMessage not in lock eclipse time(ms)=616, bodyLength=10013
      2017-03-22 14:40:16 WARN SendMessageThread_112 - putMessage not in lock eclipse time(ms)=515, bodyLength=10013
      2017-03-22 14:40:16 WARN SendMessageThread_59 - putMessage not in lock eclipse time(ms)=1301, bodyLength=10013
      2017-03-22 14:40:16 WARN SendMessageThread_96 - putMessage not in lock eclipse time(ms)=635, bodyLength=10013

      2017-03-22 14:40:16 WARN SendMessageThread_90 - ********lockForPutMessage cost:317
      2017-03-22 14:40:16 WARN SendMessageThread_10 - ********lockForPutMessage cost:447
      2017-03-22 14:40:16 WARN SendMessageThread_62 - ********lockForPutMessage cost:450
      2017-03-22 14:40:16 WARN SendMessageThread_93 - ********lockForPutMessage cost:131
      2017-03-22 14:40:16 WARN SendMessageThread_8 - ********lockForPutMessage cost:171
      2017-03-22 14:40:16 WARN SendMessageThread_93 - ********lockForPutMessage cost:4
      2017-03-22 14:40:16 WARN SendMessageThread_105 - ********lockForPutMessage cost:666
      2017-03-22 14:40:16 WARN SendMessageThread_127 - ********lockForPutMessage cost:689
      2017-03-22 14:40:16 WARN SendMessageThread_30 - ********lockForPutMessage cost:341
      2017-03-22 14:40:16 WARN SendMessageThread_70 - ********lockForPutMessage cost:317
      2017-03-22 14:40:16 WARN SendMessageThread_92 - ********lockForPutMessage cost:58
      2017-03-22 14:40:16 WARN SendMessageThread_63 - ********lockForPutMessage cost:261
      2017-03-22 14:40:16 WARN SendMessageThread_98 - ********lockForPutMessage cost:261
      2017-03-22 14:40:16 WARN SendMessageThread_27 - ********lockForPutMessage cost:546
      2017-03-22 14:40:16 WARN SendMessageThread_53 - ********lockForPutMessage cost:257
      2017-03-22 14:40:16 WARN SendMessageThread_115 - ********lockForPutMessage cost:1047
      2017-03-22 14:40:16 WARN SendMessageThread_82 - ********lockForPutMessage cost:1891
      2017-03-22 14:40:16 WARN SendMessageThread_57 - ********lockForPutMessage cost:559
      2017-03-22 14:40:16 WARN SendMessageThread_40 - ********lockForPutMessage cost:235
      2017-03-22 14:40:16 WARN SendMessageThread_77 - ********lockForPutMessage cost:265
      2017-03-22 14:40:16 WARN SendMessageThread_35 - ********lockForPutMessage cost:909
      2017-03-22 14:40:16 WARN SendMessageThread_113 - ********lockForPutMessage cost:244
      2017-03-22 14:40:16 WARN SendMessageThread_97 - ********lockForPutMessage cost:922
      2017-03-22 14:40:16 WARN SendMessageThread_114 - ********lockForPutMessage cost:40
      2017-03-22 14:40:16 WARN SendMessageThread_43 - ********lockForPutMessage cost:351
      2017-03-22 14:40:16 WARN SendMessageThread_126 - ********lockForPutMessage cost:346
      2017-03-22 14:40:16 WARN SendMessageThread_110 - ********lockForPutMessage cost:887
      2017-03-22 14:40:16 WARN SendMessageThread_15 - ********lockForPutMessage cost:320
      2017-03-22 14:40:16 WARN SendMessageThread_3 - ********lockForPutMessage cost:566
      2017-03-22 14:40:16 WARN SendMessageThread_68 - ********lockForPutMessage cost:748
      2017-03-22 14:40:16 WARN SendMessageThread_22 - ********lockForPutMessage cost:308
      2017-03-22 14:40:16 WARN SendMessageThread_34 - ********lockForPutMessage cost:1926
      2017-03-22 14:40:16 WARN SendMessageThread_91 - ********lockForPutMessage cost:358
      2017-03-22 14:40:16 WARN SendMessageThread_25 - ********lockForPutMessage cost:107
      2017-03-22 14:40:16 WARN SendMessageThread_101 - ********lockForPutMessage cost:1024
      2017-03-22 14:40:16 WARN SendMessageThread_89 - ********lockForPutMessage cost:326
      2017-03-22 14:40:16 WARN SendMessageThread_43 - ********lockForPutMessage cost:5
      2017-03-22 14:40:16 WARN SendMessageThread_30 - ********lockForPutMessage cost:1
      2017-03-22 14:40:16 WARN SendMessageThread_42 - ********lockForPutMessage cost:326
      2017-03-22 14:40:16 WARN SendMessageThread_95 - ********lockForPutMessage cost:58
      2017-03-22 14:40:16 WARN SendMessageThread_19 - ********lockForPutMessage cost:1363
      2017-03-22 14:40:16 WARN SendMessageThread_86 - ********lockForPutMessage cost:92
      2017-03-22 14:40:16 WARN SendMessageThread_1 - ********lockForPutMessage cost:1640
      2017-03-22 14:40:16 WARN SendMessageThread_82 - ********lockForPutMessage cost:30
      2017-03-22 14:40:16 WARN SendMessageThread_115 - ********lockForPutMessage cost:33
      2017-03-22 14:40:16 WARN SendMessageThread_58 - ********lockForPutMessage cost:582
      2017-03-22 14:40:16 WARN SendMessageThread_53 - ********lockForPutMessage cost:34
      2017-03-22 14:40:16 WARN SendMessageThread_123 - ********lockForPutMessage cost:2224
      2017-03-22 14:40:16 WARN SendMessageThread_78 - ********lockForPutMessage cost:286
      2017-03-22 14:40:16 WARN SendMessageThread_63 - ********lockForPutMessage cost:27
      2017-03-22 14:40:16 WARN SendMessageThread_32 - ********lockForPutMessage cost:1874
      2017-03-22 14:40:16 WARN SendMessageThread_87 - ********lockForPutMessage cost:312
      2017-03-22 14:40:16 WARN SendMessageThread_53 - ********lockForPutMessage cost:14
      2017-03-22 14:40:16 WARN SendMessageThread_123 - ********lockForPutMessage cost:21
      2017-03-22 14:40:16 WARN SendMessageThread_45 - ********lockForPutMessage cost:382
      2017-03-22 14:40:16 WARN SendMessageThread_89 - ********lockForPutMessage cost:37
      2017-03-22 14:40:16 WARN SendMessageThread_15 - ********lockForPutMessage cost:51
      2017-03-22 14:40:16 WARN SendMessageThread_2 - ********lockForPutMessage cost:1015
      2017-03-22 14:40:16 WARN SendMessageThread_68 - ********lockForPutMessage cost:50
      2017-03-22 14:40:16 WARN SendMessageThread_22 - ********lockForPutMessage cost:51
      2017-03-22 14:40:16 WARN SendMessageThread_84 - ********lockForPutMessage cost:606
      2017-03-22 14:40:16 WARN SendMessageThread_3 - ********lockForPutMessage cost:42
      2017-03-22 14:40:16 WARN SendMessageThread_122 - ********lockForPutMessage cost:652
      2017-03-22 14:40:16 WARN SendMessageThread_115 - ********lockForPutMessage cost:19
      2017-03-22 14:40:16 WARN SendMessageThread_126 - ********lockForPutMessage cost:24
      2017-03-22 14:40:16 WARN SendMessageThread_57 - ********lockForPutMessage cost:3
      2017-03-22 14:40:16 WARN SendMessageThread_110 - ********lockForPutMessage cost:3
      2017-03-22 14:40:16 WARN SendMessageThread_70 - ********lockForPutMessage cost:40
      2017-03-22 14:40:16 WARN SendMessageThread_118 - ********lockForPutMessage cost:279
      2017-03-22 14:40:16 WARN SendMessageThread_103 - ********lockForPutMessage cost:301
      2017-03-22 14:40:16 WARN SendMessageThread_36 - ********lockForPutMessage cost:384
      2017-03-22 14:40:16 WARN SendMessageThread_120 - ********lockForPutMessage cost:587
      2017-03-22 14:40:16 WARN SendMessageThread_88 - ********lockForPutMessage cost:554
      2017-03-22 14:40:16 WARN SendMessageThread_124 - ********lockForPutMessage cost:324
      2017-03-22 14:40:16 WARN SendMessageThread_44 - ********lockForPutMessage cost:888
      2017-03-22 14:40:16 WARN SendMessageThread_49 - ********lockForPutMessage cost:311
      2017-03-22 14:40:16 WARN SendMessageThread_41 - ********lockForPutMessage cost:312
      2017-03-22 14:40:16 WARN SendMessageThread_5 - ********lockForPutMessage cost:115
      2017-03-22 14:40:16 WARN SendMessageThread_4 - ********lockForPutMessage cost:151
      2017-03-22 14:40:16 WARN SendMessageThread_119 - ********lockForPutMessage cost:79
      2017-03-22 14:40:16 WARN SendMessageThread_65 - ********lockForPutMessage cost:107
      2017-03-22 14:40:16 WARN SendMessageThread_128 - ********lockForPutMessage cost:112
      2017-03-22 14:40:16 WARN SendMessageThread_96 - ********lockForPutMessage cost:609
      2017-03-22 14:40:16 WARN SendMessageThread_79 - ********lockForPutMessage cost:697
      2017-03-22 14:40:16 WARN SendMessageThread_7 - ********lockForPutMessage cost:417
      2017-03-22 14:40:16 WARN SendMessageThread_94 - ********lockForPutMessage cost:419
      2017-03-22 14:40:16 WARN SendMessageThread_72 - ********lockForPutMessage cost:208
      2017-03-22 14:40:16 WARN SendMessageThread_8 - ********lockForPutMessage cost:42
      2017-03-22 14:40:16 WARN SendMessageThread_99 - ********lockForPutMessage cost:237
      2017-03-22 14:40:16 WARN SendMessageThread_16 - ********lockForPutMessage cost:123
      2017-03-22 14:40:16 WARN SendMessageThread_85 - ********lockForPutMessage cost:133
      2017-03-22 14:40:16 WARN SendMessageThread_111 - ********lockForPutMessage cost:139
      2017-03-22 14:40:16 WARN SendMessageThread_112 - ********lockForPutMessage cost:510
      2017-03-22 14:40:16 WARN SendMessageThread_45 - ********lockForPutMessage cost:2
      2017-03-22 14:40:16 WARN SendMessageThread_2 - ********lockForPutMessage cost:3
      2017-03-22 14:40:16 WARN SendMessageThread_59 - ********lockForPutMessage cost:1291
      2017-03-22 14:40:16 WARN SendMessageThread_64 - ********lockForPutMessage cost:105
      2017-03-22 14:40:16 WARN SendMessageThread_59 - ********lockForPutMessage cost:48
      2017-03-22 14:40:16 WARN SendMessageThread_46 - ********lockForPutMessage cost:255
      2017-03-22 14:40:16 WARN SendMessageThread_49 - ********lockForPutMessage cost:95
      2017-03-22 14:40:16 WARN SendMessageThread_44 - ********lockForPutMessage cost:7
      2017-03-22 14:40:16 WARN SendMessageThread_112 - ********lockForPutMessage cost:37
      2017-03-22 14:40:16 WARN SendMessageThread_7 - ********lockForPutMessage cost:75

        Attachments

          Activity

            People

            • Assignee:
              Yukon yukon
              Reporter:
              zhaoziyan zhaoziyan
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: