Traffic Server
  1. Traffic Server
  2. TS-822

make cluster thread number configable

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 3.1.0
    • Fix Version/s: 3.1.0
    • Component/s: Clustering
    • Labels:
    • Environment:

      heavy clustering env

      Description

      when my cluster working at a heavy load, the thread seems need more cpu time, as the cluster thread number is hard coded to be just one. we should release it or find out other solutions.

      Tasks: 149 total,   1 running, 148 sleeping,   0 stopped,   0 zombie
      Cpu0  : 13.2%us, 11.6%sy,  0.0%ni, 24.5%id, 34.1%wa,  0.0%hi, 16.6%si,  0.0%st
      Cpu1  : 14.3%us,  9.6%sy,  0.0%ni, 55.5%id,  3.0%wa,  1.0%hi, 16.6%si,  0.0%st
      Cpu2  : 15.4%us,  9.7%sy,  0.0%ni, 39.8%id, 19.1%wa,  1.3%hi, 14.7%si,  0.0%st
      Cpu3  : 12.3%us,  9.0%sy,  0.0%ni, 57.0%id,  5.0%wa,  2.3%hi, 14.3%si,  0.0%st
      Mem:  16427532k total, 13547884k used,  2879648k free,   289068k buffers
      Swap:  2096472k total,        0k used,  2096472k free,   451684k cached
      
        PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
      28210 nobody    15   0 12.7g  12g 4508 S 55.9 77.0 731:00.78 [ET_CLUSTER 0]
      28145 nobody    15   0 12.7g  12g 4508 S  8.0 77.0 104:39.41 [ET_NET 1]
      28149 nobody    15   0 12.7g  12g 4508 S  7.7 77.0 108:31.92 [ET_NET 5]
      28143 nobody    15   0 12.7g  12g 4508 S  7.3 77.0 110:25.50 [ET_NET 0]
      28146 nobody    15   0 12.7g  12g 4508 S  7.3 77.0 107:19.08 [ET_NET 2]
      28147 nobody    15   0 12.7g  12g 4508 S  7.3 77.0 107:04.85 [ET_NET 3]
      28148 nobody    15   0 12.7g  12g 4508 S  7.0 77.0 105:01.84 [ET_NET 4]
      28164 nobody    16   0 12.7g  12g 4508 S  4.7 77.0   0:26.85 [ET_AIO 3]
      28188 nobody    15   0 12.7g  12g 4508 S  3.7 77.0  49:19.35 [ACCEPT]
      28135 nobody    16   0  363m  15m 2068 S  1.0  0.1   6:49.62 traffic_manager
      28132 nobody    15   0  363m  15m 2068 S  0.7  0.1  14:08.49 traffic_manager
      28178 nobody    16   0 12.7g  12g 4508 S  0.7 77.0   4:42.76 [ET_AIO 1]
      28156 nobody    16   0 12.7g  12g 4508 S  0.3 77.0   0:29.01 [ET_AIO 3]
      28166 nobody    16   0 12.7g  12g 4508 S  0.3 77.0   0:26.50 [ET_AIO 5]
      28173 nobody    16   0 12.7g  12g 4508 S  0.3 77.0   0:34.62 [ET_AIO 4]
      28177 nobody    16   0 12.7g  12g 4508 S  0.3 77.0   4:42.74 [ET_AIO 0]
      28180 nobody    16   0 12.7g  12g 4508 S  0.3 77.0   4:42.75 [ET_AIO 3]
      28181 nobody    16   0 12.7g  12g 4508 S  0.3 77.0   4:40.95 [ET_AIO 4]
      28183 nobody    16   0 12.7g  12g 4508 S  0.3 77.0   4:42.44 [ET_AIO 6]
          1 root      15   0 10348  632  540 S  0.0  0.0   0:00.69 init
          2 root      RT  -5     0    0    0 S  0.0  0.0   0:00.11 migration/0
          3 root      34  19     0    0    0 S  0.0  0.0   0:04.68 ksoftirqd/0
          4 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 watchdog/0
          5 root      RT  -5     0    0    0 S  0.0  0.0   0:00.09 migration/1
          6 root      34  19     0    0    0 S  0.0  0.0   0:06.99 ksoftirqd/1
          7 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 watchdog/1
          8 root      RT  -5     0    0    0 S  0.0  0.0   0:00.09 migration/2                                  
      
      1. TS-822.patch
        4 kB
        Zhao Yongming

        Activity

        Hide
        mohan_zl added a comment -

        fixed, and closed

        Show
        mohan_zl added a comment - fixed, and closed
        Hide
        Zhao Yongming added a comment -

        yeah, code commit as r1139153. will need to update DOCs later.

        Show
        Zhao Yongming added a comment - yeah, code commit as r1139153. will need to update DOCs later.
        Hide
        Leif Hedstrom added a comment -

        Looks good to me, I assume you have tested it ?

        Show
        Leif Hedstrom added a comment - Looks good to me, I assume you have tested it ?
        Hide
        Zhao Yongming added a comment -

        the patch tested to be able to setup more cluster threads, and here is the 'top' result.

        I am not sure WHY the highest id get more cpu usage(time), this happens in ET_NET too.

        Tasks: 145 total,   8 running, 137 sleeping,   0 stopped,   0 zombie
        Cpu0  : 27.2%us, 20.6%sy,  0.0%ni, 10.0%id,  9.0%wa,  0.0%hi, 33.2%si,  0.0%st
        Cpu1  : 27.0%us, 21.3%sy,  0.0%ni, 11.7%id,  7.0%wa,  2.0%hi, 31.0%si,  0.0%st
        Cpu2  : 30.6%us, 22.3%sy,  0.0%ni, 12.3%id,  8.3%wa,  2.0%hi, 24.6%si,  0.0%st
        Cpu3  : 26.8%us, 20.2%sy,  0.0%ni, 12.3%id,  3.6%wa,  4.0%hi, 33.1%si,  0.0%st
        Mem:  16427532k total, 11886960k used,  4540572k free,   317704k buffers
        Swap:  2096472k total,        0k used,  2096472k free,   688876k cached
        
          PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
        23389 nobody    16   0 11.4g  10g 5380 R 50.8 65.2 102:26.85 [ET_NET 5]
        23443 nobody    16   0 11.4g  10g 5380 R 40.5 65.2  69:32.35 [ET_CLUSTER 1]
        23383 nobody    16   0 11.4g  10g 5380 R 36.9 65.2  75:02.92 [ET_NET 0]
        23385 nobody    16   0 11.4g  10g 5380 D 36.2 65.2  80:23.42 [ET_NET 1]
        23388 nobody    16   0 11.4g  10g 5380 D 35.5 65.2  72:56.18 [ET_NET 4]
        23387 nobody    15   0 11.4g  10g 5380 R 29.9 65.2  58:57.08 [ET_NET 3]
        23386 nobody    15   0 11.4g  10g 5380 R 29.6 65.2  62:40.59 [ET_NET 2]
        23442 nobody    15   0 11.4g  10g 5380 S 18.3 65.2  52:03.32 [ET_CLUSTER 0]
        23428 nobody    15   0 11.4g  10g 5380 R  8.3 65.2  17:29.07 [ACCEPT]
        23372 nobody    15   0  308m  16m 2716 S  2.3  0.1   5:37.94 traffic_manager
        23375 nobody    15   0  308m  16m 2716 S  1.3  0.1   2:22.13 traffic_manager
        23423 nobody    16   0 11.4g  10g 5380 S  1.0 65.2   1:43.24 [ET_AIO 6]
        23418 nobody    16   0 11.4g  10g 5380 S  0.7 65.2   1:42.15 [ET_AIO 1]
        23419 nobody    16   0 11.4g  10g 5380 S  0.7 65.2   1:43.76 [ET_AIO 2]
        23420 nobody    16   0 11.4g  10g 5380 S  0.7 65.2   1:42.63 [ET_AIO 3]
        23421 nobody    16   0 11.4g  10g 5380 S  0.7 65.2   1:43.07 [ET_AIO 4]
        23422 nobody    16   0 11.4g  10g 5380 S  0.7 65.2   1:43.11 [ET_AIO 5]
        23424 nobody    16   0 11.4g  10g 5380 S  0.7 65.2   1:43.78 [ET_AIO 7]
         8059 root      15   0 12728 1124  824 R  0.3  0.0   0:00.99 top
        
        Show
        Zhao Yongming added a comment - the patch tested to be able to setup more cluster threads, and here is the 'top' result. I am not sure WHY the highest id get more cpu usage(time), this happens in ET_NET too. Tasks: 145 total, 8 running, 137 sleeping, 0 stopped, 0 zombie Cpu0 : 27.2%us, 20.6%sy, 0.0%ni, 10.0%id, 9.0%wa, 0.0%hi, 33.2%si, 0.0%st Cpu1 : 27.0%us, 21.3%sy, 0.0%ni, 11.7%id, 7.0%wa, 2.0%hi, 31.0%si, 0.0%st Cpu2 : 30.6%us, 22.3%sy, 0.0%ni, 12.3%id, 8.3%wa, 2.0%hi, 24.6%si, 0.0%st Cpu3 : 26.8%us, 20.2%sy, 0.0%ni, 12.3%id, 3.6%wa, 4.0%hi, 33.1%si, 0.0%st Mem: 16427532k total, 11886960k used, 4540572k free, 317704k buffers Swap: 2096472k total, 0k used, 2096472k free, 688876k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 23389 nobody 16 0 11.4g 10g 5380 R 50.8 65.2 102:26.85 [ET_NET 5] 23443 nobody 16 0 11.4g 10g 5380 R 40.5 65.2 69:32.35 [ET_CLUSTER 1] 23383 nobody 16 0 11.4g 10g 5380 R 36.9 65.2 75:02.92 [ET_NET 0] 23385 nobody 16 0 11.4g 10g 5380 D 36.2 65.2 80:23.42 [ET_NET 1] 23388 nobody 16 0 11.4g 10g 5380 D 35.5 65.2 72:56.18 [ET_NET 4] 23387 nobody 15 0 11.4g 10g 5380 R 29.9 65.2 58:57.08 [ET_NET 3] 23386 nobody 15 0 11.4g 10g 5380 R 29.6 65.2 62:40.59 [ET_NET 2] 23442 nobody 15 0 11.4g 10g 5380 S 18.3 65.2 52:03.32 [ET_CLUSTER 0] 23428 nobody 15 0 11.4g 10g 5380 R 8.3 65.2 17:29.07 [ACCEPT] 23372 nobody 15 0 308m 16m 2716 S 2.3 0.1 5:37.94 traffic_manager 23375 nobody 15 0 308m 16m 2716 S 1.3 0.1 2:22.13 traffic_manager 23423 nobody 16 0 11.4g 10g 5380 S 1.0 65.2 1:43.24 [ET_AIO 6] 23418 nobody 16 0 11.4g 10g 5380 S 0.7 65.2 1:42.15 [ET_AIO 1] 23419 nobody 16 0 11.4g 10g 5380 S 0.7 65.2 1:43.76 [ET_AIO 2] 23420 nobody 16 0 11.4g 10g 5380 S 0.7 65.2 1:42.63 [ET_AIO 3] 23421 nobody 16 0 11.4g 10g 5380 S 0.7 65.2 1:43.07 [ET_AIO 4] 23422 nobody 16 0 11.4g 10g 5380 S 0.7 65.2 1:43.11 [ET_AIO 5] 23424 nobody 16 0 11.4g 10g 5380 S 0.7 65.2 1:43.78 [ET_AIO 7] 8059 root 15 0 12728 1124 824 R 0.3 0.0 0:00.99 top
        Hide
        Zhao Yongming added a comment -

        this patch will add a records config proxy.config.cluster.threads, it will need to restart server process to get active.

        the init script may not pass the args to server process, that should be filed in another bug if it is a problem.

        Show
        Zhao Yongming added a comment - this patch will add a records config proxy.config.cluster.threads, it will need to restart server process to get active. the init script may not pass the args to server process, that should be filed in another bug if it is a problem.

          People

          • Assignee:
            Zhao Yongming
            Reporter:
            Zhao Yongming
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development