HBase
  1. HBase
  2. HBASE-5516

GZip leading to memory leak in 0.90. Fix similar to HBASE-5387 needed for 0.90.

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 0.90.5
    • Fix Version/s: 0.90.7
    • Component/s: None
    • Labels:
      None

      Description

      Usage of GZip is leading to resident memory leak in 0.90.
      We need to have something similar to HBASE-5387 in 0.90.

      1. HBASE-5516_3_0.90.patch
        12 kB
        ramkrishna.s.vasudevan
      2. HBASE-5516_2_0.90.patch
        12 kB
        Laxman

        Activity

        Hide
        ramkrishna.s.vasudevan added a comment -

        @Jon
        Currently am not working on 0.90. So i may not find time on that. But i would say that you can take a look at the patch? Actually in our 0.90 cluster while using GZIp compression we found memory leak frequently and that occured due to GZip Streams.
        Thanks Jon.

        Show
        ramkrishna.s.vasudevan added a comment - @Jon Currently am not working on 0.90. So i may not find time on that. But i would say that you can take a look at the patch? Actually in our 0.90 cluster while using GZIp compression we found memory leak frequently and that occured due to GZip Streams. Thanks Jon.
        Hide
        Jonathan Hsieh added a comment -

        Hm.. no tests, going to bump to 0.90.8 unless action taken.

        Show
        Jonathan Hsieh added a comment - Hm.. no tests, going to bump to 0.90.8 unless action taken.
        Hide
        ramkrishna.s.vasudevan added a comment -

        Pls review this patch. Will commit it tomorrow if it is fine.

        Show
        ramkrishna.s.vasudevan added a comment - Pls review this patch. Will commit it tomorrow if it is fine.
        Hide
        Ted Yu added a comment -

        +1 on patch v3.

        Show
        Ted Yu added a comment - +1 on patch v3.
        Hide
        ramkrishna.s.vasudevan added a comment - - edited

        What if blockBegin is > 0 but less than HEADER_SIZE ?

        This will not happen Ted. The new block will be created only after one block is completed. So the blcok begin should be > 0 and not less than header size. Correct me if am wrong.

        Show
        ramkrishna.s.vasudevan added a comment - - edited What if blockBegin is > 0 but less than HEADER_SIZE ? This will not happen Ted. The new block will be created only after one block is completed. So the blcok begin should be > 0 and not less than header size. Correct me if am wrong.
        Hide
        ramkrishna.s.vasudevan added a comment -

        Updated patch addressing comments.

        Show
        ramkrishna.s.vasudevan added a comment - Updated patch addressing comments.
        Hide
        ramkrishna.s.vasudevan added a comment - - edited
        top - 08:41:24 up 28 days, 17:46,  4 users,  load average: 3.23, 2.73, 2.52
        Tasks: 308 total,   1 running, 307 sleeping,   0 stopped,   0 zombie
        Cpu(s): 10.3%us,  2.4%sy,  0.0%ni, 78.8%id,  7.8%wa,  0.0%hi,  0.7%si,  0.0%st
        Mem:     48264M total,    48125M used,      139M free,     4370M buffers
        Swap:    51199M total,       53M used,    51146M free,    20926M cached
        
          PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                              
        30934 root      20   0 23.2g  20g  12m S  144 42.6 314:25.74 java                                                                                                                  
        30717 root      20   0 4859m 200m  10m S   50  0.4  66:09.54 java                                                                                                                  
         7351 root      20   0     0    0    0 D    2  0.0  14:30.49 kjournald                                                                                                             
           38 root      20   0     0    0    0 S    1  0.0   0:36.31 events/3                                                                                                              
          127 root      20   0     0    0    0 S    1  0.0  48:30.54 kswapd0                                                                                                               
          128 root      20   0     0    0    0 S    1  0.0  47:48.75 kswapd1                                                                                                               
         4877 root      20   0  8968  400  260 S    1  0.0   8:48.69 irqbalance                                                                                                            
        12644 root      20   0  8900 1356  852 R    1  0.0   0:00.91 top                  
        
        ====================================================
        
        

        The above reports are before patch. You can see the native memory going beyond 20g.

        top - 13:41:13 up 28 days, 22:46,  4 users,  load average: 0.00, 0.02, 0.00
        Tasks: 298 total,   1 running, 297 sleeping,   0 stopped,   0 zombie
        Cpu(s):  0.0%us,  0.0%sy,  0.0%ni, 99.9%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
        Mem:     48264M total,    27960M used,    20304M free,     4861M buffers
        Swap:    51199M total,       53M used,    51146M free,    12556M cached
        
          PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                              
         6822 root      20   0  382m 7916 4116 S    0  0.0  31:22.40 nautilus                                                                                                              
        13197 root      20   0 4828m 189m  10m S    0  0.4  60:49.90 java                                                                                                                  
        13424 root      20   0 8901m 8.1g  12m S    0 17.2 407:56.80 java                                                                                                                  
        27929 root      20   0  3620 1072  600 S    0  0.0   0:05.99 nmon                                                                                                                  
        28225 root      20   0  8900 1344  856 R    0  0.0   0:00.04 top                                                                                                                   
            1 root      20   0 10376  340  304 S    0  0.0   0:13.82 init                                                                                                                  
            2 root      20   0     0    0    0 S    0  0.0   0:00.25 kthreadd                                                                                                              
            3 root      RT   0     0    0    0 S    0  0.0   0:01.22 migration/0                                                                                                           
            4 root      20   0     0    0    0 S    0  0.0   2:45.12 ksoftirqd/0                                                                                                           
            5 root      RT   0     0    0    0 S    0  0.0   0:00.66 migration/1                                                                                                           
            6 root      20   0     0    0    0 S    0  0.0   0:22.16 ksoftirqd/1 
        

        In the above the process with 8.1 g is the RS process and we can see there is no increase.

        The test was done over a period of 3 hours with write load.

        Show
        ramkrishna.s.vasudevan added a comment - - edited top - 08:41:24 up 28 days, 17:46, 4 users, load average: 3.23, 2.73, 2.52 Tasks: 308 total, 1 running, 307 sleeping, 0 stopped, 0 zombie Cpu(s): 10.3%us, 2.4%sy, 0.0%ni, 78.8%id, 7.8%wa, 0.0%hi, 0.7%si, 0.0%st Mem: 48264M total, 48125M used, 139M free, 4370M buffers Swap: 51199M total, 53M used, 51146M free, 20926M cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 30934 root 20 0 23.2g 20g 12m S 144 42.6 314:25.74 java 30717 root 20 0 4859m 200m 10m S 50 0.4 66:09.54 java 7351 root 20 0 0 0 0 D 2 0.0 14:30.49 kjournald 38 root 20 0 0 0 0 S 1 0.0 0:36.31 events/3 127 root 20 0 0 0 0 S 1 0.0 48:30.54 kswapd0 128 root 20 0 0 0 0 S 1 0.0 47:48.75 kswapd1 4877 root 20 0 8968 400 260 S 1 0.0 8:48.69 irqbalance 12644 root 20 0 8900 1356 852 R 1 0.0 0:00.91 top ==================================================== The above reports are before patch. You can see the native memory going beyond 20g. top - 13:41:13 up 28 days, 22:46, 4 users, load average: 0.00, 0.02, 0.00 Tasks: 298 total, 1 running, 297 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0%us, 0.0%sy, 0.0%ni, 99.9%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 48264M total, 27960M used, 20304M free, 4861M buffers Swap: 51199M total, 53M used, 51146M free, 12556M cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 6822 root 20 0 382m 7916 4116 S 0 0.0 31:22.40 nautilus 13197 root 20 0 4828m 189m 10m S 0 0.4 60:49.90 java 13424 root 20 0 8901m 8.1g 12m S 0 17.2 407:56.80 java 27929 root 20 0 3620 1072 600 S 0 0.0 0:05.99 nmon 28225 root 20 0 8900 1344 856 R 0 0.0 0:00.04 top 1 root 20 0 10376 340 304 S 0 0.0 0:13.82 init 2 root 20 0 0 0 0 S 0 0.0 0:00.25 kthreadd 3 root RT 0 0 0 0 S 0 0.0 0:01.22 migration/0 4 root 20 0 0 0 0 S 0 0.0 2:45.12 ksoftirqd/0 5 root RT 0 0 0 0 S 0 0.0 0:00.66 migration/1 6 root 20 0 0 0 0 S 0 0.0 0:22.16 ksoftirqd/1 In the above the process with 8.1 g is the RS process and we can see there is no increase. The test was done over a period of 3 hours with write load.
        Hide
        ramkrishna.s.vasudevan added a comment -

        We are facing the same problem in the 0.90 also when using GZIP. Below is a snapshot of the 'top' command

        13236 root      20   0 21.9g  21g  12m S    1 68.4 450:56.37 /opt/nn/jdk1.6.0_22
        13236 root      20   0 21.9g  21g  12m S    1 68.4 450:56.71 /opt/nn/jdk1.6.0_22
        13236 root      20   0 21.9g  21g  12m S    1 68.4 450:57.06 /opt/nn/jdk1.6.0_22
        

        This value we got before the patch was created. After the patch no leak was found. Anyway precise reports will update tomorrow as today our test clusters are busy with other task.
        Configured heap is 4G for RS.

        Show
        ramkrishna.s.vasudevan added a comment - We are facing the same problem in the 0.90 also when using GZIP. Below is a snapshot of the 'top' command 13236 root 20 0 21.9g 21g 12m S 1 68.4 450:56.37 /opt/nn/jdk1.6.0_22 13236 root 20 0 21.9g 21g 12m S 1 68.4 450:56.71 /opt/nn/jdk1.6.0_22 13236 root 20 0 21.9g 21g 12m S 1 68.4 450:57.06 /opt/nn/jdk1.6.0_22 This value we got before the patch was created. After the patch no leak was found. Anyway precise reports will update tomorrow as today our test clusters are busy with other task. Configured heap is 4G for RS.
        Hide
        Ted Yu added a comment -

        Can test results be described here ?

        +      if (this.compressAlgo.equals(Compression.Algorithm.GZ) && blockBegin > 0) {
        +        blockBegin -= HEADER_SIZE;
        +      }
        

        What if blockBegin is > 0 but less than HEADER_SIZE ?

        +      if (compressionBos == null) {
        +        if (this.compressAlgo.equals(Compression.Algorithm.GZ)) {
        +          createCompressionStream();
        +        }
        +      }
        

        The nested if statements can be condensed into one if statement.

        Show
        Ted Yu added a comment - Can test results be described here ? + if ( this .compressAlgo.equals(Compression.Algorithm.GZ) && blockBegin > 0) { + blockBegin -= HEADER_SIZE; + } What if blockBegin is > 0 but less than HEADER_SIZE ? + if (compressionBos == null ) { + if ( this .compressAlgo.equals(Compression.Algorithm.GZ)) { + createCompressionStream(); + } + } The nested if statements can be condensed into one if statement.
        Hide
        Laxman added a comment -

        Please review the patch and share your comments.

        Show
        Laxman added a comment - Please review the patch and share your comments.
        Hide
        ramkrishna.s.vasudevan added a comment -

        Test cases are running. Will upload the patch after that.

        Show
        ramkrishna.s.vasudevan added a comment - Test cases are running. Will upload the patch after that.
        Hide
        ramkrishna.s.vasudevan added a comment -

        We have an internal patch. But it requires some more formatting while writing the header. Will upload once done and tested.

        Show
        ramkrishna.s.vasudevan added a comment - We have an internal patch. But it requires some more formatting while writing the header. Will upload once done and tested.

          People

          • Assignee:
            ramkrishna.s.vasudevan
            Reporter:
            ramkrishna.s.vasudevan
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:

              Development