[PHOENIX-4625] memory leak in PhoenixConnection if scanner renew lease thread is not enabled

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.14.0
• Fix Version/s: 4.14.0, 5.0.0
• Assignee: Vikas Vishwakarma
• Reporter: Vikas Vishwakarma
• Created: 22/Feb/18 04:54
• Updated: 26/Jul/18 01:14
• Resolved: 22/Feb/18 18:46

Description

We have two different code path

1. In ConnectionQueryServicesImpl RenewLeaseTasks is scheduled based on the following checks  if renew lease feature is supported and if the renew lease config is enabled supportsFeature(ConnectionQueryServices.Feature.RENEW_LEASE) && renewLeaseEnabled
2. In PhoenixConnection for every scan iterator is added to a Queue for lease renewal based on just the check if the renew lease feature is supported services.supportsFeature(Feature.RENEW_LEASE)

In PhoenixConnection we however miss the check whether renew lease config is enabled (phoenix.scanner.lease.renew.enabled)

 

Now consider a situation where Renew lease feature is supported but phoenix.scanner.lease.renew.enabled is set to false in hbase-site.xml . In this case PhoenixConnection will keep adding the iterators for every scan into the scannerQueue for renewal based on the feature supported check but the renewal task is not running because phoenix.scanner.lease.renew.enabled is set to false, so the scannerQueue will keep growing as long as the PhoenixConnection is alive and multiple scans requests are coming on this connection.

 

We have a use case that uses a single PhoenixConnection that is perpetual and does billions of scans on this connection. In this case scannerQueue is growing to several GB's and ultimately leading to Consecutive Full GC's/OOM

 

Add iterators for Lease renewal in PhoenixConnection

=============================================

 

public void addIteratorForLeaseRenewal(@Nonnull TableResultIterator itr) {
 if (services.supportsFeature(Feature.RENEW_LEASE))
 { 
   checkNotNull(itr); scannerQueue.add(new WeakReference<TableResultIterator>(itr)); 
 }
}

 

Starting the RenewLeaseTask

=============================

checks if Feature.RENEW_LEASE is supported and if phoenix.scanner.lease.renew.enabled is true and starts the RenewLeaseTask

 

ConnectionQueryServicesImpl {

....

this.renewLeaseEnabled = config.getBoolean(RENEW_LEASE_ENABLED, DEFAULT_RENEW_LEASE_ENABLED);

.....

@Override
 public boolean isRenewingLeasesEnabled(){ 
   return supportsFeature(ConnectionQueryServices.Feature.RENEW_LEASE) && renewLeaseEnabled; 
 }

private void scheduleRenewLeaseTasks() {
 if (isRenewingLeasesEnabled()) {
   renewLeaseExecutor =
   Executors.newScheduledThreadPool(renewLeasePoolSize, renewLeaseThreadFactory);
   for (LinkedBlockingQueue<WeakReference<PhoenixConnection>> q : connectionQueues) { 
     renewLeaseExecutor.scheduleAtFixedRate(new RenewLeaseTask(q), 0, renewLeaseTaskFrequency, TimeUnit.MILLISECONDS); 
   }

  }
}

.......

}

 

To solve this We must add both checks in PhoenixConnection if the feature is supported and if the config is enabled before adding the iterators to scannerQueue

ConnectionQueryServices.Feature.RENEW_LEASE is true  &&  phoenix.scanner.lease.renew.enabled is true 

instead of just checking if the feature ConnectionQueryServices.Feature.RENEW_LEASE is supported