[HBASE-25581] BUG - HBase native client memory leak - threads keep running after connection is closed

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Incomplete
• Assignee: Unassigned
• Reporter: Julia Emelianova
• Created: 16/Feb/21 09:49
• Updated: 24/May/23 14:39
• Resolved: 23/Jun/22 20:39

Description

Hello everyone, 

First of all, thank you for creating such an amazing product

I'm trying to implement an open-source PHP-extension based on HBase native client. It seems more promising performance-wise than Thrifts we're currently using

In terms of PHP-fpm, we need to be able to open and close connection multiple times in one process, and here's when the library leaks memory

Calling 'Close' methods on Table and Clients is not enough for closing all of the threads.

A simple program

 

int main() {
    zoo_set_debug_level( (ZooLogLevel)4);
    google::SetCommandLineOption("GLOG_minloglevel", "0");


    std::shared_ptr<hbase::Configuration> conf = std::make_shared<hbase::Configuration>();
    conf->Set(hbase::ZKUtil::kHBaseZookeeperQuorum_, "quorum");
    conf->Set(hbase::ZKUtil::kHBaseZookeeperClientPort_, "2181");
    conf->Set(hbase::ZKUtil::kHBaseZnodeParent_, "/hbase-unsecure");

    createClient(conf);  
    int i = 0;
    while (true) {
        i++;
        std::cout << "Slept " << i << std::endl;
        sleep(5);
    }

    return 0;
}

void createClient(const std::shared_ptr<hbase::Configuration>& conf)
{
    std::cout << "opening " << std::endl << std::endl << std::endl;
    auto tn = std::make_shared<hbase::pb::TableName>(folly::to<hbase::pb::TableName>("table"));
    auto client = std::make_unique<hbase::Client>(*conf);
    auto table = client->Table(*tn);
    std::cout << "closing " << std::endl << std::endl << std::endl;
    table->Close();
    client->Close();
}

 

Logs :

[julia@fdf346c7aa2b myhbase]$ ./myhbase
opening

WARNING: Logging before InitGoogleLogging() is written to STDERR
I0216 08:54:44.391023  5259 location-cache.cc:77] Connecting to ZooKeeper. Quorum:quorum
2021-02-16 08:54:44,391:5259(0x7fa88b38d3c0):ZOO_INFO@log_env@726: Client environment:zookeeper.version=zookeeper C client 3.4.8
2021-02-16 08:54:44,391:5259(0x7fa88b38d3c0):ZOO_INFO@log_env@730: Client environment:host.name=fdf346c7aa2b
2021-02-16 08:54:44,391:5259(0x7fa88b38d3c0):ZOO_INFO@log_env@737: Client environment:os.name=Linux
2021-02-16 08:54:44,391:5259(0x7fa88b38d3c0):ZOO_INFO@log_env@738: Client environment:os.arch=5.8.1-1.el7.elrepo.x86_64
2021-02-16 08:54:44,391:5259(0x7fa88b38d3c0):ZOO_INFO@log_env@739: Client environment:os.version=#1 SMP Tue Aug 11 12:01:11 EDT 2020
2021-02-16 08:54:44,391:5259(0x7fa88b38d3c0):ZOO_INFO@log_env@747: Client environment:user.name=(null)
2021-02-16 08:54:44,391:5259(0x7fa88b38d3c0):ZOO_INFO@log_env@755: Client environment:user.home=/home/julia
2021-02-16 08:54:44,391:5259(0x7fa88b38d3c0):ZOO_INFO@log_env@767: Client environment:user.dir=/usr/src/hbase/myhbase
2021-02-16 08:54:44,391:5259(0x7fa88b38d3c0):ZOO_INFO@zookeeper_init@800: Initiating client connection, host=quorum sessionTimeout=90000 watcher=(nil) sessionId=0 sessionPasswd=<null> context=(nil) flags=0
2021-02-16 08:54:44,400:5259(0x7fa88b38d3c0):ZOO_DEBUG@start_threads@221: starting threads...
2021-02-16 08:54:44,400:5259(0x7fa7a77fe700):ZOO_DEBUG@do_io@367: started IO thread
2021-02-16 08:54:44,400:5259(0x7fa7a6ffd700):ZOO_DEBUG@do_completion@459: started completion thread
closing

I0216 08:54:44.400823  5259 client.cc:56] called destructor on Client
2021-02-16 08:54:44,400:5259(0x7fa7a77fe700):ZOO_INFO@check_events@1728: initiated connection to server [192.168.197.12:2181]
2021-02-16 08:54:44,403:5259(0x7fa7a77fe700):ZOO_INFO@check_events@1775: session establishment complete on server [192.168.197.12:2181], sessionId=0x27640b8a9bd0932, negotiated timeout=60000
2021-02-16 08:54:44,403:5259(0x7fa7a77fe700):ZOO_DEBUG@check_events@1781: Calling a watcher for a ZOO_SESSION_EVENT and the state=ZOO_CONNECTED_STATE
2021-02-16 08:54:44,403:5259(0x7fa7a6ffd700):ZOO_DEBUG@process_completions@2132: Calling a watcher for node [], type = -1 event=ZOO_SESSION_EVENT
I0216 08:54:44.403533  5259 table.cc:50] called destructor on Table
Slept 1
Slept 2
Slept 3
Slept 4
Slept 5
2021-02-16 08:55:04,421:5259(0x7fa7a77fe700):ZOO_WARN@zookeeper_interest@1570: Exceeded deadline by 19ms
2021-02-16 08:55:04,421:5259(0x7fa7a77fe700):ZOO_DEBUG@zookeeper_process@2218: Got ping response in 0 ms
Slept 6
Slept 7
Slept 8
Slept 9
2021-02-16 08:55:24,436:5259(0x7fa7a77fe700):ZOO_WARN@zookeeper_interest@1570: Exceeded deadline by 14ms
2021-02-16 08:55:24,436:5259(0x7fa7a77fe700):ZOO_DEBUG@zookeeper_process@2218: Got ping response in 0 ms
Slept 10
Slept 11
Slept 12

As you can see, after calling destructors on Table and Client, a connection to Zookeeper is still open, no other destructors are called.

If I'm calling createClient()  function multiple times, I can see consumed memory is growing and being released only after the program exits

 

Best regards,

Julia