Details
-
Sub-task
-
Status: Closed
-
Major
-
Resolution: Incomplete
-
None
-
None
-
None
-
None
Description
Hello everyone,
First of all, thank you for creating such an amazing product
I'm trying to implement an open-source PHP-extension based on HBase native client. It seems more promising performance-wise than Thrifts we're currently using
In terms of PHP-fpm, we need to be able to open and close connection multiple times in one process, and here's when the library leaks memory
Calling 'Close' methods on Table and Clients is not enough for closing all of the threads.
A simple program
int main() { zoo_set_debug_level( (ZooLogLevel)4); google::SetCommandLineOption("GLOG_minloglevel", "0"); std::shared_ptr<hbase::Configuration> conf = std::make_shared<hbase::Configuration>(); conf->Set(hbase::ZKUtil::kHBaseZookeeperQuorum_, "quorum"); conf->Set(hbase::ZKUtil::kHBaseZookeeperClientPort_, "2181"); conf->Set(hbase::ZKUtil::kHBaseZnodeParent_, "/hbase-unsecure"); createClient(conf); int i = 0; while (true) { i++; std::cout << "Slept " << i << std::endl; sleep(5); } return 0; } void createClient(const std::shared_ptr<hbase::Configuration>& conf) { std::cout << "opening " << std::endl << std::endl << std::endl; auto tn = std::make_shared<hbase::pb::TableName>(folly::to<hbase::pb::TableName>("table")); auto client = std::make_unique<hbase::Client>(*conf); auto table = client->Table(*tn); std::cout << "closing " << std::endl << std::endl << std::endl; table->Close(); client->Close(); }
Logs :
[julia@fdf346c7aa2b myhbase]$ ./myhbase opening WARNING: Logging before InitGoogleLogging() is written to STDERR I0216 08:54:44.391023 5259 location-cache.cc:77] Connecting to ZooKeeper. Quorum:quorum 2021-02-16 08:54:44,391:5259(0x7fa88b38d3c0):ZOO_INFO@log_env@726: Client environment:zookeeper.version=zookeeper C client 3.4.8 2021-02-16 08:54:44,391:5259(0x7fa88b38d3c0):ZOO_INFO@log_env@730: Client environment:host.name=fdf346c7aa2b 2021-02-16 08:54:44,391:5259(0x7fa88b38d3c0):ZOO_INFO@log_env@737: Client environment:os.name=Linux 2021-02-16 08:54:44,391:5259(0x7fa88b38d3c0):ZOO_INFO@log_env@738: Client environment:os.arch=5.8.1-1.el7.elrepo.x86_64 2021-02-16 08:54:44,391:5259(0x7fa88b38d3c0):ZOO_INFO@log_env@739: Client environment:os.version=#1 SMP Tue Aug 11 12:01:11 EDT 2020 2021-02-16 08:54:44,391:5259(0x7fa88b38d3c0):ZOO_INFO@log_env@747: Client environment:user.name=(null) 2021-02-16 08:54:44,391:5259(0x7fa88b38d3c0):ZOO_INFO@log_env@755: Client environment:user.home=/home/julia 2021-02-16 08:54:44,391:5259(0x7fa88b38d3c0):ZOO_INFO@log_env@767: Client environment:user.dir=/usr/src/hbase/myhbase 2021-02-16 08:54:44,391:5259(0x7fa88b38d3c0):ZOO_INFO@zookeeper_init@800: Initiating client connection, host=quorum sessionTimeout=90000 watcher=(nil) sessionId=0 sessionPasswd=<null> context=(nil) flags=0 2021-02-16 08:54:44,400:5259(0x7fa88b38d3c0):ZOO_DEBUG@start_threads@221: starting threads... 2021-02-16 08:54:44,400:5259(0x7fa7a77fe700):ZOO_DEBUG@do_io@367: started IO thread 2021-02-16 08:54:44,400:5259(0x7fa7a6ffd700):ZOO_DEBUG@do_completion@459: started completion thread closing I0216 08:54:44.400823 5259 client.cc:56] called destructor on Client 2021-02-16 08:54:44,400:5259(0x7fa7a77fe700):ZOO_INFO@check_events@1728: initiated connection to server [192.168.197.12:2181] 2021-02-16 08:54:44,403:5259(0x7fa7a77fe700):ZOO_INFO@check_events@1775: session establishment complete on server [192.168.197.12:2181], sessionId=0x27640b8a9bd0932, negotiated timeout=60000 2021-02-16 08:54:44,403:5259(0x7fa7a77fe700):ZOO_DEBUG@check_events@1781: Calling a watcher for a ZOO_SESSION_EVENT and the state=ZOO_CONNECTED_STATE 2021-02-16 08:54:44,403:5259(0x7fa7a6ffd700):ZOO_DEBUG@process_completions@2132: Calling a watcher for node [], type = -1 event=ZOO_SESSION_EVENT I0216 08:54:44.403533 5259 table.cc:50] called destructor on Table Slept 1 Slept 2 Slept 3 Slept 4 Slept 5 2021-02-16 08:55:04,421:5259(0x7fa7a77fe700):ZOO_WARN@zookeeper_interest@1570: Exceeded deadline by 19ms 2021-02-16 08:55:04,421:5259(0x7fa7a77fe700):ZOO_DEBUG@zookeeper_process@2218: Got ping response in 0 ms Slept 6 Slept 7 Slept 8 Slept 9 2021-02-16 08:55:24,436:5259(0x7fa7a77fe700):ZOO_WARN@zookeeper_interest@1570: Exceeded deadline by 14ms 2021-02-16 08:55:24,436:5259(0x7fa7a77fe700):ZOO_DEBUG@zookeeper_process@2218: Got ping response in 0 ms Slept 10 Slept 11 Slept 12
As you can see, after calling destructors on Table and Client, a connection to Zookeeper is still open, no other destructors are called.
If I'm calling createClient() function multiple times, I can see consumed memory is growing and being released only after the program exits
Best regards,
Julia