[SAMZA-984] Upgrade to RocksDB to 4.8.0

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.13.0
• Assignee: Prateek Maheshwari
• Reporter: Chris Pettitt
• Created: 29/Jul/16 17:43
• Updated: 17/Apr/17 03:18
• Resolved: 17/Apr/17 03:17

Description

We're currently on a ~9 month old build of RocksDB.

I was experimenting with setting limits on the amount of logging done from RocksDB with the keys:

• rocksdb.max.log.file.size.bytes
• rocksdb.keep.log.file.num

With these keys enabled I consistently get the java process to SIGABRT within about a minute with the following backtrace:

Thread 3 Crashed:
0   libsystem_kernel.dylib        	0x00007fff849cf002 __pthread_kill + 10
1   libsystem_pthread.dylib       	0x00007fff96c365c5 pthread_kill + 90
2   libsystem_c.dylib             	0x00007fff862446e7 abort + 129
3   libc++abi.dylib               	0x00007fff911dff81 abort_message + 257
4   libc++abi.dylib               	0x00007fff91205a2f default_terminate_handler() + 243
5   libobjc.A.dylib               	0x00007fff957d36c3 _objc_terminate() + 124
6   libc++abi.dylib               	0x00007fff9120319e std::__terminate(void (*)()) + 8
7   libc++abi.dylib               	0x00007fff91202c12 __cxa_throw + 121
8   libc++.1.dylib                	0x00007fff94e79781 std::__1::__vector_base_common<true>::__throw_out_of_range() const + 71
9   librocksdbjni213472405106490933..jnilib	0x000000012c51503f rocksdb::DBImpl::PurgeObsoleteFiles(rocksdb::JobContext const&) + 3535
10  librocksdbjni213472405106490933..jnilib	0x000000012c516403 rocksdb::DBImpl::DeleteObsoleteFiles() + 531

Upgrading to a more recent version of RocksDB (i.e. 4.8.0) resolves this issue. The upgrade worked with a drop in replacement.

To properly do upgrade we'd need:

• Integration testing (do we have something for this?)
• Perf testing

Issue Links

• contains: SAMZA-955 Upgrade RocksDB JNI to 4.5.1