[IMPALA-7145] Leak of memory from OpenSSL when spill-to-disk encryption is enabled

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: Impala 2.0, Impala 2.3.0, Impala 2.5.0, Impala 2.4.0, Impala 2.6.0, Impala 2.7.0, Impala 2.8.0, Impala 2.9.0, Impala 2.10.0, Impala 2.11.0, Impala 2.12.0, Impala 2.13.0, Impala 3.1.0, Impala 2.2.0, Impala 2.1.0
• Fix Version/s: Impala 2.13.0, Impala 3.1.0
• Component/s: Backend
• Labels: resource-management
• Target Version: Impala 2.13.0, Impala 3.1.0
• Epic Color: ghx-label-5
• Assignee: Tim Armstrong
• Reporter: Tim Armstrong
• Created: 07/Jun/18 17:49
• Updated: 13/Jun/18 15:48
• Resolved: 12/Jun/18 22:41

Description

I tried looping this test for a while to flush out a different issue and discovered a memory leak:

while impala-py.test tests/query_test/test_spilling.py -k naaj -n4 --verbose; do date; done

Eventually the test fails with "Memory Limit Exceeded" and there is a huge amount of untracked memory reported on /memz.

I tried running the heap growth profiler and the only suspicious thing I saw was CRYPTO_malloc, similar to IMPALA-6792. My current hypothesis is that it's something related to disk spill encryption and we haven't noticed it before because we had disk spill encryption off.

If I disable encryption "start-impala-cluster.py -impalad_args=-disk_spill_encryption=false" then I don't see the leak.

Issue Links

• relates to: IMPALA-2746 Backend tests should pass with leak sanitizer enabled