[DRILL-5160] Memory leak in Parquet async reader when Snappy fails

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.9.0
• Fix Version/s: 1.11.0
• Assignee: Kunal Khatua
• Reporter: Paul Rogers
• Reviewer: Parth Chandra
• Created: 24/Dec/16 00:59
• Updated: 26/Jul/17 21:45
• Resolved: 26/Jul/17 21:40

Description

See the details in DRILL-5157. The Parquet async reader uses the Snappy library. If the call into Snappy fails (in my case, due to missing dependency management in the Drill pom.xml file), the Parquet reader fails (as it should.)

Unfortunately, the Parquet reader leaks memory which cause IllegalStateException errors in production, assertion failures in unit tests.

To reproduce this the easy way (to avoid the need to undo the fix for DRILL-5157 in Snappy dependencies), use Drill's exception injector in inject an exception at the line where we call into Snappy (See DRILL-5157 for details.)

Set a breakpoint exception for IllegalStateException you will see the memory leak.

Issue Links

• is related to: DRILL-5569 NullPointerException in Async Parquet reader
• relates to: DRILL-5435 Using Limit causes Memory Leaked Error since 1.10
• relates to: DRILL-5420 all cores at 100% of all servers