[DRILL-7641] Convert Excel Reader to Use Streaming Reader

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.17.0
• Fix Version/s: 1.18.0
• Component/s: Storage - Text & CSV
• Labels: ready-to-commit
• Assignee: Charles Givre
• Reporter: Charles Givre
• Reviewer: Arina Ielchiieva
• Created: 12/Mar/20 23:56
• Updated: 30/Mar/20 20:44
• Resolved: 30/Mar/20 20:44

Description

The current implementation of the Excel reader uses the Apache POI reader, which uses excessive amounts of memory. As a result, attempting to read large Excel files will cause out of memory errors. 

This PR converts the format plugin to use a streaming reader, based still on the POI library.  The documentation for the streaming reader can be found here. [1]

All unit tests pass and I tested the plugin with some large Excel files on my computer.

[1]: https://github.com/pjfanning/excel-streaming-reader

 

Issue Links

• fixes: DRILL-7423 Create More Efficient Way to Read Excel Cells
• links to: GitHub Pull Request #2024