[IMPALA-7556] Clean up ScanRange

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: Impala 4.1.0
• Component/s: Backend
• Labels: ramp-up
• Epic Color: ghx-label-8
• Assignee: Amogh Margoor
• Reporter: Zoltán Borók-Nagy
• Created: 11/Sep/18 09:59
• Updated: 10/Jun/21 16:55
• Resolved: 10/Jun/21 16:55

Description

For IMPALA-7543 I want to add some additional functionality to scan ranges.

However, the code of the ScanRange class is already quite messy. It handles different types of files, does some buffer management, updates all kinds of counters.

So, instead of complicating the code further, let's refactor the ScanRange class a bit.

• Do the file operations in separate classes
  • A new, abstract class could be invented to provide an API for file operations, i.e. Open(), ReadFromPos(), Close(), etc.
    • Keep in mind that the interface must be a good fit for IMPALA-7543, i.e. we need positional reads from files
  • Operations for local files and HDFS files could be implemented in child classes
• Buffer management
  • A new BufferStore class could be created
  • This new class would be responsible for managing the unused buffers
    • if possible, it would also handle the client and cached buffers as well
• Counters and metrics would be updated by the corresponding new classes
  • E.g. ImpaladMetrics::IO_MGR_NUM_OPEN_FILES would be updated by the file handling classes

Issue Links

• Blocked: IMPALA-7543 Enhance scan ranges to support sub-ranges