[IMPALA-3885] Parquet files with multiple blocks cause remote reads

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Information Provided
• Affects Version/s: Impala 2.7.0
• Component/s: Backend
• Target Version: Impala 2.7.0
• Assignee: Sailesh Mukil
• Reporter: Lars Volker
• Created: 20/Jul/16 12:35
• Updated: 30/Aug/18 18:52
• Resolved: 31/Aug/16 18:52

Description

For parquet files with multiple blocks we schedule the scan ranges across all replicas of the blocks, aiming for local execution on impalads, which are running on the datanodes where the blocks are located. However there seems to be a high number of remote reads in these scenarios.

The scheduler makes local assignments:

I0613 16:43:01.741288 36424 simple-scheduler.cc:605] Total remote scan volume = 0
I0613 16:43:01.741441 36424 simple-scheduler.cc:607] Total local scan volume = 1426.95 GB
I0613 16:43:01.741576 36424 simple-scheduler.cc:609] Total cached scan volume = 0

However the profile shows remote scans:

         - RemoteScanRanges: 283 (283)
         - RowsRead: 139.36M (139355074)
         - RowsReturned: 239 (239)
         - RowsReturnedRate: 127.00 /sec
         - ScanRangesComplete: 304 (304)

Somehow Impala seems to read data from the wrong datanodes, possibly reading everything from the one where it read the footer from.

sailesh - We briefly talked about this before in person. Do you have time to look at this? If not feel free to re-assign it to me.

Issue Links

• relates to: IMPALA-3989 Display skew warning for poorly formatted Parquet files