[IMPALA-3453] S3 : Uneven split sizes are generated for Parquet causing execution skew

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: Impala 2.6.0
• Fix Version/s: Impala 2.6.0
• Component/s: Backend
• Labels: performance s3
• Target Version: Impala 2.6.0
• Assignee: Mostafa Mokhtar
• Reporter: Mostafa Mokhtar
• Created: 29/Apr/16 03:27
• Updated: 23/Sep/19 23:37
• Resolved: 27/May/16 05:11

Description

With Impala on S3 unevenly sized splits are assigned to the scan nodes which introduces execution skew

  Averaged Fragment F00:(Total: 1m17s, non-child: 0.000ns, % non-child: 0.00%)
      split sizes:  min: 5.01 GB, max: 11.63 GB, avg: 5.91 GB, stddev: 1.08 GB
      completion times: min:5s442ms  max:2m17s  mean: 1m17s  stddev:48s312ms
      execution rates: min:47.64 MB/sec  max:1.06 GB/sec  mean:324.41 MB/sec  stddev:406.41 MB/sec
      num instances: 32

Running the same query against the exact HDFS layout doesn't produce skew.

Issue Links

• relates to: IMPALA-8942 Set file format specific values for split sizes on non-block stores