[IMPALA-7363] Spurious error generated by sequence file scanner with weird scan range length

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: Impala 3.1.0
• Fix Version/s: Impala 3.1.0
• Component/s: Backend
• Labels: avro
• Target Version: Product Backlog
• Epic Color: ghx-label-3
• Assignee: Pooja Nilangekar
• Reporter: Tim Armstrong
• Created: 26/Jul/18 21:17
• Updated: 01/Nov/18 16:12
• Resolved: 01/Nov/18 16:12

Description

Repro on master

tarmstrong@tarmstrong-box:~/Impala/incubator-impala$ impala-shell.sh
Starting Impala Shell without Kerberos authentication
Connected to localhost:21000
Server version: impalad version 3.1.0-SNAPSHOT DEBUG (build cec33fa0ae75392668273d40b5a1bc4bbd7e9e2e)
***********************************************************************************
Welcome to the Impala shell.
(Impala Shell v3.1.0-SNAPSHOT (cec33fa) built on Thu Jul 26 09:50:10 PDT 2018)

To see a summary of a query's progress that updates in real-time, run 'set
LIVE_PROGRESS=1;'.
***********************************************************************************
[localhost:21000] default> use tpch_seq_snap;
Query: use tpch_seq_snap
[localhost:21000] tpch_seq_snap> SET max_scan_range_length=5377;
MAX_SCAN_RANGE_LENGTH set to 5377
[localhost:21000] tpch_seq_snap> select count(*)
                               > from lineitem;
Query: select count(*)
from lineitem
Query submitted at: 2018-07-26 14:10:18 (Coordinator: http://tarmstrong-box:25000)
Query progress can be monitored at: http://tarmstrong-box:25000/query_plan?query_id=e9428efe173ad2f4:84b66bdb00000000

+----------+
| count(*) |
+----------+
| 5993651  |
+----------+
WARNINGS: SkipText: length is negative
Problem parsing file hdfs://localhost:20500/test-warehouse/tpch.lineitem_seq_snap/000000_0 at 36472193

Found while adding a test for IMPALA-7360

Issue Links

• relates to: IMPALA-7360 Avro scanner sometimes skips blocks when skip marker is on HDFS block boundary