[IMPALA-6394] GVO failed in "Waiting for HDFS replication"

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: Impala 2.12.0
• Fix Version/s: Impala 2.12.0
• Component/s: Infrastructure
• Labels: broken-build flaky
• Target Version: Impala 2.12.0
• Epic Color: ghx-label-7
• Assignee: Tianyi Wang
• Reporter: Tim Armstrong
• Created: 12/Jan/18 23:45
• Updated: 12/Jun/18 22:40
• Resolved: 17/Apr/18 18:41

Description

A GVO failed at this step in data loading: https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/942/console . I attached the log tail is:

/test-warehouse/zipcode_incomes_seq_gzip/000000_0:  Under replicated BP-611512709-127.0.0.1-1515787500736:blk_1073751054_10230. Target Replicas is 3 but found 2 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).
..
/test-warehouse/zipcode_incomes_seq_record_def/000000_0:  Under replicated BP-611512709-127.0.0.1-1515787500736:blk_1073750916_10092. Target Replicas is 3 but found 2 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).
..
/test-warehouse/zipcode_incomes_seq_record_snap/000000_0:  Under replicated BP-611512709-127.0.0.1-1515787500736:blk_1073751714_10890. Target Replicas is 3 but found 2 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).
...
/test-warehouse/zipcode_incomes_text_gzip/000000_0.gz:  Under replicated BP-611512709-127.0.0.1-1515787500736:blk_1073748020_7196. Target Replicas is 3 but found 2 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).
..
/test-warehouse/zipcode_incomes_text_lzo/000000_0.lzo.index:  Under replicated BP-611512709-127.0.0.1-1515787500736:blk_1073752971_12147. Target Replicas is 3 but found 2 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).
.Status: HEALTHY
 Total size:    8422045399 B
 Total dirs:    8441
 Total files:   10228
 Total symlinks:                0
 Total blocks (validated):      10249 (avg. block size 821743 B)
 Minimally replicated blocks:   10249 (100.0 %)
 Over-replicated blocks:        0 (0.0 %)
 Under-replicated blocks:       1523 (14.859986 %)
 Mis-replicated blocks:         0 (0.0 %)
 Default replication factor:    3
 Average block replication:     2.8514001
 Corrupt blocks:                0
 Missing replicas:              1523 (4.9533286 %)
 Number of data-nodes:          3
 Number of racks:               1
FSCK ended at Fri Jan 12 20:57:37 UTC 2018 in 140 milliseconds


The filesystem under path '/test-warehouse' is HEALTHY
Some HDFS blocks are still under replicated after 30s.
Some tests cannot pass without fully replicated blocks (IMPALA-3887).
Failing the data loading.

Issue Links

• relates to: IMPALA-3887 Planner tests failing due to metadata loading race with HDFS, fewer #hosts than expected