Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-6394

GVO failed in "Waiting for HDFS replication"

    Details

      Description

      A GVO failed at this step in data loading: https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/942/console . I attached the log tail is:

      /test-warehouse/zipcode_incomes_seq_gzip/000000_0:  Under replicated BP-611512709-127.0.0.1-1515787500736:blk_1073751054_10230. Target Replicas is 3 but found 2 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).
      ..
      /test-warehouse/zipcode_incomes_seq_record_def/000000_0:  Under replicated BP-611512709-127.0.0.1-1515787500736:blk_1073750916_10092. Target Replicas is 3 but found 2 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).
      ..
      /test-warehouse/zipcode_incomes_seq_record_snap/000000_0:  Under replicated BP-611512709-127.0.0.1-1515787500736:blk_1073751714_10890. Target Replicas is 3 but found 2 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).
      ...
      /test-warehouse/zipcode_incomes_text_gzip/000000_0.gz:  Under replicated BP-611512709-127.0.0.1-1515787500736:blk_1073748020_7196. Target Replicas is 3 but found 2 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).
      ..
      /test-warehouse/zipcode_incomes_text_lzo/000000_0.lzo.index:  Under replicated BP-611512709-127.0.0.1-1515787500736:blk_1073752971_12147. Target Replicas is 3 but found 2 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).
      .Status: HEALTHY
       Total size:    8422045399 B
       Total dirs:    8441
       Total files:   10228
       Total symlinks:                0
       Total blocks (validated):      10249 (avg. block size 821743 B)
       Minimally replicated blocks:   10249 (100.0 %)
       Over-replicated blocks:        0 (0.0 %)
       Under-replicated blocks:       1523 (14.859986 %)
       Mis-replicated blocks:         0 (0.0 %)
       Default replication factor:    3
       Average block replication:     2.8514001
       Corrupt blocks:                0
       Missing replicas:              1523 (4.9533286 %)
       Number of data-nodes:          3
       Number of racks:               1
      FSCK ended at Fri Jan 12 20:57:37 UTC 2018 in 140 milliseconds
      
      
      The filesystem under path '/test-warehouse' is HEALTHY
      Some HDFS blocks are still under replicated after 30s.
      Some tests cannot pass without fully replicated blocks (IMPALA-3887).
      Failing the data loading.
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                tianyiwang Tianyi Wang
                Reporter:
                tarmstrong Tim Armstrong
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: