Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-7122

Data load failure: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Cannot Reproduce
    • Affects Version/s: Impala 3.1.0
    • Fix Version/s: Not Applicable
    • Component/s: Infrastructure
    • Labels:

      Description

      20:58:29 Started Loading functional-query data in background; pid 6813.
      20:58:29 Loading functional-query data (logging to /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/load-functional-query.log)... 
      20:58:29 Started Loading TPC-H data in background; pid 6814.
      20:58:29 Loading TPC-H data (logging to /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/load-tpch.log)... 
      20:58:29 Started Loading TPC-DS data in background; pid 6815.
      20:58:29 Loading TPC-DS data (logging to /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/load-tpcds.log)... 
      21:35:26     FAILED (Took: 36 min 57 sec)
      21:35:26     'load-data functional-query exhaustive' failed. Tail of log:
      21:35:26 	at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:213)
      21:35:26 	at org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1086)
      21:35:26 18/06/04 21:20:29 WARN hdfs.DataStreamer: Error Recovery for BP-1407206351-127.0.0.1-1528170335185:blk_1073743620_2799 in pipeline [DatanodeInfoWithStorage[127.0.0.1:31000,DS-37cfc57c-ab39-443c-80c9-e440cb18b63d,DISK], DatanodeInfoWithStorage[127.0.0.1:31001,DS-2bc41558-4f2c-460f-ae87-5d1a6acbf42f,DISK], DatanodeInfoWithStorage[127.0.0.1:31002,DS-4ba4d3a0-af31-4eaf-b43d-89b408231481,DISK]]: datanode 0(DatanodeInfoWithStorage[127.0.0.1:31000,DS-37cfc57c-ab39-443c-80c9-e440cb18b63d,DISK]) is bad.
      21:35:26 18/06/04 21:21:29 INFO hdfs.DataStreamer: Exception in createBlockOutputStream blk_1073743620_2799
      21:35:26 java.io.IOException: Got error, status=ERROR, status message , ack with firstBadLink as 127.0.0.1:31002
      21:35:26 	at org.apache.hadoop.hdfs.protocol.datatransfer.DataTransferProtoUtil.checkBlockOpStatus(DataTransferProtoUtil.java:134)
      21:35:26 	at org.apache.hadoop.hdfs.protocol.datatransfer.DataTransferProtoUtil.checkBlockOpStatus(DataTransferProtoUtil.java:110)
      21:35:26 	at org.apache.hadoop.hdfs.DataStreamer.createBlockOutputStream(DataStreamer.java:1778)
      21:35:26 	at org.apache.hadoop.hdfs.DataStreamer.setupPipelineInternal(DataStreamer.java:1507)
      21:35:26 	at org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1481)
      21:35:26 	at org.apache.hadoop.hdfs.DataStreamer.processDatanodeOrExternalError(DataStreamer.java:1256)
      21:35:26 	at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:667)
      21:35:26 18/06/04 21:21:29 WARN hdfs.DataStreamer: Error Recovery for BP-1407206351-127.0.0.1-1528170335185:blk_1073743620_2799 in pipeline [DatanodeInfoWithStorage[127.0.0.1:31001,DS-2bc41558-4f2c-460f-ae87-5d1a6acbf42f,DISK], DatanodeInfoWithStorage[127.0.0.1:31002,DS-4ba4d3a0-af31-4eaf-b43d-89b408231481,DISK]]: datanode 1(DatanodeInfoWithStorage[127.0.0.1:31002,DS-4ba4d3a0-af31-4eaf-b43d-89b408231481,DISK]) is bad.
      21:35:26 18/06/04 21:21:29 WARN hdfs.DataStreamer: DataStreamer Exception
      21:35:26 java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[DatanodeInfoWithStorage[127.0.0.1:31001,DS-2bc41558-4f2c-460f-ae87-5d1a6acbf42f,DISK]], original=[DatanodeInfoWithStorage[127.0.0.1:31001,DS-2bc41558-4f2c-460f-ae87-5d1a6acbf42f,DISK]]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration.
      21:35:26 	at org.apache.hadoop.hdfs.DataStreamer.findNewDatanode(DataStreamer.java:1304)
      21:35:26 	at org.apache.hadoop.hdfs.DataStreamer.addDatanode2ExistingPipeline(DataStreamer.java:1372)
      21:35:26 	at org.apache.hadoop.hdfs.DataStreamer.handleDatanodeReplacement(DataStreamer.java:1598)
      21:35:26 	at org.apache.hadoop.hdfs.DataStreamer.setupPipelineInternal(DataStreamer.java:1499)
      21:35:26 	at org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1481)
      21:35:26 	at org.apache.hadoop.hdfs.DataStreamer.processDatanodeOrExternalError(DataStreamer.java:1256)
      21:35:26 	at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:667)
      21:35:26 put: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[DatanodeInfoWithStorage[127.0.0.1:31001,DS-2bc41558-4f2c-460f-ae87-5d1a6acbf42f,DISK]], original=[DatanodeInfoWithStorage[127.0.0.1:31001,DS-2bc41558-4f2c-460f-ae87-5d1a6acbf42f,DISK]]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration.
      21:35:26 18/06/04 21:24:25 INFO hdfs.DFSClient: Could not complete /test-warehouse/testescape_17_crlf/126._COPYING_ retrying...
      21:35:26 be loaded.
      21:35:26 Empty base table load for chars_tiny. Skipping load generation
      21:35:26 HDFS path: /test-warehouse/widetable_250_cols does not exists or is empty. Data will be loaded.
      21:35:26 HDFS path: /test-warehouse/widetable_500_cols does not exists or is empty. Data will be loaded.
      21:35:26 HDFS path: /test-warehouse/widetable_1000_cols does not exists or is empty. Data will be loaded.
      21:35:26 Skipping 'functional.avro_decimal_tbl' due to include constraint match.
      21:35:26 Skipping 'functional.no_avro_schema' due to include constraint match.
      21:35:26 HDFS path: /test-warehouse/table_no_newline does not exists or is empty. Data will be loaded.
      21:35:26 Empty base table load for table_no_newline. Skipping load generation
      21:35:26 HDFS path: /test-warehouse/table_no_newline_part does not exists or is empty. Data will be loaded.
      21:35:26 Empty base table load for table_no_newline_part. Skipping load generation
      21:35:26 HDFS path: /test-warehouse/testescape_16_lf does not exists or is empty. Data will be loaded.
      21:35:26 Empty base table load for testescape_16_lf. Skipping load generation
      21:35:26 HDFS path: /test-warehouse/testescape_16_crlf does not exists or is empty. Data will be loaded.
      21:35:26 Empty base table load for testescape_16_crlf. Skipping load generation
      21:35:26 HDFS path: /test-warehouse/testescape_17_lf does not exists or is empty. Data will be loaded.
      21:35:26 Empty base table load for testescape_17_lf. Skipping load generation
      21:35:26 Traceback (most recent call last):
      21:35:26   File "/data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/testdata/bin/generate-schema-statements.py", line 836, in <module>
      21:35:26     test_vectors, sections, include_constraints, exclude_constraints, only_constraints)
      21:35:26   File "/data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/testdata/bin/generate-schema-statements.py", line 595, in generate_statements
      21:35:26     load = eval_section(section['LOAD'])
      21:35:26   File "/data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/testdata/bin/generate-schema-statements.py", line 533, in eval_section
      21:35:26     assert p.returncode == 0
      21:35:26 AssertionError
      21:35:26 21:35:26 Error generating schema statements for workload: functional-query
      21:35:26 Background task Loading functional-query data (pid 6813) failed.
      21:48:12     FAILED (Took: 49 min 43 sec)
      21:48:12     'load-data tpch core' failed. Tail of log:
      21:48:13 20:59:36 Beginning execution of impala SQL: /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/create-tpch-core-impala-generated-text-none-none.sql
      21:48:13 20:59:36 Beginning execution of impala SQL: /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/create-tpch-core-impala-generated-text-gzip-block.sql
      21:48:13 20:59:36 Beginning execution of impala SQL: /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/create-tpch-core-impala-generated-seq-snap-block.sql
      21:48:13 20:59:36 Beginning execution of impala SQL: /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/create-tpch-core-impala-generated-avro-none-none.sql
      21:48:13 20:59:36 Beginning execution of impala SQL: /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/create-tpch-core-impala-generated-seq-gzip-block.sql
      21:48:13 20:59:36 Beginning execution of impala SQL: /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/create-tpch-core-impala-generated-avro-snap-block.sql
      21:48:13 20:59:36 Beginning execution of impala SQL: /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/create-tpch-core-impala-generated-parquet-none-none.sql
      21:48:13 20:59:36 Beginning execution of impala SQL: /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/create-tpch-core-impala-generated-rc-none-none.sql
      21:48:13 20:59:53 Finished execution of impala SQL: /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/create-tpch-core-impala-generated-parquet-none-none.sql
      21:48:13 20:59:53 Beginning execution of impala SQL: /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/create-tpch-core-impala-generated-orc-def-block.sql
      21:48:13 20:59:53 Finished execution of impala SQL: /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/create-tpch-core-impala-generated-text-none-none.sql
      21:48:13 20:59:53 Beginning execution of impala SQL: /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/create-tpch-core-impala-generated-kudu-none-none.sql
      21:48:13 20:59:53 Finished execution of impala SQL: /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/create-tpch-core-impala-generated-text-gzip-block.sql
      21:48:13 20:59:53 Finished execution of impala SQL: /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/create-tpch-core-impala-generated-seq-gzip-block.sql
      21:48:13 20:59:53 Finished execution of impala SQL: /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/create-tpch-core-impala-generated-avro-snap-block.sql
      21:48:13 20:59:54 Finished execution of impala SQL: /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/create-tpch-core-impala-generated-avro-none-none.sql
      21:48:13 20:59:59 Finished execution of impala SQL: /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/create-tpch-core-impala-generated-seq-snap-block.sql
      21:48:13 20:59:59 Finished execution of impala SQL: /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/create-tpch-core-impala-generated-rc-none-none.sql
      21:48:13 20:59:59 Finished execution of impala SQL: /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/create-tpch-core-impala-generated-orc-def-block.sql
      21:48:13 21:00:21 Finished execution of impala SQL: /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/create-tpch-core-impala-generated-kudu-none-none.sql
      21:48:13 21:00:21 Beginning execution of hive SQL: /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/load-tpch-core-hive-generated-text-none-none.sql
      21:48:13 21:01:21 Finished execution of hive SQL: /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/load-tpch-core-hive-generated-text-none-none.sql
      21:48:13 21:01:21 Beginning execution of hive SQL: /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/load-tpch-core-hive-generated-text-gzip-block.sql
      21:48:13 21:01:21 Beginning execution of hive SQL: /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/load-tpch-core-hive-generated-seq-gzip-block.sql
      21:48:13 21:01:21 Beginning execution of hive SQL: /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/load-tpch-core-hive-generated-rc-none-none.sql
      21:48:13 21:01:21 Beginning execution of hive SQL: /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/load-tpch-core-hive-generated-avro-none-none.sql
      21:48:13 21:01:21 Beginning execution of hive SQL: /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/load-tpch-core-hive-generated-seq-snap-block.sql
      21:48:13 21:01:21 Beginning execution of hive SQL: /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/load-tpch-core-hive-generated-avro-snap-block.sql
      21:48:13 21:01:21 Beginning execution of hive SQL: /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/load-tpch-core-hive-generated-orc-def-block.sql
      21:48:13 21:21:55 Finished execution of hive SQL: /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/load-tpch-core-hive-generated-seq-snap-block.sql
      21:48:13 21:22:22 Finished execution of hive SQL: /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/load-tpch-core-hive-generated-orc-def-block.sql
      21:48:13 21:26:17 Finished execution of hive SQL: /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/load-tpch-core-hive-generated-text-gzip-block.sql
      21:48:13 21:28:15 Finished execution of hive SQL: /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/load-tpch-core-hive-generated-rc-none-none.sql
      21:48:13 21:29:13 Finished execution of hive SQL: /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/load-tpch-core-hive-generated-seq-gzip-block.sql
      21:48:13 21:29:43 Finished execution of hive SQL: /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/load-tpch-core-hive-generated-avro-snap-block.sql
      21:48:13 21:37:08 Finished execution of hive SQL: /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/load-tpch-core-hive-generated-avro-none-none.sql
      21:48:13 21:37:08 Beginning execution of impala SQL: /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/invalidate-tpch-core-impala-generated.sql
      21:48:13 21:37:31 Finished execution of impala SQL: /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/invalidate-tpch-core-impala-generated.sql
      21:48:13 21:37:31 Beginning execution of impala SQL: /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/load-tpch-core-impala-generated-kudu-none-none.sql
      21:48:13 21:37:31 Beginning execution of impala SQL: /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/load-tpch-core-impala-generated-parquet-none-none.sql
      21:48:13 21:48:12 Error executing impala SQL: /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/load-tpch-core-impala-generated-parquet-none-none.sql See: /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/logs/data_loading/sql/tpch/load-tpch-core-impala-generated-parquet-none-none.sql.log
      

        Attachments

        1. impalad.ec2-m2-4xlarge-centos-6-4-0570.vpc.cloudera.com.jenkins.log.INFO.20180604-205755.5587
          2.20 MB
          Tim Armstrong
        2. load-functional-query.log
          6 kB
          Tim Armstrong
        3. data-load-functional-exhaustive.log
          15 kB
          Tim Armstrong
        4. hdfs-logs.tar.gz
          1.85 MB
          Tim Armstrong

          Activity

            People

            • Assignee:
              joemcdonnell Joe McDonnell
              Reporter:
              tarmstrong Tim Armstrong
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: