[AMBARI-12135] After HDP installation, hdfs fsck reporting CORRUPT status (Missing blocks)

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Duplicate
• Affects Version/s: 2.1.0
• Component/s: blueprints

Environment

ambari-server-2.1.0-1231

• Assignee: Unassigned
• Reporter: Sudhir Prakash
• Created: 24/Jun/15 20:40
• Updated: 23/Jul/15 17:08
• Resolved: 23/Jul/15 17:07

Description

This issue is 100% reproducible and has been happening on Ambari/HDP builds for at least a couple weeks. After completing an installation (Using blueprint), the hdfs fsck / reports as CORRUPT. Even after letting the cluster sit idle for a couple hours, it is still in the corrupt state

hades1:/var/opt/teradata/packages # su hdfs
hdfs@hades1:/var/opt/teradata/packages> hdfs fsck /
Connecting to namenode via http://hades1.labs.teradata.com:50070/fsck?ugi=hdfs&path=%2F
FSCK started by hdfs (auth:SIMPLE) from /39.0.8.2 for path / at Wed Jun 24 20:40:17 GMT 2015
...
/apps/hbase/data/WALs/hades4.labs.teradata.com,16020,1435168292684/hades4.labs.teradata.com%2C16020%2C1435168292684.default.1435175500556: MISSING 1 blocks of total size 83 B.
/apps/hbase/data/WALs/hades5.labs.teradata.com,16020,1435168290466/hades5.labs.teradata.com%2C16020%2C1435168290466..meta.1435175562144.meta: MISSING 1 blocks of total size 83 B.
/apps/hbase/data/WALs/hades5.labs.teradata.com,16020,1435168290466/hades5.labs.teradata.com%2C16020%2C1435168290466.default.1435175498500: MISSING 1 blocks of total size 83 B.
/apps/hbase/data/WALs/hades6.labs.teradata.com,16020,1435168292373/hades6.labs.teradata.com%2C16020%2C1435168292373.default.1435175500301: MISSING 1 blocks of total size 83 B..................................................................................................
....................................................................................................
....................................................................................................
........................................................................................Status: CORRUPT
 Total size:    723977553 B (Total open files size: 332 B)
 Total dirs:    79
 Total files:   388
 Total symlinks:                0 (Files currently being written: 5)
 Total blocks (validated):      387 (avg. block size 1870743 B) (Total open file blocks (not validated): 4)
  ********************************
  UNDER MIN REPL'D BLOCKS:      4 (1.0335917 %)
  dfs.namenode.replication.min: 1
  CORRUPT FILES:        4
  MISSING BLOCKS:       4
  MISSING SIZE:         332 B
  ********************************
 Minimally replicated blocks:   387 (100.0 %)
 Over-replicated blocks:        0 (0.0 %)
 Under-replicated blocks:       0 (0.0 %)
 Mis-replicated blocks:         0 (0.0 %)
 Default replication factor:    3
 Average block replication:     3.0
 Corrupt blocks:                0
 Missing replicas:              0 (0.0 %)
 Number of data-nodes:          3
 Number of racks:               1
FSCK ended at Wed Jun 24 20:40:17 GMT 2015 in 7 milliseconds


The filesystem under path '/' is CORRUPT
hdfs@hades1:/var/opt/teradata/packages>

Issue Links

• duplicates: HDFS-8809 HDFS fsck reports under construction blocks as "CORRUPT"