Details
-
New Feature
-
Status: Open
-
Minor
-
Resolution: Unresolved
-
None
-
None
-
None
-
None
Description
We recently run into an issue where during an upgrade from Hadoop2 to Hadoop3, the filesystem under DataNode data directories was left in an intermediate state, and part of the directories were in Hadoop2 format other parts in Hadoop3 format.
At first we had to rollback the upgrade, and after the rollback we started to see FileNotFoundExceptions for particular block files.
The exception was logged on the DataNodes and failed the jobs as well sometimes. HDFS NameNode did not show any missing blocks, and we found the block files and meta files also in the DataNode's data directories, but at a different location.
This was the point when we realized that something went wrong during the rollback, and some of the data directories had blocks placed according to Hadoop3 rules, while other were placed according to Hadoop2 rules. We suspect a possible premature DataNode shutdown or an unknown failure during the rollback, but at the point when we realized what is the issue and could check into things, we already ran out of the logs that would have been able to show us the cause.
This JIRA is to suggest two new commands that can help administrators in this situation, to validate the data directory and ensure that blocks are placed correctly according to the rules, and to fix the data directory layout if needed.