[HDFS-5458] Datanode failed volume threshold ignored if exception is thrown in getDataDirsFromURIs

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0-alpha1
• Fix Version/s: 2.3.0
• Component/s: datanode
• Hadoop Flags: Reviewed
• Assignee: Mike Mellenthin
• Reporter: Andrew Wang
• Created: 04/Nov/13 23:34
• Updated: 12/May/16 18:12
• Resolved: 05/Nov/13 18:27

Description

Saw a stacktrace of datanode startup with a bad volume, where even listing directories would throw an IOException. The failed volume threshold was set to 1, but it would fatally error out in File#getCanonicalPath in getDataDirsFromURIs:

      File dir = new File(dirURI.getPath());
      try {
        dataNodeDiskChecker.checkDir(localFS, new Path(dir.toURI()));
        dirs.add(dir);
      } catch (IOException ioe) {
        LOG.warn("Invalid " + DFS_DATANODE_DATA_DIR_KEY + " "
            + dir + " : ", ioe);
        invalidDirs.append("\"").append(dir.getCanonicalPath()).append("\" ");
      }

Since getCanonicalPath can need to do I/O and thus throw an IOException, this catch clause doesn't properly protect startup from a failed volume.