Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-2704

NameNodeResouceChecker#checkAvailableResources should check for inodes

    Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 0.24.0
    • Fix Version/s: None
    • Component/s: namenode
    • Labels:
      None

      Description

      NameNodeResouceChecker#checkAvailableResources currently just checks for free space. However inodes are also a file system resource that needs to be available (you can run out of inodes but have free space).

        Issue Links

          Activity

          Hide
          SreeHari added a comment -

          Will give a patch for this .

          Show
          SreeHari added a comment - Will give a patch for this .
          Hide
          Robert Joseph Evans added a comment -

          That is a great point. I would suggest that we output a warning/error the first time this happens for each disk and possibly include a metric for it. For all file systems that I know of that have a static number of inodes the ratio of inodes to disk space is configurable and even if it requires reformatting the drive, knowing that it is a problem will allow operations to possibly adjust that ratio appropriately.

          Show
          Robert Joseph Evans added a comment - That is a great point. I would suggest that we output a warning/error the first time this happens for each disk and possibly include a metric for it. For all file systems that I know of that have a static number of inodes the ratio of inodes to disk space is configurable and even if it requires reformatting the drive, knowing that it is a problem will allow operations to possibly adjust that ratio appropriately.
          Hide
          Uma Maheswara Rao G added a comment -

          Hey SreeHari, Any progress on this issue?

          Show
          Uma Maheswara Rao G added a comment - Hey SreeHari, Any progress on this issue?
          Hide
          SreeHari added a comment -

          Is a low number of fs inodes a problem for namenode ? Since namenode doesn't create new files in the local filesystem (unlike datanode) while it is running , will it be fine if we put nn into safe mode when system run out of inodes ? Or are we talking about just logging a warning ?

          Show
          SreeHari added a comment - Is a low number of fs inodes a problem for namenode ? Since namenode doesn't create new files in the local filesystem (unlike datanode) while it is running , will it be fine if we put nn into safe mode when system run out of inodes ? Or are we talking about just logging a warning ?
          Hide
          Robert Joseph Evans added a comment -

          I personally would like to see it go into safemode, just because you never know what in the system will start to have problems when you run out of inodes, or any other resource for that matter. I would hope most standard things would fail gracefully, but you never know and having a corrupted edit log because we tried to roll them when there were no inodes available is a very bad thing.

          Show
          Robert Joseph Evans added a comment - I personally would like to see it go into safemode, just because you never know what in the system will start to have problems when you run out of inodes, or any other resource for that matter. I would hope most standard things would fail gracefully, but you never know and having a corrupted edit log because we tried to roll them when there were no inodes available is a very bad thing.
          Hide
          Harsh J added a comment -

          NameNode also does create local image/edits files upon every checkpointing and such - so the inode monitoring is surely relevant.

          Show
          Harsh J added a comment - NameNode also does create local image/edits files upon every checkpointing and such - so the inode monitoring is surely relevant.
          Hide
          Aaron T. Myers added a comment -

          having a corrupted edit log because we tried to roll them when there were no inodes available is a very bad thing.

          Certainly true, though it seems unlikely that this would result in a truly corrupt edit log that the NN couldn't restart from, since running out of inodes would presumably result in no file being created at all, which isn't an invalid edit log.

          NameNode also does create local image/edits files upon every checkpointing and such - so the inode monitoring is surely relevant.

          +1

          Show
          Aaron T. Myers added a comment - having a corrupted edit log because we tried to roll them when there were no inodes available is a very bad thing. Certainly true, though it seems unlikely that this would result in a truly corrupt edit log that the NN couldn't restart from, since running out of inodes would presumably result in no file being created at all, which isn't an invalid edit log. NameNode also does create local image/edits files upon every checkpointing and such - so the inode monitoring is surely relevant. +1
          Hide
          Robert Joseph Evans added a comment -

          Good point Aaron

          Show
          Robert Joseph Evans added a comment - Good point Aaron

            People

            • Assignee:
              SreeHari
              Reporter:
              Eli Collins
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:

                Development