Hadoop Common
  1. Hadoop Common
  2. HADOOP-240

namenode should not log failed mkdirs at warning level

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.2.1
    • Fix Version/s: 0.3.2
    • Component/s: None
    • Labels:
      None

      Description

      when a namenode creates a directory, it also recursively tries to creates its parent directories. If they already exist, the lastSuccess is false and the "error" is logged at the warning level.

      The right approach is to set the logging leve lower to fine or simply ignore the the "error".

      1. mkdir.patch
        3 kB
        Hairong Kuang
      2. mkdir.patch
        9 kB
        Hairong Kuang
      3. mkdir_new.patch
        15 kB

        Activity

        Hide
        Hairong Kuang added a comment -

        This patch fixes the problem by removing the logging when a namenode "fails" to create a parent directory. It also adjusts the logging level of some logs.

        Show
        Hairong Kuang added a comment - This patch fixes the problem by removing the logging when a namenode "fails" to create a parent directory. It also adjusts the logging level of some logs.
        Hide
        Hairong Kuang added a comment -

        On a second thought, the right approach to this problem should distiguish different cases when create a directory and then log them differently. When the mkdirs function is called, either the directory is successfully created, or the directory already existed, or the directory is not able to created due to its parent's problem.

        Currently FSDirectory.addNode returns null when a directory exists or a directory is not able to be created. I am thinking to add a PathNotFound exception. When a directory is not able to be created, addNode throws a PathNotFound exception. When a directory exists, it returns null. Otherwise, returns a new node.

        When the function mkdirs catches a PathNotFound exception, it logs it as a failure and returns. When it gets a null, simply continues.

        Show
        Hairong Kuang added a comment - On a second thought, the right approach to this problem should distiguish different cases when create a directory and then log them differently. When the mkdirs function is called, either the directory is successfully created, or the directory already existed, or the directory is not able to created due to its parent's problem. Currently FSDirectory.addNode returns null when a directory exists or a directory is not able to be created. I am thinking to add a PathNotFound exception. When a directory is not able to be created, addNode throws a PathNotFound exception. When a directory exists, it returns null. Otherwise, returns a new node. When the function mkdirs catches a PathNotFound exception, it logs it as a failure and returns. When it gets a null, simply continues.
        Hide
        Hairong Kuang added a comment -

        Just attached a new patch to this issue. Sorry that I forgot to log in.

        Show
        Hairong Kuang added a comment - Just attached a new patch to this issue. Sorry that I forgot to log in.
        Hide
        Doug Cutting added a comment -

        I'd like to apply this now, but it has conflicts. I'll try to work through them...

        Show
        Doug Cutting added a comment - I'd like to apply this now, but it has conflicts. I'll try to work through them...
        Hide
        Doug Cutting added a comment -

        Sorry. I am unable to get these into 0.3.1, they'll have to wait for 0.4.

        Show
        Doug Cutting added a comment - Sorry. I am unable to get these into 0.3.1, they'll have to wait for 0.4.
        Hide
        Hairong Kuang added a comment -

        Doug, I rebuilt the patch and here it is. I hope that you are able to apply it.

        Show
        Hairong Kuang added a comment - Doug, I rebuilt the patch and here it is. I hope that you are able to apply it.
        Hide
        Doug Cutting added a comment -

        I just committed this. Thanks, Hairong.

        Show
        Doug Cutting added a comment - I just committed this. Thanks, Hairong.

          People

          • Assignee:
            Hairong Kuang
            Reporter:
            Hairong Kuang
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development