Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-8153

Error Message points to wrong parent directory in case of path component name length error

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.5.2
    • Fix Version/s: 2.8.0, 2.7.1, 3.0.0-alpha1
    • Component/s: namenode
    • Labels:
      None
    • Target Version/s:

      Description

      If the name component length is greater than the permitted length, the error message points to wrong parent directory for mkdir and touchz.

      Here are examples where the parent directory name is in error message. In this example dfs.namenode.fs-limits.max-component-length is set to 19.

      hdfs dfs -mkdir /user/hrt_qa/FileNameLength/really_big_name_dir01
      mkdir: The maximum path component name limit of really_big_name_dir01 in directory /user/hrt_qa/ is exceeded: limit=19 length=21
      

      The expected value for the directory was /user/hrt_qa/FileNameLength. The same behavior is observed for touchz

      hdfs dfs -touchz /user/hrt_qa/FileNameLength/really_big_name_0004
      touchz: The maximum path component name limit of really_big_name_0004 in directory /user/hrt_qa/ is exceeded: limit=19 length=20
      

        Activity

        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2116 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2116/)
        HDFS-8153. Error Message points to wrong parent directory in case of path component name length error. Contributed by Anu Engineer. (jitendra: rev 369ddc67bdaf61cca3f2f766ab504e2932f6fb72)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsLimits.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2116 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2116/ ) HDFS-8153 . Error Message points to wrong parent directory in case of path component name length error. Contributed by Anu Engineer. (jitendra: rev 369ddc67bdaf61cca3f2f766ab504e2932f6fb72) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsLimits.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #167 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/167/)
        HDFS-8153. Error Message points to wrong parent directory in case of path component name length error. Contributed by Anu Engineer. (jitendra: rev 369ddc67bdaf61cca3f2f766ab504e2932f6fb72)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsLimits.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #167 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/167/ ) HDFS-8153 . Error Message points to wrong parent directory in case of path component name length error. Contributed by Anu Engineer. (jitendra: rev 369ddc67bdaf61cca3f2f766ab504e2932f6fb72) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsLimits.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #157 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/157/)
        HDFS-8153. Error Message points to wrong parent directory in case of path component name length error. Contributed by Anu Engineer. (jitendra: rev 369ddc67bdaf61cca3f2f766ab504e2932f6fb72)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsLimits.java
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #157 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/157/ ) HDFS-8153 . Error Message points to wrong parent directory in case of path component name length error. Contributed by Anu Engineer. (jitendra: rev 369ddc67bdaf61cca3f2f766ab504e2932f6fb72) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsLimits.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk #2098 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2098/)
        HDFS-8153. Error Message points to wrong parent directory in case of path component name length error. Contributed by Anu Engineer. (jitendra: rev 369ddc67bdaf61cca3f2f766ab504e2932f6fb72)

        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsLimits.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2098 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2098/ ) HDFS-8153 . Error Message points to wrong parent directory in case of path component name length error. Contributed by Anu Engineer. (jitendra: rev 369ddc67bdaf61cca3f2f766ab504e2932f6fb72) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsLimits.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk #900 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/900/)
        HDFS-8153. Error Message points to wrong parent directory in case of path component name length error. Contributed by Anu Engineer. (jitendra: rev 369ddc67bdaf61cca3f2f766ab504e2932f6fb72)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsLimits.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #900 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/900/ ) HDFS-8153 . Error Message points to wrong parent directory in case of path component name length error. Contributed by Anu Engineer. (jitendra: rev 369ddc67bdaf61cca3f2f766ab504e2932f6fb72) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsLimits.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #166 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/166/)
        HDFS-8153. Error Message points to wrong parent directory in case of path component name length error. Contributed by Anu Engineer. (jitendra: rev 369ddc67bdaf61cca3f2f766ab504e2932f6fb72)

        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsLimits.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #166 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/166/ ) HDFS-8153 . Error Message points to wrong parent directory in case of path component name length error. Contributed by Anu Engineer. (jitendra: rev 369ddc67bdaf61cca3f2f766ab504e2932f6fb72) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsLimits.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
        Hide
        jnp Jitendra Nath Pandey added a comment -

        I have committed this to trunk, branch-2 and branch-2.7. Thanks Anu Engineer.

        Show
        jnp Jitendra Nath Pandey added a comment - I have committed this to trunk, branch-2 and branch-2.7. Thanks Anu Engineer .
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #7603 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7603/)
        HDFS-8153. Error Message points to wrong parent directory in case of path component name length error. Contributed by Anu Engineer. (jitendra: rev 369ddc67bdaf61cca3f2f766ab504e2932f6fb72)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsLimits.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #7603 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7603/ ) HDFS-8153 . Error Message points to wrong parent directory in case of path component name length error. Contributed by Anu Engineer. (jitendra: rev 369ddc67bdaf61cca3f2f766ab504e2932f6fb72) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsLimits.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Hide
        anu Anu Engineer added a comment -

        I re-ran these tests on my local machine and they did pass. The error on these test says Time out, and it quite possible that the failure is due to test infra issues.

        --Anu

        Show
        anu Anu Engineer added a comment - I re-ran these tests on my local machine and they did pass. The error on these test says Time out, and it quite possible that the failure is due to test infra issues. --Anu
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12725698/hdfs-8153.001.patch
        against trunk revision 80a2a12.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

        org.apache.hadoop.hdfs.TestEncryptionZonesWithKMS

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/10288//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/10288//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12725698/hdfs-8153.001.patch against trunk revision 80a2a12. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.TestEncryptionZonesWithKMS Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/10288//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/10288//console This message is automatically generated.
        Hide
        jnp Jitendra Nath Pandey added a comment -

        I will commit it after test results come out satisfactorily.

        Show
        jnp Jitendra Nath Pandey added a comment - I will commit it after test results come out satisfactorily.
        Hide
        jnp Jitendra Nath Pandey added a comment -

        +1. I will commit it shortly.

        Show
        jnp Jitendra Nath Pandey added a comment - +1. I will commit it shortly.
        Hide
        anu Anu Engineer added a comment -

        It is not really broken, the reason why that code looks different is the semantics of getINode seems to be different from getPath(int pos). In the second call we send the pos as the max length of path component, and if you call getPath() without any argument then the pos parameter passed to DFSUtil.byteArray2Path string is the length of the path components.

        It took me a while to notice this subtle difference, I did another fix in DFSUtil but that cause some test failure, hence this minimal invasive fix.

        Show
        anu Anu Engineer added a comment - It is not really broken, the reason why that code looks different is the semantics of getINode seems to be different from getPath(int pos). In the second call we send the pos as the max length of path component, and if you call getPath() without any argument then the pos parameter passed to DFSUtil.byteArray2Path string is the length of the path components. It took me a while to notice this subtle difference, I did another fix in DFSUtil but that cause some test failure, hence this minimal invasive fix.
        Hide
        jnp Jitendra Nath Pandey added a comment -

        Anu Engineer,
        The fix looks ok to me. However, this means that following line is also broken
        final INodeDirectory parent = existing.getINode(pos - 1).asDirectory();
        which is used for verifyMaxDirItems(parent, parentPath); in the same method.

        If that is the case, we should file another jira to track the fix for that too.

        Show
        jnp Jitendra Nath Pandey added a comment - Anu Engineer , The fix looks ok to me. However, this means that following line is also broken final INodeDirectory parent = existing.getINode(pos - 1).asDirectory(); which is used for verifyMaxDirItems(parent, parentPath); in the same method. If that is the case, we should file another jira to track the fix for that too.
        Hide
        anu Anu Engineer added a comment -

        With this patch applied the directory names are as expected.

        $ ./bin/hdfs dfs -mkdir /user/hrt_qa

        $ ./bin/hdfs dfs -mkdir /user/hrt_qa/FileNameLength

        $ ./bin/hdfs dfs -mkdir /user/hrt_qa/FileNameLength/ReallyLongFileName_003_Fail
        mkdir: The maximum path component name limit of ReallyLongFileName_003_Fail in directory /user/hrt_qa/FileNameLength is exceeded: limit=20 length=27

        $ ./bin/hdfs dfs -touchz /user/hrt_qa/FileNameLength/ReallyLongFileName_003_Fail
        touchz: The maximum path component name limit of ReallyLongFileName_003_Fail in directory /user/hrt_qa/FileNameLength is exceeded: limit=20 length=27

        Show
        anu Anu Engineer added a comment - With this patch applied the directory names are as expected. $ ./bin/hdfs dfs -mkdir /user/hrt_qa $ ./bin/hdfs dfs -mkdir /user/hrt_qa/FileNameLength $ ./bin/hdfs dfs -mkdir /user/hrt_qa/FileNameLength/ReallyLongFileName_003_Fail mkdir: The maximum path component name limit of ReallyLongFileName_003_Fail in directory /user/hrt_qa/FileNameLength is exceeded: limit=20 length=27 $ ./bin/hdfs dfs -touchz /user/hrt_qa/FileNameLength/ReallyLongFileName_003_Fail touchz: The maximum path component name limit of ReallyLongFileName_003_Fail in directory /user/hrt_qa/FileNameLength is exceeded: limit=20 length=27

          People

          • Assignee:
            anu Anu Engineer
            Reporter:
            anu Anu Engineer
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development