Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-8299

ViewFileSystem link slash mount point crashes with IndexOutOfBoundsException

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0-alpha1
    • Fix Version/s: 3.0.0-alpha2
    • Component/s: viewfs
    • Labels:
      None

      Description

      We currently assume a typical viewfs client configuration is a set of non-overlapping mounts. This means every time you want to add a new top-level directory you need to update the client-side mountable config. If users could specify a slash mount, and then add additional mounts as necessary they could add a new top-level directory without updating all client configs (as long as the new top-level directory was being created on the NN the slash mount points to). This could be achieved by HADOOP-8298 (merge mounts, since we're effectively merging all new mount points with slash) or having the notion of a "default NN" for a mount table.

      1. HADOOP-8299.02.patch
        3 kB
        Manoj Govindassamy
      2. HADOOP-8299.01.patch
        3 kB
        Manoj Govindassamy

        Issue Links

          Activity

          Hide
          eli2 Eli Collins added a comment -

          Here's the exception you get btw if you try to create a slash mount entry. We should make this a more friendly error message in the mean time.

          java.lang.ArrayIndexOutOfBoundsException: 1
          at org.apache.hadoop.fs.viewfs.InodeTree.createLink(InodeTree.java:237)
          at org.apache.hadoop.fs.viewfs.InodeTree.<init>(InodeTree.java:334)
          at org.apache.hadoop.fs.viewfs.ViewFileSystem$1.<init>(ViewFileSystem.java:165)
          at org.apache.hadoop.fs.viewfs.ViewFileSystem.initialize(ViewFileSystem.java:165)
          at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2041)
          at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:78)
          at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2075)
          at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2057)
          at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:284)
          at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:151)
          
          Show
          eli2 Eli Collins added a comment - Here's the exception you get btw if you try to create a slash mount entry. We should make this a more friendly error message in the mean time. java.lang.ArrayIndexOutOfBoundsException: 1 at org.apache.hadoop.fs.viewfs.InodeTree.createLink(InodeTree.java:237) at org.apache.hadoop.fs.viewfs.InodeTree.<init>(InodeTree.java:334) at org.apache.hadoop.fs.viewfs.ViewFileSystem$1.<init>(ViewFileSystem.java:165) at org.apache.hadoop.fs.viewfs.ViewFileSystem.initialize(ViewFileSystem.java:165) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2041) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:78) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2075) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2057) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:284) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:151)
          Hide
          manojg Manoj Govindassamy added a comment -

          ViewFileSystem support for LinkMergeSlash is now tracked by HADOOP-13055.

          Will make use of this bug to address the issue discussed in the previous comment – "link./" mount point throwing IndexOutOfBoundsException.

          Show
          manojg Manoj Govindassamy added a comment - ViewFileSystem support for LinkMergeSlash is now tracked by HADOOP-13055 . Will make use of this bug to address the issue discussed in the previous comment – "link./" mount point throwing IndexOutOfBoundsException.
          Hide
          manojg Manoj Govindassamy added a comment -

          Attached v01 patch to address the following:

          1. fs.viewfs.mounttable.<mtTableName>.link./ is no more an allowed mount point. ViewFileSystem construction will throw out an IOException recommending to use fs.viewfs.mounttable.<mtTableName>.linkMergeSlash instead
          2. Test to verify link./ configuration throwing IOException with recommendation.
          Show
          manojg Manoj Govindassamy added a comment - Attached v01 patch to address the following: fs.viewfs.mounttable.<mtTableName>.link./ is no more an allowed mount point. ViewFileSystem construction will throw out an IOException recommending to use fs.viewfs.mounttable.<mtTableName>.linkMergeSlash instead Test to verify link./ configuration throwing IOException with recommendation.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 14s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 6m 54s trunk passed
          +1 compile 6m 54s trunk passed
          +1 checkstyle 0m 25s trunk passed
          +1 mvnsite 0m 58s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 1m 20s trunk passed
          +1 javadoc 0m 44s trunk passed
          +1 mvninstall 0m 39s the patch passed
          +1 compile 6m 52s the patch passed
          +1 javac 6m 52s the patch passed
          +1 checkstyle 0m 24s the patch passed
          +1 mvnsite 0m 54s the patch passed
          +1 mvneclipse 0m 13s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 28s the patch passed
          +1 javadoc 0m 42s the patch passed
          +1 unit 8m 5s hadoop-common in the patch passed.
          +1 asflicense 0m 21s The patch does not generate ASF License warnings.
          38m 44s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HADOOP-8299
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12835054/HADOOP-8299.01.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 979a8b738b01 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 0a166b1
          Default Java 1.8.0_101
          findbugs v3.0.0
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10885/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10885/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 54s trunk passed +1 compile 6m 54s trunk passed +1 checkstyle 0m 25s trunk passed +1 mvnsite 0m 58s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 20s trunk passed +1 javadoc 0m 44s trunk passed +1 mvninstall 0m 39s the patch passed +1 compile 6m 52s the patch passed +1 javac 6m 52s the patch passed +1 checkstyle 0m 24s the patch passed +1 mvnsite 0m 54s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 28s the patch passed +1 javadoc 0m 42s the patch passed +1 unit 8m 5s hadoop-common in the patch passed. +1 asflicense 0m 21s The patch does not generate ASF License warnings. 38m 44s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HADOOP-8299 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12835054/HADOOP-8299.01.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 979a8b738b01 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 0a166b1 Default Java 1.8.0_101 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10885/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10885/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          manojg Manoj Govindassamy added a comment -

          Andrew Wang, can you please review v01 patch ?

          Show
          manojg Manoj Govindassamy added a comment - Andrew Wang , can you please review v01 patch ?
          Hide
          andrew.wang Andrew Wang added a comment -

          Thanks for working on this Manoj. Seems like this is dependent on HADOOP-13055, since the linkMergeSlash config won't work until that goes in. If you want to get this in first though, we can have the message say it's unsupported, and update the message in HADOOP-13055.

          Show
          andrew.wang Andrew Wang added a comment - Thanks for working on this Manoj. Seems like this is dependent on HADOOP-13055 , since the linkMergeSlash config won't work until that goes in. If you want to get this in first though, we can have the message say it's unsupported, and update the message in HADOOP-13055 .
          Hide
          manojg Manoj Govindassamy added a comment -

          Thats true, HADOOP-13055 is under code review and might not be in before this. So, attaching v02 patch with unsupported message for LinkMergeSlash and the test case modified accordingly. Thanks for the review.

          Show
          manojg Manoj Govindassamy added a comment - Thats true, HADOOP-13055 is under code review and might not be in before this. So, attaching v02 patch with unsupported message for LinkMergeSlash and the test case modified accordingly. Thanks for the review.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 15s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 7m 7s trunk passed
          +1 compile 6m 55s trunk passed
          +1 checkstyle 0m 25s trunk passed
          +1 mvnsite 0m 57s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 1m 20s trunk passed
          +1 javadoc 0m 43s trunk passed
          +1 mvninstall 0m 35s the patch passed
          +1 compile 6m 52s the patch passed
          +1 javac 6m 52s the patch passed
          +1 checkstyle 0m 23s the patch passed
          +1 mvnsite 0m 55s the patch passed
          +1 mvneclipse 0m 13s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 28s the patch passed
          +1 javadoc 0m 41s the patch passed
          +1 unit 8m 18s hadoop-common in the patch passed.
          +1 asflicense 0m 21s The patch does not generate ASF License warnings.
          39m 4s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HADOOP-8299
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12835239/HADOOP-8299.02.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux f2f58d2a1f85 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / d88dca8
          Default Java 1.8.0_101
          findbugs v3.0.0
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10897/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10897/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 7s trunk passed +1 compile 6m 55s trunk passed +1 checkstyle 0m 25s trunk passed +1 mvnsite 0m 57s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 20s trunk passed +1 javadoc 0m 43s trunk passed +1 mvninstall 0m 35s the patch passed +1 compile 6m 52s the patch passed +1 javac 6m 52s the patch passed +1 checkstyle 0m 23s the patch passed +1 mvnsite 0m 55s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 28s the patch passed +1 javadoc 0m 41s the patch passed +1 unit 8m 18s hadoop-common in the patch passed. +1 asflicense 0m 21s The patch does not generate ASF License warnings. 39m 4s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HADOOP-8299 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12835239/HADOOP-8299.02.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux f2f58d2a1f85 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / d88dca8 Default Java 1.8.0_101 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10897/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10897/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          andrew.wang Andrew Wang added a comment -

          LGTM will commit shortly

          Show
          andrew.wang Andrew Wang added a comment - LGTM will commit shortly
          Hide
          andrew.wang Andrew Wang added a comment -

          Committed to trunk, thanks Manoj for the patch!

          Show
          andrew.wang Andrew Wang added a comment - Committed to trunk, thanks Manoj for the patch!
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10692 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10692/)
          HADOOP-8299. ViewFileSystem link slash mount point crashes with (wang: rev 22ff0eff4d58ac0beda7a5a3ae0e5d108da14f7f)

          • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/InodeTree.java
          • (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFileSystemBaseTest.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10692 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10692/ ) HADOOP-8299 . ViewFileSystem link slash mount point crashes with (wang: rev 22ff0eff4d58ac0beda7a5a3ae0e5d108da14f7f) (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/InodeTree.java (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFileSystemBaseTest.java

            People

            • Assignee:
              manojg Manoj Govindassamy
              Reporter:
              eli2 Eli Collins
            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development