Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-13412

filesystem will throw out-of-bounds access for webhdfs using DirectoryListing

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 3.0.0
    • Fix Version/s: 3.0.0
    • Component/s: webhdfs
    • Labels:
      None
    • Target Version/s:

      Description

      In a big directory, there are a lot of entries; so we use LISTSTATUS_BATCH over webhdfs;

      and if the first batch is returned and preparing to query with startAfter but the left entries are deleted from the directory.

      for this scenario,the hadoop client will throw out-of-bounds access at line 'return (T)entries.getEntries()[i++];'

      @Override
      @SuppressWarnings("unchecked")
      public T next() throws IOException {
      Preconditions.checkState(hasNext(), "No more items in iterator");
      if (i == entries.getEntries().length)

      { fetchMore(); }

      return (T)entries.getEntries()[i++];
      }

      here is the exception stack:

       

      java.lang.ArrayIndexOutOfBoundsException: 0
      at org.apache.hadoop.fs.FileSystem$DirListingIterator.next(FileSystem.java:2120)
      at org.apache.hadoop.fs.FileSystem$DirListingIterator.next(FileSystem.java:2084)
      at org.apache.hadoop.hdfs.web.TestWebHdfsListBatch$1.run(TestWebHdfsListBatch.java:149)
      at org.apache.hadoop.hdfs.web.TestWebHdfsListBatch$1.run(TestWebHdfsListBatch.java:135)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:422)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
      at org.apache.hadoop.hdfs.web.TestWebHdfsListBatch.testLargeDirectory(TestWebHdfsListBatch.java:135)
      at org.apache.hadoop.hdfs.web.TestWebHdfsListBatch.main(TestWebHdfsListBatch.java:102)
      2018-03-30 22:53:00,731 DEBUG util.ShutdownHookManager: ShutdownHookManger complete shutdown.

       

       

       

       

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              nshen Shen, Neal
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:

                Time Tracking

                Estimated:
                Original Estimate - 96h
                96h
                Remaining:
                Remaining Estimate - 96h
                96h
                Logged:
                Time Spent - Not Specified
                Not Specified