Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-16840

AliyunOSS: getFileStatus throws FileNotFoundException in versioning bucket

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.10.0, 2.9.2, 3.0.3, 3.2.1, 3.1.3
    • Fix Version/s: 3.0.4, 3.3.0, 2.9.3, 3.1.4, 3.2.2, 2.10.1
    • Component/s: fs/oss
    • Labels:
      None

      Description

      When hadoop lists object in versioning bucket with many delete marker in it, OSS will return

      <?xml version="1.0" encoding="UTF-8"?>
      <ListBucketResult>
        <Name>select-us-east-1</Name>
        <Prefix>test/hadoop/file/</Prefix>
        <Marker></Marker>
        <MaxKeys>100</MaxKeys>
        <Delimiter>/</Delimiter>
        <IsTruncated>true</IsTruncated>
        <NextMarker>test/hadoop/file/sub2</NextMarker>
      </ListBucketResult>
      

      It sets IsTruncated to true and without ObjectSummaries or CommonPrefixes, and will throw FileNotFoundException

      // code placeholder
      java.io.FileNotFoundException: oss://select-us-east-1/test/hadoop/file: No such file or directory!java.io.FileNotFoundException: oss://select-us-east-1/test/hadoop/file: No such file or directory!
       at org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystem.getFileStatus(AliyunOSSFileSystem.java:281) at org.apache.hadoop.fs.aliyun.oss.TestAliyunOSSFileSystemContract.testGetFileStatusInVersioningBucket

       

      // code placeholder
      ObjectListing listing = store.listObjects(key, 1, null, false);
      if (CollectionUtils.isNotEmpty(listing.getObjectSummaries()) ||
          CollectionUtils.isNotEmpty(listing.getCommonPrefixes())) {
        return new OSSFileStatus(0, true, 1, 0, 0, qualifiedPath, username);
      } else {
        throw new FileNotFoundException(path + ": No such file or directory!");
      }
      

       In this case, we should call listObjects until IsTruncated is false or ObjectSummaries is not empty or CommonPrefixes is not empty.

        Attachments

        1. HADOOP-16840.002.patch
          3 kB
          wujinhu
        2. HADOOP-16840.001.patch
          3 kB
          wujinhu

          Activity

            People

            • Assignee:
              wujinhu wujinhu
              Reporter:
              wujinhu wujinhu
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: