Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-15619 Über-JIRA: S3Guard Phase IV: Hadoop 3.3 features
  3. HADOOP-16433

S3Guard: Filter expired entries and tombstones when listing with MetadataStore#listChildren

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 3.3.0
    • Fix Version/s: None
    • Component/s: fs/s3
    • Labels:
      None
    • Target Version/s:

      Description

      Currently, we don't filter out entries in listChildren implementations.

      This can cause bugs and inconsistencies, so this should be fixed.
      It can lead to a status where we can't recover from the following:
      guarded and raw (OOB op) clients are doing ops to S3

      Guarded: touch /AAAA
      Guarded: touch /ZZZZ
      Guarded: rm /AAAA {{-> tombstone in MS}}
      RAW: touch /AAAA/file.ext {{-> file is hidden with a tombstone}}
      Guarded: ls / {{-> only ZZZZ will show up in the listing. }}
      

      After we change the following code

                final List<PathMetadata> metas = new ArrayList<>();
                for (Item item : items) {
                  DDBPathMetadata meta = itemToPathMetadata(item, username);
                  metas.add(meta);
                }
      

      to

                  // handle expiry - only add not expired entries to listing.
                  if (meta.getLastUpdated() == 0 ||
                      !meta.isExpired(ttlTimeProvider.getMetadataTtl(),
                      ttlTimeProvider.getNow())) {
                    metas.add(meta);
                  }
      

      we will filter out expired entries from the listing, so we can recover form these kind of OOB ops.

      Note: we have to handle the lastUpdated == 0 case, where the lastUpdated field is not filled in!
      Note: this can only be fixed cleanly after HADOOP-16383 is fixed because we need to have the TTLtimeProvider in MS to handle this internally.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                gabor.bota Gabor Bota
                Reporter:
                gabor.bota Gabor Bota
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: