Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-12761 Ozone: Merge Ozone to trunk
  3. HDFS-12768

Ozone: Qualify OzoneFileSystem paths in Filesystem APIs

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Open
    • Major
    • Resolution: Unresolved
    • HDFS-7240
    • HDFS-7240
    • ozone
    • None

    Description

      This is based on stevel@apache.org's comments on HDFS-7240. This jira will be used to qualify filesystem paths before they are used, this jira will also address rest of the review comments for filesystem api as well.

      General
      1) various places use LOG.info("text " + something). they should all move to LOG.info("text {}", something)
      2) Once OzoneException -> IOE, you can cut the catch and translate here.
      3) qualify path before all uses. That's needed to stop them being relative, and to catch things like someone calling ozfs.rename("o3://bucket/src", "s3a://bucket/dest"), delete("s3a://bucket/path"), etc, as well as problems with validation happening before paths are made absolute.
      4) RenameIterator.iterate() it's going to log @ warn whenever it can't delete a temp file because it doesn't exist, which may be a distraction in failures. Better: if(!tmpFile.delete() && tmpFile.exists()), as that will only warn if the temp file is actually there.

      OzoneFileSystem.rename().
      1) Qualify all the paths before doing directory validation. Otherwise you can defeat the "don't rename into self checks" rename("/path/src", "/path/../path/src/dest").
      2) Log @ debu all the paths taken before returning so you can debug if needed.
      3) S3A rename ended up having a special RenameFailedException() which innerRename() raises, with text and return code. Outer rename logs the text and returns the return code. This means that all failing paths have an exception clearly thrown, and when we eventually make rename/3 public, it's lined up to throw exceptions back to the caller. Consider copying this code.

      OzoneFileSystem.delete
      1) qualify path before use
      2) dont' log at error if you can't delete a nonexistent path, it is used everywhere for silent cleanup. Cut it

      OzoneFileSystem.ListStatusIterator
      1) make status field final

      OzoneFileSystem.mkdir
      1) do qualify path first.

      OzoneFileSystem.getFileStatus
      1) getKeyInfo() catches all exceptions and maps to null, which is interpreted not found and eventually surfaces as FNFE. This is misleading if the failure is for any other reason.
      2) Once OzoneException -> IOException, getKeyInfo() should only catch & downgrade the explicit not found (404?) responses.

      OzoneFileSystem.listKeys()
      unless this needs to be tagged as VisibleForTesting, make private.

      OzoneFileSystem.getDefaultBlockSize()
      implement getDefaultBlockSize(); add a config option to let people set it. add a sensible default like 64 or 128 MB.

      Attachments

        Activity

          People

            msingh Mukul Kumar Singh
            msingh Mukul Kumar Singh
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated: