Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.7.0
    • Fix Version/s: 2.7.0
    • Component/s: documentation
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      With the addition of a major new feature to filesystems, the filesystem specification in hadoop-common/site is now out of sync.

      This means that

      1. there's no strict specification of what it should do
      2. you can't derive tests from that specification
      3. other people trying to implement the API will have to infer what to do from the HDFS source
      4. there's no way to decide whether or not the HDFS implementation does what it is intended.
      5. without matching tests against the raw local FS, differences between the HDFS impl and the Posix standard one won't be caught until it is potentially too late to fix.

      The operation should be relatively easy to define (after a truncate, the files bytes [0...len-1] must equal the original bytes, length(file)==len, etc)

      The truncate tests already written could then be pulled up into contract tests which any filesystem implementation can run against.

      1. HDFS-7665.patch
        3 kB
        Konstantin Shvachko
      2. HADOOP-11579-01.patch
        3 kB
        Konstantin Shvachko
      3. HADOOP-11579-branch-2.patch
        3 kB
        Konstantin Shvachko

        Issue Links

          Activity

          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Hdfs-trunk-Java8 #97 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/97/)
          HADOOP-11579. Documentation for truncate. Contributed by Konstantin Shvachko. (shv: rev 6f6737b3bfda0f5c0b0a508dc8523f6636f3f4ab)

          • hadoop-common-project/hadoop-common/src/site/markdown/FileSystemShell.md
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/site/markdown/filesystem/filesystem.md
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk-Java8 #97 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/97/ ) HADOOP-11579 . Documentation for truncate. Contributed by Konstantin Shvachko. (shv: rev 6f6737b3bfda0f5c0b0a508dc8523f6636f3f4ab) hadoop-common-project/hadoop-common/src/site/markdown/FileSystemShell.md hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/site/markdown/filesystem/filesystem.md
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2053 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2053/)
          HADOOP-11579. Documentation for truncate. Contributed by Konstantin Shvachko. (shv: rev 6f6737b3bfda0f5c0b0a508dc8523f6636f3f4ab)

          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/site/markdown/filesystem/filesystem.md
          • hadoop-common-project/hadoop-common/src/site/markdown/FileSystemShell.md
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2053 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2053/ ) HADOOP-11579 . Documentation for truncate. Contributed by Konstantin Shvachko. (shv: rev 6f6737b3bfda0f5c0b0a508dc8523f6636f3f4ab) hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/site/markdown/filesystem/filesystem.md hadoop-common-project/hadoop-common/src/site/markdown/FileSystemShell.md
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #103 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/103/)
          HADOOP-11579. Documentation for truncate. Contributed by Konstantin Shvachko. (shv: rev 6f6737b3bfda0f5c0b0a508dc8523f6636f3f4ab)

          • hadoop-common-project/hadoop-common/src/site/markdown/FileSystemShell.md
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/site/markdown/filesystem/filesystem.md
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #103 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/103/ ) HADOOP-11579 . Documentation for truncate. Contributed by Konstantin Shvachko. (shv: rev 6f6737b3bfda0f5c0b0a508dc8523f6636f3f4ab) hadoop-common-project/hadoop-common/src/site/markdown/FileSystemShell.md hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/site/markdown/filesystem/filesystem.md
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2034 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2034/)
          HADOOP-11579. Documentation for truncate. Contributed by Konstantin Shvachko. (shv: rev 6f6737b3bfda0f5c0b0a508dc8523f6636f3f4ab)

          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/site/markdown/filesystem/filesystem.md
          • hadoop-common-project/hadoop-common/src/site/markdown/FileSystemShell.md
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2034 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2034/ ) HADOOP-11579 . Documentation for truncate. Contributed by Konstantin Shvachko. (shv: rev 6f6737b3bfda0f5c0b0a508dc8523f6636f3f4ab) hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/site/markdown/filesystem/filesystem.md hadoop-common-project/hadoop-common/src/site/markdown/FileSystemShell.md
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk #836 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/836/)
          HADOOP-11579. Documentation for truncate. Contributed by Konstantin Shvachko. (shv: rev 6f6737b3bfda0f5c0b0a508dc8523f6636f3f4ab)

          • hadoop-common-project/hadoop-common/src/site/markdown/FileSystemShell.md
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/site/markdown/filesystem/filesystem.md
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #836 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/836/ ) HADOOP-11579 . Documentation for truncate. Contributed by Konstantin Shvachko. (shv: rev 6f6737b3bfda0f5c0b0a508dc8523f6636f3f4ab) hadoop-common-project/hadoop-common/src/site/markdown/FileSystemShell.md hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/site/markdown/filesystem/filesystem.md
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #102 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/102/)
          HADOOP-11579. Documentation for truncate. Contributed by Konstantin Shvachko. (shv: rev 6f6737b3bfda0f5c0b0a508dc8523f6636f3f4ab)

          • hadoop-common-project/hadoop-common/src/site/markdown/filesystem/filesystem.md
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/site/markdown/FileSystemShell.md
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #102 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/102/ ) HADOOP-11579 . Documentation for truncate. Contributed by Konstantin Shvachko. (shv: rev 6f6737b3bfda0f5c0b0a508dc8523f6636f3f4ab) hadoop-common-project/hadoop-common/src/site/markdown/filesystem/filesystem.md hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/site/markdown/FileSystemShell.md
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #7075 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7075/)
          HADOOP-11579. Documentation for truncate. Contributed by Konstantin Shvachko. (shv: rev 6f6737b3bfda0f5c0b0a508dc8523f6636f3f4ab)

          • hadoop-common-project/hadoop-common/src/site/markdown/filesystem/filesystem.md
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/site/markdown/FileSystemShell.md
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #7075 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7075/ ) HADOOP-11579 . Documentation for truncate. Contributed by Konstantin Shvachko. (shv: rev 6f6737b3bfda0f5c0b0a508dc8523f6636f3f4ab) hadoop-common-project/hadoop-common/src/site/markdown/filesystem/filesystem.md hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/site/markdown/FileSystemShell.md
          Hide
          shv Konstantin Shvachko added a comment -

          I just committed this.

          Show
          shv Konstantin Shvachko added a comment - I just committed this.
          Hide
          shv Konstantin Shvachko added a comment -

          Patch for branch-2.

          Show
          shv Konstantin Shvachko added a comment - Patch for branch-2.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          +1
          Thanks for doing this!

          Show
          stevel@apache.org Steve Loughran added a comment - +1 Thanks for doing this!
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12698014/HADOOP-11579-01.patch
          against trunk revision 7c6b654.

          +1 @author. The patch does not contain any @author tags.

          +0 tests included. The patch appears to be a documentation patch that doesn't require tests.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 2 new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5653//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/5653//artifact/patchprocess/newPatchFindbugsWarningshadoop-common.html
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5653//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12698014/HADOOP-11579-01.patch against trunk revision 7c6b654. +1 @author . The patch does not contain any @author tags. +0 tests included . The patch appears to be a documentation patch that doesn't require tests. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 2 new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5653//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/5653//artifact/patchprocess/newPatchFindbugsWarningshadoop-common.html Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5653//console This message is automatically generated.
          Hide
          shv Konstantin Shvachko added a comment -

          New patch to reflect current trunk.

          Show
          shv Konstantin Shvachko added a comment - New patch to reflect current trunk.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12697861/HDFS-7665.patch
          against trunk revision 7c6b654.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5649//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12697861/HDFS-7665.patch against trunk revision 7c6b654. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5649//console This message is automatically generated.
          Hide
          shv Konstantin Shvachko added a comment -

          Moved to hadoop-common.
          Changed the Summary, which was
          "Add definition of truncate preconditions/postconditions to filesystem specification"

          Show
          shv Konstantin Shvachko added a comment - Moved to hadoop-common. Changed the Summary, which was "Add definition of truncate preconditions/postconditions to filesystem specification"
          Hide
          shv Konstantin Shvachko added a comment -

          This are changes to the documentation related to truncate.
          All changes are in hadoop-common. So it would make sense to convert this to a HADOOP issue.

          Show
          shv Konstantin Shvachko added a comment - This are changes to the documentation related to truncate. All changes are in hadoop-common. So it would make sense to convert this to a HADOOP issue.
          Hide
          shv Konstantin Shvachko added a comment -

          Makes sense. Do you want to implement this? Seems like you had experience.

          Show
          shv Konstantin Shvachko added a comment - Makes sense. Do you want to implement this? Seems like you had experience.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          The second of those, src/site/markdown/filesystem is where I'd like it done; something which says "this is what truncate()" does, this is the valid state before and after, here are the exceptions you should/must throw, and here is the state afterwards.

          Once that's done, deriving a new subclass of AbstractFSContract purely to test truncate operations becomes straightforward: all a new contract test option to declare whether or not an FS supports truncate, and if it does, feed it valid data & expect a valid response, then feed it invalid & corner case data and expect failures.

          It's probably wise to mention the lack of guarantees about concurrent access, e.g. "if an input stream is open when truncate() occurs, the outcome of all operations other than close() on that stream are undefined.

          Show
          stevel@apache.org Steve Loughran added a comment - The second of those, src/site/markdown/filesystem is where I'd like it done; something which says "this is what truncate()" does, this is the valid state before and after, here are the exceptions you should/must throw, and here is the state afterwards. Once that's done, deriving a new subclass of AbstractFSContract purely to test truncate operations becomes straightforward: all a new contract test option to declare whether or not an FS supports truncate, and if it does, feed it valid data & expect a valid response, then feed it invalid & corner case data and expect failures. It's probably wise to mention the lack of guarantees about concurrent access, e.g. "if an input stream is open when truncate() occurs, the outcome of all operations other than close() on that stream are undefined.
          Hide
          shv Konstantin Shvachko added a comment -

          Steve, could you mention where exactly the specifications should be added.
          I understand we need to add truncate operation to HDFS documentation here:

          I guess you meant something else.

          Show
          shv Konstantin Shvachko added a comment - Steve, could you mention where exactly the specifications should be added. I understand we need to add truncate operation to HDFS documentation here: http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/FileSystemShell.html http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/filesystem/filesystem.html I guess you meant something else.

            People

            • Assignee:
              shv Konstantin Shvachko
              Reporter:
              stevel@apache.org Steve Loughran
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development