Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.7.0
    • Component/s: datanode, namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Systems with transaction support often need to undo changes made to the underlying storage when a transaction is aborted. Currently HDFS does not support truncate (a standard Posix operation) which is a reverse operation of append, which makes upper layer applications use ugly workarounds (such as keeping track of the discarded byte range per file in a separate metadata store, and periodically running a vacuum process to rewrite compacted files) to overcome this limitation of HDFS.

      1. HDFS_truncate_semantics_Mar15.pdf
        13 kB
        Tsz Wo Nicholas Sze
      2. HDFS_truncate_semantics_Mar21.pdf
        13 kB
        Tsz Wo Nicholas Sze
      3. HDFS_truncate.pdf
        478 kB
        Konstantin Shvachko
      4. HDFS-3107.15_branch2.patch
        59 kB
        Plamen Jeliazkov
      5. editsStored.xml
        30 kB
        Plamen Jeliazkov
      6. editsStored
        6 kB
        Plamen Jeliazkov
      7. HDFS-3107-15.patch
        59 kB
        Konstantin Shvachko
      8. HDFS-3107-14.patch
        59 kB
        Konstantin Shvachko
      9. HDFS-3107-13.patch
        59 kB
        Konstantin Shvachko
      10. HDFS-3107.patch
        60 kB
        Konstantin Shvachko
      11. HDFS-3107.patch
        59 kB
        Konstantin Shvachko
      12. HDFS-3107.patch
        60 kB
        Konstantin Shvachko
      13. HDFS_truncate.pdf
        479 kB
        Konstantin Shvachko
      14. HDFS-3107-HDFS-7056-combined.patch
        124 kB
        Konstantin Shvachko
      15. HDFS-3107.patch
        59 kB
        Plamen Jeliazkov
      16. HDFS-3107.008.patch
        36 kB
        Colin Patrick McCabe
      17. HDFS-3107.patch
        59 kB
        Konstantin Shvachko
      18. HDFS-3107.patch
        58 kB
        Plamen Jeliazkov
      19. HDFS_truncate.pdf
        473 kB
        Konstantin Shvachko
      20. HDFS-3107.patch
        83 kB
        Plamen Jeliazkov
      21. HDFS-3107.patch
        75 kB
        Plamen Jeliazkov
      22. HDFS-3107.patch
        50 kB
        Plamen Jeliazkov
      23. HDFS-3107.patch
        50 kB
        Plamen Jeliazkov
      24. HDFS_truncate.pdf
        337 kB
        Konstantin Shvachko
      25. HDFS-3107.patch
        50 kB
        Plamen Jeliazkov
      26. HDFS_truncate_semantics_Mar21.pdf
        189 kB
        Lei Chang
      27. HDFS_truncate_semantics_Mar15.pdf
        265 kB
        Lei Chang

        Issue Links

          Activity

          Hide
          Tsz Wo Nicholas Sze added a comment -

          HDFS_truncate_semantics_Mar15.pdf: obsoletes the previous doc.

          Show
          Tsz Wo Nicholas Sze added a comment - HDFS_truncate_semantics_Mar15.pdf: obsoletes the previous doc.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          HDFS_truncate_semantics_Mar21.pdf: obsoletes the previous doc.

          Show
          Tsz Wo Nicholas Sze added a comment - HDFS_truncate_semantics_Mar21.pdf: obsoletes the previous doc.
          Hide
          Konstantin Shvachko added a comment - - edited

          Attaching new version of the design document.
          You guys keep referring to the document attached to HDFS-7056, which I am really not in control of. It was uploaded by Guo Ruijing, and it reflects according to the jira comments Pivotal's internal implementation of truncate. It is quite close to what we did, but not the exact match.
          I think my design doc has all necessary details for the truncate-snapshot integration. So I just removed the reference to Guo's doc, because it keeps causing confusion. It is still a good source of introduction to snapshots impl. in general.

          Show
          Konstantin Shvachko added a comment - - edited Attaching new version of the design document. You guys keep referring to the document attached to HDFS-7056 , which I am really not in control of. It was uploaded by Guo Ruijing , and it reflects according to the jira comments Pivotal's internal implementation of truncate. It is quite close to what we did, but not the exact match. I think my design doc has all necessary details for the truncate-snapshot integration. So I just removed the reference to Guo's doc, because it keeps causing confusion. It is still a good source of introduction to snapshots impl. in general.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          We should also have a truncate test with cached data to that client won't read beyond truncated length from the cached data.

          BTW, Section 5.2.9 Snapshot operation during truncate in the design doc is out dated. It is still talking about creating a transient file. We should update it.

          Show
          Tsz Wo Nicholas Sze added a comment - We should also have a truncate test with cached data to that client won't read beyond truncated length from the cached data. BTW, Section 5.2.9 Snapshot operation during truncate in the design doc is out dated. It is still talking about creating a transient file. We should update it.
          Hide
          Konstantin Shvachko added a comment -

          Nicholas, you are right we don't have those tests.

          • For truncate with HA as I mentioned in HDFS-7738 we can incorporate truncate to TestHAAppend. LMK if you want to add it to your patch, otherwise I'll ad it to HDFS-7740.
          • Created HDFS-7740 for adding truncate test with DNs restarts. Described two scenarios there. Feel free to add other scenarios you have in mind there.
          Show
          Konstantin Shvachko added a comment - Nicholas, you are right we don't have those tests. For truncate with HA as I mentioned in HDFS-7738 we can incorporate truncate to TestHAAppend. LMK if you want to add it to your patch, otherwise I'll ad it to HDFS-7740 . Created HDFS-7740 for adding truncate test with DNs restarts. Described two scenarios there. Feel free to add other scenarios you have in mind there.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Konstantin Shvachko, how is the remaining work? Hope that we still have progress after merging to branch-2.

          BTW, it seems that we do not have truncate tests with HA nor datanode restart. Is it true?

          Show
          Tsz Wo Nicholas Sze added a comment - Konstantin Shvachko , how is the remaining work? Hope that we still have progress after merging to branch-2. BTW, it seems that we do not have truncate tests with HA nor datanode restart. Is it true?
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Thanks a lot for all the great works!

          Show
          Tsz Wo Nicholas Sze added a comment - Thanks a lot for all the great works!
          Hide
          Yi Liu added a comment -

          Thanks Konstantin Shvachko, I find another two things:

          • DistributedFileSystem#truncate should resolve symlinks
          • Expose truncate API via FileContext

          I open two new JIRAs HDFS-7677, HADOOP-11510, and will fix them there.

          Show
          Yi Liu added a comment - Thanks Konstantin Shvachko , I find another two things: DistributedFileSystem#truncate should resolve symlinks Expose truncate API via FileContext I open two new JIRAs HDFS-7677 , HADOOP-11510 , and will fix them there.
          Hide
          Konstantin Shvachko added a comment -

          I just merged the following jiras to branch-2:
          HDFS-3107, HDFS-7056, HDFS-7606, HDFS-7638, HDFS-7634, HDFS-7643, HADOOP-11490, HDFS-7659.

          Show
          Konstantin Shvachko added a comment - I just merged the following jiras to branch-2: HDFS-3107 , HDFS-7056 , HDFS-7606 , HDFS-7638 , HDFS-7634 , HDFS-7643 , HADOOP-11490 , HDFS-7659 .
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Sure. I think it is fine to merge.

          Show
          Tsz Wo Nicholas Sze added a comment - Sure. I think it is fine to merge.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Sure. I think it is fine to merge.

          Show
          Tsz Wo Nicholas Sze added a comment - Sure. I think it is fine to merge.
          Hide
          Konstantin Shvachko added a comment -

          Yes HDFS-7659 is ready.
          Documentation needs to be updates. I mentioned it in my email on the dev list some days ago. Other things to do are adding truncate to DFSIO and SLive.
          I don't think we should wait for them to merge. My main concern is that it increases the work for developers. Branches being substancially diverged means that it is harder to merge new code into branch-2, which is not related to truncate.
          Also it will be easier to implement TestCLI for example or the documentation update and then merge it into both branches at once.
          In the end it is not that we have a release planned next week.

          Would it be ok with you if we commit HDFS-7659, fix TestFileTruncate as I proposed there, and then merge?

          Show
          Konstantin Shvachko added a comment - Yes HDFS-7659 is ready. Documentation needs to be updates. I mentioned it in my email on the dev list some days ago. Other things to do are adding truncate to DFSIO and SLive. I don't think we should wait for them to merge. My main concern is that it increases the work for developers. Branches being substancially diverged means that it is harder to merge new code into branch-2, which is not related to truncate. Also it will be easier to implement TestCLI for example or the documentation update and then merge it into both branches at once. In the end it is not that we have a release planned next week. Would it be ok with you if we commit HDFS-7659 , fix TestFileTruncate as I proposed there, and then merge?
          Hide
          Tsz Wo Nicholas Sze added a comment -

          BTW, have we updated user documentation for the truncate CLI change?

          Show
          Tsz Wo Nicholas Sze added a comment - BTW, have we updated user documentation for the truncate CLI change?
          Hide
          Tsz Wo Nicholas Sze added a comment -

          There is a list of unresolved JIRAs. Let's discuss it.

          • HDFS-7341 Add initial snapshot support based on pipeline recovery
            Is it still relevant?
          • HDFS-7058 Tests for truncate CLI
            Let's finish it before merging since CLI is user facing. Is anyone working on it?
          • HDFS-7655/HDFS-7656 Expose truncate API for Web HDFS/httpfs
            It seems that we should not wait for them before merging. Agree?
          • HDFS-7659 We should check the new length of truncate can't be a negative value
            Look like that this is going to be committed soon.
          • HDFS-7665 Add definition of truncate preconditions/postconditions to filesystem specification
            This is simple a documentation change. Let's finish it before merging?

          As a summary, how about finishing HDFS-7058, HDFS-7659 and HDFS-7665 before merging it to branch-2?

          Show
          Tsz Wo Nicholas Sze added a comment - There is a list of unresolved JIRAs. Let's discuss it. HDFS-7341 Add initial snapshot support based on pipeline recovery Is it still relevant? HDFS-7058 Tests for truncate CLI Let's finish it before merging since CLI is user facing. Is anyone working on it? HDFS-7655 / HDFS-7656 Expose truncate API for Web HDFS/httpfs It seems that we should not wait for them before merging. Agree? HDFS-7659 We should check the new length of truncate can't be a negative value Look like that this is going to be committed soon. HDFS-7665 Add definition of truncate preconditions/postconditions to filesystem specification This is simple a documentation change. Let's finish it before merging? As a summary, how about finishing HDFS-7058 , HDFS-7659 and HDFS-7665 before merging it to branch-2?
          Hide
          Konstantin Shvachko added a comment -

          Hey guys, Plamen's ports to branch-2 look good to me.
          I'll schedule time during next weekend to commit these and related jiras. This is almost two weeks from the trunk commit.
          LMK if there are any issues. Looks like we are making good progress.
          I think we've reached the stage when further truncate related changes can be done both on trunk and branch-2 after the merge.
          Nichloas, are you ok doing HDFS-7058, which has only TestCLI remaining, after the merge?

          Show
          Konstantin Shvachko added a comment - Hey guys, Plamen's ports to branch-2 look good to me. I'll schedule time during next weekend to commit these and related jiras. This is almost two weeks from the trunk commit. LMK if there are any issues. Looks like we are making good progress. I think we've reached the stage when further truncate related changes can be done both on trunk and branch-2 after the merge. Nichloas, are you ok doing HDFS-7058 , which has only TestCLI remaining, after the merge?
          Hide
          Yi Liu added a comment -

          I just noticed we have not exposed truncate API for Web HDFS and HDFS http fs, I created two JIRAs HDFS-7655, HDFS-7656 for them.
          Of course, they will not block the merging of this feature to branch-2.

          Show
          Yi Liu added a comment - I just noticed we have not exposed truncate API for Web HDFS and HDFS http fs, I created two JIRAs HDFS-7655 , HDFS-7656 for them. Of course, they will not block the merging of this feature to branch-2.
          Hide
          Plamen Jeliazkov added a comment -

          Attaching patch for branch-2 port. Ran tests locally with latest editsStored already attached to this JIRA. Everything passed locally.

          Show
          Plamen Jeliazkov added a comment - Attaching patch for branch-2 port. Ran tests locally with latest editsStored already attached to this JIRA. Everything passed locally.
          Hide
          Konstantin Shvachko added a comment -

          This is tracked under HDFS-7611. Working on it.

          Show
          Konstantin Shvachko added a comment - This is tracked under HDFS-7611 . Working on it.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          BTW, the truncate unit test is failing occasionally; see build #9264, build #9257, build #9255 and build #9244.

          Show
          Tsz Wo Nicholas Sze added a comment - BTW, the truncate unit test is failing occasionally; see build #9264 , build #9257 , build #9255 and build #9244 .
          Hide
          Tsz Wo Nicholas Sze added a comment -

          It is great to merge this to branch-2. How about finishing the tests (HDFS-7058) first? I am also going to test it this week.

          Show
          Tsz Wo Nicholas Sze added a comment - It is great to merge this to branch-2. How about finishing the tests ( HDFS-7058 ) first? I am also going to test it this week.
          Hide
          Yi Liu added a comment -

          I have done some functionality tests in my cluster, and do some code review from my point of view (I'm not very good at snapshot part, so just did some basic review for that part ).

          +1 (non-binding) for merging into branch-2, it's good to me, except some small nits, and I filed them in HDFS-7634 and HDFS-7638. Certainly we can file follow ups if I find some new issues.

          Show
          Yi Liu added a comment - I have done some functionality tests in my cluster, and do some code review from my point of view (I'm not very good at snapshot part, so just did some basic review for that part ). +1 (non-binding) for merging into branch-2, it's good to me, except some small nits, and I filed them in HDFS-7634 and HDFS-7638 . Certainly we can file follow ups if I find some new issues.
          Hide
          Yi Liu added a comment -

          Konstantin Shvachko, this is a good feature and nice work. Agree with Colin that we should give it a work or two before merging to branch-2, then others can have enough test.
          Personally I will get some time to verify the functionality and do some review next week.

          Show
          Yi Liu added a comment - Konstantin Shvachko , this is a good feature and nice work. Agree with Colin that we should give it a work or two before merging to branch-2, then others can have enough test. Personally I will get some time to verify the functionality and do some review next week.
          Hide
          Uma Maheswara Rao G added a comment -

          Thanks Plamen, Konstantin for the great efforts on this feature. I did not get chance to review this fully.
          +1 for merging this feature into branch-2 in few days. By that time others can test and raise the follow ups if there are any issue. I will look into it with my review and file followups if any things I could notice. Thanks

          Show
          Uma Maheswara Rao G added a comment - Thanks Plamen, Konstantin for the great efforts on this feature. I did not get chance to review this fully. +1 for merging this feature into branch-2 in few days. By that time others can test and raise the follow ups if there are any issue. I will look into it with my review and file followups if any things I could notice. Thanks
          Hide
          Uma Maheswara Rao G added a comment -

          Thanks Plamen, Konstantin for the great efforts on this feature. I did not get chance to review this fully.
          +1 for merging this feature into branch-2 in few days. By that time others can test and raise the follow ups if there are any issue. I will look into it with my review and file followups if any things I could notice. Thanks

          Show
          Uma Maheswara Rao G added a comment - Thanks Plamen, Konstantin for the great efforts on this feature. I did not get chance to review this fully. +1 for merging this feature into branch-2 in few days. By that time others can test and raise the follow ups if there are any issue. I will look into it with my review and file followups if any things I could notice. Thanks
          Hide
          Konstantin Boudnik added a comment -

          I think it is a reasonable expectation to do the merge in a few days or a week. Most importantly, the merge might require certain changes resulting from the conflicts resolution - so it'd be some dev. effort anyway.

          Show
          Konstantin Boudnik added a comment - I think it is a reasonable expectation to do the merge in a few days or a week. Most importantly, the merge might require certain changes resulting from the conflicts resolution - so it'd be some dev. effort anyway.
          Hide
          Colin Patrick McCabe added a comment -

          I can't speak for anyone else on this thread, but I personally think we should give it a week or two and then merge to branch-2. I hope anyone with unresolved concerns will post follow-ups by then.

          In the future we should probably do this kind of work in a branch. It would have sped up the work, because we could have committed things quickly to the branch. git has made branches easier than ever. The requirement to get three +1s in a branch might seem onerous, but a feature of this size needs several eyes on it anyway, so the review requirement ended up being about the same (or even more).

          Show
          Colin Patrick McCabe added a comment - I can't speak for anyone else on this thread, but I personally think we should give it a week or two and then merge to branch-2. I hope anyone with unresolved concerns will post follow-ups by then. In the future we should probably do this kind of work in a branch. It would have sped up the work, because we could have committed things quickly to the branch. git has made branches easier than ever. The requirement to get three +1s in a branch might seem onerous, but a feature of this size needs several eyes on it anyway, so the review requirement ended up being about the same (or even more).
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2023 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2023/)
          HDFS-3107. Introduce truncate. Contributed by Plamen Jeliazkov. (shv: rev 7e9358feb326d48b8c4f00249e7af5023cebd2e2)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/proto/ClientNamenodeProtocol.proto
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/metrics/NameNodeMetrics.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/HdfsServerConstants.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/proto/hdfs.proto
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/BlockRecoveryCommand.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOpCodes.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestRetryCacheWithHA.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSInotifyEventInputStream.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoUnderConstruction.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FileWithSnapshotFeature.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStorageInfo.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored.xml
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNamenodeRetryCache.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFileTruncate.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolTranslatorPB.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolServerSideTranslatorPB.java
          Show
          Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2023 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2023/ ) HDFS-3107 . Introduce truncate. Contributed by Plamen Jeliazkov. (shv: rev 7e9358feb326d48b8c4f00249e7af5023cebd2e2) hadoop-hdfs-project/hadoop-hdfs/src/main/proto/ClientNamenodeProtocol.proto hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/metrics/NameNodeMetrics.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/HdfsServerConstants.java hadoop-hdfs-project/hadoop-hdfs/src/main/proto/hdfs.proto hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/BlockRecoveryCommand.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOpCodes.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestRetryCacheWithHA.java hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSInotifyEventInputStream.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoUnderConstruction.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FileWithSnapshotFeature.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStorageInfo.java hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored.xml hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNamenodeRetryCache.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFileTruncate.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolTranslatorPB.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolServerSideTranslatorPB.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #73 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/73/)
          HDFS-3107. Introduce truncate. Contributed by Plamen Jeliazkov. (shv: rev 7e9358feb326d48b8c4f00249e7af5023cebd2e2)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestRetryCacheWithHA.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSInotifyEventInputStream.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStorageInfo.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored.xml
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/proto/hdfs.proto
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/BlockRecoveryCommand.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOpCodes.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolTranslatorPB.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FileWithSnapshotFeature.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolServerSideTranslatorPB.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoUnderConstruction.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/HdfsServerConstants.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/proto/ClientNamenodeProtocol.proto
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNamenodeRetryCache.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/metrics/NameNodeMetrics.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFileTruncate.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #73 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/73/ ) HDFS-3107 . Introduce truncate. Contributed by Plamen Jeliazkov. (shv: rev 7e9358feb326d48b8c4f00249e7af5023cebd2e2) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestRetryCacheWithHA.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSInotifyEventInputStream.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStorageInfo.java hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored.xml hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java hadoop-hdfs-project/hadoop-hdfs/src/main/proto/hdfs.proto hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/BlockRecoveryCommand.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOpCodes.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolTranslatorPB.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FileWithSnapshotFeature.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolServerSideTranslatorPB.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoUnderConstruction.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/HdfsServerConstants.java hadoop-hdfs-project/hadoop-hdfs/src/main/proto/ClientNamenodeProtocol.proto hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNamenodeRetryCache.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/metrics/NameNodeMetrics.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFileTruncate.java
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Hdfs-trunk #2004 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2004/)
          HDFS-3107. Introduce truncate. Contributed by Plamen Jeliazkov. (shv: rev 7e9358feb326d48b8c4f00249e7af5023cebd2e2)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFileTruncate.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/proto/ClientNamenodeProtocol.proto
          • hadoop-hdfs-project/hadoop-hdfs/src/main/proto/hdfs.proto
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOpCodes.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSInotifyEventInputStream.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolTranslatorPB.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestRetryCacheWithHA.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolServerSideTranslatorPB.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored.xml
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/HdfsServerConstants.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNamenodeRetryCache.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoUnderConstruction.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FileWithSnapshotFeature.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStorageInfo.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/metrics/NameNodeMetrics.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/BlockRecoveryCommand.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java
          Show
          Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk #2004 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2004/ ) HDFS-3107 . Introduce truncate. Contributed by Plamen Jeliazkov. (shv: rev 7e9358feb326d48b8c4f00249e7af5023cebd2e2) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFileTruncate.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java hadoop-hdfs-project/hadoop-hdfs/src/main/proto/ClientNamenodeProtocol.proto hadoop-hdfs-project/hadoop-hdfs/src/main/proto/hdfs.proto hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOpCodes.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSInotifyEventInputStream.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolTranslatorPB.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestRetryCacheWithHA.java hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolServerSideTranslatorPB.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored.xml hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/HdfsServerConstants.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNamenodeRetryCache.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoUnderConstruction.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FileWithSnapshotFeature.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStorageInfo.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/metrics/NameNodeMetrics.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/BlockRecoveryCommand.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #69 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/69/)
          HDFS-3107. Introduce truncate. Contributed by Plamen Jeliazkov. (shv: rev 7e9358feb326d48b8c4f00249e7af5023cebd2e2)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoUnderConstruction.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSInotifyEventInputStream.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolServerSideTranslatorPB.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFileTruncate.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FileWithSnapshotFeature.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/proto/ClientNamenodeProtocol.proto
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/BlockRecoveryCommand.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStorageInfo.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolTranslatorPB.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored.xml
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/metrics/NameNodeMetrics.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNamenodeRetryCache.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestRetryCacheWithHA.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/HdfsServerConstants.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOpCodes.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/proto/hdfs.proto
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #69 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/69/ ) HDFS-3107 . Introduce truncate. Contributed by Plamen Jeliazkov. (shv: rev 7e9358feb326d48b8c4f00249e7af5023cebd2e2) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoUnderConstruction.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSInotifyEventInputStream.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolServerSideTranslatorPB.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFileTruncate.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FileWithSnapshotFeature.java hadoop-hdfs-project/hadoop-hdfs/src/main/proto/ClientNamenodeProtocol.proto hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/BlockRecoveryCommand.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStorageInfo.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolTranslatorPB.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored.xml hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/metrics/NameNodeMetrics.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNamenodeRetryCache.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestRetryCacheWithHA.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/HdfsServerConstants.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOpCodes.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java hadoop-hdfs-project/hadoop-hdfs/src/main/proto/hdfs.proto
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk #806 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/806/)
          HDFS-3107. Introduce truncate. Contributed by Plamen Jeliazkov. (shv: rev 7e9358feb326d48b8c4f00249e7af5023cebd2e2)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolServerSideTranslatorPB.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestRetryCacheWithHA.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/proto/hdfs.proto
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoUnderConstruction.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/metrics/NameNodeMetrics.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/proto/ClientNamenodeProtocol.proto
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/BlockRecoveryCommand.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNamenodeRetryCache.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSInotifyEventInputStream.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FileWithSnapshotFeature.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFileTruncate.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/HdfsServerConstants.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolTranslatorPB.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored.xml
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOpCodes.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStorageInfo.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored
          Show
          Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #806 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/806/ ) HDFS-3107 . Introduce truncate. Contributed by Plamen Jeliazkov. (shv: rev 7e9358feb326d48b8c4f00249e7af5023cebd2e2) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolServerSideTranslatorPB.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestRetryCacheWithHA.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java hadoop-hdfs-project/hadoop-hdfs/src/main/proto/hdfs.proto hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoUnderConstruction.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/metrics/NameNodeMetrics.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java hadoop-hdfs-project/hadoop-hdfs/src/main/proto/ClientNamenodeProtocol.proto hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/BlockRecoveryCommand.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNamenodeRetryCache.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSInotifyEventInputStream.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FileWithSnapshotFeature.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFileTruncate.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/HdfsServerConstants.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolTranslatorPB.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored.xml hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOpCodes.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStorageInfo.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #72 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/72/)
          HDFS-3107. Introduce truncate. Contributed by Plamen Jeliazkov. (shv: rev 7e9358feb326d48b8c4f00249e7af5023cebd2e2)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolServerSideTranslatorPB.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored.xml
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStorageInfo.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/HdfsServerConstants.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSInotifyEventInputStream.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/BlockRecoveryCommand.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FileWithSnapshotFeature.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/proto/hdfs.proto
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNamenodeRetryCache.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoUnderConstruction.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestRetryCacheWithHA.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/metrics/NameNodeMetrics.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolTranslatorPB.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFileTruncate.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/proto/ClientNamenodeProtocol.proto
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOpCodes.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #72 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/72/ ) HDFS-3107 . Introduce truncate. Contributed by Plamen Jeliazkov. (shv: rev 7e9358feb326d48b8c4f00249e7af5023cebd2e2) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolServerSideTranslatorPB.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored.xml hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStorageInfo.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/HdfsServerConstants.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSInotifyEventInputStream.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/BlockRecoveryCommand.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FileWithSnapshotFeature.java hadoop-hdfs-project/hadoop-hdfs/src/main/proto/hdfs.proto hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNamenodeRetryCache.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoUnderConstruction.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestRetryCacheWithHA.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/metrics/NameNodeMetrics.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolTranslatorPB.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFileTruncate.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java hadoop-hdfs-project/hadoop-hdfs/src/main/proto/ClientNamenodeProtocol.proto hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOpCodes.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #6853 (See https://builds.apache.org/job/Hadoop-trunk-Commit/6853/)
          HDFS-3107. Introduce truncate. Contributed by Plamen Jeliazkov. (shv: rev 7e9358feb326d48b8c4f00249e7af5023cebd2e2)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolServerSideTranslatorPB.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FileWithSnapshotFeature.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStorageInfo.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFileTruncate.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored.xml
          • hadoop-hdfs-project/hadoop-hdfs/src/main/proto/hdfs.proto
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSInotifyEventInputStream.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNamenodeRetryCache.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoUnderConstruction.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOpCodes.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolTranslatorPB.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/BlockRecoveryCommand.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestRetryCacheWithHA.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/HdfsServerConstants.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/proto/ClientNamenodeProtocol.proto
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/metrics/NameNodeMetrics.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #6853 (See https://builds.apache.org/job/Hadoop-trunk-Commit/6853/ ) HDFS-3107 . Introduce truncate. Contributed by Plamen Jeliazkov. (shv: rev 7e9358feb326d48b8c4f00249e7af5023cebd2e2) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolServerSideTranslatorPB.java hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FileWithSnapshotFeature.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStorageInfo.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFileTruncate.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored.xml hadoop-hdfs-project/hadoop-hdfs/src/main/proto/hdfs.proto hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSInotifyEventInputStream.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNamenodeRetryCache.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoUnderConstruction.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOpCodes.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolTranslatorPB.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/BlockRecoveryCommand.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestRetryCacheWithHA.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/HdfsServerConstants.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java hadoop-hdfs-project/hadoop-hdfs/src/main/proto/ClientNamenodeProtocol.proto hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/metrics/NameNodeMetrics.java
          Hide
          Konstantin Shvachko added a comment -

          I just committed this to trunk. Thank you Plamen.

          Should we port it to branch 2 along with HDFS-7056? What people think?

          Show
          Konstantin Shvachko added a comment - I just committed this to trunk. Thank you Plamen. Should we port it to branch 2 along with HDFS-7056 ? What people think?
          Hide
          Plamen Jeliazkov added a comment -

          Attaching updated versions of the editsStored files necessary to make TestOfflineEditsViewer pass.

          Show
          Plamen Jeliazkov added a comment - Attaching updated versions of the editsStored files necessary to make TestOfflineEditsViewer pass.
          Hide
          Konstantin Shvachko added a comment -

          Haohui Mai do you have comments or suggestions on the patch? Or do you have anybody in particular in mind to return from vacation? I mean we cannot wait for everybody as the Hadoop community is pretty big - there is always somebody on vacation.
          I don't think we are rushing here. The patch is sitting here from September. People expressed desire to have the snapshot support implemented before committing the feature, which is now also done under HDFS-7056. And went through a series of good review rounds by Jing, Collin, Cos, which improved the implementation substantially. Byron independently tested the feature on a cluster with and without snapshots.

          So let's plan to finally commit both issues eod tomorrow Monday. If there are no more comments.

          Show
          Konstantin Shvachko added a comment - Haohui Mai do you have comments or suggestions on the patch? Or do you have anybody in particular in mind to return from vacation? I mean we cannot wait for everybody as the Hadoop community is pretty big - there is always somebody on vacation. I don't think we are rushing here. The patch is sitting here from September. People expressed desire to have the snapshot support implemented before committing the feature, which is now also done under HDFS-7056 . And went through a series of good review rounds by Jing, Collin, Cos, which improved the implementation substantially. Byron independently tested the feature on a cluster with and without snapshots. So let's plan to finally commit both issues eod tomorrow Monday. If there are no more comments.
          Hide
          Konstantin Boudnik added a comment -

          It's not rush, but having the patch sitting on the JIRA with very low comments activity - and none of them are really touching on the design or the integration of the feature to the rest of the HDFS - just increases the maintenance cost: the patch maintainer just keeps rebasing the patch over the current trunk.

          Show
          Konstantin Boudnik added a comment - It's not rush, but having the patch sitting on the JIRA with very low comments activity - and none of them are really touching on the design or the integration of the feature to the rest of the HDFS - just increases the maintenance cost: the patch maintainer just keeps rebasing the patch over the current trunk.
          Hide
          Haohui Mai added a comment -

          I don't see any reasons to rush to commit the patch. It is a big feature and it makes sense to ensure all feedbacks are properly address.

          I think it is worth holding off until people are back from the vacations.

          Show
          Haohui Mai added a comment - I don't see any reasons to rush to commit the patch. It is a big feature and it makes sense to ensure all feedbacks are properly address. I think it is worth holding off until people are back from the vacations.
          Hide
          Konstantin Boudnik added a comment -

          I think it is ready for commit.
          Any other comments from any of the reviewers?

          Show
          Konstantin Boudnik added a comment - I think it is ready for commit. Any other comments from any of the reviewers?
          Hide
          Konstantin Shvachko added a comment -

          Another rebase.

          Show
          Konstantin Shvachko added a comment - Another rebase.
          Hide
          Konstantin Shvachko added a comment -

          Fixed findbugs warning.
          Updated to current trunk.

          Show
          Konstantin Shvachko added a comment - Fixed findbugs warning. Updated to current trunk.
          Hide
          Konstantin Shvachko added a comment -

          Updating. This was rboken by logging changes of HADOOP-11430.

          Show
          Konstantin Shvachko added a comment - Updating. This was rboken by logging changes of HADOOP-11430 .
          Hide
          Colin Patrick McCabe added a comment -

          Now, I am not posting any patches into this JIRA nor HDFS-7056. You are confusing me with someone else.

          Sorry, Konstantin Boudnik, for some reason I thought you had posted the latest patch on this JIRA. But now I see it was Konstantin Shvachko.

          Anyway, I will review HDFS-7056 in a sec

          Show
          Colin Patrick McCabe added a comment - Now, I am not posting any patches into this JIRA nor HDFS-7056 . You are confusing me with someone else. Sorry, Konstantin Boudnik , for some reason I thought you had posted the latest patch on this JIRA. But now I see it was Konstantin Shvachko . Anyway, I will review HDFS-7056 in a sec
          Hide
          Konstantin Boudnik added a comment -

          Colin Patrick McCabe, I have repeated +1 on the latest version of the patch which was a simple rebase on top of latest changes in the trunk.
          Now, I am not posting any patches into this JIRA nor HDFS-7056. You are confusing me with someone else.

          Show
          Konstantin Boudnik added a comment - Colin Patrick McCabe , I have repeated +1 on the latest version of the patch which was a simple rebase on top of latest changes in the trunk. Now, I am not posting any patches into this JIRA nor HDFS-7056 . You are confusing me with someone else.
          Hide
          Colin Patrick McCabe added a comment -

          Konstantin Boudnik, can you answer my question about what you are +1 on? I find it confusing that you are continuing to post patches to this JIRA as well as to HDFS-7056. Based on the earlier comments, I thought we were going to move this code into subtasks. We also discussed giving patches revision numbers on hdfs-dev@, (and your last comment on that thread was "Oh well. Revision numbers is the way to go then for now"), but still I see all these patches have the same name.

          thanks

          Show
          Colin Patrick McCabe added a comment - Konstantin Boudnik , can you answer my question about what you are +1 on? I find it confusing that you are continuing to post patches to this JIRA as well as to HDFS-7056 . Based on the earlier comments, I thought we were going to move this code into subtasks. We also discussed giving patches revision numbers on hdfs-dev@, (and your last comment on that thread was "Oh well. Revision numbers is the way to go then for now"), but still I see all these patches have the same name. thanks
          Hide
          Colin Patrick McCabe added a comment -

          I'll take a look tomorrow. Was busy on HDFS-7443, which is a blocker for upgrades.

          Show
          Colin Patrick McCabe added a comment - I'll take a look tomorrow. Was busy on HDFS-7443 , which is a blocker for upgrades.
          Hide
          Jing Zhao added a comment -

          Yeah, I will review the latest patch in HDFS-7056 today and tomorrow.

          Show
          Jing Zhao added a comment - Yeah, I will review the latest patch in HDFS-7056 today and tomorrow.
          Hide
          Konstantin Shvachko added a comment -

          Colin, please do review. For your convenience here is a direct link to Plamen's comment, addressing your review comments.

          Show
          Konstantin Shvachko added a comment - Colin, please do review. For your convenience here is a direct link to Plamen's comment, addressing your review comments.
          Hide
          Colin Patrick McCabe added a comment -

          Hi Konstantin Shvachko, I see that you have submitted a patch on this JIRA, and also one on HDFS-7056. Can you clarify which one you are +1 on, and proposing to commit? Jing Zhao, can you clarify if the latest revision on HDFS-7056 addresses your comments? I will look through it and see if mine were addressed, and put that comment on HDFS-7056. Thanks, all.

          Show
          Colin Patrick McCabe added a comment - Hi Konstantin Shvachko , I see that you have submitted a patch on this JIRA, and also one on HDFS-7056 . Can you clarify which one you are +1 on, and proposing to commit? Jing Zhao , can you clarify if the latest revision on HDFS-7056 addresses your comments? I will look through it and see if mine were addressed, and put that comment on HDFS-7056 . Thanks, all.
          Hide
          Konstantin Boudnik added a comment -

          The diff between the two seems to be quite small, yet I guess it requires a formal review again. Hence +1 again.

          Show
          Konstantin Boudnik added a comment - The diff between the two seems to be quite small, yet I guess it requires a formal review again. Hence +1 again.
          Hide
          Konstantin Shvachko added a comment -

          Incorporated latest trunk.

          Show
          Konstantin Shvachko added a comment - Incorporated latest trunk.
          Hide
          Konstantin Boudnik added a comment -

          I actually quite like it. I think over the last few iterations the patch was polished enough and the test coverage is quite decent.

          +1

          Show
          Konstantin Boudnik added a comment - I actually quite like it. I think over the last few iterations the patch was polished enough and the test coverage is quite decent. +1
          Hide
          Konstantin Shvachko added a comment -

          Looks like HDFS-7506 broke this again.

          Show
          Konstantin Shvachko added a comment - Looks like HDFS-7506 broke this again.
          Hide
          Konstantin Shvachko added a comment -

          Updating the patches once again.
          This update is mostly related to Jing Zhao's refactoring of HDFS-7509.
          There are no changes to the truncate logic itself.

          Just to remind people here. The snapshot part of truncate is being maintained under HDFS-7056. And the combined patch for the two issues is also submitted there (per Colin Patrick McCabe's request).

          Show
          Konstantin Shvachko added a comment - Updating the patches once again. This update is mostly related to Jing Zhao 's refactoring of HDFS-7509 . There are no changes to the truncate logic itself. Just to remind people here. The snapshot part of truncate is being maintained under HDFS-7056 . And the combined patch for the two issues is also submitted there (per Colin Patrick McCabe 's request).
          Hide
          Konstantin Shvachko added a comment -

          Updating patch to current trunk.

          Show
          Konstantin Shvachko added a comment - Updating patch to current trunk.
          Hide
          Colin Patrick McCabe added a comment -

          Thanks, I will take a look at HDFS-7056. I suppose this means we can mark HDFS-7341 as a duplicate.

          Show
          Colin Patrick McCabe added a comment - Thanks, I will take a look at HDFS-7056 . I suppose this means we can mark HDFS-7341 as a duplicate.
          Hide
          Plamen Jeliazkov added a comment -

          I checked FindBugs since Jenkins messed up the patch build.

          I xml-diff compared the XMLs generated with and without the combined patch and found no new FindBugs were introduced.

          Show
          Plamen Jeliazkov added a comment - I checked FindBugs since Jenkins messed up the patch build. I xml-diff compared the XMLs generated with and without the combined patch and found no new FindBugs were introduced.
          Hide
          Plamen Jeliazkov added a comment -

          I will attach it to HDFS-7056 since it has the design doc attached to it and is assigned to me. Thanks Colin Patrick McCabe.

          Show
          Plamen Jeliazkov added a comment - I will attach it to HDFS-7056 since it has the design doc attached to it and is assigned to me. Thanks Colin Patrick McCabe .
          Hide
          Colin Patrick McCabe added a comment -

          There is the option to treat the combined patch HDFS-3107-&-7056 as the first patch, which accounts for upgrade and rollback functionality as well as snapshot support, demonstrated in unit test.

          That's fine with me. It can go into trunk directly if it doesn't break rollback + snapshots.

          I am not objecting to do work on a branch but I am unsure it is necessary given the combined patch seems to meet the support requirements asked for this work.

          I suggested a branch since I thought it would let us commit things quicker. But I don't think it's necessary if you can do things without breaking trunk. It is going to be no more than 3-4 patches anyway as I understand. Whatever is easiest for you guys.

          Just one request: Can you post the combined patch on a subtask rather than this JIRA? I think having patches on this umbrella jira is very confusing. If you're going to combine the patches, post the combined patch on either HDFS-7341 or HDFS-7056 please. Thanks.

          Show
          Colin Patrick McCabe added a comment - There is the option to treat the combined patch HDFS-3107 -&-7056 as the first patch, which accounts for upgrade and rollback functionality as well as snapshot support, demonstrated in unit test. That's fine with me. It can go into trunk directly if it doesn't break rollback + snapshots. I am not objecting to do work on a branch but I am unsure it is necessary given the combined patch seems to meet the support requirements asked for this work. I suggested a branch since I thought it would let us commit things quicker. But I don't think it's necessary if you can do things without breaking trunk. It is going to be no more than 3-4 patches anyway as I understand. Whatever is easiest for you guys. Just one request: Can you post the combined patch on a subtask rather than this JIRA? I think having patches on this umbrella jira is very confusing. If you're going to combine the patches, post the combined patch on either HDFS-7341 or HDFS-7056 please. Thanks.
          Hide
          Plamen Jeliazkov added a comment -

          Colin Patrick McCabe,

          At the time Konstantin Shvachko talked about my new patch there was nothing posted yet in HDFS-7056 minus Konstantin's design doc.
          We only uploaded even newer patches yesterday around noon.

          Please be careful not to confuse Konstantin Shvachko and Konstantin Boudnik.
          The snapshot support patch (for HDFS-7056) was not ready yet when Konstantin Boudnik made his comment.

          We don't have to commit HDFS-3107 on its own.
          There is the option to treat the combined patch HDFS-3107-&-7056 as the first patch, which accounts for upgrade and rollback functionality as well as snapshot support, demonstrated in unit test.
          This should address your comment: "My reasoning is that if the first patch breaks rollback, it's tough to see it getting into trunk."
          I am not objecting to do work on a branch but I am unsure it is necessary given the combined patch seems to meet the support requirements asked for this work.

          I'll investigate the FindBugs.

          Show
          Plamen Jeliazkov added a comment - Colin Patrick McCabe , At the time Konstantin Shvachko talked about my new patch there was nothing posted yet in HDFS-7056 minus Konstantin's design doc. We only uploaded even newer patches yesterday around noon. Please be careful not to confuse Konstantin Shvachko and Konstantin Boudnik . The snapshot support patch (for HDFS-7056 ) was not ready yet when Konstantin Boudnik made his comment. We don't have to commit HDFS-3107 on its own. There is the option to treat the combined patch HDFS-3107 -&-7056 as the first patch, which accounts for upgrade and rollback functionality as well as snapshot support, demonstrated in unit test. This should address your comment: "My reasoning is that if the first patch breaks rollback, it's tough to see it getting into trunk." I am not objecting to do work on a branch but I am unsure it is necessary given the combined patch seems to meet the support requirements asked for this work. I'll investigate the FindBugs.
          Hide
          Colin Patrick McCabe added a comment -

          Hi Konstantin,

          Yes, I was aware of HDFS-7056 and had commented about it earlier. However, when you announced on this JIRA: "So, but the new patch from Plamen Jeliazkov already have snapshot support," the patch that I checked was HDFS-3107.patch, not a patch on a different JIRA. After all, Plamen Jeliazkov is the assignee on this JIRA as well as HDFS-7056, so "the new patch from Plamen Jeliazkov" could refer to either patch. It seems logical to assume that what is being discussed on a particular JIRA is the patch attached to that JIRA, unless specified otherwise.

          I think this highlights one confusing thing: that HDFS-3107 is now an umbrella JIRA as well as a JIRA with a (non-rollup) patch. This creates confusion in people's minds because questions like "is HDFS-3107 done?" become ambiguous. It could be interpreted as either "is the patch on HDFS-3107 done?" or "is the feature discussed in HDFS-3107 done?" To remove this confusion, I created the HDFS-7341 subtask to implement the part we've been discussing here. That is, the pipeline-recovery based solution which the other subtasks build on top of.

          I would like to create the HDFS-3107 branch as soon as possible. I would have already created it, but as per my comment above, I want to make sure you are not objecting. The quicker we can get this stuff into the branch, the quicker we can get this feature polished and merged into trunk.

          As I commented earlier, it's not up to me to determine if something gets into 2.6 or not. It's up to the release manager (currently Arun C Murthy) and the community to vote. Since 2.6 is so far along (it should have been released weeks ago, if the original schedule had been followed), I doubt that most people will welcome a big new feature getting added at this stage. I don't think branch versus since commit matters in this regard-- a big new feature is a big new feature. It's not going to "sneak in the back door"-- nor should it, given the problems we've had in the past (like with HDFS append). We have time to do a thorough review. In the meantime, distributions that are already shipping a variant of append can continue to do so, knowing that eventually the feature has a path to mainline.

          Let me know if you have any objections to creating the branch, otherwise I'll do it tomorrow. Thanks

          Show
          Colin Patrick McCabe added a comment - Hi Konstantin, Yes, I was aware of HDFS-7056 and had commented about it earlier. However, when you announced on this JIRA: "So, but the new patch from Plamen Jeliazkov already have snapshot support," the patch that I checked was HDFS-3107 .patch , not a patch on a different JIRA. After all, Plamen Jeliazkov is the assignee on this JIRA as well as HDFS-7056 , so "the new patch from Plamen Jeliazkov" could refer to either patch. It seems logical to assume that what is being discussed on a particular JIRA is the patch attached to that JIRA, unless specified otherwise. I think this highlights one confusing thing: that HDFS-3107 is now an umbrella JIRA as well as a JIRA with a (non-rollup) patch. This creates confusion in people's minds because questions like "is HDFS-3107 done?" become ambiguous. It could be interpreted as either "is the patch on HDFS-3107 done?" or "is the feature discussed in HDFS-3107 done?" To remove this confusion, I created the HDFS-7341 subtask to implement the part we've been discussing here. That is, the pipeline-recovery based solution which the other subtasks build on top of. I would like to create the HDFS-3107 branch as soon as possible. I would have already created it, but as per my comment above, I want to make sure you are not objecting. The quicker we can get this stuff into the branch, the quicker we can get this feature polished and merged into trunk. As I commented earlier, it's not up to me to determine if something gets into 2.6 or not. It's up to the release manager (currently Arun C Murthy ) and the community to vote. Since 2.6 is so far along (it should have been released weeks ago, if the original schedule had been followed), I doubt that most people will welcome a big new feature getting added at this stage. I don't think branch versus since commit matters in this regard-- a big new feature is a big new feature. It's not going to "sneak in the back door"-- nor should it, given the problems we've had in the past (like with HDFS append). We have time to do a thorough review. In the meantime, distributions that are already shipping a variant of append can continue to do so, knowing that eventually the feature has a path to mainline. Let me know if you have any objections to creating the branch, otherwise I'll do it tomorrow. Thanks
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12679004/HDFS-3107-HDFS-7056-combined.patch
          against trunk revision 35d353e.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 9 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to cause Findbugs (version 2.0.3) to fail.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8630//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8630//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12679004/HDFS-3107-HDFS-7056-combined.patch against trunk revision 35d353e. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 9 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to cause Findbugs (version 2.0.3) to fail. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in . +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8630//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8630//console This message is automatically generated.
          Hide
          Konstantin Shvachko added a comment -

          Updated design document.

          Show
          Konstantin Shvachko added a comment - Updated design document.
          Hide
          Konstantin Boudnik added a comment -

          Would love to see it in a feature branch

          Roman Shaposhnik how a feature branch helps to solve your issue at hands? I thought you guys want to have it in the next release, right?

          Show
          Konstantin Boudnik added a comment - Would love to see it in a feature branch Roman Shaposhnik how a feature branch helps to solve your issue at hands? I thought you guys want to have it in the next release, right?
          Hide
          Konstantin Shvachko added a comment -

          The combined patch. It is a merge of HDFS-3107 and HDFS-7056.

          Show
          Konstantin Shvachko added a comment - The combined patch. It is a merge of HDFS-3107 and HDFS-7056 .
          Hide
          Plamen Jeliazkov added a comment -

          Attaching refreshed set of new editsStored files.

          Show
          Plamen Jeliazkov added a comment - Attaching refreshed set of new editsStored files.
          Hide
          Konstantin Shvachko added a comment -

          Submitted two patches separetly. One here and the follow up patch in HDFS-7056. We will post a combined patch here for jenkins once the edits are regenerated.

          > I wasn't aware of HDFS-7056.

          I am glad you are now. It was created almots two months ago and was mentioned in this jira 8 times, including one of your own comments.
          Colin, it is not a problem by itself when people catch up on what they missed. But I am confused about your objections. You seem to be putting pressure on the development of this feature, but doing it without having full information at hand.

          Show
          Konstantin Shvachko added a comment - Submitted two patches separetly. One here and the follow up patch in HDFS-7056 . We will post a combined patch here for jenkins once the edits are regenerated. > I wasn't aware of HDFS-7056 . I am glad you are now. It was created almots two months ago and was mentioned in this jira 8 times, including one of your own comments. Colin, it is not a problem by itself when people catch up on what they missed. But I am confused about your objections. You seem to be putting pressure on the development of this feature, but doing it without having full information at hand.
          Hide
          Plamen Jeliazkov added a comment -

          Attaching refreshed patch for latest trunk.

          Show
          Plamen Jeliazkov added a comment - Attaching refreshed patch for latest trunk.
          Hide
          Colin Patrick McCabe added a comment -

          Great. If there are no objections, later today I will create a feature branch for this and commit the patch on this JIRA to it. (Since it has my +1 for committing to feature branch and we've discussed it a bunch). I'm looking forward to seeing the other patches which add snapshot support and shell support. Thanks, everyone.

          Show
          Colin Patrick McCabe added a comment - Great. If there are no objections, later today I will create a feature branch for this and commit the patch on this JIRA to it. (Since it has my +1 for committing to feature branch and we've discussed it a bunch). I'm looking forward to seeing the other patches which add snapshot support and shell support. Thanks, everyone.
          Hide
          Roman Shaposhnik added a comment -

          Would love to see it in a feature branch.

          Show
          Roman Shaposhnik added a comment - Would love to see it in a feature branch.
          Hide
          Colin Patrick McCabe added a comment -

          Thanks for the clarification. I wasn't aware of HDFS-7056. The design there looks reasonable.

          I would prefer to see this work in a feature branch. My reasoning is that if the first patch breaks rollback, it's tough to see it getting into trunk. What do others think?

          Show
          Colin Patrick McCabe added a comment - Thanks for the clarification. I wasn't aware of HDFS-7056 . The design there looks reasonable. I would prefer to see this work in a feature branch. My reasoning is that if the first patch breaks rollback, it's tough to see it getting into trunk. What do others think?
          Hide
          Konstantin Shvachko added a comment -

          Let me try to clarify.

          1. The latest attached patch (Oct 17) does not support snapshots. We are targeting to post a patch for truncate with snapshots under HDFS-7056. We are in the testing stage there.
          2. I did comment on rollbacks just yesterday. Copy replica on truncate solves the issue. Also addressed in the comming patch under HDFS-7056.
          3. The solution you described in your comment is exactly the one we are implementing here. Or I do not understand how it differs from the design document.

          Can we put numbers on patch files? I find it difficult to keep track of them when they all have the same file name.
          Colin I usually sort attachments by date. That way you do need to track any file names or numbers. Actually was asking many times to change the default for sorting the attachements, but never got a solution for that. Sould have been so much easier to just look at them in the order of submission, same as comments.

          Show
          Konstantin Shvachko added a comment - Let me try to clarify. The latest attached patch (Oct 17) does not support snapshots. We are targeting to post a patch for truncate with snapshots under HDFS-7056 . We are in the testing stage there. I did comment on rollbacks just yesterday . Copy replica on truncate solves the issue. Also addressed in the comming patch under HDFS-7056 . The solution you described in your comment is exactly the one we are implementing here. Or I do not understand how it differs from the design document. Can we put numbers on patch files? I find it difficult to keep track of them when they all have the same file name. Colin I usually sort attachments by date. That way you do need to track any file names or numbers. Actually was asking many times to change the default for sorting the attachements, but never got a solution for that. Sould have been so much easier to just look at them in the order of submission, same as comments.
          Hide
          Colin Patrick McCabe added a comment -

          So, but the new patch from Plamen Jeliazkov already have snapshot support

          Am I looking at the wrong version? The patch posted on 17 Oct does not seem to have snapshot support. The relevant code is here:

          2242	  boolean truncateInternal(String src, long newLength,
          2243	                           String clientName, String clientMachine,
          2244	                           long mtime, FSPermissionChecker pc)
          2245	      throws IOException, UnresolvedLinkException {
          2246	    assert hasWriteLock();
          2247	    if (isPermissionEnabled) {
          2248	      checkPathAccess(pc, src, FsAction.WRITE);
          2249	    }
          2250	    INodesInPath iip = dir.getINodesInPath4Write(src, true);
          2251	    final int latestSnapshotId = iip.getLatestSnapshotId();
          2252	    INodeFile inodeFile = INodeFile.valueOf(iip.getLastINode(), src);
          2253	    // Data will be lost after truncate occurs so it cannot support snapshots.
          2254	    if(inodeFile.isInLatestSnapshot(latestSnapshotId))
          2255	      throw new HadoopIllegalArgumentException(
          2256	          "Cannot truncate file with snapshot.");
          

          I think it's misleading to refer to throwing an exception when there are snapshots present as "support." But whatever we call it, my patch doesn't throw an exception here-- it always allows truncation even when the admin is using snapshots.

          I was trying to be helpful and point out an approach that would let you respond to the comments here. For example, Suresh Srinivas 's comment here:
          https://issues.apache.org/jira/browse/HDFS-3107?focusedCommentId=14155537&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14155537 Or Tsz Wo Nicholas Sze's comment about the need to support rollback. You need to address the comments that people have made.

          P.S. Can we put numbers on patch files? I find it difficult to keep track of them when they all have the same file name.

          Show
          Colin Patrick McCabe added a comment - So, but the new patch from Plamen Jeliazkov already have snapshot support Am I looking at the wrong version? The patch posted on 17 Oct does not seem to have snapshot support. The relevant code is here: 2242 boolean truncateInternal( String src, long newLength, 2243 String clientName, String clientMachine, 2244 long mtime, FSPermissionChecker pc) 2245 throws IOException, UnresolvedLinkException { 2246 assert hasWriteLock(); 2247 if (isPermissionEnabled) { 2248 checkPathAccess(pc, src, FsAction.WRITE); 2249 } 2250 INodesInPath iip = dir.getINodesInPath4Write(src, true ); 2251 final int latestSnapshotId = iip.getLatestSnapshotId(); 2252 INodeFile inodeFile = INodeFile.valueOf(iip.getLastINode(), src); 2253 // Data will be lost after truncate occurs so it cannot support snapshots. 2254 if (inodeFile.isInLatestSnapshot(latestSnapshotId)) 2255 throw new HadoopIllegalArgumentException( 2256 "Cannot truncate file with snapshot." ); I think it's misleading to refer to throwing an exception when there are snapshots present as "support." But whatever we call it, my patch doesn't throw an exception here-- it always allows truncation even when the admin is using snapshots. I was trying to be helpful and point out an approach that would let you respond to the comments here. For example, Suresh Srinivas 's comment here: https://issues.apache.org/jira/browse/HDFS-3107?focusedCommentId=14155537&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14155537 Or Tsz Wo Nicholas Sze 's comment about the need to support rollback. You need to address the comments that people have made. P.S. Can we put numbers on patch files? I find it difficult to keep track of them when they all have the same file name.
          Hide
          Konstantin Boudnik added a comment -

          So, but the new patch from Plamen Jeliazkov already have snapshot support. Hence I'll repeat: what's the point of having an alternative version of the original patch?

          Show
          Konstantin Boudnik added a comment - So, but the new patch from Plamen Jeliazkov already have snapshot support. Hence I'll repeat: what's the point of having an alternative version of the original patch?
          Hide
          Colin Patrick McCabe added a comment -

          The point is that the original implementation doesn't support snapshots and rollback. It needs to support those two things before it can be committed to trunk.

          Show
          Colin Patrick McCabe added a comment - The point is that the original implementation doesn't support snapshots and rollback. It needs to support those two things before it can be committed to trunk.
          Hide
          Konstantin Boudnik added a comment -

          Sorry, perhaps I missing something, but what's the point of the alternative implementation of the original patch? As far as I see Plamen Jeliazkov's version doesn't have any shortcomings as per the current design. What are you trying to achieve exactly?

          Show
          Konstantin Boudnik added a comment - Sorry, perhaps I missing something, but what's the point of the alternative implementation of the original patch? As far as I see Plamen Jeliazkov 's version doesn't have any shortcomings as per the current design. What are you trying to achieve exactly?
          Hide
          Colin Patrick McCabe added a comment -

          And it needs to be atomic e.g. not involving 5 RPC calls, otherwise recovery would be a nightmare.

          The block-copying solution could be done with 1 RPC. The NN receives the truncate RPC. It marks the file as open-for-write, and creates a new block ID for the last block. Then it returns to the client. Later, when the DNs which have the last block check in with the NN via heartbeats, the NN sends them a (new) command that leads them to duplicate the block file with a new ID. Then when the NN receives word that these blocks have been duplicated, it closes the file and truncate is complete.

          This is pretty similar to the original patch, except the original patch had the NN send a command that led the DNs to truncate the block on-disk (destroying the part of it after the truncation point). Just like in the original patch, the client may need to call isFileClosed a few times to wait for the truncation to finish.

          Show
          Colin Patrick McCabe added a comment - And it needs to be atomic e.g. not involving 5 RPC calls, otherwise recovery would be a nightmare. The block-copying solution could be done with 1 RPC. The NN receives the truncate RPC. It marks the file as open-for-write, and creates a new block ID for the last block. Then it returns to the client. Later, when the DNs which have the last block check in with the NN via heartbeats, the NN sends them a (new) command that leads them to duplicate the block file with a new ID. Then when the NN receives word that these blocks have been duplicated, it closes the file and truncate is complete. This is pretty similar to the original patch, except the original patch had the NN send a command that led the DNs to truncate the block on-disk (destroying the part of it after the truncation point). Just like in the original patch, the client may need to call isFileClosed a few times to wait for the truncation to finish.
          Hide
          Konstantin Shvachko added a comment -

          Yes, good point, Nicholas.
          During upgrade old blocks are hard linked and the hard links are stored in a separate directory, which allows the blocks themselves be deleted while the file system is updated. So when we roll back the deleted blocks are still available via those hard links.
          With truncate the same is applied to the blocks that were deleted. For the blocks that are truncated to a smaller length we will need to do copy-on-truncate recovery, same as we do for snapshots. That is if upgrade is in progress NN will schedule copy-on-truncate wether snapshots are present or not.
          The bottom line is that implementing copy-on-truncate is needed both for snapshots and upgrades. I'll make a note to update the design.

          Show
          Konstantin Shvachko added a comment - Yes, good point, Nicholas. During upgrade old blocks are hard linked and the hard links are stored in a separate directory, which allows the blocks themselves be deleted while the file system is updated. So when we roll back the deleted blocks are still available via those hard links. With truncate the same is applied to the blocks that were deleted. For the blocks that are truncated to a smaller length we will need to do copy-on-truncate recovery, same as we do for snapshots. That is if upgrade is in progress NN will schedule copy-on-truncate wether snapshots are present or not. The bottom line is that implementing copy-on-truncate is needed both for snapshots and upgrades. I'll make a note to update the design.
          Hide
          Konstantin Boudnik added a comment -

          I posted it as a demonstration. I think to make it more robust we would want

          And it needs to be atomic e.g. not involving 5 RPC calls, otherwise recovery would be a nightmare.

          Show
          Konstantin Boudnik added a comment - I posted it as a demonstration. I think to make it more robust we would want And it needs to be atomic e.g. not involving 5 RPC calls, otherwise recovery would be a nightmare.
          Hide
          Colin Patrick McCabe added a comment -

          Good point, Nicholas.

          Note that the patch I posted does handle rollback correctly, since it never modifies any existing block files.

          I posted it as a demonstration. I think to make it more robust we would want to avoid having the client write out the last block and concat it, and instead have some other mechanism for duplicating + shortening the final block of the file-- possibly a new DN command similar to COPY_BLOCK, but taking a length argument.

          Show
          Colin Patrick McCabe added a comment - Good point, Nicholas. Note that the patch I posted does handle rollback correctly, since it never modifies any existing block files. I posted it as a demonstration. I think to make it more robust we would want to avoid having the client write out the last block and concat it, and instead have some other mechanism for duplicating + shortening the final block of the file-- possibly a new DN command similar to COPY_BLOCK, but taking a length argument.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          In a HDFS upgrade, how to roll back a truncated file? The current design seems not covering it.

          Show
          Tsz Wo Nicholas Sze added a comment - In a HDFS upgrade, how to roll back a truncated file? The current design seems not covering it.
          Hide
          Plamen Jeliazkov added a comment -

          M. C. Srivas,
          There is no plan to grow the file by padding it with zeroes as general-purpose truncate does. Both Konstantin Shvachko and Lei Chang mentioned this in their design docs, I believe.

          Colin Patrick McCabe,
          While the copying the last block up to its truncate point and doing a delete/concat is definitely a simpler overall approach, the full truncate implementation has the benefit of being a single NameNode RPC call that can both truncate in-place and copy-on-truncate, preserving the original last block and moving the 'copy&truncate' work to the DataNodes themselves (as opposed to having to pass data through the network / client). I am not intending to debate either implementation – I like both personally; just wanted to explain as briefly as I could why Konstantin and I are taking our approach.

          Show
          Plamen Jeliazkov added a comment - M. C. Srivas , There is no plan to grow the file by padding it with zeroes as general-purpose truncate does. Both Konstantin Shvachko and Lei Chang mentioned this in their design docs, I believe. Colin Patrick McCabe , While the copying the last block up to its truncate point and doing a delete/concat is definitely a simpler overall approach, the full truncate implementation has the benefit of being a single NameNode RPC call that can both truncate in-place and copy-on-truncate, preserving the original last block and moving the 'copy&truncate' work to the DataNodes themselves (as opposed to having to pass data through the network / client). I am not intending to debate either implementation – I like both personally; just wanted to explain as briefly as I could why Konstantin and I are taking our approach.
          Hide
          M. C. Srivas added a comment -

          Note that a general-purpose truncate can be used to also increase the size of the file. Used very often, for example, to implement a database and growing the file if it isn't large enough. Are you planning to implement truncate to behave so too?

          Show
          M. C. Srivas added a comment - Note that a general-purpose truncate can be used to also increase the size of the file. Used very often, for example, to implement a database and growing the file if it isn't large enough. Are you planning to implement truncate to behave so too?
          Hide
          Colin Patrick McCabe added a comment -

          fix log message which should be trace, not info

          Show
          Colin Patrick McCabe added a comment - fix log message which should be trace, not info
          Hide
          Colin Patrick McCabe added a comment -

          Hi all,

          Here's a patch which implements truncate in such a way that it works with snapshots.

          This doesn't modify the last replica file of the truncated file in place. Instead, it writes out a new file with the new (shorter) contents of the last replica file, and uses concat to combine it with the first part of the file.

          Show
          Colin Patrick McCabe added a comment - Hi all, Here's a patch which implements truncate in such a way that it works with snapshots. This doesn't modify the last replica file of the truncated file in place. Instead, it writes out a new file with the new (shorter) contents of the last replica file, and uses concat to combine it with the first part of the file.
          Hide
          Todd Lipcon added a comment -

          Dhruba, Indeed. Lack of concurrent writes to a single HDFS file means that there will be only a single outstanding transaction against a file (unless the concurrency is implemented at a higher level.) A database can consist of multiple files, though, and one can have multiple outstanding transactions against the database (one per file.) In either case, rollback is achieved by truncating the file to position prior to beginning of transaction.

          I don't know the architecture of this particular database, but it seems to me that this is not the method that most databases use to issue an abort. Instead, an abort record is logged. Abort is a rare operation, and optimizing it by truncating the log doesn't seem worth the complexity to me.

          Do you have some reference which explains why the truncate is advantageous for a database log vs logging a new abort record? Typical database logging architectures based on ARIES couldn't really work this way, since aborting a transaction may involve rolling back changes which have already been persisted to the on-disk state (ie applying UNDO records). Truncation can only work in a REDO-only logging scheme (what ARIES calls "no-steal" iirc).

          Show
          Todd Lipcon added a comment - Dhruba, Indeed. Lack of concurrent writes to a single HDFS file means that there will be only a single outstanding transaction against a file (unless the concurrency is implemented at a higher level.) A database can consist of multiple files, though, and one can have multiple outstanding transactions against the database (one per file.) In either case, rollback is achieved by truncating the file to position prior to beginning of transaction. I don't know the architecture of this particular database, but it seems to me that this is not the method that most databases use to issue an abort. Instead, an abort record is logged. Abort is a rare operation, and optimizing it by truncating the log doesn't seem worth the complexity to me. Do you have some reference which explains why the truncate is advantageous for a database log vs logging a new abort record? Typical database logging architectures based on ARIES couldn't really work this way, since aborting a transaction may involve rolling back changes which have already been persisted to the on-disk state (ie applying UNDO records). Truncation can only work in a REDO-only logging scheme (what ARIES calls "no-steal" iirc).
          Hide
          Konstantin Shvachko added a comment -

          A more snapshot-friendly version of Plamen's patch.

          • Changed parameter types for some internal methods to accomodate snapshot changes.
          • Moved collectBlocksBeyondMax() into INodeFile in order to use it in and simplify unprotectedTruncate().
          • added disk usage verification to the test.
          Show
          Konstantin Shvachko added a comment - A more snapshot-friendly version of Plamen's patch. Changed parameter types for some internal methods to accomodate snapshot changes. Moved collectBlocksBeyondMax() into INodeFile in order to use it in and simplify unprotectedTruncate() . added disk usage verification to the test.
          Hide
          Plamen Jeliazkov added a comment -

          Refreshing patch.

          Made a couple changes:

          1. TruncateOp now acts like an AddOp (it opens the file for write).
          2. Added updateSpaceConsumed() call in FSDirectory.unprotectedTruncate().
          3. Replaced asserts in tests with assertThat() calls.
          4. Also attaching new editsStored and editsStored.xml files.
          Show
          Plamen Jeliazkov added a comment - Refreshing patch. Made a couple changes: TruncateOp now acts like an AddOp (it opens the file for write). Added updateSpaceConsumed() call in FSDirectory.unprotectedTruncate(). Replaced asserts in tests with assertThat() calls. Also attaching new editsStored and editsStored.xml files.
          Hide
          dhruba borthakur added a comment -

          Thanks for the clarification Milind. I was just making sure that i understand the limitations of such a database the uses the HDFS truncate feature. Given this fact, it is unlikely that HBase can use it (in future) to support transactions.

          Thanks anyways.

          Show
          dhruba borthakur added a comment - Thanks for the clarification Milind. I was just making sure that i understand the limitations of such a database the uses the HDFS truncate feature. Given this fact, it is unlikely that HBase can use it (in future) to support transactions. Thanks anyways.
          Hide
          Milind Bhandarkar added a comment -

          Dhruba, Indeed. Lack of concurrent writes to a single HDFS file means that there will be only a single outstanding transaction against a file (unless the concurrency is implemented at a higher level.) A database can consist of multiple files, though, and one can have multiple outstanding transactions against the database (one per file.) In either case, rollback is achieved by truncating the file to position prior to beginning of transaction.

          Show
          Milind Bhandarkar added a comment - Dhruba, Indeed. Lack of concurrent writes to a single HDFS file means that there will be only a single outstanding transaction against a file (unless the concurrency is implemented at a higher level.) A database can consist of multiple files, though, and one can have multiple outstanding transactions against the database (one per file.) In either case, rollback is achieved by truncating the file to position prior to beginning of transaction.
          Hide
          dhruba borthakur added a comment -

          Thanks KonstantinS. I get it now.

          Just to make it clear, my questions/comments are neither to support this patch nor to reject it. I just want to understand if the 'proposed truncate' feature can be used by databases like HBase.

          KonstantinS/Milind: If the database depends on the truncate feature of HDFS to rollback a transaction, won't it mean that there can be only one outstanding transaction per hdfs-file? Won't it be better for the database to support multiple outstanding transactions in parallel and have the ability to rollback (if needed) only one without having to rollback the others?

          Show
          dhruba borthakur added a comment - Thanks KonstantinS. I get it now. Just to make it clear, my questions/comments are neither to support this patch nor to reject it. I just want to understand if the 'proposed truncate' feature can be used by databases like HBase. KonstantinS/Milind: If the database depends on the truncate feature of HDFS to rollback a transaction, won't it mean that there can be only one outstanding transaction per hdfs-file? Won't it be better for the database to support multiple outstanding transactions in parallel and have the ability to rollback (if needed) only one without having to rollback the others?
          Hide
          Konstantin Shvachko added a comment -

          Dhruba, this is not my story to tell. The use case was introduced earlier by Milind and he discussed it with Hairong based on the comments above. So you may have a local reference there.
          I personally see it useful for transaction management tools / systems, which use HDFS as a reliable storage for their journal.
          When processing begin-transaction op the transaction manager (TM) writes this op into the journal and remembers the offset (Offset1) in the journal file where it was written. Then it processes operations in the transaction and writes them into the journal file. If commit-transaction happens TM just keeps writing to the journal. If TM gets and abort-transaction op, then TM truncates the journal file to Offset1 and starts appending to it for further transactions. That way if TM is restarted it sees only committed transactions.
          Hope this makes sense.

          Show
          Konstantin Shvachko added a comment - Dhruba, this is not my story to tell. The use case was introduced earlier by Milind and he discussed it with Hairong based on the comments above. So you may have a local reference there. I personally see it useful for transaction management tools / systems, which use HDFS as a reliable storage for their journal. When processing begin-transaction op the transaction manager (TM) writes this op into the journal and remembers the offset (Offset1) in the journal file where it was written. Then it processes operations in the transaction and writes them into the journal file. If commit-transaction happens TM just keeps writing to the journal. If TM gets and abort-transaction op, then TM truncates the journal file to Offset1 and starts appending to it for further transactions. That way if TM is restarted it sees only committed transactions. Hope this makes sense.
          Hide
          dhruba borthakur added a comment -

          Hi KonstantinS, I have been following this jira, mostly as a passive observer.

          Can you pl explain me the use-case for truncate? You might have already explained this earlier, but if you could again elaborate the reason why you need truncate. I would appreciate it a lot.

          from your comments, it feels that you have a database layer on top of hdfs and the database is using an hdfs file as the transaction log. But I am not able to understand the rest of the story.

          Show
          dhruba borthakur added a comment - Hi KonstantinS, I have been following this jira, mostly as a passive observer. Can you pl explain me the use-case for truncate? You might have already explained this earlier, but if you could again elaborate the reason why you need truncate. I would appreciate it a lot. from your comments, it feels that you have a database layer on top of hdfs and the database is using an hdfs file as the transaction log. But I am not able to understand the rest of the story.
          Hide
          Konstantin Shvachko added a comment -

          Even with truncate, snapshot creation does not require moving data. The data movement happens at truncate time. Is it correct?

          Good point, Nicholas. You are right the data moves on truncate in option 3 or op append in option 2. I'll rephrase this.

          Jing Zhao thanks for offering your help. I'll reach out offline to check on your availability.

          Suresh Srinivas I thought you would notice that I am not arguing about the integration with snapshots or postponing it, or exposing it to the users until done.
          I am arguing that the faster way is to commit this patch since it is ready and get into snapshots integration.

          Setting aside commercial distributions and superiority of features.
          It used to be a normal development path to split a new feature into parts and commit them to trunk as long as the parts are consistent and the feature is small, since nobody is running trunk in production. I wonder what happened with that practice. Truncate has only 3 subtasks. It is nowhere near 136 tasks that went into HDFS-2802, which was done on its branch.

          Show
          Konstantin Shvachko added a comment - Even with truncate, snapshot creation does not require moving data. The data movement happens at truncate time. Is it correct ? Good point, Nicholas. You are right the data moves on truncate in option 3 or op append in option 2. I'll rephrase this. Jing Zhao thanks for offering your help. I'll reach out offline to check on your availability. Suresh Srinivas I thought you would notice that I am not arguing about the integration with snapshots or postponing it, or exposing it to the users until done. I am arguing that the faster way is to commit this patch since it is ready and get into snapshots integration. Setting aside commercial distributions and superiority of features. It used to be a normal development path to split a new feature into parts and commit them to trunk as long as the parts are consistent and the feature is small, since nobody is running trunk in production. I wonder what happened with that practice. Truncate has only 3 subtasks. It is nowhere near 136 tasks that went into HDFS-2802 , which was done on its branch.
          Hide
          Colin Patrick McCabe added a comment - - edited

          So to summarize all this above, myself, Jing Zhao, and Suresh Srinivas have made the point that this feature is still incomplete until it supports snapshots. Since snapshot support may involve fundamentally changing the design (e.g. copying the final partial block file versus using block recovery), we need to figure it out before merging to trunk or branch-2.

          As far as I can see, there are two options here. We could roll the snapshots support into this patch, or start a feature branch with the above commit... and then do snapshots support (and whatever else is needed) in that feature branch. I'm fine with either option, I have absolutely no preference. I'm happy to review anything and Jing has offered to help with snapshots support.

          Roman Shaposhnik: I realize that getting truncate into a release is important to you. If people get this done by 2.6, I wouldn't oppose putting it in. But you will have to convince the release manager for 2.6, and propose it to the community. Since 2.6 is already a very big release, I think you will get pushback.

          I also think you should evaluate writing length-delimited records instead of using truncate. Truncate is an operation that can fail, and relying on truncate to clean up mistakes will always be more fragile than writing in a format that can ignore torn records automatically. If a client gets an error from DFSOutputStream#write because it has dropped off the network, truncate is also going to fail for the same reason. And then you have a torn record.

          Show
          Colin Patrick McCabe added a comment - - edited So to summarize all this above, myself, Jing Zhao , and Suresh Srinivas have made the point that this feature is still incomplete until it supports snapshots. Since snapshot support may involve fundamentally changing the design (e.g. copying the final partial block file versus using block recovery), we need to figure it out before merging to trunk or branch-2. As far as I can see, there are two options here. We could roll the snapshots support into this patch, or start a feature branch with the above commit... and then do snapshots support (and whatever else is needed) in that feature branch. I'm fine with either option, I have absolutely no preference. I'm happy to review anything and Jing has offered to help with snapshots support. Roman Shaposhnik : I realize that getting truncate into a release is important to you. If people get this done by 2.6, I wouldn't oppose putting it in. But you will have to convince the release manager for 2.6, and propose it to the community. Since 2.6 is already a very big release, I think you will get pushback. I also think you should evaluate writing length-delimited records instead of using truncate. Truncate is an operation that can fail, and relying on truncate to clean up mistakes will always be more fragile than writing in a format that can ignore torn records automatically. If a client gets an error from DFSOutputStream#write because it has dropped off the network, truncate is also going to fail for the same reason. And then you have a torn record.
          Hide
          Colin Patrick McCabe added a comment -

          I think the approach that one new feature is treated should be equally used for another. I am looking at HDFS-6994 and see subtasks getting committed despite the fact that they were rejected in the parent. Am I missing some subtle differences between two features?

          Subtasks were not "rejected in the parent." Allen Wittenauer made a comment that he doesn't like the name of the library, and he doesn't like the fact that it uses boost. We agreed to consider these points once the code is in the branch since they can easily be changed later. It's clear that nobody likes the current name, and it will almost certainly be renamed before it gets merged. That's the point of a feature branch-- the code and the design are flexible while they're in a feature branch.

          Show
          Colin Patrick McCabe added a comment - I think the approach that one new feature is treated should be equally used for another. I am looking at HDFS-6994 and see subtasks getting committed despite the fact that they were rejected in the parent. Am I missing some subtle differences between two features? Subtasks were not "rejected in the parent." Allen Wittenauer made a comment that he doesn't like the name of the library, and he doesn't like the fact that it uses boost. We agreed to consider these points once the code is in the branch since they can easily be changed later. It's clear that nobody likes the current name, and it will almost certainly be renamed before it gets merged. That's the point of a feature branch-- the code and the design are flexible while they're in a feature branch.
          Hide
          Konstantin Boudnik added a comment -

          I think the approach that one new feature is treated should be equally used for another. I am looking at HDFS-6994 and see subtasks getting committed despite the fact that they were rejected in the parent. Am I missing some subtle differences between two features?

          Show
          Konstantin Boudnik added a comment - I think the approach that one new feature is treated should be equally used for another. I am looking at HDFS-6994 and see subtasks getting committed despite the fact that they were rejected in the parent. Am I missing some subtle differences between two features?
          Hide
          Konstantin Boudnik added a comment -

          Sorry, what I meant so say is that yes, you're right but the actual use of the feature is questionable - sounded better in my head

          Show
          Konstantin Boudnik added a comment - Sorry, what I meant so say is that yes, you're right but the actual use of the feature is questionable - sounded better in my head
          Hide
          Konstantin Boudnik added a comment -

          Most users of commercial distros are using snapshots

          This is a hearsay as far as I know

          Show
          Konstantin Boudnik added a comment - Most users of commercial distros are using snapshots This is a hearsay as far as I know
          Hide
          Suresh Srinivas added a comment -

          In fact, if I were to rank the importance of snapshots vs. truncate to the majority of our users, I'd say truncate functionality is way more important.

          But the snapshots feature is already in the older releases (available since 2.1.0). All Colin Patrick McCabe is saying is, it is an existing feature and is currently used by many deployments. So debating a feature that already exists vs. a new feature is not useful.

          I agree with Jing Zhao's comment. We should address the interaction of this feature with Snapshots. Having a feature that fails in unpredictable ways when a file has been snapshotted vs. not is confusing. Postponing that will also result in an API that changes its behavior.

          Show
          Suresh Srinivas added a comment - In fact, if I were to rank the importance of snapshots vs. truncate to the majority of our users, I'd say truncate functionality is way more important. But the snapshots feature is already in the older releases (available since 2.1.0). All Colin Patrick McCabe is saying is, it is an existing feature and is currently used by many deployments. So debating a feature that already exists vs. a new feature is not useful. I agree with Jing Zhao 's comment. We should address the interaction of this feature with Snapshots. Having a feature that fails in unpredictable ways when a file has been snapshotted vs. not is confusing. Postponing that will also result in an API that changes its behavior.
          Hide
          Roman Shaposhnik added a comment -

          Most users of commercial distros are using snapshots, so even if we pulled this into 2.6 and then into the commercial releases based on it, the feature still wouldn't get tested. And anyway it's not going to make 2.6, so let's not get an artificial sense of urgency here. We have time to get the design right and test it.

          I think we can only speak on behalf of the companies that we represent. Making a generic statement that starts with "Most users of commercial distros" is just bad form. With that in mind, I can assure you that regardless of the use cases you see around CDH, we (PHD) see it differently. In fact, if I were to rank the importance of snapshots vs. truncate to the majority of our users, I'd say truncate functionality is way more important.

          Now, this is not to say that we should be basing decisions affecting an open source project on the immediate needs of any commercial distribution. This is just pointing out that the statement above is false.

          Show
          Roman Shaposhnik added a comment - Most users of commercial distros are using snapshots, so even if we pulled this into 2.6 and then into the commercial releases based on it, the feature still wouldn't get tested. And anyway it's not going to make 2.6, so let's not get an artificial sense of urgency here. We have time to get the design right and test it. I think we can only speak on behalf of the companies that we represent. Making a generic statement that starts with "Most users of commercial distros" is just bad form. With that in mind, I can assure you that regardless of the use cases you see around CDH, we (PHD) see it differently. In fact, if I were to rank the importance of snapshots vs. truncate to the majority of our users, I'd say truncate functionality is way more important. Now, this is not to say that we should be basing decisions affecting an open source project on the immediate needs of any commercial distribution. This is just pointing out that the statement above is false.
          Hide
          Colin Patrick McCabe added a comment -

          [The boolean] is an optimization for the case when truncate happens on the block boundary. Clients will save one RPC call in this particular case. From NameNode perspective returning the boolean does not require any extra processing.

          Fair enough. Can we change it to return an enum of { TRUNCATE_IN_PROGRESS, TRUNCATE_COMPLETED }? A lot of developers just don't read documentation and will assume "it returned false, that means it failed."

          [concurrent readers discussion]

          It's true that concurrent unlink has the same problem with making readers get mysterious IOExceptions. I don't consider this a good behavior to copy, though (it's more like a bug). Perhaps in a follow-up change, we could add some faculty for clients to ask the NN for information about whether the file has been unlinked or truncated after an unrecoverable IOException happened during a read? That's probably better as a follow-up, though.

          as HDFS-7056 indicated it will take more time to come up with design and implementation of the complimentary functionality that would extend truncate to snapshotted files

          Clearly, we're still at the stage where the design isn't complete. This is exactly what feature branches are for.

          in its current form, this is an extremely useful self-contained feature that allows various vendors of solutions running on Hadoop to build products having having much easier time running on HDFS.... we all know that features sitting in a branch don't get exposed to commercial distributions and workloads as much as the ones hitting trunk do. This is, of course, a totally right approach to features that are half-baked or not self-contained, but it feels that in this particular case committing the patch would benefit us all by giving customers access to the self-contained feature AND start receiving feedback for the more extended functionality much earlier.

          Most users of commercial distros are using snapshots, so even if we pulled this into 2.6 and then into the commercial releases based on it, the feature still wouldn't get tested. And anyway it's not going to make 2.6, so let's not get an artificial sense of urgency here. We have time to get the design right and test it.

          Show
          Colin Patrick McCabe added a comment - [The boolean] is an optimization for the case when truncate happens on the block boundary. Clients will save one RPC call in this particular case. From NameNode perspective returning the boolean does not require any extra processing. Fair enough. Can we change it to return an enum of { TRUNCATE_IN_PROGRESS , TRUNCATE_COMPLETED }? A lot of developers just don't read documentation and will assume "it returned false, that means it failed." [concurrent readers discussion] It's true that concurrent unlink has the same problem with making readers get mysterious IOExceptions. I don't consider this a good behavior to copy, though (it's more like a bug). Perhaps in a follow-up change, we could add some faculty for clients to ask the NN for information about whether the file has been unlinked or truncated after an unrecoverable IOException happened during a read? That's probably better as a follow-up, though. as HDFS-7056 indicated it will take more time to come up with design and implementation of the complimentary functionality that would extend truncate to snapshotted files Clearly, we're still at the stage where the design isn't complete. This is exactly what feature branches are for. in its current form, this is an extremely useful self-contained feature that allows various vendors of solutions running on Hadoop to build products having having much easier time running on HDFS.... we all know that features sitting in a branch don't get exposed to commercial distributions and workloads as much as the ones hitting trunk do. This is, of course, a totally right approach to features that are half-baked or not self-contained, but it feels that in this particular case committing the patch would benefit us all by giving customers access to the self-contained feature AND start receiving feedback for the more extended functionality much earlier. Most users of commercial distros are using snapshots, so even if we pulled this into 2.6 and then into the commercial releases based on it, the feature still wouldn't get tested. And anyway it's not going to make 2.6, so let's not get an artificial sense of urgency here. We have time to get the design right and test it.
          Hide
          Jing Zhao added a comment - - edited

          Actually I agree with Colin Patrick McCabe that it's better to finish HDFS-7056 before committing the current patch into trunk, especially considering the patch contains a DistributedFileSystem API already. Thus a feature branch makes sense to me.

          I have not thought through the details about HDFS-7056 yet. But it does not seem to be a very hard problem. I will also be happy to help there.

          Show
          Jing Zhao added a comment - - edited Actually I agree with Colin Patrick McCabe that it's better to finish HDFS-7056 before committing the current patch into trunk, especially considering the patch contains a DistributedFileSystem API already. Thus a feature branch makes sense to me. I have not thought through the details about HDFS-7056 yet. But it does not seem to be a very hard problem. I will also be happy to help there.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Checked the design doc. It looks quite good. Just a minor comment below:

          Current implementation of snapshots does not require moving any data. Both approaches 2 and 3 mean that this property will be lost. Some data will need to be moved for the snapshots to work with truncates and appends.

          Even with truncate, snapshot creation does not require moving data. The data movement happens at truncate time. Is it correct?

          Show
          Tsz Wo Nicholas Sze added a comment - Checked the design doc. It looks quite good. Just a minor comment below: Current implementation of snapshots does not require moving any data. Both approaches 2 and 3 mean that this property will be lost. Some data will need to be moved for the snapshots to work with truncates and appends. Even with truncate, snapshot creation does not require moving data. The data movement happens at truncate time. Is it correct?
          Hide
          Konstantin Shvachko added a comment -

          Colin Patrick McCabe feels like some clarification is needed from you here, so that we could avoid misunderstanding like in the other jira.

          1. I assume that your veto of Sept 17 conditional on the design document had been addressed. Now with subsequent corrections in line with your suggestions.
          2. Earlier this week you stated
            we should hold off on committing to trunk until we figure out the snapshot story.
            Given the design in place, the subtask HDFS-7056 opened, and the general acceptance of the approach by other contributors, the snapshot story seem to be clear.

          So if your statement is a veto could you please clarify the reason(s)? If not please say so, so that people could proceed with the subtasks? I hope you can reply by tomorrow.

          Show
          Konstantin Shvachko added a comment - Colin Patrick McCabe feels like some clarification is needed from you here, so that we could avoid misunderstanding like in the other jira. I assume that your veto of Sept 17 conditional on the design document had been addressed. Now with subsequent corrections in line with your suggestions. Earlier this week you stated we should hold off on committing to trunk until we figure out the snapshot story. Given the design in place, the subtask HDFS-7056 opened, and the general acceptance of the approach by other contributors, the snapshot story seem to be clear. So if your statement is a veto could you please clarify the reason(s)? If not please say so, so that people could proceed with the subtasks? I hope you can reply by tomorrow.
          Hide
          Roman Shaposhnik added a comment -

          FWIW I would like to provide a few additional datapoints to what Konstantin Shvachko has said:

          1. in its current form, this is an extremely useful self-contained feature that allows various vendors of solutions running on Hadoop to build products having having much easier time running on HDFS.
          2. it is true that currently there's not immediate integration with snapshot functionality, but the way the current patch is implemented makes it extremely easy to expand the scope of the feature to snapshots. In other words, if this current implementation gets committed it will NOT create a migration opportunity. The snapshot+truncate can be added into later releases of HDFS and applications targeting truncate as it is implemented currently will continue to run unmodified.
          3. as HDFS-7056 indicated it will take more time to come up with design and implementation of the complimentary functionality that would extend truncate to snapshotted files. It feels unfortunate if we had to hold the current patch hostage, even though today it delivers a very much needed functionality AND it allows for smooth migration for when snapshot+truncate gets implemented.
          4. we all know that features sitting in a branch don't get exposed to commercial distributions and workloads as much as the ones hitting trunk do. This is, of course, a totally right approach to features that are half-baked or not self-contained, but it feels that in this particular case committing the patch would benefit us all by giving customers access to the self-contained feature AND start receiving feedback for the more extended functionality much earlier.

          Hope this provides additional food for thought to reconsider this patch for inclusion. Also, FWIW, based on our testing, this feels like an extremely useful and important feature to get into Hadoop now and extend to cover snapshots later.

          Show
          Roman Shaposhnik added a comment - FWIW I would like to provide a few additional datapoints to what Konstantin Shvachko has said: in its current form, this is an extremely useful self-contained feature that allows various vendors of solutions running on Hadoop to build products having having much easier time running on HDFS. it is true that currently there's not immediate integration with snapshot functionality, but the way the current patch is implemented makes it extremely easy to expand the scope of the feature to snapshots. In other words, if this current implementation gets committed it will NOT create a migration opportunity. The snapshot+truncate can be added into later releases of HDFS and applications targeting truncate as it is implemented currently will continue to run unmodified. as HDFS-7056 indicated it will take more time to come up with design and implementation of the complimentary functionality that would extend truncate to snapshotted files. It feels unfortunate if we had to hold the current patch hostage, even though today it delivers a very much needed functionality AND it allows for smooth migration for when snapshot+truncate gets implemented. we all know that features sitting in a branch don't get exposed to commercial distributions and workloads as much as the ones hitting trunk do. This is, of course, a totally right approach to features that are half-baked or not self-contained, but it feels that in this particular case committing the patch would benefit us all by giving customers access to the self-contained feature AND start receiving feedback for the more extended functionality much earlier. Hope this provides additional food for thought to reconsider this patch for inclusion. Also, FWIW, based on our testing, this feels like an extremely useful and important feature to get into Hadoop now and extend to cover snapshots later.
          Hide
          Konstantin Shvachko added a comment -

          Updated the design doc.

          Show
          Konstantin Shvachko added a comment - Updated the design doc.
          Hide
          Konstantin Shvachko added a comment - - edited

          Thanks Dhruba and Colin for your reviews of the design document.
          Colin, I'll incorporate your suggestions. But it looks that you got everything right from the current edition.

          boolean truncate(Path src, long newLength). do we really need the boolean here?

          • This is an optimization for the case when truncate happens on the block boundary. Clients will save one RPC call in this particular case.
            From NameNode perspective returning the boolean does not require any extra processing.

          DFSInputStream#locatedBlocks will continue to have the block information it had prior to truncation.

          • Don't we have the same behaviour with deletes.
            Somebody can delete a file on the NameNode, but readers will keep reading old blocks until they are deleted.
            Truncate doesn't add anything new in that regard.

          I don't think we should commit anything to trunk until we figure out how this integrates with snapshots.

          • You should have seen HDFS-7056 subtask. Mentioning it again to reassure there is no intension to avoid the snapshot issue.
          • People agreed above that they are OK implementing snapshot integration in a separate jira.
          • We also agreed not to port it to branch 2 until this is completed.
          • And there was a request to commit this sooner rather than later.
          • Besides, it seems from your comments that you yourself are in favour of option 3 for snapshots from the design.

          So the question arise what is not clear in the truncate-snapshot story and why you object committing anything to trunk?

          Show
          Konstantin Shvachko added a comment - - edited Thanks Dhruba and Colin for your reviews of the design document. Colin, I'll incorporate your suggestions. But it looks that you got everything right from the current edition. boolean truncate(Path src, long newLength) . do we really need the boolean here ? This is an optimization for the case when truncate happens on the block boundary. Clients will save one RPC call in this particular case. From NameNode perspective returning the boolean does not require any extra processing. DFSInputStream#locatedBlocks will continue to have the block information it had prior to truncation. Don't we have the same behaviour with deletes . Somebody can delete a file on the NameNode, but readers will keep reading old blocks until they are deleted. Truncate doesn't add anything new in that regard. I don't think we should commit anything to trunk until we figure out how this integrates with snapshots. You should have seen HDFS-7056 subtask. Mentioning it again to reassure there is no intension to avoid the snapshot issue. People agreed above that they are OK implementing snapshot integration in a separate jira. We also agreed not to port it to branch 2 until this is completed. And there was a request to commit this sooner rather than later. Besides, it seems from your comments that you yourself are in favour of option 3 for snapshots from the design. So the question arise what is not clear in the truncate-snapshot story and why you object committing anything to trunk?
          Hide
          Colin Patrick McCabe added a comment -

          Thanks for waiting. I'm checking out the design doc.

          In proposed approach truncate is performed only on a closed file. If the file is opened for write an 
          attempt to truncate fails.
          

          Just a style change, but maybe "Truncate cannot be performed on a file which is currently open for writing" would be clearer.

          Conceptually, truncate removes all full blocks of the file and then starts a recovery process for the 
          last block if it is not fully truncated. The truncate recovery is similar to standard HDFS lease recovery
          procedure. That is, NameNode sends a DatanodeCommand to one of the DataNodes containing block 
          replicas. The primary DataNode synchronizes the new length among the replicas, and then confirms it to 
          the NameNode by sending commitBlockSynchronization() message, which completes the 
          truncate. Until the truncate recovery is complete the file is assigned a lease, which revokes the ability for 
          other clients to modify that file.
          

          I think a diagram might help here. The impression I'm getting is that we have some "truncation point" like this:

                         truncation point
                              |
                              V 
          +-----+-----+-----+-----+-----+-----+
          | A   | B   | C   | D   | E   | F   |
          +-----+-----+-----+-----+-----+-----+
          

          In this case, blocks E and F would be invalidated by the NameNode, and block recovery would begin on block D?

          "Conceptually, truncate removes all full blocks of the file" seems to suggest we're removing all blocks, so it might be nice to rewrite this as "Truncate removes all full blocks after the truncation point."

           Full blocks if any are deleted instantaneously. And if there is nothing more to truncate NameNode returns success to the client.
          

          They're invalidated instantly, but not deleted instantly, right? Clients may still be reading from them on the various datanodes.

          public boolean truncate(Path src, long newLength)
          throws IOException;
          Truncate file src to the specified newLength.
          Returns:
          - true if the file have been truncated to the desired newLength and is immediately available to 
          be reused for write operations such as append, or
          - false if a background process of adjusting the length of the last block has been started, and 
          clients should wait for it to complete before they can proceed with further file updates.
          

          Hmm, do we really need the boolean here? It seems like the client could simply try to reopen the file until it no longer got an RecoveryInProgressException. (or lease exception, as the case may be.) The client will have to do this anyway most of the time, since most truncates don't fall on even block boundaries.

           It should be noted that applications that cache data may still see old bytes of the file stored 
          in the cache. It is advised for such applications to incorporate techniques, which would retire cache 
          when the data is truncated.
          

          One issue that I see here is that DFSInputStream users will continue to see the old, longer length for a long time potentially. DFSInputStream#locatedBlocks will continue to have the block information it had prior to truncation. And eventually, whenever they try to read from that longer length, they'll get read failures since the blocks will actually be unlinked. These will look like IOExceptions to the user. I don't know if there's a good way around this problem with the design proposed here.

          [truncate with snapshots]

          I don't think we should commit anything to trunk until we figure out how this integrates with snapshots. It just impacts the design too much. When you start seriously thinking about snapshots, integrating this with block recovery (by adding BEING_TRUNCATED, etc.) does not look like a very good option. A better option would be simply to copy the partial block and have the snapshotted version reference the old block, and the new version reference the (shorter) copy. That corresponds to your approach #3, right? truncate is presumably a rare operation and doing the truncation in-place for non-snapshotted files is an optimization we could do later.

          The copy approach is also nice for DFSInputStream, since readers can continue reading from the old (longer) copy until the readers close. If we truncated that copy directly, this would not work.

          We could commit this to a branch, but I think we should hold off on committing to trunk until we figure out the snapshot story.

          Show
          Colin Patrick McCabe added a comment - Thanks for waiting. I'm checking out the design doc. In proposed approach truncate is performed only on a closed file. If the file is opened for write an attempt to truncate fails. Just a style change, but maybe "Truncate cannot be performed on a file which is currently open for writing" would be clearer. Conceptually, truncate removes all full blocks of the file and then starts a recovery process for the last block if it is not fully truncated. The truncate recovery is similar to standard HDFS lease recovery procedure. That is, NameNode sends a DatanodeCommand to one of the DataNodes containing block replicas. The primary DataNode synchronizes the new length among the replicas, and then confirms it to the NameNode by sending commitBlockSynchronization() message, which completes the truncate. Until the truncate recovery is complete the file is assigned a lease, which revokes the ability for other clients to modify that file. I think a diagram might help here. The impression I'm getting is that we have some "truncation point" like this: truncation point | V +-----+-----+-----+-----+-----+-----+ | A | B | C | D | E | F | +-----+-----+-----+-----+-----+-----+ In this case, blocks E and F would be invalidated by the NameNode, and block recovery would begin on block D? "Conceptually, truncate removes all full blocks of the file" seems to suggest we're removing all blocks, so it might be nice to rewrite this as "Truncate removes all full blocks after the truncation point." Full blocks if any are deleted instantaneously. And if there is nothing more to truncate NameNode returns success to the client. They're invalidated instantly, but not deleted instantly, right? Clients may still be reading from them on the various datanodes. public boolean truncate(Path src, long newLength) throws IOException; Truncate file src to the specified newLength. Returns: - true if the file have been truncated to the desired newLength and is immediately available to be reused for write operations such as append, or - false if a background process of adjusting the length of the last block has been started, and clients should wait for it to complete before they can proceed with further file updates. Hmm, do we really need the boolean here? It seems like the client could simply try to reopen the file until it no longer got an RecoveryInProgressException. (or lease exception, as the case may be.) The client will have to do this anyway most of the time, since most truncates don't fall on even block boundaries. It should be noted that applications that cache data may still see old bytes of the file stored in the cache. It is advised for such applications to incorporate techniques, which would retire cache when the data is truncated. One issue that I see here is that DFSInputStream users will continue to see the old, longer length for a long time potentially. DFSInputStream#locatedBlocks will continue to have the block information it had prior to truncation. And eventually, whenever they try to read from that longer length, they'll get read failures since the blocks will actually be unlinked. These will look like IOExceptions to the user. I don't know if there's a good way around this problem with the design proposed here. [truncate with snapshots] I don't think we should commit anything to trunk until we figure out how this integrates with snapshots. It just impacts the design too much. When you start seriously thinking about snapshots, integrating this with block recovery (by adding BEING_TRUNCATED , etc.) does not look like a very good option. A better option would be simply to copy the partial block and have the snapshotted version reference the old block, and the new version reference the (shorter) copy. That corresponds to your approach #3, right? truncate is presumably a rare operation and doing the truncation in-place for non-snapshotted files is an optimization we could do later. The copy approach is also nice for DFSInputStream , since readers can continue reading from the old (longer) copy until the readers close. If we truncated that copy directly, this would not work. We could commit this to a branch, but I think we should hold off on committing to trunk until we figure out the snapshot story.
          Hide
          dhruba borthakur added a comment -

          The design-doc looks cool. And thanks for waiting to commit, Konstantin.

          Show
          dhruba borthakur added a comment - The design-doc looks cool. And thanks for waiting to commit, Konstantin.
          Hide
          Konstantin Shvachko added a comment -

          Hi Dhruba, good to hear from you.
          The design was actually outlined on Sept 5. The idea is rather simple. We just formalized it in a document per Colin's request last week. More reviews is better. So if people need time lets wait until end of day Monday Sept 29.
          Hope Roman is OK with that?

          Show
          Konstantin Shvachko added a comment - Hi Dhruba, good to hear from you. The design was actually outlined on Sept 5 . The idea is rather simple. We just formalized it in a document per Colin's request last week. More reviews is better. So if people need time lets wait until end of day Monday Sept 29. Hope Roman is OK with that?
          Hide
          dhruba borthakur added a comment - - edited

          hi konstantin, is it possible to wait for a few more days before committing? from the comments, it feels that aaron might want to do a detailed review of the design. Especially since the design doc was added only 4 days ago and the 'truncate' feature is not trivial.

          also, feel free to ignore my comment in case u feel strongly about committing it today.

          Show
          dhruba borthakur added a comment - - edited hi konstantin, is it possible to wait for a few more days before committing? from the comments, it feels that aaron might want to do a detailed review of the design. Especially since the design doc was added only 4 days ago and the 'truncate' feature is not trivial. also, feel free to ignore my comment in case u feel strongly about committing it today.
          Hide
          Konstantin Boudnik added a comment -

          Don't miss the hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFileTruncate.java that needs to be explicitely added to the commit as it's a new file.

          Show
          Konstantin Boudnik added a comment - Don't miss the hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFileTruncate.java that needs to be explicitely added to the commit as it's a new file.
          Hide
          Konstantin Shvachko added a comment -

          +1 works for me. Will commit a bit later today.

          Show
          Konstantin Shvachko added a comment - +1 works for me. Will commit a bit later today.
          Hide
          Plamen Jeliazkov added a comment -

          Seems the FindBugs was introduced in HDFS-4257, which added that condition field in ReplaceDatanodeOnFailure.java.
          We should not handle it as part of this work.

          I see 2 builds before mine was also with the same FindBugs warning:
          https://builds.apache.org/job/PreCommit-HDFS-Build/8216//artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html

          Show
          Plamen Jeliazkov added a comment - Seems the FindBugs was introduced in HDFS-4257 , which added that condition field in ReplaceDatanodeOnFailure.java. We should not handle it as part of this work. I see 2 builds before mine was also with the same FindBugs warning: https://builds.apache.org/job/PreCommit-HDFS-Build/8216//artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
          Hide
          Konstantin Boudnik added a comment -

          but it seems that it is unrelated to the truncate patch.

          I looked it up and it doesn't seem to be related, indeed. I am ok with leaving it as is.

          be aware that you have to also commit the attached 'editsStored' file

          Any particular reason why the binary file wasn't simply added to the patch? Just curious...

          Show
          Konstantin Boudnik added a comment - but it seems that it is unrelated to the truncate patch. I looked it up and it doesn't seem to be related, indeed. I am ok with leaving it as is. be aware that you have to also commit the attached 'editsStored' file Any particular reason why the binary file wasn't simply added to the patch? Just curious...
          Hide
          Plamen Jeliazkov added a comment -

          The FindBugs addition seems to be for a class that is unmodified by my patch:
          "Class org.apache.hadoop.hdfs.protocol.datatransfer.ReplaceDatanodeOnFailure$Policy defines non-transient non-serializable instance field condition"

          I am not sure how that happened. I could remedy it by adding a transient modifier to the Condition field 'condition', but it seems that it is unrelated to the truncate patch.
          Probably best to keep the separation of concerns.

          The failure in TestEncryptionZonesWithKMS was due to a Mini HDFS Cluster setup timing out.
          I have ran TestEncryptionZonesWithKMS locally and seen it succeed with my patch applied.

          If you are going to commit be aware that you have to also commit the attached 'editsStored' file into the hadoop-hdfs-project/hadoop-hdfs/src/test/resources directory and overwrite the file that is already there.

          Show
          Plamen Jeliazkov added a comment - The FindBugs addition seems to be for a class that is unmodified by my patch: "Class org.apache.hadoop.hdfs.protocol.datatransfer.ReplaceDatanodeOnFailure$Policy defines non-transient non-serializable instance field condition" I am not sure how that happened. I could remedy it by adding a transient modifier to the Condition field 'condition', but it seems that it is unrelated to the truncate patch. Probably best to keep the separation of concerns. The failure in TestEncryptionZonesWithKMS was due to a Mini HDFS Cluster setup timing out. I have ran TestEncryptionZonesWithKMS locally and seen it succeed with my patch applied. If you are going to commit be aware that you have to also commit the attached 'editsStored' file into the hadoop-hdfs-project/hadoop-hdfs/src/test/resources directory and overwrite the file that is already there.
          Hide
          Konstantin Shvachko added a comment -

          Thanks for testing, Roman. We also tested on our side.
          TestEncryptionZonesWithKMS has been unstable for a while and is not related to the patch.
          TestOfflineEditsViewer fails as expected.
          Waiting for Findbugs resolution. Ready to commit after that, imho.

          Show
          Konstantin Shvachko added a comment - Thanks for testing, Roman. We also tested on our side. TestEncryptionZonesWithKMS has been unstable for a while and is not related to the patch. TestOfflineEditsViewer fails as expected. Waiting for Findbugs resolution. Ready to commit after that, imho.
          Hide
          Roman Shaposhnik added a comment -

          FWIW: we've tested it in whatever limited capacity and it feels like a great bit of functionality to have in HDFS.

          Any chance this can be decoupled from the subtasks and committed as-is now that all concerns have been taken care of?

          Show
          Roman Shaposhnik added a comment - FWIW: we've tested it in whatever limited capacity and it feels like a great bit of functionality to have in HDFS. Any chance this can be decoupled from the subtasks and committed as-is now that all concerns have been taken care of?
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12671508/HDFS-3107.patch
          against trunk revision 1147b9a.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 6 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
          org.apache.hadoop.hdfs.TestEncryptionZonesWithKMS

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8219//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/8219//artifact/PreCommit-HADOOP-Build-patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8219//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12671508/HDFS-3107.patch against trunk revision 1147b9a. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 6 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 1 new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer org.apache.hadoop.hdfs.TestEncryptionZonesWithKMS +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8219//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/8219//artifact/PreCommit-HADOOP-Build-patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8219//console This message is automatically generated.
          Hide
          Plamen Jeliazkov added a comment -

          Seems the bot only tried to apply the binary file, editsStored, not my patch.

          Show
          Plamen Jeliazkov added a comment - Seems the bot only tried to apply the binary file, editsStored, not my patch.
          Hide
          Konstantin Boudnik added a comment -

          Plamen Jeliazkov, try to generate the patch using git format-patch or just git diff - it seems that the extra info added by IDEA is confusing the hell out of patch utility.

          Show
          Konstantin Boudnik added a comment - Plamen Jeliazkov , try to generate the patch using git format-patch or just git diff - it seems that the extra info added by IDEA is confusing the hell out of patch utility.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12671332/editsStored
          against trunk revision 8269bfa.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8214//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12671332/editsStored against trunk revision 8269bfa. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8214//console This message is automatically generated.
          Hide
          Plamen Jeliazkov added a comment -

          Attaching a new patch. I have included a separate file called "editsStored" to the attachments. This is a binary editLog segment that belongs in test/resources directory.
          Without that binary file in place we can expect TestOfflineEditsViewer.testStored() to fail. I cannot seem to add in a binary file to a patch.

          Turns out the following tests also needed to be fixed up to account for a new FSEditLogOp:

          1. TestNameNodeRetryCache
          2. TestRetryCacheWithHA
          1. I added in Jing Zhao's catch for the snapshot check. I now make use of INode.isInLatestSnapshot() instead.
          Show
          Plamen Jeliazkov added a comment - Attaching a new patch. I have included a separate file called "editsStored" to the attachments. This is a binary editLog segment that belongs in test/resources directory. Without that binary file in place we can expect TestOfflineEditsViewer.testStored() to fail. I cannot seem to add in a binary file to a patch. Turns out the following tests also needed to be fixed up to account for a new FSEditLogOp: TestNameNodeRetryCache TestRetryCacheWithHA I added in Jing Zhao 's catch for the snapshot check. I now make use of INode.isInLatestSnapshot() instead.
          Hide
          Konstantin Shvachko added a comment -

          (3) Cos is right the disabling config option for append was added to actually disable it on running clusters. It was a controversial but inavoidable decision.
          I am not sure if we need it now in case of truncate. If we will it can be introduced later?
          (4) As discussed earlier, indeed snapshots need to be addressed. Makes sense not to port it to branch 2 until HDFS-7056 and other subtasks are done.

          Show
          Konstantin Shvachko added a comment - (3) Cos is right the disabling config option for append was added to actually disable it on running clusters. It was a controversial but inavoidable decision. I am not sure if we need it now in case of truncate. If we will it can be introduced later? (4) As discussed earlier, indeed snapshots need to be addressed. Makes sense not to port it to branch 2 until HDFS-7056 and other subtasks are done.
          Hide
          Plamen Jeliazkov added a comment -

          Thanks for the points, Jing Zhao!

          1. I had a lurking suspicion that something was wrong with my snapshot check. Thanks for catching that.
          2. The BEING_TRUNCATED state is basically just an extension of UNDER_RECOVERY state; there should not be an issue with block reports then; the main purpose of BEING_TRUNCATED is to "force" the DataNode to truncate the block to the new length rather than let the DataNodes attempt to negotiate the size of the block (as they would in regular recovery).
          3. I am OK with adding configuration option if general opinion is that I should. I take it I should throw OperationNotSupportedException in such case?

          I will upload a new patch with those points addressed.

          Show
          Plamen Jeliazkov added a comment - Thanks for the points, Jing Zhao ! I had a lurking suspicion that something was wrong with my snapshot check. Thanks for catching that. The BEING_TRUNCATED state is basically just an extension of UNDER_RECOVERY state; there should not be an issue with block reports then; the main purpose of BEING_TRUNCATED is to "force" the DataNode to truncate the block to the new length rather than let the DataNodes attempt to negotiate the size of the block (as they would in regular recovery). I am OK with adding configuration option if general opinion is that I should. I take it I should throw OperationNotSupportedException in such case? I will upload a new patch with those points addressed.
          Hide
          Konstantin Boudnik added a comment -

          # Maybe we should also consider adding a configuration key to disable the functionality just like what we did for append in the past

          Wasn't that configuration option added post-factum to deal with issues of the first append implementation?

          Show
          Konstantin Boudnik added a comment - # Maybe we should also consider adding a configuration key to disable the functionality just like what we did for append in the past Wasn't that configuration option added post-factum to deal with issues of the first append implementation?
          Hide
          Jing Zhao added a comment -

          Thanks for working on this, Plamen! I have not done a detailed review yet, so far some quick comments on the design doc and patch:

          1. The following code is not the correct way to check if a file is contained in snapshots. isWithSnapshot only checks if an INodeFile has a snapshot feature, and a snapshot feature is added to this file only when the file is modified. Thus a file can be contained in snapshots but without snapshot feature on itself. So here we should use INode#isInLatestSnapshot for the checking.
            +    // Data will be lost after truncate occurs so it cannot support snapshots.
            +    if(inodeFile.isWithSnapshot())
            +      throw new HadoopIllegalArgumentException("Cannot truncate file with " +
            +          "snapshots enabled.");
            
          2. Maybe I'm missing something here, but do we also need to take the new BEING_TRUNCATED state into account when processing block reports (I guess the logic should be similar with UNDER_CONSTRUCTION)?
          3. Maybe we should also consider adding a configuration key to disable the functionality just like what we did for append in the past.
          4. In the design doc, for handling snapshots, either approach 2 or 3 looks good to me. However, to disallow truncate only when snapshot is taken does not look like a good semantic to me. In practice, this will cause a lot of trouble for both admin and applications using truncate. I suggest we finish this work before committing this work to trunk or branch-2.
          Show
          Jing Zhao added a comment - Thanks for working on this, Plamen! I have not done a detailed review yet, so far some quick comments on the design doc and patch: The following code is not the correct way to check if a file is contained in snapshots. isWithSnapshot only checks if an INodeFile has a snapshot feature, and a snapshot feature is added to this file only when the file is modified. Thus a file can be contained in snapshots but without snapshot feature on itself. So here we should use INode#isInLatestSnapshot for the checking. + // Data will be lost after truncate occurs so it cannot support snapshots. + if (inodeFile.isWithSnapshot()) + throw new HadoopIllegalArgumentException( "Cannot truncate file with " + + "snapshots enabled." ); Maybe I'm missing something here, but do we also need to take the new BEING_TRUNCATED state into account when processing block reports (I guess the logic should be similar with UNDER_CONSTRUCTION)? Maybe we should also consider adding a configuration key to disable the functionality just like what we did for append in the past. In the design doc, for handling snapshots, either approach 2 or 3 looks good to me. However, to disallow truncate only when snapshot is taken does not look like a good semantic to me. In practice, this will cause a lot of trouble for both admin and applications using truncate. I suggest we finish this work before committing this work to trunk or branch-2.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12671267/HDFS-3107.patch
          against trunk revision e0b1dc5.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8205//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12671267/HDFS-3107.patch against trunk revision e0b1dc5. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8205//console This message is automatically generated.
          Hide
          Plamen Jeliazkov added a comment -

          Attached new patch.

          In order to fix TestOfflineEditsViewer I needed to modify test code and resources/ files, namely editsStored binary file and editsStored.xml.
          TestPipelinesFailover and TestWebHdfsFileSystemContract passed on my local machine.

          Show
          Plamen Jeliazkov added a comment - Attached new patch. In order to fix TestOfflineEditsViewer I needed to modify test code and resources/ files, namely editsStored binary file and editsStored.xml. TestPipelinesFailover and TestWebHdfsFileSystemContract passed on my local machine.
          Hide
          Plamen Jeliazkov added a comment -
          1. I forgot to include the Apache License header in TestFileTruncate.
          2. TestOfflineEditsViewer and TestDFSInotifyEventInputStream appear to be caused by my introduction of a new FSEditLogOp; not sure why I didn't catch that in my run.

          I'll have these both fixed and a new patch generated.

          I am unsure about the failures in TestPipelinesFailover and TestWebHdfsFileSystemContract.

          Show
          Plamen Jeliazkov added a comment - I forgot to include the Apache License header in TestFileTruncate. TestOfflineEditsViewer and TestDFSInotifyEventInputStream appear to be caused by my introduction of a new FSEditLogOp; not sure why I didn't catch that in my run. I'll have these both fixed and a new patch generated. I am unsure about the failures in TestPipelinesFailover and TestWebHdfsFileSystemContract.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12671164/HDFS-3107.patch
          against trunk revision 428a766.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          -1 release audit. The applied patch generated 1 release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.web.TestWebHdfsFileSystemContract
          org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
          org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
          org.apache.hadoop.hdfs.TestDFSInotifyEventInputStream

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8199//testReport/
          Release audit warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/8199//artifact/PreCommit-HADOOP-Build-patchprocess/patchReleaseAuditProblems.txt
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8199//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12671164/HDFS-3107.patch against trunk revision 428a766. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. -1 release audit . The applied patch generated 1 release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.web.TestWebHdfsFileSystemContract org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer org.apache.hadoop.hdfs.TestDFSInotifyEventInputStream +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8199//testReport/ Release audit warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/8199//artifact/PreCommit-HADOOP-Build-patchprocess/patchReleaseAuditProblems.txt Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8199//console This message is automatically generated.
          Hide
          Plamen Jeliazkov added a comment -

          Refreshing patch. Trunk had moved on since my last patch.

          Thanks for the advice Aaron T. Myers. I added the line to checkOperation(OpCat.WRITE) after acquiring the writeLock as well now.

          Some other enhancements:

          1. Made use of the getAuditFileInfo() call instead of doing getFileInfo myself.
          2. Made use of the checkNameNodeSafeMode() call instead of doing the check myself.
          Show
          Plamen Jeliazkov added a comment - Refreshing patch. Trunk had moved on since my last patch. Thanks for the advice Aaron T. Myers . I added the line to checkOperation(OpCat.WRITE) after acquiring the writeLock as well now. Some other enhancements: Made use of the getAuditFileInfo() call instead of doing getFileInfo myself. Made use of the checkNameNodeSafeMode() call instead of doing the check myself.
          Hide
          Aaron T. Myers added a comment -

          I have not done any detailed review of the code or design, but in a quick look at the current patch I do see one little problem:

          +    checkOperation(OperationCategory.WRITE);
          +    boolean res;
          +    writeLock();
          +    try {
          +      if (isInSafeMode()) {
          +        throw new SafeModeException("Cannot truncate for " + src, safeMode);
          +      }
          

          In the above, you should be checking for OperationCategory.WRITE both before and after taking the write lock. Before to prevent clients from failing over slowly when contacting the standby NN, after to ensure correctness.

          Show
          Aaron T. Myers added a comment - I have not done any detailed review of the code or design, but in a quick look at the current patch I do see one little problem: + checkOperation(OperationCategory.WRITE); + boolean res; + writeLock(); + try { + if (isInSafeMode()) { + throw new SafeModeException( "Cannot truncate for " + src, safeMode); + } In the above, you should be checking for OperationCategory.WRITE both before and after taking the write lock. Before to prevent clients from failing over slowly when contacting the standby NN, after to ensure correctness.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12670760/HDFS-3107.patch
          against trunk revision f48686a.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8172//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12670760/HDFS-3107.patch against trunk revision f48686a. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8172//console This message is automatically generated.
          Hide
          Plamen Jeliazkov added a comment -

          Attaching patch with updated Javadoc.

          1. Changed ClientProtocol JavaDoc to match what Konstantin pointed out.
          2. Also added an @return JavaDoc for ClientProtocol.
          3. Modified the FSDirectory unprotectedTruncate() JavaDoc and comments to remove any notion of 'schedule block for truncate'. The scheduling logic lives in FSNamesystem.

          More tests to show when dealing with competing appends / creates are necessary. I'll include some in the next patch.

          Show
          Plamen Jeliazkov added a comment - Attaching patch with updated Javadoc. Changed ClientProtocol JavaDoc to match what Konstantin pointed out. Also added an @return JavaDoc for ClientProtocol. Modified the FSDirectory unprotectedTruncate() JavaDoc and comments to remove any notion of 'schedule block for truncate'. The scheduling logic lives in FSNamesystem. More tests to show when dealing with competing appends / creates are necessary. I'll include some in the next patch.
          Hide
          Konstantin Shvachko added a comment -

          Plamen could you please update JavaDocs per the design doc. I found one thing:

          • Fails if truncate is not to block boundary and someone has lease.
            This should be removed because block boundary is irrelevant and you already said before that truncate fails if file is not closed.
          Show
          Konstantin Shvachko added a comment - Plamen could you please update JavaDocs per the design doc. I found one thing: Fails if truncate is not to block boundary and someone has lease. This should be removed because block boundary is irrelevant and you already said before that truncate fails if file is not closed.
          Hide
          Konstantin Shvachko added a comment -

          Here is the design doc.

          Show
          Konstantin Shvachko added a comment - Here is the design doc.
          Hide
          Konstantin Shvachko added a comment -

          Design doc makes sense. Will do.

          Show
          Konstantin Shvachko added a comment - Design doc makes sense. Will do.
          Hide
          Colin Patrick McCabe added a comment -

          Using a record-length prefix is not a good fix to get around this. What happens if you fail when writing your record length?

          In that case, the record is incomplete and not valid. It's pretty clear when bytes are missing from a fixed-length 4 byte record.

          I would argue that this has everything to do with append. You are absolutely correct that HDFS can write a bad file on a standard open/write. The 'undo' for this failure is the delete operation. Your data integrity is preserved regardless of any external factors (file format, metadata, applications, etc). You can't have bad data if you never write bad data.

          I don't follow. What does append have to do with writing partial records? You can write partial records without append, and append doesn't make it any more or less likely.

          As I said earlier, "append" really should have been called "reopen for write". You don't need to use "append" to create and append to a file (confusing, I know)

          Show
          Colin Patrick McCabe added a comment - Using a record-length prefix is not a good fix to get around this. What happens if you fail when writing your record length? In that case, the record is incomplete and not valid. It's pretty clear when bytes are missing from a fixed-length 4 byte record. I would argue that this has everything to do with append. You are absolutely correct that HDFS can write a bad file on a standard open/write. The 'undo' for this failure is the delete operation. Your data integrity is preserved regardless of any external factors (file format, metadata, applications, etc). You can't have bad data if you never write bad data. I don't follow. What does append have to do with writing partial records? You can write partial records without append, and append doesn't make it any more or less likely. As I said earlier, "append" really should have been called "reopen for write". You don't need to use "append" to create and append to a file (confusing, I know)
          Hide
          Ryan Maus added a comment -

          Most HDFS users have logic to handle partial records, due to this problem. For example, Flume can roll to a new file when the old one has an error. It's pretty simple to prefix your records with a length, and simply ignore partial records that result from an incomplete flush to a file.

          Using a record-length prefix is not a good fix to get around this. What happens if you fail when writing your record length?

          I don't see what any of this has to do with append, since this issue could equally well come up without HDFS append. Remember that HDFS append really means "reopen for write." HDFS can have an error on writing and write a partial record even without anyone reopening for write.

          I would argue that this has everything to do with append. You are absolutely correct that HDFS can write a bad file on a standard open/write. The 'undo' for this failure is the delete operation. Your data integrity is preserved regardless of any external factors (file format, metadata, applications, etc). You can't have bad data if you never write bad data.

          The 'undo' for an reopen/write (append) failure is the truncate operation. To preserve data integrity independent of other factors, you have to truncate to the last known good size (we are assuming here that the existing file was written correctly).

          Show
          Ryan Maus added a comment - Most HDFS users have logic to handle partial records, due to this problem. For example, Flume can roll to a new file when the old one has an error. It's pretty simple to prefix your records with a length, and simply ignore partial records that result from an incomplete flush to a file. Using a record-length prefix is not a good fix to get around this. What happens if you fail when writing your record length? I don't see what any of this has to do with append, since this issue could equally well come up without HDFS append. Remember that HDFS append really means "reopen for write." HDFS can have an error on writing and write a partial record even without anyone reopening for write. I would argue that this has everything to do with append. You are absolutely correct that HDFS can write a bad file on a standard open/write. The 'undo' for this failure is the delete operation. Your data integrity is preserved regardless of any external factors (file format, metadata, applications, etc). You can't have bad data if you never write bad data. The 'undo' for an reopen/write (append) failure is the truncate operation. To preserve data integrity independent of other factors, you have to truncate to the last known good size (we are assuming here that the existing file was written correctly).
          Hide
          Colin Patrick McCabe added a comment -

          The main use case as far as I understand from this and other conversations is transaction handling for external databases. DB writes its transactions into a HDFS file. While transactions succeeds the DB keeps writing the same file. But when tx fails it is aborted and the file is truncated to the previous successfull transaction.

          As I mentioned earlier, the external database could simply use length-prefixed records. Then, if it encounters a partial record, it is ignored. Flume has been doing something like this for a while. So I don't see this as a very important use case.

          There is no divergent history. If you truncate you loose data that you truncated. You will not be able to open file for append until truncate is catually comleted and DNs shrink the last block replicas. Then file can be opened for append and add new data.

          OK, that's fair. The single-writer nature of HDFS makes this easier.

          There are also interactions with files open for read. The NameNode doesn't know which files are open for read so you cannot forbid this. Keep in mind that there is a generous amount of buffering inside DFSInputStream. So following a truncate + append new data, we may continue to read the "old" truncated data from the buffer inside the DFSInputStream's RemoteBlockReader2 for a while. That is partly what I meant by a "divergent history." This is probably ok, but the semantics need to be spelled out in the design doc.

          How this interacts with snapshots... is something yet to be designed

          OK.

          There is a patch attached. Did you have a chance to review? It is much simpler than append, but it does not allow to truncate files in snapshots. If we decide to implement copy-on-write approach for truncated files in snapshots, then we may end up creating a branch.

          I'm -1 on committing anything without a design doc. I apologize if this seems harsh, but I don't want there to be any ambiguity. I think you are on the right track, but let's see a complete design and then get started with committing the code. Thanks, Konstantin.

          Show
          Colin Patrick McCabe added a comment - The main use case as far as I understand from this and other conversations is transaction handling for external databases. DB writes its transactions into a HDFS file. While transactions succeeds the DB keeps writing the same file. But when tx fails it is aborted and the file is truncated to the previous successfull transaction. As I mentioned earlier, the external database could simply use length-prefixed records. Then, if it encounters a partial record, it is ignored. Flume has been doing something like this for a while. So I don't see this as a very important use case. There is no divergent history. If you truncate you loose data that you truncated. You will not be able to open file for append until truncate is catually comleted and DNs shrink the last block replicas. Then file can be opened for append and add new data. OK, that's fair. The single-writer nature of HDFS makes this easier. There are also interactions with files open for read. The NameNode doesn't know which files are open for read so you cannot forbid this. Keep in mind that there is a generous amount of buffering inside DFSInputStream. So following a truncate + append new data, we may continue to read the "old" truncated data from the buffer inside the DFSInputStream's RemoteBlockReader2 for a while. That is partly what I meant by a "divergent history." This is probably ok, but the semantics need to be spelled out in the design doc. How this interacts with snapshots... is something yet to be designed OK. There is a patch attached. Did you have a chance to review? It is much simpler than append, but it does not allow to truncate files in snapshots. If we decide to implement copy-on-write approach for truncated files in snapshots, then we may end up creating a branch. I'm -1 on committing anything without a design doc. I apologize if this seems harsh, but I don't want there to be any ambiguity. I think you are on the right track, but let's see a complete design and then get started with committing the code. Thanks, Konstantin.
          Hide
          Konstantin Shvachko added a comment -

          I also like Rayn's use case. The ability to restart appending to the same file after truncating incomplete data, which was introduced by a failed append during a previous session. Anything that results in larger files helps to releave pressure from the NameNode.

          Show
          Konstantin Shvachko added a comment - I also like Rayn's use case. The ability to restart appending to the same file after truncating incomplete data, which was introduced by a failed append during a previous session. Anything that results in larger files helps to releave pressure from the NameNode.
          Hide
          Konstantin Shvachko added a comment -

          Hey Colin Patrick McCabe, most of your questions are answered in my earlier comment. This is the design in a nutshell. The snapshots issue is discussed here.
          I'll try to answer your questions in more details.

          What are the use-cases.

          • The main use case as far as I understand from this and other conversations is transaction handling for external databases. DB writes its transactions into a HDFS file. While transactions succeeds the DB keeps writing the same file. But when tx fails it is aborted and the file is truncated to the previous successfull transaction.
            Yours are also good use cases.

          if some client reads a file at position X after it has been truncted to X-1, what might the client see?

          • The client will see EOF. Should be the same as in reading beyond the file length after it was created (no truncate).

          If data is appended after a truncate, it seems like we could get divergent histories in many situations, where one client sees one timeline and another client sees another.

          • There is no divergent history. If you truncate you loose data that you truncated. You will not be able to open file for append until truncate is catually comleted and DNs shrink the last block replicas. Then file can be opened for append and add new data.

          Are we going to guarantee that files never appear to shrink while clients have them open?

          • Correct, truncate can be applied only to closed file. If the file is opened for write an attempt to truncate fails.

          How does this interact with hflush and hsync?

          • Truncate is not applicable to open files, so it does not interact with hflush and hsync, which are applicable to open files only.

          How this interacts with snapshots.

          • This is something yet to be designed as Nicholas mentioned. Targeted in HDFS-7056. Three options have been proposed. I was looking at option two, where we reduce file length, but keep the data unchanged until it is needed in snapshots. You are right, this does not work with interleaving truncates and appends.
            Current implementation prohibits truncate if file has an active snapshot.

          how it's going to be implemented?

          • There is a patch attached. Did you have a chance to review? It is much simpler than append, but it does not allow to truncate files in snapshots. If we decide to implement copy-on-write approach for truncated files in snapshots, then we may end up creating a branch.
          Show
          Konstantin Shvachko added a comment - Hey Colin Patrick McCabe , most of your questions are answered in my earlier comment . This is the design in a nutshell. The snapshots issue is discussed here . I'll try to answer your questions in more details. What are the use-cases. The main use case as far as I understand from this and other conversations is transaction handling for external databases. DB writes its transactions into a HDFS file. While transactions succeeds the DB keeps writing the same file. But when tx fails it is aborted and the file is truncated to the previous successfull transaction. Yours are also good use cases. if some client reads a file at position X after it has been truncted to X-1, what might the client see ? The client will see EOF. Should be the same as in reading beyond the file length after it was created (no truncate). If data is appended after a truncate, it seems like we could get divergent histories in many situations, where one client sees one timeline and another client sees another. There is no divergent history. If you truncate you loose data that you truncated. You will not be able to open file for append until truncate is catually comleted and DNs shrink the last block replicas. Then file can be opened for append and add new data. Are we going to guarantee that files never appear to shrink while clients have them open ? Correct, truncate can be applied only to closed file. If the file is opened for write an attempt to truncate fails. How does this interact with hflush and hsync ? Truncate is not applicable to open files, so it does not interact with hflush and hsync, which are applicable to open files only. How this interacts with snapshots. This is something yet to be designed as Nicholas mentioned . Targeted in HDFS-7056 . Three options have been proposed . I was looking at option two, where we reduce file length, but keep the data unchanged until it is needed in snapshots. You are right, this does not work with interleaving truncates and appends. Current implementation prohibits truncate if file has an active snapshot. how it's going to be implemented ? There is a patch attached. Did you have a chance to review? It is much simpler than append, but it does not allow to truncate files in snapshots. If we decide to implement copy-on-write approach for truncated files in snapshots, then we may end up creating a branch.
          Hide
          Colin Patrick McCabe added a comment -

          However, an extremely common use-case is failed appends, where the writing application dies / is killed / etc, which results in writing corrupted or incomplete data to HDFS. The big problem here is that most HDFS guides suggest storing large amounts of data in each file (e.g. in a typical log-file setup, one simply appends each day's new data onto a much larger existing file). By not having a truncate command, this is essentially not supported.

          Most HDFS users have logic to handle partial records, due to this problem. For example, Flume can roll to a new file when the old one has an error. It's pretty simple to prefix your records with a length, and simply ignore partial records that result from an incomplete flush to a file.

          Append should never have been implemented without truncate.

          I don't see what any of this has to do with append, since this issue could equally well come up without HDFS append. Remember that HDFS append really means "reopen for write." HDFS can have an error on writing and write a partial record even without anyone reopening for write.

          This is not a particularly good use case, certainly not enough to justify the change we are discussing here. You can't protect users from every little potential accident.

          Indeed. As Suresh Srinivas already pointed out, adding truncate opens up the possibility of accidental truncation, which is worse than accidental appending. Also, in general, we have not seen users complain about accidentally appending data to a file. I can't even think of one support request or email about this issue. On the other hand, we have seen many cases where people have accidentally deleted things, and ask us to try to get the data back.

          In general, the main use-case I see here is better fuse and NFS support.

          Show
          Colin Patrick McCabe added a comment - However, an extremely common use-case is failed appends, where the writing application dies / is killed / etc, which results in writing corrupted or incomplete data to HDFS. The big problem here is that most HDFS guides suggest storing large amounts of data in each file (e.g. in a typical log-file setup, one simply appends each day's new data onto a much larger existing file). By not having a truncate command, this is essentially not supported. Most HDFS users have logic to handle partial records, due to this problem. For example, Flume can roll to a new file when the old one has an error. It's pretty simple to prefix your records with a length, and simply ignore partial records that result from an incomplete flush to a file. Append should never have been implemented without truncate. I don't see what any of this has to do with append, since this issue could equally well come up without HDFS append. Remember that HDFS append really means "reopen for write." HDFS can have an error on writing and write a partial record even without anyone reopening for write. This is not a particularly good use case, certainly not enough to justify the change we are discussing here. You can't protect users from every little potential accident. Indeed. As Suresh Srinivas already pointed out, adding truncate opens up the possibility of accidental truncation, which is worse than accidental appending. Also, in general, we have not seen users complain about accidentally appending data to a file. I can't even think of one support request or email about this issue. On the other hand, we have seen many cases where people have accidentally deleted things, and ask us to try to get the data back. In general, the main use-case I see here is better fuse and NFS support.
          Hide
          Ryan Maus added a comment -

          > What are the use-cases. Right now we have discussed only one use-case: allowing users to remove data they accidentially appended.

          This is not a particularly good use case, certainly not enough to justify the change we are discussing here. You can't protect users from every little potential accident.

          However, an extremely common use-case is failed appends, where the writing application dies / is killed / etc, which results in writing corrupted or incomplete data to HDFS. The big problem here is that most HDFS guides suggest storing large amounts of data in each file (e.g. in a typical log-file setup, one simply appends each day's new data onto a much larger existing file). By not having a truncate command, this is essentially not supported.

          Sooner or later you are going to have some kind of error during an append, and it will require you to copy-truncate a massive file to fix it. Append should never have been implemented without truncate.

          Show
          Ryan Maus added a comment - > What are the use-cases. Right now we have discussed only one use-case: allowing users to remove data they accidentially appended. This is not a particularly good use case, certainly not enough to justify the change we are discussing here. You can't protect users from every little potential accident. However, an extremely common use-case is failed appends, where the writing application dies / is killed / etc, which results in writing corrupted or incomplete data to HDFS. The big problem here is that most HDFS guides suggest storing large amounts of data in each file (e.g. in a typical log-file setup, one simply appends each day's new data onto a much larger existing file). By not having a truncate command, this is essentially not supported. Sooner or later you are going to have some kind of error during an append, and it will require you to copy-truncate a massive file to fix it. Append should never have been implemented without truncate.
          Hide
          Colin Patrick McCabe added a comment -

          Konstantin Shvachko, Plamen Jeliazkov, do you have a design doc for this? It's a pretty big change to HDFS semantics (previously, file data was immutable once written.)

          At minimum, we need to decide:

          • What the consistency semantics are (if some client reads a file at position X after it has been truncted to X-1, what might the client see?) If data is appended after a truncate, it seems like we could get divergent histories in many situations, where one client sees one timeline and another client sees another. Is this acceptable? If so, we'll need to describe what the behavior actually is. Are we going to guarantee that files never appear to shrink while clients have them open? How does this interact with hflush and hsync?
          • How this interacts with snapshots. When truncating a block and then re-appending, a divergent block will be created, invalidating previous snapshots. It seems that someone will have to copy this block, so the question is when. I see some discussion of that here in the comments, but I don't see any conclusions.
          • What are the use-cases. Right now we have discussed only one use-case: allowing users to remove data they accidentially appended. I would add one more use-case: improving support for fuse-dfs and NFS, which both require truncate to be implemented. Are there any additional use-cases?
          • And, of course... how it's going to be implemented

          I think this should be a branch since as the HDFS append experience showed, it may take some time to get everything right.

          Show
          Colin Patrick McCabe added a comment - Konstantin Shvachko , Plamen Jeliazkov , do you have a design doc for this? It's a pretty big change to HDFS semantics (previously, file data was immutable once written.) At minimum, we need to decide: What the consistency semantics are (if some client reads a file at position X after it has been truncted to X-1, what might the client see?) If data is appended after a truncate, it seems like we could get divergent histories in many situations, where one client sees one timeline and another client sees another. Is this acceptable? If so, we'll need to describe what the behavior actually is. Are we going to guarantee that files never appear to shrink while clients have them open? How does this interact with hflush and hsync? How this interacts with snapshots. When truncating a block and then re-appending, a divergent block will be created, invalidating previous snapshots. It seems that someone will have to copy this block, so the question is when. I see some discussion of that here in the comments, but I don't see any conclusions. What are the use-cases. Right now we have discussed only one use-case: allowing users to remove data they accidentially appended. I would add one more use-case: improving support for fuse-dfs and NFS, which both require truncate to be implemented. Are there any additional use-cases? And, of course... how it's going to be implemented I think this should be a branch since as the HDFS append experience showed, it may take some time to get everything right.
          Hide
          Konstantin Shvachko added a comment -

          Created three subtasks to cover remaining work related to truncate.

          Show
          Konstantin Shvachko added a comment - Created three subtasks to cover remaining work related to truncate.
          Hide
          Plamen Jeliazkov added a comment -

          Attaching an initial patch for discussion.

          I looked at the PDFs by Lei Chang and I do not see this breaking the API and ideas proposed there.

          Konstantin's comment above with the steps of truncate are how this patch works.

          Some additional notes I'd like to make here about the patch:

          1. I am making use of FsDatasetImpl's truncateBlock() method for performing the actual truncation. That method does not appear to do copy-on-write; it just truncates the original block.
          2. When truncation is completed on the DNs it is reported thru the primary DN's commitBlockSynchronization call.
          3. The file length on the INodeFile will change with the truncate call.

          I think we will be able to support snapshots if I can keep the old last blocks as well as any truncated data around. Currently, I can see us keeping the old blocks around (if truncate is done on block-boundary), but I do not have a way to keep the actual truncated data around. Probably more work needs to be done on FsDatasetImpl and FSNamesystem side. As Tsz Wo Nicholas Sze suggested, a full design will be required.

          Let me know what you guys think.
          I appreciate all feedback and will try to incorporate it carefully.

          Show
          Plamen Jeliazkov added a comment - Attaching an initial patch for discussion. I looked at the PDFs by Lei Chang and I do not see this breaking the API and ideas proposed there. Konstantin's comment above with the steps of truncate are how this patch works. Some additional notes I'd like to make here about the patch: I am making use of FsDatasetImpl's truncateBlock() method for performing the actual truncation. That method does not appear to do copy-on-write; it just truncates the original block. When truncation is completed on the DNs it is reported thru the primary DN's commitBlockSynchronization call. The file length on the INodeFile will change with the truncate call. I think we will be able to support snapshots if I can keep the old last blocks as well as any truncated data around. Currently, I can see us keeping the old blocks around (if truncate is done on block-boundary), but I do not have a way to keep the actual truncated data around. Probably more work needs to be done on FsDatasetImpl and FSNamesystem side. As Tsz Wo Nicholas Sze suggested, a full design will be required. Let me know what you guys think. I appreciate all feedback and will try to incorporate it carefully.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          > Can we add snapshot support in separate JIRA?

          Implementing it in a separate JIRA is okay. However, we must come up a design first.

          Show
          Tsz Wo Nicholas Sze added a comment - > Can we add snapshot support in separate JIRA? Implementing it in a separate JIRA is okay. However, we must come up a design first.
          Hide
          Plamen Jeliazkov added a comment -

          I have a patch with failure scenarios however I do not have snapshot support.
          Can we add snapshot support in separate JIRA?

          Show
          Plamen Jeliazkov added a comment - I have a patch with failure scenarios however I do not have snapshot support. Can we add snapshot support in separate JIRA?
          Hide
          Lei Chang added a comment -

          Method #3 does not work for this use case which is very important for database on HDFS. Users do a snapshot when there are some running data loading transactions. But after the snapshot is done, a transaction fails. But the transaction cannot be rollbacked. This is a big limitation. So, IMO, keeping the blocks until snapshot is removed is a better way to handle this.

          Show
          Lei Chang added a comment - Method #3 does not work for this use case which is very important for database on HDFS. Users do a snapshot when there are some running data loading transactions. But after the snapshot is done, a transaction fails. But the transaction cannot be rollbacked. This is a big limitation. So, IMO, keeping the blocks until snapshot is removed is a better way to handle this.
          Hide
          Konstantin Shvachko added a comment -

          I see your point. I'll let Plamen speak about current state of the art. Let's talk how it should be.

          1. The documentation on snapshots explicetly states "there is no data copying". So may be copy-on-write is not not desirable here although appealing.
          2. Another way is not to remove data during truncate if the file is in a snapshot. Just reduce the length, and deal with block removal / truncation when snapshot is removed. Sort of symmetrical to append.
          3. The simplest is to disallow truncate on files that are in a snapshot, as you indicated.
            May be we should do this first and add one of the above when a use case emerges?
          Show
          Konstantin Shvachko added a comment - I see your point. I'll let Plamen speak about current state of the art. Let's talk how it should be. The documentation on snapshots explicetly states "there is no data copying". So may be copy-on-write is not not desirable here although appealing. Another way is not to remove data during truncate if the file is in a snapshot. Just reduce the length, and deal with block removal / truncation when snapshot is removed. Sort of symmetrical to append. The simplest is to disallow truncate on files that are in a snapshot, as you indicated. May be we should do this first and add one of the above when a use case emerges?
          Hide
          Jing Zhao added a comment -

          Then the user cannot truncate the file to a length smaller than the largest length in snapshots? Or say whether a truncate can or cannot work may depend on whether there is a snapshot on the file. Maybe a better way to handle truncate + snapshot is to support copy-on-write snapshot on block replicas?

          Show
          Jing Zhao added a comment - Then the user cannot truncate the file to a length smaller than the largest length in snapshots? Or say whether a truncate can or cannot work may depend on whether there is a snapshot on the file. Maybe a better way to handle truncate + snapshot is to support copy-on-write snapshot on block replicas?
          Hide
          Konstantin Shvachko added a comment -

          Yes, I think it is. You cannot modify files (including truncate) inside a snapshot. Outside of it it is a regular operation. Do you see any restrictions?

          Show
          Konstantin Shvachko added a comment - Yes, I think it is. You cannot modify files (including truncate) inside a snapshot. Outside of it it is a regular operation. Do you see any restrictions?
          Hide
          Lei Chang added a comment -

          Is it compatible with snapshot?

          Show
          Lei Chang added a comment - Is it compatible with snapshot?
          Hide
          Lei Chang added a comment -

          Is it compatible with snapshot?

          Show
          Lei Chang added a comment - Is it compatible with snapshot?
          Hide
          Konstantin Shvachko added a comment -

          No, while under recovery the file has a lease so nobody can open it for append. Same as with lease recovery.

          Show
          Konstantin Shvachko added a comment - No, while under recovery the file has a lease so nobody can open it for append. Same as with lease recovery.
          Hide
          Jing Zhao added a comment -

          While the file remains in the under_recovery state while truncation, can the file still be appended?

          Show
          Jing Zhao added a comment - While the file remains in the under_recovery state while truncation, can the file still be appended?
          Hide
          Konstantin Shvachko added a comment -

          Nicholas in his comment proposed three approaches to implement truncate. Here is another one, which was mentioned in this comment of HDFS-6087.
          Conceptually, truncate removes all full blocks and then starts a recovery process for the last block which is not fully truncated. The truncate recovery is similar to lease recovery. That is, NN sends truncate-DatanodeCommand to one of the DNs containing block replicas. The primary DN synchronizes the new length between replicas, and then sends commitBlockSynchronization() to NN, which completes the truncate.
          Truncate will work only for closed files. If the file is opened for write an attempt to truncate fails.

          Here are the truncate steps in more details:

          • NN receives a truncate(src, newLength) call from a client.
          • Full blocks are deleted instantaneously. And if there is nothing more to truncate NN returns success to the client.
          • If not on the block boundary, then NN converts INode to INodeUnderConstruction and set file length to newLength.
          • The last blocks state is set to BEING_TRUNCATED.
          • Truncate operation is persisted in editLog.
          • NN triggers last block length recovery by sending DatanodeCommand and waits for the DN to report back.
          • File remains UNDER_RECOVERY until the recovery completes.
          • Lease expiration (soft or hard) will trigger last block recovery for truncate.
          • If NN restarts it will restart the recovery

          Assigning to Plamen, he seems to be almost ready with the patch.

          Show
          Konstantin Shvachko added a comment - Nicholas in his comment proposed three approaches to implement truncate. Here is another one, which was mentioned in this comment of HDFS-6087 . Conceptually, truncate removes all full blocks and then starts a recovery process for the last block which is not fully truncated. The truncate recovery is similar to lease recovery. That is, NN sends truncate-DatanodeCommand to one of the DNs containing block replicas. The primary DN synchronizes the new length between replicas, and then sends commitBlockSynchronization() to NN, which completes the truncate. Truncate will work only for closed files. If the file is opened for write an attempt to truncate fails. Here are the truncate steps in more details: NN receives a truncate(src, newLength) call from a client. Full blocks are deleted instantaneously. And if there is nothing more to truncate NN returns success to the client. If not on the block boundary, then NN converts INode to INodeUnderConstruction and set file length to newLength. The last blocks state is set to BEING_TRUNCATED. Truncate operation is persisted in editLog. NN triggers last block length recovery by sending DatanodeCommand and waits for the DN to report back. File remains UNDER_RECOVERY until the recovery completes. Lease expiration (soft or hard) will trigger last block recovery for truncate. If NN restarts it will restart the recovery Assigning to Plamen, he seems to be almost ready with the patch.
          Hide
          Lei Chang added a comment -

          > The proposed way to go about #3 by creating copies at the DN level and truncating there seems messy, but if you think about it as a variant of #2 that leaks less information into the API (block boundaries, contents of last segment), it seems simpler to me.

          Agree with you, if only looking at the simplicity of the internal RPC APIs, #3 is simpler. However, from the implementation part, in #3, clients need to work with both NN and DNs. There are many cases clients should take care when some nodes fails in the copy/truncate phase and some nodes succeed. For example:
          1) The client should work with NN to handle the failures and do some recovery when DN fails. It is somewhat like the pipeline rebuild and recovery in the APPEND case.
          2) Client fail introduces some extra work too. (#1 also has to deal with this case, but simpler)

          Thus, the implementation of #3 should be easier.

          You mentioned a good point about the security part of the temporary file. It should be created with the same access privilege to the file being truncated.

          Show
          Lei Chang added a comment - > The proposed way to go about #3 by creating copies at the DN level and truncating there seems messy, but if you think about it as a variant of #2 that leaks less information into the API (block boundaries, contents of last segment), it seems simpler to me. Agree with you, if only looking at the simplicity of the internal RPC APIs, #3 is simpler. However, from the implementation part, in #3, clients need to work with both NN and DNs. There are many cases clients should take care when some nodes fails in the copy/truncate phase and some nodes succeed. For example: 1) The client should work with NN to handle the failures and do some recovery when DN fails. It is somewhat like the pipeline rebuild and recovery in the APPEND case. 2) Client fail introduces some extra work too. (#1 also has to deal with this case, but simpler) Thus, the implementation of #3 should be easier. You mentioned a good point about the security part of the temporary file. It should be created with the same access privilege to the file being truncated.
          Hide
          Scott Carey added a comment -

          I think #2 and #3 can be the same thing, except that the system creates "concatFile" by reading some number of bytes from the last block, and that it does not throw an error if the passed in length is not on a block boundary in #3.

          The proposed way to go about #3 by creating copies at the DN level and truncating there seems messy, but if you think about it as a variant of #2 that leaks less information into the API (block boundaries, contents of last segment), it seems simpler to me.

          Show
          Scott Carey added a comment - I think #2 and #3 can be the same thing, except that the system creates "concatFile" by reading some number of bytes from the last block, and that it does not throw an error if the passed in length is not on a block boundary in #3. The proposed way to go about #3 by creating copies at the DN level and truncating there seems messy, but if you think about it as a variant of #2 that leaks less information into the API (block boundaries, contents of last segment), it seems simpler to me.
          Hide
          Lei Chang added a comment -

          Indicating EOF is reasonable, and this is exactly the behavior in the preliminary prototype built before.

          Show
          Lei Chang added a comment - Indicating EOF is reasonable, and this is exactly the behavior in the preliminary prototype built before.
          Hide
          M. C. Srivas added a comment -

          @Zhanwei and @TszWo: A comment on truncate & read interaction: The behavior of the read() system call in Posix is to return fewer number of bytes than asked for if EOF is encountered early. For example, if a file is of length 100 bytes, and a thread comes along and tries to read 200 bytes starting at offset 20, then read() should return 80. Subsequent calls to read() then return 0, to indicate EOF. The same principle can be applied to a file that gets truncated after it is opened for read ... treat it like a file that got shortened, ie, do a short read the first time, and raise the EOF exception subsequently.

          Show
          M. C. Srivas added a comment - @Zhanwei and @TszWo: A comment on truncate & read interaction: The behavior of the read() system call in Posix is to return fewer number of bytes than asked for if EOF is encountered early. For example, if a file is of length 100 bytes, and a thread comes along and tries to read 200 bytes starting at offset 20, then read() should return 80. Subsequent calls to read() then return 0, to indicate EOF. The same principle can be applied to a file that gets truncated after it is opened for read ... treat it like a file that got shortened, ie, do a short read the first time, and raise the EOF exception subsequently.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          > ... I also think it is better to not change the API ...

          Agree. Milind's idea on changing only the rpc and keeping the API unchanged is great.

          Show
          Tsz Wo Nicholas Sze added a comment - > ... I also think it is better to not change the API ... Agree. Milind's idea on changing only the rpc and keeping the API unchanged is great.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          > A problem of truncate is the "visibility". ...

          @Zhanwei, The behavior of truncate should be similar to delete. For delete, readers can continue reading the file after the file is deleted. The reader will fail if it further talks to the Namenode. For truncate, it seems okay to let the reader continue reading even beyond the truncated length and it will fail when it talks to the Namenode. What do you think?

          > One minor comment: please think about maintenance problems when you expose funky semantics ...

          @M.C., I got your point. We probably should annotate the new API as unstable or evolving in the first release.

          Show
          Tsz Wo Nicholas Sze added a comment - > A problem of truncate is the "visibility". ... @Zhanwei, The behavior of truncate should be similar to delete. For delete, readers can continue reading the file after the file is deleted. The reader will fail if it further talks to the Namenode. For truncate, it seems okay to let the reader continue reading even beyond the truncated length and it will fail when it talks to the Namenode. What do you think? > One minor comment: please think about maintenance problems when you expose funky semantics ... @M.C., I got your point. We probably should annotate the new API as unstable or evolving in the first release.
          Hide
          Lei Chang added a comment -

          > Lei, do you see any issues with the proposal (i.e option 2) ?

          I do like the second option , for its simplicity to implement atomicity. I also think it is better to not change the API which should be much easier to use for end users. truncate(file, length, concatFile) can be used internally.

          For the third option (1), it indeed introduces a lot of difficulties in implementation, such as fault tolerance implementation issues. This has been seen in our first try to implement truncate which takes a more strong semantic.

          IMO, for visibility, the weak consistency for concurrent read is ok for upper layer applications. For instance, most database systems already use their own locks to synchronize the concurrent access to files or file blocks.

          Show
          Lei Chang added a comment - > Lei, do you see any issues with the proposal (i.e option 2) ? I do like the second option , for its simplicity to implement atomicity. I also think it is better to not change the API which should be much easier to use for end users. truncate(file, length, concatFile) can be used internally. For the third option (1), it indeed introduces a lot of difficulties in implementation, such as fault tolerance implementation issues. This has been seen in our first try to implement truncate which takes a more strong semantic. IMO, for visibility, the weak consistency for concurrent read is ok for upper layer applications. For instance, most database systems already use their own locks to synchronize the concurrent access to files or file blocks.
          Hide
          Zhanwei Wang added a comment -

          Add more detail to my previous question, how to define "may read content of a file that will be truncated", that is the "visibility" problem. If a file is opened and read just before truncation, should the truncated data be visible? Or just depends on the process of truncation? What if a file is opened before truncation and read after truncation?

          Show
          Zhanwei Wang added a comment - Add more detail to my previous question, how to define "may read content of a file that will be truncated", that is the "visibility" problem. If a file is opened and read just before truncation, should the truncated data be visible? Or just depends on the process of truncation? What if a file is opened before truncation and read after truncation?
          Hide
          M. C. Srivas added a comment -

          One minor comment: please think about maintenance problems when you expose funky semantics that have been tacked on to truncate() ... people will start using it, and it will be hard/impossible to change. It is easy to add code, but very difficult to remove it later.

          I see that you need something like what's being proposed to implement snapshots, but it should be an internal-only API and not exposed.

          Show
          M. C. Srivas added a comment - One minor comment: please think about maintenance problems when you expose funky semantics that have been tacked on to truncate() ... people will start using it, and it will be hard/impossible to change. It is easy to add code, but very difficult to remove it later. I see that you need something like what's being proposed to implement snapshots, but it should be an internal-only API and not exposed.
          Hide
          Zhanwei Wang added a comment -

          A problem of truncate is the "visibility". Since to truncate a file needs to get the lease first, we do not need to take care of concurrent write, but we need to take care of concurrent read when we truncate a file. Hdfs client will buffer some block info when open and read a file, while these blocks may be truncated. Furthermore, socket and Hdfs client may buffer some data which may will be truncated.

          When I implement the first edition of truncate prototype, if the block or data the client required is truncated, datanode will throw a exception and client will update the metadata to check if the data is truncated or the real error happened. But this cannot prevent the client reading buffered data.

          Any comment and suggestion?

          Show
          Zhanwei Wang added a comment - A problem of truncate is the "visibility". Since to truncate a file needs to get the lease first, we do not need to take care of concurrent write, but we need to take care of concurrent read when we truncate a file. Hdfs client will buffer some block info when open and read a file, while these blocks may be truncated. Furthermore, socket and Hdfs client may buffer some data which may will be truncated. When I implement the first edition of truncate prototype, if the block or data the client required is truncated, datanode will throw a exception and client will update the metadata to check if the data is truncated or the real error happened. But this cannot prevent the client reading buffered data. Any comment and suggestion?
          Hide
          Milind Bhandarkar added a comment -

          We don't need unowned block. The block should be owned by the user since the data of the block is.

          Sorry, by "unowned", I meant block that does not belong to any file. If a separate block management becomes an external first class citizen, one could have existing blocks added to a new namespace (from what I remember in the vision discussions with Sanjay last year), and concat could take a list of blocks instead of a file.

          Lei, do you see any issues with the proposal (i.e option 2) ?

          Show
          Milind Bhandarkar added a comment - We don't need unowned block. The block should be owned by the user since the data of the block is. Sorry, by "unowned", I meant block that does not belong to any file. If a separate block management becomes an external first class citizen, one could have existing blocks added to a new namespace (from what I remember in the vision discussions with Sanjay last year), and concat could take a list of blocks instead of a file. Lei, do you see any issues with the proposal (i.e option 2) ?
          Hide
          Tsz Wo Nicholas Sze added a comment -

          > ... The namenode RPC should have length as exactly a multiple of blocksize ...

          Agree.

          > Is this futureproof ? What happens where block storage becomes first class citizen ? Will we have an ability to create unowned block, and append it to an existing (truncated to block boundary) file ?

          I think you mean separating namespace management and block management. The beauty of truncate with concat is that it does not introduce new block level operations so that there is no block management change (arguably, it makes sense adding a new operation for copying an existing block to a new block so that the copy can be done locally in datanodes but this not a big deal.) It shouldn't cause any problem.

          We don't need unowned block. The block should be owned by the user since the data of the block is.

          Show
          Tsz Wo Nicholas Sze added a comment - > ... The namenode RPC should have length as exactly a multiple of blocksize ... Agree. > Is this futureproof ? What happens where block storage becomes first class citizen ? Will we have an ability to create unowned block, and append it to an existing (truncated to block boundary) file ? I think you mean separating namespace management and block management. The beauty of truncate with concat is that it does not introduce new block level operations so that there is no block management change (arguably, it makes sense adding a new operation for copying an existing block to a new block so that the copy can be done locally in datanodes but this not a big deal.) It shouldn't cause any problem. We don't need unowned block. The block should be owned by the user since the data of the block is.
          Hide
          Milind Bhandarkar added a comment -

          Thought a little more about this. The namenode RPC should have length as exactly a multiple of blocksize, otherwise it should throw exception. (so, option 1 in my comment above.) And concatFile could be null, so that case where files need to be truncated to exactly block boundary (option 1 in Nicholas's comment) can be trivially supported.

          Show
          Milind Bhandarkar added a comment - Thought a little more about this. The namenode RPC should have length as exactly a multiple of blocksize, otherwise it should throw exception. (so, option 1 in my comment above.) And concatFile could be null, so that case where files need to be truncated to exactly block boundary (option 1 in Nicholas's comment) can be trivially supported.
          Hide
          Milind Bhandarkar added a comment -

          I really like Sanjay's idea (i.e. 2) for it's simplicity, atomicity, and ease of implementation. However, I would like the client API to be truncate(file, length), and namenode RPC to be truncate(file, length, concatFile).

          In the client API, i.e. method in DFSClient, length could be anything less than the file size, whereas in the namenode RPC, we have two options:

          1. length must be a multiple of blocksize for file, less than filesize

          2. length could be anything less than filesize, but namenode will do length = (length - (length%blocksize)).

          For the client API to support any length, without specifying concatFile, we need to add o.a.h.fs.FileUtils.createTempFile.

          Thoughts ?

          Is this futureproof ? What happens where block storage becomes first class citizen ? Will we have an ability to create unowned block, and append it to an existing (truncated to block boundary) file ?

          Show
          Milind Bhandarkar added a comment - I really like Sanjay's idea (i.e. 2) for it's simplicity, atomicity, and ease of implementation. However, I would like the client API to be truncate(file, length), and namenode RPC to be truncate(file, length, concatFile). In the client API, i.e. method in DFSClient, length could be anything less than the file size, whereas in the namenode RPC, we have two options: 1. length must be a multiple of blocksize for file, less than filesize 2. length could be anything less than filesize, but namenode will do length = (length - (length%blocksize)). For the client API to support any length, without specifying concatFile, we need to add o.a.h.fs.FileUtils.createTempFile. Thoughts ? Is this futureproof ? What happens where block storage becomes first class citizen ? Will we have an ability to create unowned block, and append it to an existing (truncated to block boundary) file ?
          Hide
          Tsz Wo Nicholas Sze added a comment -

          The full truncate feature may be hard to implement. Below are some ideas.

          (1) Support only block boundary truncate.

          When the length is not a multiple of the block size, throw an exception.

          This is very easy to implement. The client could use it to support full truncate by (i ) copy the data in the truncated block, (ii) truncate to block boundary and (iii) append the data back.

          Example 1: Suppose file size is 290 and block size is 100. Then, truncate(file, 180) can be done by (i ) reading bytes from position 101 to 180, (ii) truncate(file, 100) and (iii) append the 80 bytes back.

          (2) truncate with concat. (Sanjay’s idea)

          A problem of (1) is that it is not atomic. It may end up with finishing (ii) but failing at (iii). A remedy is to add a parameter and the API becomes truncate(file, length, concateFile), where length must be a multiple of the block size. HDFS will first truncate file to length and then concatenate the block in concateFile to the end of file. Note that this is a namenode-only operation and can be easily implemented as atomic.

          Example 2: For the case in e.g. 1, truncate can be done by first copying bytes 101 to 180 to a new file f and then truncate(file, 100, f).

          (3) Copy on truncate.

          Support full truncate by first copying the last block to a new block, then truncate the replicas of the new block at the datanodes and then commit the truncate once the datanodes report to new block to the namenode. Rollback is possible since the old block is still around. This is harder to implement than (1) or (2).

          Show
          Tsz Wo Nicholas Sze added a comment - The full truncate feature may be hard to implement. Below are some ideas. (1) Support only block boundary truncate. When the length is not a multiple of the block size, throw an exception. This is very easy to implement. The client could use it to support full truncate by (i ) copy the data in the truncated block, (ii) truncate to block boundary and (iii) append the data back. Example 1: Suppose file size is 290 and block size is 100. Then, truncate(file, 180) can be done by (i ) reading bytes from position 101 to 180, (ii) truncate(file, 100) and (iii) append the 80 bytes back. (2) truncate with concat. (Sanjay’s idea) A problem of (1) is that it is not atomic. It may end up with finishing (ii) but failing at (iii). A remedy is to add a parameter and the API becomes truncate(file, length, concateFile), where length must be a multiple of the block size. HDFS will first truncate file to length and then concatenate the block in concateFile to the end of file. Note that this is a namenode-only operation and can be easily implemented as atomic. Example 2: For the case in e.g. 1, truncate can be done by first copying bytes 101 to 180 to a new file f and then truncate(file, 100, f). (3) Copy on truncate. Support full truncate by first copying the last block to a new block, then truncate the replicas of the new block at the datanodes and then commit the truncate once the datanodes report to new block to the namenode. Rollback is possible since the old block is still around. This is harder to implement than (1) or (2).
          Hide
          Milind Bhandarkar added a comment -

          Thanks Lei for addressing Nicholas's comments

          Show
          Milind Bhandarkar added a comment - Thanks Lei for addressing Nicholas's comments
          Hide
          Lei Chang added a comment -

          Good comments . Nicolas.

          Attached file revised. Looks more clean now.

          Show
          Lei Chang added a comment - Good comments . Nicolas. Attached file revised. Looks more clean now.
          Hide
          Milind Bhandarkar added a comment -

          Thanks Nicolas.

          yes, we are implementing it. If there is agreement (pending discussion on hdfs-dev list) that it is a useful feature, we will upload the design, and patch will follow.

          Show
          Milind Bhandarkar added a comment - Thanks Nicolas. yes, we are implementing it. If there is agreement (pending discussion on hdfs-dev list) that it is a useful feature, we will upload the design, and patch will follow.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          > Suresh, Nicolas, Eli; Any opinions about the proposed API and semantics ?

          The API looks reasonable to me. How about the implementation? Do you have plan to implement it?

          A minor comment: the usual practice for referring other documentation is to include links or a list of references but not copy & paste.

          Show
          Tsz Wo Nicholas Sze added a comment - > Suresh, Nicolas, Eli; Any opinions about the proposed API and semantics ? The API looks reasonable to me. How about the implementation? Do you have plan to implement it? A minor comment: the usual practice for referring other documentation is to include links or a list of references but not copy & paste.
          Hide
          Eli Collins added a comment -

          Is the proposal to remove appends from all 1.x+ versions of Hadoop or just the 1.x versions ?

          All versions. See the mail I just sent to hdfs-dev@, let's continue the discussion there.

          Show
          Eli Collins added a comment - Is the proposal to remove appends from all 1.x+ versions of Hadoop or just the 1.x versions ? All versions. See the mail I just sent to hdfs-dev@, let's continue the discussion there.
          Hide
          Milind Bhandarkar added a comment -

          I understand that truncate adds more complexity, and have discussed the design at length offline with Sanjay and Hairong. We plan to reuse the append-pipeline for this, and therefore restricted the API to only work with closed files. (We have submitted the exact use case presentation proposal to Hadoop summit, without exposing it to public voting currently, but hopefully would be able to publicly announce it in a few weeks.) The transaction feature is not HDFS-specific, but is at application-level, and works with other file systems that support truncate.

          I don't follow... we don't even expose append() via the shell.

          Indeed, I was not talking about Apache Hadoop, but a distribution that includes this feature.

          Otherwise I'm more inclined to agree with Eli's suggestion to remove append entirely (please continue that discussion on-list, though).

          Is the proposal to remove appends from all 1.x+ versions of Hadoop or just the 1.x versions ?

          Show
          Milind Bhandarkar added a comment - I understand that truncate adds more complexity, and have discussed the design at length offline with Sanjay and Hairong. We plan to reuse the append-pipeline for this, and therefore restricted the API to only work with closed files. (We have submitted the exact use case presentation proposal to Hadoop summit, without exposing it to public voting currently, but hopefully would be able to publicly announce it in a few weeks.) The transaction feature is not HDFS-specific, but is at application-level, and works with other file systems that support truncate. I don't follow... we don't even expose append() via the shell. Indeed, I was not talking about Apache Hadoop, but a distribution that includes this feature. Otherwise I'm more inclined to agree with Eli's suggestion to remove append entirely (please continue that discussion on-list, though). Is the proposal to remove appends from all 1.x+ versions of Hadoop or just the 1.x versions ?
          Hide
          Todd Lipcon added a comment -

          IMO adding truncate() adds a bunch of non-trivial complexity. It's not so much because truncating a block is that hard – but rather because it breaks a serious invariant we have elsewhere that blocks only get longer after they are created. This means that we have to revisit code all over HDFS – in particular some of the trickiest bits around block synchronization – to get this to work. It's not insurmountable, but I would like to know a lot more about the use case before commenting on the API/semantics.

          Maybe you can open a JIRA or upload a design about your transactional HDFS feature, so we can understand the motivation better? Otherwise I'm more inclined to agree with Eli's suggestion to remove append entirely (please continue that discussion on-list, though).

          After appends were enabled in HDFS, we have seen a lot of cases where a lot of (mainly text, or even compressed text) datasets were merged using appends.

          This is where customers realize their mistake immediately after starting to append, and do a ctrl-c.

          I don't follow... we don't even expose append() via the shell. And if we did, would users actually be using "fs -append" to manually write new lines of data into their Hadoop systems??

          Show
          Todd Lipcon added a comment - IMO adding truncate() adds a bunch of non-trivial complexity. It's not so much because truncating a block is that hard – but rather because it breaks a serious invariant we have elsewhere that blocks only get longer after they are created. This means that we have to revisit code all over HDFS – in particular some of the trickiest bits around block synchronization – to get this to work. It's not insurmountable, but I would like to know a lot more about the use case before commenting on the API/semantics. Maybe you can open a JIRA or upload a design about your transactional HDFS feature, so we can understand the motivation better? Otherwise I'm more inclined to agree with Eli's suggestion to remove append entirely (please continue that discussion on-list, though). After appends were enabled in HDFS, we have seen a lot of cases where a lot of (mainly text, or even compressed text) datasets were merged using appends. This is where customers realize their mistake immediately after starting to append, and do a ctrl-c. I don't follow... we don't even expose append() via the shell. And if we did, would users actually be using "fs -append" to manually write new lines of data into their Hadoop systems??
          Hide
          Milind Bhandarkar added a comment -

          Suresh, Nicolas, Eli; Any opinions about the proposed API and semantics ?

          Show
          Milind Bhandarkar added a comment - Suresh, Nicolas, Eli; Any opinions about the proposed API and semantics ?
          Hide
          Milind Bhandarkar added a comment -

          Yes, I am using the term "append" loosely, because of FB's 20-append branch. Our transaction work is done with 0.23.x.

          Show
          Milind Bhandarkar added a comment - Yes, I am using the term "append" loosely, because of FB's 20-append branch. Our transaction work is done with 0.23.x.
          Hide
          Eli Collins added a comment -

          Since appends were enabled very recently, only those with the facebook's version of hadoop, or hadoop 1.0 are users doing this now.

          Append doesn't work on hadoop 1.0, see HDFS-3120. I'm actually going to start a discussion about removing append entirely on hdfs-dev@.

          Show
          Eli Collins added a comment - Since appends were enabled very recently, only those with the facebook's version of hadoop, or hadoop 1.0 are users doing this now. Append doesn't work on hadoop 1.0, see HDFS-3120 . I'm actually going to start a discussion about removing append entirely on hdfs-dev@.
          Hide
          Milind Bhandarkar added a comment -

          .bq Thats okay. You missed the smiley in the tweet too.

          I just copy-pasted, so it was expected

          .bq I see I was not aware it was that common.

          Since appends were enabled very recently, only those with the facebook's version of hadoop, or hadoop 1.0 are users doing this now. Before this, users were creating multiple files.

          In any case, my interest in this feature is for implementing transactions over HDFS (as Lei and I have already discussed with Sanjay Radia and Hairong.) And aborting a transaction means truncating to the last known good data across multiple files.

          Show
          Milind Bhandarkar added a comment - .bq Thats okay. You missed the smiley in the tweet too. I just copy-pasted, so it was expected .bq I see I was not aware it was that common. Since appends were enabled very recently, only those with the facebook's version of hadoop, or hadoop 1.0 are users doing this now. Before this, users were creating multiple files. In any case, my interest in this feature is for implementing transactions over HDFS (as Lei and I have already discussed with Sanjay Radia and Hairong.) And aborting a transaction means truncating to the last known good data across multiple files.
          Hide
          Suresh Srinivas added a comment -

          I must have missed a smiley

          Thats okay. You missed the smiley in the tweet too.

          This is very common.

          I see I was not aware it was that common.

          Show
          Suresh Srinivas added a comment - I must have missed a smiley Thats okay. You missed the smiley in the tweet too. This is very common. I see I was not aware it was that common.
          Hide
          Milind Bhandarkar added a comment -

          I must have missed a smiley

          Nicolas,

          After appends were enabled in HDFS, we have seen a lot of cases where a lot of (mainly text, or even compressed text) datasets were merged using appends.

          This is where customers realize their mistake immediately after starting to append, and do a ctrl-c.

          This is very common.


          Milind Bhandarkar
          Chief Architect, Greenplum Labs,
          Data Computing Division, EMC
          +1-650-523-3858 (W)
          +1-408-666-8483 (C)

          Show
          Milind Bhandarkar added a comment - I must have missed a smiley Nicolas, After appends were enabled in HDFS, we have seen a lot of cases where a lot of (mainly text, or even compressed text) datasets were merged using appends. This is where customers realize their mistake immediately after starting to append, and do a ctrl-c. This is very common. – Milind Bhandarkar Chief Architect, Greenplum Labs, Data Computing Division, EMC +1-650-523-3858 (W) +1-408-666-8483 (C)
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Easy, Milind. I do agree with Suresh that (2) is not a very good reason to have truncate. I think such accidence is rare. However, you made a good point that having append without truncate is a deficiency.

          Show
          Tsz Wo Nicholas Sze added a comment - Easy, Milind. I do agree with Suresh that (2) is not a very good reason to have truncate. I think such accidence is rare. However, you made a good point that having append without truncate is a deficiency.
          Hide
          Milind Bhandarkar added a comment -

          What if user accidentally deletes a directory ? You guys never supported me when I asked for a file-by-file deletion, that could be aborted in time to save 70 pct of users' time, right? Instead you have always supported a directory deletion with a single misdirected RPC.

          Anyway, to answer your question, if user accidentally truncates, he/she can always append again, without losing any efficiency.

          Can we have some mature discussions on this jira please ?


          Milind Bhandarkar
          Chief Architect, Greenplum Labs,
          Data Computing Division, EMC
          +1-650-523-3858 (W)
          +1-408-666-8483 (C)

          Show
          Milind Bhandarkar added a comment - What if user accidentally deletes a directory ? You guys never supported me when I asked for a file-by-file deletion, that could be aborted in time to save 70 pct of users' time, right? Instead you have always supported a directory deletion with a single misdirected RPC. Anyway, to answer your question, if user accidentally truncates, he/she can always append again, without losing any efficiency. Can we have some mature discussions on this jira please ? – Milind Bhandarkar Chief Architect, Greenplum Labs, Data Computing Division, EMC +1-650-523-3858 (W) +1-408-666-8483 (C)
          Hide
          Suresh Srinivas added a comment -

          if a user mistakenly starts to append data to an existing large file, and discovers the mistake, the only recourse is to recreate that file, by rewriting the contents. This is very inefficient.

          What if user accidentally truncates a file

          Show
          Suresh Srinivas added a comment - if a user mistakenly starts to append data to an existing large file, and discovers the mistake, the only recourse is to recreate that file, by rewriting the contents. This is very inefficient. What if user accidentally truncates a file
          Hide
          Milind Bhandarkar added a comment -

          This will be a great addition to HDFS for a couple of reasons:

          1. Having an append without a truncate is a serious deficiency.
          2. If a user mistakenly starts to append data to an existing large file, and discovers the mistake, the only recourse is to recreate that file, by rewriting the contents. This is very inefficient.

          Show
          Milind Bhandarkar added a comment - This will be a great addition to HDFS for a couple of reasons: 1. Having an append without a truncate is a serious deficiency. 2. If a user mistakenly starts to append data to an existing large file, and discovers the mistake, the only recourse is to recreate that file, by rewriting the contents. This is very inefficient.

            People

            • Assignee:
              Plamen Jeliazkov
              Reporter:
              Lei Chang
            • Votes:
              4 Vote for this issue
              Watchers:
              51 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - 1,344h
                1,344h
                Remaining:
                Remaining Estimate - 1,344h
                1,344h
                Logged:
                Time Spent - Not Specified
                Not Specified

                  Development