Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0.0, 2.1.0-beta
    • Component/s: namenode
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Incompatible change

      Description

      Snapshots are point in time images of parts of the filesystem or the entire filesystem. Snapshots can be a read-only or a read-write point in time copy of the filesystem. There are several use cases for snapshots in HDFS. I will post a detailed write-up soon with with more information.

      1. 2802.diff
        867 kB
        Suresh Srinivas
      2. 2802.patch
        819 kB
        Tsz Wo Nicholas Sze
      3. 2802.patch
        818 kB
        Suresh Srinivas
      4. h2802_20130417.patch
        988 kB
        Tsz Wo Nicholas Sze
      5. h2802_20130422.patch
        1015 kB
        Tsz Wo Nicholas Sze
      6. h2802_20130423.patch
        1015 kB
        Tsz Wo Nicholas Sze
      7. h2802_20130425.patch
        1.03 MB
        Tsz Wo Nicholas Sze
      8. h2802_20130426.patch
        1.03 MB
        Tsz Wo Nicholas Sze
      9. h2802_20130430.patch
        1.05 MB
        Tsz Wo Nicholas Sze
      10. h2802_20130430b.patch
        1.05 MB
        Tsz Wo Nicholas Sze
      11. h2802_20130508.patch
        1.06 MB
        Tsz Wo Nicholas Sze
      12. h2802_20130509.patch
        1.07 MB
        Tsz Wo Nicholas Sze
      13. HDFS-2802.20121101.patch
        137 kB
        Suresh Srinivas
      14. HDFS-2802.branch-2.0510.patch
        1.07 MB
        Jing Zhao
      15. HDFS-2802-meeting-minutes-121101.txt
        5 kB
        David S. Wang
      16. HDFSSnapshotsDesign.pdf
        73 kB
        Aaron T. Myers
      17. snap.patch
        122 kB
        Hari Mankude
      18. snapshot-design.pdf
        189 kB
        Aaron T. Myers
      19. snapshot-design.tex
        25 kB
        Aaron T. Myers
      20. snapshot-one-pager.pdf
        99 kB
        Hari Mankude
      21. Snapshots.pdf
        76 kB
        Suresh Srinivas
      22. Snapshots20120429.pdf
        386 kB
        Suresh Srinivas
      23. Snapshots20121018.pdf
        372 kB
        Tsz Wo Nicholas Sze
      24. Snapshots20121030.pdf
        289 kB
        Suresh Srinivas
      25. snapshot-testplan.pdf
        51 kB
        Jing Zhao

        Issue Links

        1.
        Snapshot of Being Written Files Sub-task Resolved Jing Zhao
         
        2.
        Support snapshot of single files Sub-task Resolved Tsz Wo Nicholas Sze
         
        3.
        Support snapshottable INodeDirectory Sub-task Resolved Tsz Wo Nicholas Sze
         
        4.
        Handle replication in snapshots Sub-task Resolved Tsz Wo Nicholas Sze
         
        5.
        Add SnapshotManager Sub-task Resolved Tsz Wo Nicholas Sze
         
        6.
        Add editlog opcodes for snapshot create and delete operations Sub-task Resolved Suresh Srinivas
         
        7.
        Protocol changes for snapshots Sub-task Resolved Suresh Srinivas
         
        8.
        provide CLI support for allow and disallow snapshot on a directory Sub-task Resolved Brandon Li
         
        9.
        Add editlog opcodes to allow and disallow snapshots on a directory Sub-task Resolved Brandon Li
         
        10.
        Protocol changes for listSnapshots functionality Sub-task Resolved Brandon Li
         
        11.
        Add snapshot quota to limit the number of snapshots Sub-task Resolved Tsz Wo Nicholas Sze
         
        12.
        Update file deletion logic to support snapshot files Sub-task Resolved Tsz Wo Nicholas Sze
         
        13.
        Add snapshot related metrics Sub-task Resolved Jing Zhao
         
        14.
        Add snapshot information to namenode WebUI Sub-task Closed Haohui Mai
         
        15.
        provide CLI support for createSnapshot Sub-task Resolved Brandon Li
         
        16.
        Support append to original files which are snapshotted Sub-task Resolved Tsz Wo Nicholas Sze
         
        17.
        Test replication with snapshots Sub-task Resolved Tsz Wo Nicholas Sze
         
        18.
        Support O(1) snapshot creation Sub-task Resolved Tsz Wo Nicholas Sze
         
        19.
        Support snapshot of subtrees Sub-task Resolved Tsz Wo Nicholas Sze
         
        20.
        Add audit log for snapshot operations Sub-task Resolved Jing Zhao
         
        21.
        Change INodeDirectory.getExistingPathINodes(..) to work with snapshots Sub-task Resolved Jing Zhao
         
        22.
        Complete the allowSnapshot code and add a test for it Sub-task Resolved Tsz Wo Nicholas Sze
         
        23.
        Use a persistent data structure for snapshots Sub-task Resolved Tsz Wo Nicholas Sze
         
        24.
        Add reading/writing snapshot information to FSImage Sub-task Resolved Jing Zhao
         
        25.
        Add capability to namenode to get snapshot diff Sub-task Resolved Jing Zhao
         
        26.
        Add testcases for testing basic snapshot functionalities Sub-task Resolved Jing Zhao
         
        27.
        Directory diff in snapshots Sub-task Resolved Tsz Wo Nicholas Sze
         
        28.
        Create test for all snapshot-related metrics Sub-task Resolved Jing Zhao
         
        29.
        Further work on changing INodeFile.blocks to private in Snapshot branch Sub-task Resolved Tsz Wo Nicholas Sze
         
        30.
        Deletion of snapshottable dir with snapshots should fail Sub-task Resolved Jing Zhao
         
        31.
        Disallow write/modify operations on files and directories in a snapshot Sub-task Resolved Brandon Li
         
        32.
        Complete the code for disallowSnapshot Sub-task Resolved Tsz Wo Nicholas Sze
         
        33.
        Update inode in blocksMap when deleting original/snapshot file Sub-task Resolved Jing Zhao
         
        34.
        FSDirectory#hasSnapshot will NPE when deleting an empty directory Sub-task Resolved Aaron T. Myers
         
        35.
        Rename should fail when the destination dir is snapshottable and has snapshots Sub-task Resolved Jing Zhao
         
        36.
        Add support for scheduled automatic snapshots Sub-task Resolved Suresh Srinivas
         
        37. Add support for restoring/rolling back to a snapshot Sub-task Patch Available Jing Zhao
         
        38.
        Add snapshot information to INodesInPath Sub-task Resolved Tsz Wo Nicholas Sze
         
        39.
        Additional snapshot tests for more complicated directory structure and modifications. Sub-task Resolved Jing Zhao
         
        40.
        Add a snapshot parameter to INodeDirectory.getChildrenList() Sub-task Resolved Tsz Wo Nicholas Sze
         
        41.
        Add testcases for replication handling in snapshots Sub-task Resolved Jing Zhao
         
        42.
        Add a snapshot id comparator Sub-task Resolved Tsz Wo Nicholas Sze
         
        43.
        rename getter method getMutableX and getXMutable to getXAndEnsureMutable Sub-task Resolved Tsz Wo Nicholas Sze
         
        44.
        Support renaming of snapshots Sub-task Resolved Jing Zhao
         
        45.
        Listing all the current snapshottable directories Sub-task Resolved Jing Zhao
         
        46.
        Support deleting snapshots Sub-task Resolved Jing Zhao
         
        47.
        Include snapshot related operations in OfflineEditsViewerHelper#runOperations() to fix test failures in TestOfflineEditsViewer Sub-task Resolved Jing Zhao
         
        48.
        TestSnapshot fails Sub-task Resolved Jing Zhao
         
        49.
        Change INode to support HDFS-4103 Sub-task Resolved Tsz Wo Nicholas Sze
         
        50.
        Support snapshots up to the snapshot limit Sub-task Resolved Tsz Wo Nicholas Sze
         
        51.
        After calling replaceSelf, further operations that should be applied on the new INode may be wrongly applied to the original INode Sub-task Resolved Jing Zhao
         
        52.
        When listing snapshottable directories, only return those where the user has permission to take snapshots Sub-task Resolved Jing Zhao
         
        53.
        INodeDirectorySnapshottable's constructor should take into account the scenario when passed-in dir is of the type INodeDirectoryWithSnapshot Sub-task Resolved Jing Zhao
         
        54.
        Combined diff may not be identical to the diffs applied sequentially Sub-task Resolved Tsz Wo Nicholas Sze
         
        55.
        In INodeDirectoryWithSnapshot, change combinePostDiff to merge-sort like Sub-task Resolved Tsz Wo Nicholas Sze
         
        56.
        Add support for getting snapshot diff from DistributedFileSystem Sub-task Resolved Jing Zhao
         
        57.
        Add unit tests for taking snapshots while file appending Sub-task Resolved Jing Zhao
         
        58.
        Support snapshot in OfflineImageViewer Sub-task Resolved Jing Zhao
         
        59.
        Support INodeFileUnderConstructionWithSnapshot in FSImage saving/loading Sub-task Resolved Jing Zhao
         
        60.
        The snapshot copy INode.recordModification(..) returned is never used Sub-task Resolved Tsz Wo Nicholas Sze
         
        61.
        Refactor INodeDirectoryWithSnapshot.Diff as a top level generic class Sub-task Resolved Tsz Wo Nicholas Sze
         
        62.
        Support file snapshot with diff lists Sub-task Resolved Tsz Wo Nicholas Sze
         
        63.
        Refactor INodeDirectoryWithSnapshot for sharing the code with INodeFileWithSnapshot Sub-task Resolved Tsz Wo Nicholas Sze
         
        64.
        Combine collectSubtreeBlocksAndClear with deleteDiffsForSnapshot Sub-task Resolved Tsz Wo Nicholas Sze
         
        65.
        Eliminate the file snapshot circular linked list Sub-task Resolved Tsz Wo Nicholas Sze
         
        66.
        Fix snapshot fsimage for file diffs Sub-task Resolved Tsz Wo Nicholas Sze
         
        67.
        Fix snapshot diff report Sub-task Resolved Jing Zhao
         
        68.
        Fix file/directory/snapshot deletion for file diff Sub-task Resolved Jing Zhao
         
        69.
        Refactor snapshot INode methods Sub-task Resolved Tsz Wo Nicholas Sze
         
        70.
        Update computeContentSummary, spaceConsumedInTree and diskspaceConsumed for snapshot Sub-task Resolved Tsz Wo Nicholas Sze
         
        71.
        Update quota verification for snapshot Sub-task Resolved Tsz Wo Nicholas Sze
         
        72.
        Add CLI support for snapshot rename, diff report, and snapshottable directory listing Sub-task Resolved Jing Zhao
         
        73.
        Support listing snapshots under a snapshottable directory using "ls" Sub-task Resolved Jing Zhao
         
        74.
        Fix INodeFile replacement, TestQuota and javac error Sub-task Resolved Tsz Wo Nicholas Sze
         
        75.
        Update SnapshotManager#snapshottables when loading fsimage Sub-task Resolved Jing Zhao
         
        76.
        Decide the semantic of concat with snapshots Sub-task Resolved Tsz Wo Nicholas Sze
         
        77.
        Update rename for snapshots Sub-task Resolved Tsz Wo Nicholas Sze
         
        78.
        Snapshot: unprotectedSetReplication may update diskspace count incorrectly Sub-task Resolved Tsz Wo Nicholas Sze
         
        79.
        Refactor INodesInPath to a standalone class Sub-task Resolved Tsz Wo Nicholas Sze
         
        80.
        Fix FSDirectory#delete when INode#cleanSubtree returns 0 Sub-task Resolved Jing Zhao
         
        81.
        Update namespace/diskspace after deleting snapshots Sub-task Resolved Jing Zhao
         
        82.
        Move Diff and EnumCounters to util package Sub-task Resolved Tsz Wo Nicholas Sze
         
        83.
        Annotate snapshot tests Sub-task Resolved Arpit Agarwal
         
        84.
        Restrict snapshot IDs to 24-bits wide Sub-task Resolved Arpit Agarwal
         
        85.
        Add snapshotdiff and LsSnapshottableDir tools to hdfs script Sub-task Resolved Arpit Agarwal
         
        86.
        Snapshot: Update FSImage for INodeReference Sub-task Resolved Tsz Wo Nicholas Sze
         
        87.
        Not to use INode.getParent() when generating snapshot diff report Sub-task Resolved Jing Zhao
         
        88.
        Update the FilesDeleted metric while deleting file/dir in the current tree Sub-task Resolved Jing Zhao
         
        89.
        Fix FSImageFormat#Loader NPE and synchronization issues Sub-task Resolved Jing Zhao
         
        90.
        Update quota usage when deleting files/dirs that were created after taking the latest snapshot Sub-task Resolved Jing Zhao
         
        91.
        INodeDirectory#replaceSelf4Quota may convert a newly created directory (which is not included in the latest snapshot) to an INodeDirectoryWithSnapshot Sub-task Resolved Jing Zhao
         
        92.
        For snapshot deletion, when merging the diff from to-delete snapshot to the prior snapshot, make sure files/directories created after the prior snapshot get deleted Sub-task Resolved Jing Zhao
         
        93.
        Add rename test in TestSnapshot Sub-task Resolved Jing Zhao
         
        94.
        llegalStateException renaming file under snapshottable directory Sub-task Resolved Arpit Agarwal
         
        95.
        Disallow users to create file/dir named ".snapshot" Sub-task Resolved Tsz Wo Nicholas Sze
         
        96.
        Capture renamed files/directories in snapshot diff report Sub-task Resolved Jing Zhao
         
        97.
        Renaming file in subdirectory of a snapshotted directory does not work. Sub-task Resolved Arpit Agarwal
         
        98.
        Fix rename across snapshottable directories Sub-task Resolved Jing Zhao
         
        99.
        Snapshot: Use INode id for image serialization Sub-task Resolved Tsz Wo Nicholas Sze
         
        100.
        Fix quota computation for rename with snapshots Sub-task Resolved Jing Zhao
         
        101.
        Use timestamp as the default snapshot name Sub-task Resolved Tsz Wo Nicholas Sze
         
        102.
        Fix the undo section of rename with snapshots Sub-task Resolved Jing Zhao
         
        103. Add a transient flag to file so that transient files won't be included in any snapshot Sub-task Open Tsz Wo Nicholas Sze
         
        104.
        disallowSnapshot does not work for root Sub-task Resolved Tsz Wo Nicholas Sze
         
        105.
        Fix FilterFileSystem and findbugs warning in Snapshot branch Sub-task Resolved Tsz Wo Nicholas Sze
         
        106.
        Add snapshot user guide Sub-task Resolved Tsz Wo Nicholas Sze
         
        107.
        Change the parameter type of the snapshot methods in HdfsAdmin to Path Sub-task Resolved Tsz Wo Nicholas Sze
         
        108.
        Minor simplifications to snapshot code Sub-task Resolved Arpit Agarwal
         
        109.
        Fix test failures after merging the mapping from INodeId to INode Sub-task Resolved Jing Zhao
         
        110.
        Update inodeMap after deleting files/directories/snapshots Sub-task Resolved Jing Zhao
         
        111.
        Snapshot tests broken after merge from trunk Sub-task Resolved Arpit Agarwal
         
        112.
        Fix OfflineImageViewer and permission checking for snapshot operations Sub-task Resolved Jing Zhao
         
        113.
        DisallowSnapshot throws IllegalStateException Sub-task Resolved Jing Zhao
         
        114.
        Fix javadoc and other warnings in the snapshot branch Sub-task Resolved Tsz Wo Nicholas Sze
         
        115.
        Fix appending to a renamed file with snapshot Sub-task Resolved Jing Zhao
         
        116.
        Convert snapshot user guide to APT from XDOC Sub-task Resolved Aaron T. Myers
         
        117.
        Use INodeId to identify the corresponding directory node for FSImage saving/loading Sub-task Resolved Jing Zhao
         
        118.
        AccessControlException message is changed in snapshot branch Sub-task Resolved Tsz Wo Nicholas Sze
         
        119.
        Disallow nested snapshottable directories Sub-task Resolved Tsz Wo Nicholas Sze
         
        120.
        snapshotDiff of two invalid snapshots but with same name returns success Sub-task Resolved Jing Zhao
         
        121.
        Update inodeMap after node replacement Sub-task Resolved Jing Zhao
         
        122.
        directory is not snapshottable after clrQuota Sub-task Resolved Jing Zhao
         
        123.
        Fix bugs in quota usage updating/computation Sub-task Resolved Jing Zhao
         
        124.
        File listing of .snapshot under a non-existing dir throws NullPointer Sub-task Resolved Tsz Wo Nicholas Sze
         
        125.
        Update and fix deletion of reference inode Sub-task Resolved Jing Zhao
         
        126.
        Update computeContentSummary() for the reference nodes in snapshots Sub-task Resolved Tsz Wo Nicholas Sze
         
        127.
        Fix INodeDirectoryWithSnapshot#cleanDeletedINode Sub-task Resolved Jing Zhao
         
        128.
        lsSnapshottableDir throws IllegalArgumentException when root is made snapshottable Sub-task Resolved Jing Zhao
         
        129.
        Disallowing snapshot on / twice throws error Sub-task Resolved Jing Zhao
         
        130.
        For an INodeDirectoryWithSnapshot, use INode#isInLatestSnapshot() to check if the added/removed child should be recorded in snapshot diff Sub-task Resolved Jing Zhao
         
        131.
        Add more unit tests for the undo section of rename Sub-task Resolved Jing Zhao
         
        132.
        Snapshot: identify the correct prior snapshot when deleting a snapshot under a renamed subtree Sub-task Closed Jing Zhao
         
        133.
        The javadoc of FileWithSnapshot is incorrect Sub-task Closed Tsz Wo Nicholas Sze
         
        134.
        Add a test for testing snapshot file length Sub-task Closed Arpit Agarwal
         
        135.
        Update Snapshot doc for HDFS-4758 Sub-task Closed Tsz Wo Nicholas Sze
         
        136.
        Reduce snapshot inode memory usage Sub-task Closed Tsz Wo Nicholas Sze
         

          Activity

          Hide
          Jeff Hammerbacher added a comment -

          Duplicate of HDFS-233

          Show
          Jeff Hammerbacher added a comment - Duplicate of HDFS-233
          Hide
          Hari Mankude added a comment -

          I (We) are well aware of hdfs-233. This jira was opened to provide comprehensive snapshot solution (both RW/RO support) for HDFS.

          Show
          Hari Mankude added a comment - I (We) are well aware of hdfs-233. This jira was opened to provide comprehensive snapshot solution (both RW/RO support) for HDFS.
          Hide
          Jeff Hammerbacher added a comment -

          There's a ton of useful discussion on the HDFS-233 ticket that should be preserved when working on this issue. Please continue the discussion there rather than duplicating the issue.

          Show
          Jeff Hammerbacher added a comment - There's a ton of useful discussion on the HDFS-233 ticket that should be preserved when working on this issue. Please continue the discussion there rather than duplicating the issue.
          Hide
          Suresh Srinivas added a comment -

          Jeff, that is the reason why hdfs-233 is marked as related. Can you please stop marking the Jira as resolved!

          Show
          Suresh Srinivas added a comment - Jeff, that is the reason why hdfs-233 is marked as related. Can you please stop marking the Jira as resolved!
          Hide
          Hari Mankude added a comment -

          Uploaded the one pager. More detailed design doc and the first version of the patch is in the works.

          Show
          Hari Mankude added a comment - Uploaded the one pager. More detailed design doc and the first version of the patch is in the works.
          Hide
          Eli Collins added a comment -

          Hey Hari,

          Could you elaborate some on scenario #3? The motivation is test, so I assume use cases are things like (1) giving an experimental Hive/Pig user safe access /datasets/xyz or (2) running a test/dev HBase instance from an existing /hbase directory. Use case #1 seems like it could be accomplished in practice with external tables (or, in general, analysis programs that can separate their input data from directories they create/mutate). Use case #2 seems unlikely to be used in practice. Could be useful if /hbase is production (ie big) but in that case you wouldn't want to run a dev/test HBase build against a production HDFS cluster (as the production HBase instance runs there as well). Ie the usual case here is that you have a separate test/dev HBase/HDFS instance and you need/want to copy the datasets.

          What's your plan for handling append and truncate? The design document does not discuss whether you're punting on these, or if not the semantics (eg for a snapshot taken while a block is being written) or the implementation. There's relevant design and implementation discussion in HDFS-233 and Snapshots in Hadoop Distributed File System that's worth referencing.

          Looking forward to checking out your prototype patch.

          Thanks,
          Eli

          Show
          Eli Collins added a comment - Hey Hari, Could you elaborate some on scenario #3? The motivation is test, so I assume use cases are things like (1) giving an experimental Hive/Pig user safe access /datasets/xyz or (2) running a test/dev HBase instance from an existing /hbase directory. Use case #1 seems like it could be accomplished in practice with external tables (or, in general, analysis programs that can separate their input data from directories they create/mutate). Use case #2 seems unlikely to be used in practice. Could be useful if /hbase is production (ie big) but in that case you wouldn't want to run a dev/test HBase build against a production HDFS cluster (as the production HBase instance runs there as well). Ie the usual case here is that you have a separate test/dev HBase/HDFS instance and you need/want to copy the datasets. What's your plan for handling append and truncate? The design document does not discuss whether you're punting on these, or if not the semantics (eg for a snapshot taken while a block is being written) or the implementation. There's relevant design and implementation discussion in HDFS-233 and Snapshots in Hadoop Distributed File System that's worth referencing. Looking forward to checking out your prototype patch. Thanks, Eli
          Hide
          Hari Mankude added a comment -

          @Eli,
          Regarding scenario #3, consider a hbase setup with huge dataset in production. A new app has been developed which needs to be validated against production dataset. It is not feasible to copy the entire dataset to a test setup. At the same time, app is not ready for production and it is not safe to have the app modify the data in the production database. One of the solutions for these types of problems is to take a RW snapshot of the production dataset and then have the development app run against the RW snapshot. After the app testing is done, RW snap is deleted. This assumes that the cluster has sufficient compute capacity and incremental storage capacity to support RW snaps.

          Regarding appends, current prototype of snapshot relies on the filesize that is available at the namenode. So, if a file is appended after snap is taken, then it is a no-op from a snap perspective. If a snap is taken of a file which has append pipeline setup, inode is of type underconstruction in the NN. Prototype relies on filesize that is available on the NN for snaps. This might not be perfect and I have some ideas on trying to acquire more upto-date filesize.

          I thought that truncate is not supported currently in the trunk. If you are referring to deletes, prototype handles deletes correctly without issues.

          I will post a more detailed doc after I am done with HA related work.

          Show
          Hari Mankude added a comment - @Eli, Regarding scenario #3, consider a hbase setup with huge dataset in production. A new app has been developed which needs to be validated against production dataset. It is not feasible to copy the entire dataset to a test setup. At the same time, app is not ready for production and it is not safe to have the app modify the data in the production database. One of the solutions for these types of problems is to take a RW snapshot of the production dataset and then have the development app run against the RW snapshot. After the app testing is done, RW snap is deleted. This assumes that the cluster has sufficient compute capacity and incremental storage capacity to support RW snaps. Regarding appends, current prototype of snapshot relies on the filesize that is available at the namenode. So, if a file is appended after snap is taken, then it is a no-op from a snap perspective. If a snap is taken of a file which has append pipeline setup, inode is of type underconstruction in the NN. Prototype relies on filesize that is available on the NN for snaps. This might not be perfect and I have some ideas on trying to acquire more upto-date filesize. I thought that truncate is not supported currently in the trunk. If you are referring to deletes, prototype handles deletes correctly without issues. I will post a more detailed doc after I am done with HA related work.
          Hide
          Eli Collins added a comment -

          Thanks Hari. Wrt scenario #3 doesn't this imply that the user would have to run two HBase instances on the same cluster - one running from /hbase and one running from /snapshot/hbase? If not then you need to shutdown the HBase instance running on the production data and then restart it with the snapshot. Maybe I'm missing something but both options (stopping the production instance and running two instances on the same production hardware) seem unrealistic. Also, don't HBase snapshots (HBASE-50) solve this problem? Are there other use cases aside from HBase?

          Show
          Eli Collins added a comment - Thanks Hari. Wrt scenario #3 doesn't this imply that the user would have to run two HBase instances on the same cluster - one running from /hbase and one running from /snapshot/hbase? If not then you need to shutdown the HBase instance running on the production data and then restart it with the snapshot. Maybe I'm missing something but both options (stopping the production instance and running two instances on the same production hardware) seem unrealistic. Also, don't HBase snapshots ( HBASE-50 ) solve this problem? Are there other use cases aside from HBase?
          Hide
          Hari Mankude added a comment -

          I am attaching an early version of the patch based on the trunk. It took longer than I expected to rebase the patch. Code needs cleanup, further optimization of memory usage in the NN, fixes to checkpointing code to handle some border conditions and more tests. Next steps - Working on splitting the patch into smaller and easy to review code. Branch HDFS-2802 has been created for this work. Next version of the design document will be posted soon. (some we discussed during HDFS meetup).

          Show
          Hari Mankude added a comment - I am attaching an early version of the patch based on the trunk. It took longer than I expected to rebase the patch. Code needs cleanup, further optimization of memory usage in the NN, fixes to checkpointing code to handle some border conditions and more tests. Next steps - Working on splitting the patch into smaller and easy to review code. Branch HDFS-2802 has been created for this work. Next version of the design document will be posted soon. (some we discussed during HDFS meetup).
          Hide
          Hari Mankude added a comment -

          A quick user's guide

          hadoop dfsadmin -createsnap <snapname> <path where snap is to be taken> <ro/rw> will create a snap with snapname at the location mentioned

          hadoop dfsadmin -removesnap <snapname> will remove snapshot

          hadoop dfsadmin -listsnap / will list all snaps that have been taken under /

          Show
          Hari Mankude added a comment - A quick user's guide hadoop dfsadmin -createsnap <snapname> <path where snap is to be taken> <ro/rw> will create a snap with snapname at the location mentioned hadoop dfsadmin -removesnap <snapname> will remove snapshot hadoop dfsadmin -listsnap / will list all snaps that have been taken under /
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12536172/snap.patch
          against trunk revision .

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3193//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12536172/snap.patch against trunk revision . -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3193//console This message is automatically generated.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          From the last time the patch is posted for the prototype, I have been continuing work on the code. The attachment (Snapshots20121018.pdf) is the updated design document based on currently where the implementation is. It is still fairly highlevel, but should has enough details to give an idea on how the snapshot feature is being implemented. I will continue to update the design as the implementation progresses.

          Here are some of the next steps:

          1. Split the large patch and create subtasks in this jira, to make it easy to understand the implementation.
          2. Will schedule a design review meeting to go over the requirements and design in detail with the folks who are interested. Especially to get feedback from other Hadoop projects. This meeting will be in the week of Oct 29, post Hadoop World.
          Show
          Tsz Wo Nicholas Sze added a comment - From the last time the patch is posted for the prototype, I have been continuing work on the code. The attachment (Snapshots20121018.pdf) is the updated design document based on currently where the implementation is. It is still fairly highlevel, but should has enough details to give an idea on how the snapshot feature is being implemented. I will continue to update the design as the implementation progresses. Here are some of the next steps: Split the large patch and create subtasks in this jira, to make it easy to understand the implementation. Will schedule a design review meeting to go over the requirements and design in detail with the folks who are interested. Especially to get feedback from other Hadoop projects. This meeting will be in the week of Oct 29, post Hadoop World.
          Hide
          Colin Patrick McCabe added a comment -

          It's good to see that this design carefully considers how to separate the metadata of a snapshotted file (or directory) from the metadata of a later version of that file.

          When there are one or more objects (either the original file or snaplinks) under a sub-tree, the occupied space is counted as the max file size times the max replication of these object (the max calculations include only the objects under the sub-tree but exclude the objects outside the sub-tree.) Note that it is easy to determine if a given INode is under a sub-tree by traversing up with the parent references.

          In some of the most commercially popular systems which implement snapshots, snapshots do not count against the disk quotas. I think system administrators might expect this behavior by now. Some other filesystems have two kinds of quotas-- one which counts snapshots, and another which does not. This could be a good way to go.

          Show
          Colin Patrick McCabe added a comment - It's good to see that this design carefully considers how to separate the metadata of a snapshotted file (or directory) from the metadata of a later version of that file. When there are one or more objects (either the original file or snaplinks) under a sub-tree, the occupied space is counted as the max file size times the max replication of these object (the max calculations include only the objects under the sub-tree but exclude the objects outside the sub-tree.) Note that it is easy to determine if a given INode is under a sub-tree by traversing up with the parent references. In some of the most commercially popular systems which implement snapshots, snapshots do not count against the disk quotas. I think system administrators might expect this behavior by now. Some other filesystems have two kinds of quotas-- one which counts snapshots, and another which does not. This could be a good way to go.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          > In some of the most commercially popular systems which implement snapshots, snapshots do not count against the disk quotas. ...

          Thanks for the comment. I think the systems you mentioned probably only support RO snapshot. In the design, we also consider RW snapshot so that disk quotas have to be counted. Your suggestion on two kinds of quotas could be a good alternative.

          Show
          Tsz Wo Nicholas Sze added a comment - > In some of the most commercially popular systems which implement snapshots, snapshots do not count against the disk quotas. ... Thanks for the comment. I think the systems you mentioned probably only support RO snapshot. In the design, we also consider RW snapshot so that disk quotas have to be counted. Your suggestion on two kinds of quotas could be a good alternative.
          Hide
          Aaron T. Myers added a comment -

          Hey guys, thanks a lot for posting an updated design document. I have one high-level concern about the design, and a number of specific comments / questions about the design document itself.

          First, I'm concerned with the O(# of files + # of directories) nature of this design, both in terms of time taken to create a snapshot and the NN memory resources consumed.

          It seems to me that this will result in snapshot creation/retention being sufficiently costly as to make creating snapshots with this design not a viable option for large HDFS instances. I think another design requirement for this work should be "snapshot creation must be sufficiently fast as to be unnoticeable to clients" (an attempted definition of "reasonable" per high-level requirement #4 on page 1.) I don't think this design will satisfy such a requirement in large HDFS instances. Also, many large HDFS instances that I'm aware of already run with very large NN heaps, and a snapshot design which results in making copies of large portions of the working set will not be viable in these situations. At least, I think this design does not address the #3 high-level requirement on page 1 of "support for a reasonable number of snapshots." In deployments where the NameNode is already running with a large working set, this design might only allow for a single snapshot of the root of the file system, or none at all.

          I think this design document is a very good start, but we really must figure out a way to create and retain snapshots in a more efficient manner, both in terms of time to create the snapshot and memory overhead to retain the snapshot, in order for the snapshot solution to be viable for HDFS. Having an O(# of files + # of directories) system will not be acceptable for all but the smallest HDFS installations. I think that creation of a snapshot should be either O(1) or worst case O(depth of file system tree). There are many precedents for file systems supporting more efficient snapshot creation than O(# of files + # of directories), e.g. WAFL, ZFS, BTRFS, etc.

          Second, comments on the design document itself:

          1. I see that you're now considering read/write snapshots as an optional requirement. Per several of the comments on HDFS-233, it seems to me that supporting writable snapshots is an extra complexity that many folks aren't actually interested in. I suggest we explicitly punt on support for writable snapshots, and declare that snapshots are completely immutable. I think doing so may have the potential to allow us to make some simplifying design decisions.
          2. On page 2 the design mentions that "it should be possible to extend the current design to materialize the snapshot metadata and migrate it to outside the NameNode." I didn't see any other discussion of this in the rest of the document. Can you perhaps expand on what you mean by this?
          3. On page 2 you mention that there "are snapshot root directories that are configured by the system administrator to allow snapshots." How are these configured? Is it something that can be dynamically added to a running NN? Or would it need to be configured at NN startup time?
          4. In several places (e.g. the footnote on page 2, use case 2 on page 6) the design document refers to file system "volumes," in particular "Snapshots are created at the volume level simplifying administration." What "volume" are you referring to here?
          5. I think that the "detailed requirements" section on page 4 is missing a critical requirement: the snapshot must be consistent from the perspective of individual clients. The requirements state that the snapshot must be atomic, but not consistent. The consistency of the snapshot really must be well-defined and strong. For example, it would be unacceptable if files restored from an HDFS snapshot of a running hbase.rootdir resulted in a corrupted HBase instance.
          6. On page 4 you mention that a snapshot will have "a unique snapshot name for a given path." Can you expand upon that? How is this name created? Why is this necesssary in addition to the "path where the snapshot is created" which as far as I can tell should also serve the purpose of uniquely identifying the snapshot?
          7. I think that the solution described on page 5 of the document for the length of files being written does not satisfy the requirement that I mentioned above that the snapshot must be consistent. In particular, I think I can construct a scenario wherein a client which performs an NN-only metadata operation (A), then writes and hflushes some data (B), and then performs another metadata operation (C) right before a snapshot is created may result in the snapshot containing both metadata operations, but not the data hflush'ed to the DNs, i.e. A and C but not B. This would result in the snapshot representing a moment in time that never existed from the point of view of that client. We can continue the discussion of this particular issue more on HDFS-3960, if you'd like.
          8. I'm glad that the document discusses atime - that is not something that I had considered in my thinking on HDFS snapshots. I am a little leery, however, of tracking atime at all in what is ostensibly a read-only snapshot. Does anyone know what other file systems that support read-only snapshots do with regard to atime?
          9. Regarding open question #2 on page 7, I would think that this should be a hard requirement if we go with this design, and the main motivation should be taking snapshots of different parts of the tree on different schedules. For example, an administrator may want to schedule a nightly snapshot of the whole FS, but hourly snapshots of their /user directories.
          10. Regarding open question #4 on page 7, I feel confident that this design does not currently result in producing consistent HBase snapshots, because of #7 I described above.
          11. One question regarding the user experience that I don't see described in the document: will creating a snapshot require super user privileges? Or can any user create a snapshot of a subdirectory? If the latter, what permissions are required to create a snapshot? What if the user doesn't have permissions on some files under the subtree of the snapshot target? Does this result in an incomplete snapshot? Or a completely failed snapshot? My personal inclination is to limit snapshot creation to super users only, as a simplification.
          12. One high-level comment on the document: I'm a little leery of introducing this new concept of "snapshottable directories." I'm not aware of any precedent in other file systems for this sort of restriction, and I fear that the concept may be confusing for administrators and operators of HDFS.
          Show
          Aaron T. Myers added a comment - Hey guys, thanks a lot for posting an updated design document. I have one high-level concern about the design, and a number of specific comments / questions about the design document itself. First, I'm concerned with the O(# of files + # of directories) nature of this design, both in terms of time taken to create a snapshot and the NN memory resources consumed. It seems to me that this will result in snapshot creation/retention being sufficiently costly as to make creating snapshots with this design not a viable option for large HDFS instances. I think another design requirement for this work should be "snapshot creation must be sufficiently fast as to be unnoticeable to clients" (an attempted definition of "reasonable" per high-level requirement #4 on page 1.) I don't think this design will satisfy such a requirement in large HDFS instances. Also, many large HDFS instances that I'm aware of already run with very large NN heaps, and a snapshot design which results in making copies of large portions of the working set will not be viable in these situations. At least, I think this design does not address the #3 high-level requirement on page 1 of "support for a reasonable number of snapshots." In deployments where the NameNode is already running with a large working set, this design might only allow for a single snapshot of the root of the file system, or none at all. I think this design document is a very good start, but we really must figure out a way to create and retain snapshots in a more efficient manner, both in terms of time to create the snapshot and memory overhead to retain the snapshot, in order for the snapshot solution to be viable for HDFS. Having an O(# of files + # of directories) system will not be acceptable for all but the smallest HDFS installations. I think that creation of a snapshot should be either O(1) or worst case O(depth of file system tree). There are many precedents for file systems supporting more efficient snapshot creation than O(# of files + # of directories), e.g. WAFL, ZFS, BTRFS, etc. Second, comments on the design document itself: I see that you're now considering read/write snapshots as an optional requirement. Per several of the comments on HDFS-233 , it seems to me that supporting writable snapshots is an extra complexity that many folks aren't actually interested in. I suggest we explicitly punt on support for writable snapshots, and declare that snapshots are completely immutable. I think doing so may have the potential to allow us to make some simplifying design decisions. On page 2 the design mentions that "it should be possible to extend the current design to materialize the snapshot metadata and migrate it to outside the NameNode." I didn't see any other discussion of this in the rest of the document. Can you perhaps expand on what you mean by this? On page 2 you mention that there "are snapshot root directories that are configured by the system administrator to allow snapshots." How are these configured? Is it something that can be dynamically added to a running NN? Or would it need to be configured at NN startup time? In several places (e.g. the footnote on page 2, use case 2 on page 6) the design document refers to file system "volumes," in particular "Snapshots are created at the volume level simplifying administration." What "volume" are you referring to here? I think that the "detailed requirements" section on page 4 is missing a critical requirement: the snapshot must be consistent from the perspective of individual clients. The requirements state that the snapshot must be atomic, but not consistent. The consistency of the snapshot really must be well-defined and strong. For example, it would be unacceptable if files restored from an HDFS snapshot of a running hbase.rootdir resulted in a corrupted HBase instance. On page 4 you mention that a snapshot will have "a unique snapshot name for a given path." Can you expand upon that? How is this name created? Why is this necesssary in addition to the "path where the snapshot is created" which as far as I can tell should also serve the purpose of uniquely identifying the snapshot? I think that the solution described on page 5 of the document for the length of files being written does not satisfy the requirement that I mentioned above that the snapshot must be consistent. In particular, I think I can construct a scenario wherein a client which performs an NN-only metadata operation (A), then writes and hflushes some data (B), and then performs another metadata operation (C) right before a snapshot is created may result in the snapshot containing both metadata operations, but not the data hflush'ed to the DNs, i.e. A and C but not B. This would result in the snapshot representing a moment in time that never existed from the point of view of that client. We can continue the discussion of this particular issue more on HDFS-3960 , if you'd like. I'm glad that the document discusses atime - that is not something that I had considered in my thinking on HDFS snapshots. I am a little leery, however, of tracking atime at all in what is ostensibly a read-only snapshot. Does anyone know what other file systems that support read-only snapshots do with regard to atime? Regarding open question #2 on page 7, I would think that this should be a hard requirement if we go with this design, and the main motivation should be taking snapshots of different parts of the tree on different schedules. For example, an administrator may want to schedule a nightly snapshot of the whole FS, but hourly snapshots of their /user directories. Regarding open question #4 on page 7, I feel confident that this design does not currently result in producing consistent HBase snapshots, because of #7 I described above. One question regarding the user experience that I don't see described in the document: will creating a snapshot require super user privileges? Or can any user create a snapshot of a subdirectory? If the latter, what permissions are required to create a snapshot? What if the user doesn't have permissions on some files under the subtree of the snapshot target? Does this result in an incomplete snapshot? Or a completely failed snapshot? My personal inclination is to limit snapshot creation to super users only, as a simplification. One high-level comment on the document: I'm a little leery of introducing this new concept of "snapshottable directories." I'm not aware of any precedent in other file systems for this sort of restriction, and I fear that the concept may be confusing for administrators and operators of HDFS.
          Hide
          Suresh Srinivas added a comment - - edited

          Thanks for the comments guys.

          In some of the most commercially popular systems which implement snapshots, snapshots do not count against the disk quotas

          How do they handle disk quota use when the original file is deleted and only snapshots exists? That is the reason why counting the disk quota makes sense.

          First, I'm concerned with the O(# of files + # of directories) nature of this design, both in terms of time taken to create a snapshot and the NN memory resources consumed.

          I agree with you on this. We wanted to begin with this approach and then optimize it further in memory. The initial patch uploaded here tried premature optimization both for memory and snapshot creation time and thus made the code really complicated. But this is a definite goal and that part of the design we will update as we continue to work. This is covered in open issues/future work section.

          comment 1:
          Agree with this part. As we continue the work, we can make a decision on this. For supporting RW, lets not make the design/implementation more complicated.

          comment 2:
          Will address this as we continue to add more details to the design in the next update.

          Comment 3, 6:
          I want to make sure you understand this is early design and we will continue to add more details. I think some of the questions will be answered by how this works:

          • Admin can mark directories as snapshottable using CLI
          • User then can create snapshots for these directories using CLI/API. A snapshot has a snapshot name and it is unique for given snapshot root.

          comment 4:
          If you look at snapshot implementation in other systems it is done at volume level. That is the parallel we are talking about.

          Comment 5, Comment 7, comment 10:
          As regards to consistency (comment 7), a system where snapshot is taken at the namespace without involving data layer cannot provide string consistency guarantee. I also think it may not be relevant where writers are different from the client that is taking the snapshot. Not sure what guarantee such a client can expect/depend on given writers are separate. We could discuss this during design review. I also think based on discussion with few HBase folks, they should be okay with it. Some thing to discuss with them. I am also not clear on their dependency on HDFS with hbase-6055.

          comment 8:
          This could change during implementation if we think access time may not be that important to maintain.

          comment 9:
          Agreed. I am leaning towards allowing it.

          comment 11:
          Will add usecases

          comment 12:
          See the volume comment and the document sort of covers this. We could discuss this further if the document is not clear.

          Show
          Suresh Srinivas added a comment - - edited Thanks for the comments guys. In some of the most commercially popular systems which implement snapshots, snapshots do not count against the disk quotas How do they handle disk quota use when the original file is deleted and only snapshots exists? That is the reason why counting the disk quota makes sense. First, I'm concerned with the O(# of files + # of directories) nature of this design, both in terms of time taken to create a snapshot and the NN memory resources consumed. I agree with you on this. We wanted to begin with this approach and then optimize it further in memory. The initial patch uploaded here tried premature optimization both for memory and snapshot creation time and thus made the code really complicated. But this is a definite goal and that part of the design we will update as we continue to work. This is covered in open issues/future work section. comment 1: Agree with this part. As we continue the work, we can make a decision on this. For supporting RW, lets not make the design/implementation more complicated. comment 2: Will address this as we continue to add more details to the design in the next update. Comment 3, 6: I want to make sure you understand this is early design and we will continue to add more details. I think some of the questions will be answered by how this works: Admin can mark directories as snapshottable using CLI User then can create snapshots for these directories using CLI/API. A snapshot has a snapshot name and it is unique for given snapshot root. comment 4: If you look at snapshot implementation in other systems it is done at volume level. That is the parallel we are talking about. Comment 5, Comment 7, comment 10: As regards to consistency (comment 7), a system where snapshot is taken at the namespace without involving data layer cannot provide string consistency guarantee. I also think it may not be relevant where writers are different from the client that is taking the snapshot. Not sure what guarantee such a client can expect/depend on given writers are separate. We could discuss this during design review. I also think based on discussion with few HBase folks, they should be okay with it. Some thing to discuss with them. I am also not clear on their dependency on HDFS with hbase-6055. comment 8: This could change during implementation if we think access time may not be that important to maintain. comment 9: Agreed. I am leaning towards allowing it. comment 11: Will add usecases comment 12: See the volume comment and the document sort of covers this. We could discuss this further if the document is not clear.
          Hide
          Aaron T. Myers added a comment -

          I agree with you on this. We wanted to begin with this approach and then optimize it further in memory. The initial patch uploaded here tried premature optimization both for memory and snapshot creation time and thus made the code really complicated. But this is a definite goal and that part of the design we will update as we continue to work. This is covered in open issues/future work section.

          My concern is that we're going about this wrong if the plan is to implement a O(# of files + # of directories) solution and then optimize this via multi-threading, offloading to disk, etc., as the design document suggests. We should work on coming up with a design which is O(1) or something which is roughly O(log(# of files + # of directories)), both in terms of time and space efficiency. This sort of fundamental design decision is not something that can be easily improved incrementally. I hope you will be open to reconsidering a reworking of this design along these lines.

          Show
          Aaron T. Myers added a comment - I agree with you on this. We wanted to begin with this approach and then optimize it further in memory. The initial patch uploaded here tried premature optimization both for memory and snapshot creation time and thus made the code really complicated. But this is a definite goal and that part of the design we will update as we continue to work. This is covered in open issues/future work section. My concern is that we're going about this wrong if the plan is to implement a O(# of files + # of directories) solution and then optimize this via multi-threading, offloading to disk, etc., as the design document suggests. We should work on coming up with a design which is O(1) or something which is roughly O(log(# of files + # of directories)), both in terms of time and space efficiency. This sort of fundamental design decision is not something that can be easily improved incrementally. I hope you will be open to reconsidering a reworking of this design along these lines.
          Hide
          Suresh Srinivas added a comment -

          My concern is that we're going about this wrong if the plan is to implement a O(# of files + # of directories) solution and then optimize this via multi-threading, offloading to disk, etc., as the design document suggests.

          May be the design document is fairly early and might have misled you. That is not the goal. The goal is to have efficient implementation. Will post and updated design to address your concerns.

          Show
          Suresh Srinivas added a comment - My concern is that we're going about this wrong if the plan is to implement a O(# of files + # of directories) solution and then optimize this via multi-threading, offloading to disk, etc., as the design document suggests. May be the design document is fairly early and might have misled you. That is not the goal. The goal is to have efficient implementation. Will post and updated design to address your concerns.
          Hide
          Todd Lipcon added a comment -

          As regards to consistency (comment 7), a system where snapshot is taken at the namespace without involving data layer cannot provide string consistency guarantee. I also think it may not be relevant where writers are different from the client that is taking the snapshot. Not sure what guarantee such a client can expect/depend on given writers are separate

          There's a continuum between no consistency and strong consistency. In particular, I think the consistency we should seek to provide is "causal consistency from the perspective of any single writer". Aaron outlined an example earlier which is useful for discussing the consistency model:

          1) Client issues a write to a file (eg an HBase HLog)
          2) Client modifies namespace (eg creates a file)
          3) Client issues another write and hflushes
          4) Client modifies namespace again

          Because a single client issued all 4 operations, we'd like to ensure that any snapshot, when restored, has a full prefix of these operations - eg just #1, or 1-2, 1-3, or 1-4.

          This isn't trivial to implement, but it's also reasonably doable. After some discussion with Aaron and Colin we came up with a design that does provide this. Here's a sketch:

          • When the client does any namespace modification, the response from the NameNode returns the transaction ID used to record the modification in the transaction log. (eg the txn of the OP_MKDIR for a mkdir). The client remembers the max transaction ID it has seen in a variable inside DFSClient
          • When the client issues hflush(), it sticks the transaction ID into the data packet. This establishes a causality relationship between the namespace layer and the data layer.
          • The datanode maintains a simple (and compact) data structure for open files: for each byte offset in an open file, keep the transaction ID associated with the write packet that wrote it. This data structure will be small: it only needs to be maintained for open files and, since most writers don't often interleave namespace and data access, it changes rarely. We only need to store entries in the data structure at the byte offsets where the txid changes.

          The process of creating a consistent snapshot then proceeds as follows:

          • when the snapshot is created, it is assigned a transaction ID (eg the transaction id of the OP_MAKE_SNAPSHOT or whatever it is going to be called)
          • the Snapshot initially starts in a "in progress" state
          • The NN enqueues a command to all datanodes: ReportSnapshotLengths(snapshot_txid)
          • The DNs, upon receiving this command, look at their local data structures to determine the length of open blocks at the given transaction ID, and report them back. (even if the length has since grown longer)
          • Once the NN has received a reported length for each of the in-progress blocks, it uses those lengths for the completed snapshot and marks it finalized.

          Back of the envelope math seems to indicate that tracking the mapping of txid to length is feasible, and we think this provides the above consistency guarantee, which is much stronger than what has been proposed in the design document. The implementation isn't trivial but also doesn't seem out of reach.

          Hope this can spur some discussion on how we can offer stronger semantics.

          Show
          Todd Lipcon added a comment - As regards to consistency (comment 7), a system where snapshot is taken at the namespace without involving data layer cannot provide string consistency guarantee. I also think it may not be relevant where writers are different from the client that is taking the snapshot. Not sure what guarantee such a client can expect/depend on given writers are separate There's a continuum between no consistency and strong consistency. In particular, I think the consistency we should seek to provide is "causal consistency from the perspective of any single writer". Aaron outlined an example earlier which is useful for discussing the consistency model: 1) Client issues a write to a file (eg an HBase HLog) 2) Client modifies namespace (eg creates a file) 3) Client issues another write and hflushes 4) Client modifies namespace again Because a single client issued all 4 operations, we'd like to ensure that any snapshot, when restored, has a full prefix of these operations - eg just #1, or 1-2, 1-3, or 1-4. This isn't trivial to implement, but it's also reasonably doable. After some discussion with Aaron and Colin we came up with a design that does provide this. Here's a sketch: When the client does any namespace modification, the response from the NameNode returns the transaction ID used to record the modification in the transaction log. (eg the txn of the OP_MKDIR for a mkdir). The client remembers the max transaction ID it has seen in a variable inside DFSClient When the client issues hflush(), it sticks the transaction ID into the data packet. This establishes a causality relationship between the namespace layer and the data layer. The datanode maintains a simple (and compact) data structure for open files: for each byte offset in an open file, keep the transaction ID associated with the write packet that wrote it. This data structure will be small: it only needs to be maintained for open files and, since most writers don't often interleave namespace and data access, it changes rarely. We only need to store entries in the data structure at the byte offsets where the txid changes. The process of creating a consistent snapshot then proceeds as follows: when the snapshot is created, it is assigned a transaction ID (eg the transaction id of the OP_MAKE_SNAPSHOT or whatever it is going to be called) the Snapshot initially starts in a "in progress" state The NN enqueues a command to all datanodes: ReportSnapshotLengths(snapshot_txid) The DNs, upon receiving this command, look at their local data structures to determine the length of open blocks at the given transaction ID, and report them back. (even if the length has since grown longer) Once the NN has received a reported length for each of the in-progress blocks, it uses those lengths for the completed snapshot and marks it finalized. Back of the envelope math seems to indicate that tracking the mapping of txid to length is feasible, and we think this provides the above consistency guarantee, which is much stronger than what has been proposed in the design document. The implementation isn't trivial but also doesn't seem out of reach. Hope this can spur some discussion on how we can offer stronger semantics.
          Hide
          Suresh Srinivas added a comment -

          Not sure if you read the discussion in section snapshot of being written files. I will add more details to the design. My comment earlier was related to strict consistency requirements.

          Show
          Suresh Srinivas added a comment - Not sure if you read the discussion in section snapshot of being written files. I will add more details to the design. My comment earlier was related to strict consistency requirements.
          Hide
          Hari Mankude added a comment -

          Todd, another option is to look at the inodesUnderConstruction in the NN and query the DNs for the exact filesize at the time of taking snapshot. Even with this, the filesize that is obtained will be at the instant. Applications like hbase will have to deal with hlogs that could have incomplete log entries when an un-cordinated snapshot is taken at the hdfs. A better approach is to have the application reach a quiesce point and then take a snap. This is normally done for oracle (hot backup mode) and sqlserver so that an application consistent snapshot can be taken.

          Also, createSnap()/removeSnap() has the writeLock() on the FSNamesystem which will ensure that there are no other metadata updates when snap is being taken.

          Show
          Hari Mankude added a comment - Todd, another option is to look at the inodesUnderConstruction in the NN and query the DNs for the exact filesize at the time of taking snapshot. Even with this, the filesize that is obtained will be at the instant. Applications like hbase will have to deal with hlogs that could have incomplete log entries when an un-cordinated snapshot is taken at the hdfs. A better approach is to have the application reach a quiesce point and then take a snap. This is normally done for oracle (hot backup mode) and sqlserver so that an application consistent snapshot can be taken. Also, createSnap()/removeSnap() has the writeLock() on the FSNamesystem which will ensure that there are no other metadata updates when snap is being taken.
          Hide
          Todd Lipcon added a comment -

          Hi Suresh. Yes, I read the design there. In fact I think the design is based my comment on HDFS-3960 from a few weeks ago. But after further thinking, I think that design is too weak. Here's why:

          If the first DN in the pipeline has to RPC on every hflush, that would be way too many RPCs. HBase for example flushes several hundred times per second per server, so a 1000 node HBase cluster under heavy load would quickly take down a NameNode. So instead of the DN immediately RPCing on every hflush, it has to wait until the next heartbeat and report lengths with the heartbeat.

          Given this, it may be 5-10 seconds between the hflush and the report of the length to the datanode. This means that the snapshot will get a length which is either 5-10 seconds too old or 5-10 seconds too new (depending on whether it uses the last reported length or if it waits until the next heartbeat to finalize the snapshot)

          A 5-10 second inconsistency window is plenty to break the situation described above: it's quite likely to get data layer modifications 1 and 3 wthout getting namespace modifications 2 and 4, or vice versa.

          On the other hand, the design I proposed above does handle this, because the DN isn't reporting the length at the time of heartbeat. Instead it's reporting a length which is causally consistent with the namespace from the perspective of the writer of that file.

          Todd, another option is to look at the inodesUnderConstruction in the NN and query the DNs for the exact filesize at the time of taking snapshot

          We can't query the DNs while holding the NN lock. It could take several seconds or longer to contact all the DNs in a loaded 1000+ node cluster, and potentially 10s of seconds if one of the nodes is actually down. So you'd have to drop the lock, at which point we're back to the above issue with consistency against concurrent NS modifications.

          A better approach is to have the application reach a quiesce point and then take a snap. This is normally done for oracle (hot backup mode) and sqlserver so that an application consistent snapshot can be taken.

          The difference is that quiescing a single-node or small-cluster database like SQL Server or RAC is relatively easy. On the other hand, quiescing a 1000 node HBase cluster would take a while, and I don't think users will really tolerate a global stop-the-world to make a snapshot. This is especially true for use cases like DR/backup where you expect to take snapshots as often as once every few minutes.

          Show
          Todd Lipcon added a comment - Hi Suresh. Yes, I read the design there. In fact I think the design is based my comment on HDFS-3960 from a few weeks ago. But after further thinking, I think that design is too weak. Here's why: If the first DN in the pipeline has to RPC on every hflush, that would be way too many RPCs. HBase for example flushes several hundred times per second per server, so a 1000 node HBase cluster under heavy load would quickly take down a NameNode. So instead of the DN immediately RPCing on every hflush, it has to wait until the next heartbeat and report lengths with the heartbeat. Given this, it may be 5-10 seconds between the hflush and the report of the length to the datanode. This means that the snapshot will get a length which is either 5-10 seconds too old or 5-10 seconds too new (depending on whether it uses the last reported length or if it waits until the next heartbeat to finalize the snapshot) A 5-10 second inconsistency window is plenty to break the situation described above: it's quite likely to get data layer modifications 1 and 3 wthout getting namespace modifications 2 and 4, or vice versa. On the other hand, the design I proposed above does handle this, because the DN isn't reporting the length at the time of heartbeat. Instead it's reporting a length which is causally consistent with the namespace from the perspective of the writer of that file. Todd, another option is to look at the inodesUnderConstruction in the NN and query the DNs for the exact filesize at the time of taking snapshot We can't query the DNs while holding the NN lock. It could take several seconds or longer to contact all the DNs in a loaded 1000+ node cluster, and potentially 10s of seconds if one of the nodes is actually down. So you'd have to drop the lock, at which point we're back to the above issue with consistency against concurrent NS modifications. A better approach is to have the application reach a quiesce point and then take a snap. This is normally done for oracle (hot backup mode) and sqlserver so that an application consistent snapshot can be taken. The difference is that quiescing a single-node or small-cluster database like SQL Server or RAC is relatively easy. On the other hand, quiescing a 1000 node HBase cluster would take a while, and I don't think users will really tolerate a global stop-the-world to make a snapshot. This is especially true for use cases like DR/backup where you expect to take snapshots as often as once every few minutes.
          Hide
          Suresh Srinivas added a comment - - edited

          In fact I think the design is based my comment on HDFS-3960

          Actually that is not true. We have been mulling over many of these ideas for a long time. HDFS-3960 was just create to get the discussion going.

          The design we are proposing is to let DNs send the length. The length known is what goes into the snapshot instead of recording either zero length for block under construction or having to initiate communication with datanodes/implicitly getting it from DN. From what I have heard from some HBase folks 5-10 seconds lagging should be workable for them. That is why I want to to talk to few HBase folks in the design review.

          Show
          Suresh Srinivas added a comment - - edited In fact I think the design is based my comment on HDFS-3960 Actually that is not true. We have been mulling over many of these ideas for a long time. HDFS-3960 was just create to get the discussion going. The design we are proposing is to let DNs send the length. The length known is what goes into the snapshot instead of recording either zero length for block under construction or having to initiate communication with datanodes/implicitly getting it from DN. From what I have heard from some HBase folks 5-10 seconds lagging should be workable for them. That is why I want to to talk to few HBase folks in the design review.
          Hide
          Hari Mankude added a comment -

          Todd,

          I do not agree that your solution will be any beneficial to hbase than what is being proposed. Any type of txid information in DNs will be at the beginning of the transaction. If the client is writing in the middle of block, there is no way to know the exact size when snap was taken. Querying inodesUnderConstruction will give the block length at the time of the query. It is not possible to take an application consistent snapshot (one which does not require recovery) without coordination with the application.

          In fact, communication with DNs when snapshots are being taken will make the process of taking snapshots very slow while giving very little additional benefit.

          Show
          Hari Mankude added a comment - Todd, I do not agree that your solution will be any beneficial to hbase than what is being proposed. Any type of txid information in DNs will be at the beginning of the transaction. If the client is writing in the middle of block, there is no way to know the exact size when snap was taken. Querying inodesUnderConstruction will give the block length at the time of the query. It is not possible to take an application consistent snapshot (one which does not require recovery) without coordination with the application. In fact, communication with DNs when snapshots are being taken will make the process of taking snapshots very slow while giving very little additional benefit.
          Hide
          Hari Mankude added a comment -

          Sorry hit the comment early.

          Additionally, including the sizes of non-finalized blocks in snapshots has implication that if the client dies and the non-finalized section is discarded, then snapshot might have pointers to non-existent blocks.

          Show
          Hari Mankude added a comment - Sorry hit the comment early. Additionally, including the sizes of non-finalized blocks in snapshots has implication that if the client dies and the non-finalized section is discarded, then snapshot might have pointers to non-existent blocks.
          Hide
          Todd Lipcon added a comment -

          The design we are proposing is to let DNs send the length. The length known is what goes into the snapshot instead of recording either zero length for block under construction or having to initiate communication with datanodes/implicitly getting it from DN. From what I have heard from some HBase folks 5-10 seconds lagging should be workable for them. That is why I want to to talk to few HBase folks in the design review.

          I hope I qualify as an HBase folk?

          5-10 seconds lagging on the data is probably fine. But inconsistency between metadata and namespace modifications is a lot tougher. Consider for example an application which uses a write-ahead log on HDFS to make a group of namespace modifications consistent. See HBASE-2231 for an example of a place where we currently have a dataloss bug for which the proposed fix is exactly this:

          1. Write new files (compaction result)
          2. Write to WAL that compaction is finished
          3. Delete old files (compaction sources)

          On recovery, if we see the "compaction finished" entry in the WAL, then we "roll forward" the transaction and delete the source. But if the snapshot doesn't preserve ordering of the above operations, we risk either seeing the "compaction finished" when the namespace doesn't have the new files, which would result in an accidental deletion of a bunch of data.

          So I think we need a way to provide barriers between namespace and data layer modifications. The proposal I made above should achieve this.

          Another option is something that we've called "super flush". This would be a flag on hflush() or hsync() indicating that the new length of the file needs to be persisted to the NameNode, not just the datanodes. It would be used by applications like HBase to determine consistency points for file lengths.

          In fact, communication with DNs when snapshots are being taken will make the process of taking snapshots very slow while giving very little additional benefit.

          We should distinguish between two types of slowness for snapshots:
          1) Slowness while holding a lock. This is unacceptable IMO - we must hold the lock for a bounded amount of time and never make an RPC while holding the lock.
          2) Slowness before a snapshot is available for restore. This is acceptable. For example, if the user operation "create snapshot" holds the lock for 10ms, but the snapshot is initially in a "COLLECTING_LENGTHS" state while it waits for block lengths that seems acceptable. So long as the lengths are filled in by the next heartbeat (or two heartbeats from now) it should be complete (and thus ready for recovery) within the minute. Note that we don't need to wait for a heartbeat from every datanode. Instead, we just need to wait until, for each under-construction block in the snapshotted area, one of its replicas has reported. When snapshotting a subtree without any open files, it would still be instant.

          Additionally, including the sizes of non-finalized blocks in snapshots has implication that if the client dies and the non-finalized section is discarded, then snapshot might have pointers to non-existent blocks.

          I don't understand what you mean here...can you be more specific about the scenario?

          Show
          Todd Lipcon added a comment - The design we are proposing is to let DNs send the length. The length known is what goes into the snapshot instead of recording either zero length for block under construction or having to initiate communication with datanodes/implicitly getting it from DN. From what I have heard from some HBase folks 5-10 seconds lagging should be workable for them. That is why I want to to talk to few HBase folks in the design review. I hope I qualify as an HBase folk? 5-10 seconds lagging on the data is probably fine. But inconsistency between metadata and namespace modifications is a lot tougher. Consider for example an application which uses a write-ahead log on HDFS to make a group of namespace modifications consistent. See HBASE-2231 for an example of a place where we currently have a dataloss bug for which the proposed fix is exactly this: 1. Write new files (compaction result) 2. Write to WAL that compaction is finished 3. Delete old files (compaction sources) On recovery, if we see the "compaction finished" entry in the WAL, then we "roll forward" the transaction and delete the source. But if the snapshot doesn't preserve ordering of the above operations, we risk either seeing the "compaction finished" when the namespace doesn't have the new files, which would result in an accidental deletion of a bunch of data. So I think we need a way to provide barriers between namespace and data layer modifications. The proposal I made above should achieve this. Another option is something that we've called "super flush". This would be a flag on hflush() or hsync() indicating that the new length of the file needs to be persisted to the NameNode, not just the datanodes. It would be used by applications like HBase to determine consistency points for file lengths. In fact, communication with DNs when snapshots are being taken will make the process of taking snapshots very slow while giving very little additional benefit. We should distinguish between two types of slowness for snapshots: 1) Slowness while holding a lock. This is unacceptable IMO - we must hold the lock for a bounded amount of time and never make an RPC while holding the lock. 2) Slowness before a snapshot is available for restore. This is acceptable. For example, if the user operation "create snapshot" holds the lock for 10ms, but the snapshot is initially in a "COLLECTING_LENGTHS" state while it waits for block lengths that seems acceptable. So long as the lengths are filled in by the next heartbeat (or two heartbeats from now) it should be complete (and thus ready for recovery) within the minute. Note that we don't need to wait for a heartbeat from every datanode. Instead, we just need to wait until, for each under-construction block in the snapshotted area, one of its replicas has reported. When snapshotting a subtree without any open files, it would still be instant. Additionally, including the sizes of non-finalized blocks in snapshots has implication that if the client dies and the non-finalized section is discarded, then snapshot might have pointers to non-existent blocks. I don't understand what you mean here...can you be more specific about the scenario?
          Hide
          Suresh Srinivas added a comment -

          I hope I qualify as an HBase folk?

          Sure. But I want others to seek others feedback before even considering adding any more complexity to the design.

          Show
          Suresh Srinivas added a comment - I hope I qualify as an HBase folk? Sure. But I want others to seek others feedback before even considering adding any more complexity to the design.
          Hide
          Suresh Srinivas added a comment -

          Should we consider moving the consistency part of the discussion to HDFS-3960?

          Show
          Suresh Srinivas added a comment - Should we consider moving the consistency part of the discussion to HDFS-3960 ?
          Hide
          Colin Patrick McCabe added a comment -

          Suresh said:

          How do [other filesystems] handle disk quota use when the original file is deleted and only snapshots exists? That is the reason why counting the disk quota makes sense.

          ZFS has "quotas" and "refquotas". The former includes snapshot overhead; the latter does not. Based on some Googling, I think that on NetApp devices, quotas do not include snapshot overhead. (at least by default). I think it makes sense to offer both kinds of quota, although we don't have to implement them both right away, of course.

          Show
          Colin Patrick McCabe added a comment - Suresh said: How do [other filesystems] handle disk quota use when the original file is deleted and only snapshots exists? That is the reason why counting the disk quota makes sense. ZFS has "quotas" and "refquotas". The former includes snapshot overhead; the latter does not. Based on some Googling, I think that on NetApp devices, quotas do not include snapshot overhead. (at least by default). I think it makes sense to offer both kinds of quota, although we don't have to implement them both right away, of course.
          Hide
          Suresh Srinivas added a comment -

          ZFS has "quotas" and "refquotas". The former includes snapshot overhead; the latter does not.

          Good to know. Some thing we should consider as well.

          Show
          Suresh Srinivas added a comment - ZFS has "quotas" and "refquotas". The former includes snapshot overhead; the latter does not. Good to know. Some thing we should consider as well.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          > First, I'm concerned with the O(# of files + # of directories) nature of this design, both in terms of time taken to create a snapshot and the NN memory resources consumed.

          Aaron, by O(N) where N=# of files + # of directories, I guess you mean O(N) snapshot creation time and O(N) memory usage at snapshot creation. Snapshot creation can be optimized by lazy INode creation. No INode is created at snapshot creation time. Only the INode modified after snapshot will be created. Then it becomes O(1) snapshot creation time and O(1) memory usage at snapshot creation. The design does not exclude this optimization.

          Show
          Tsz Wo Nicholas Sze added a comment - > First, I'm concerned with the O(# of files + # of directories) nature of this design, both in terms of time taken to create a snapshot and the NN memory resources consumed. Aaron, by O(N) where N=# of files + # of directories, I guess you mean O(N) snapshot creation time and O(N) memory usage at snapshot creation. Snapshot creation can be optimized by lazy INode creation. No INode is created at snapshot creation time. Only the INode modified after snapshot will be created. Then it becomes O(1) snapshot creation time and O(1) memory usage at snapshot creation. The design does not exclude this optimization.
          Hide
          Aaron T. Myers added a comment -

          Aaron, by O(N) where N=# of files + # of directories, I guess you mean O(N) snapshot creation time and O(N) memory usage at snapshot creation. Snapshot creation can be optimized by lazy INode creation. No INode is created at snapshot creation time. Only the INode modified after snapshot will be created. Then it becomes O(1) snapshot creation time and O(1) memory usage at snapshot creation. The design does not exclude this optimization.

          On page 7 the design document says the following:

          The memory usage is linear to the number of INode snapped because it copies all INodes when a snapshot is created.

          Then later on page 7 the design document goes on to discuss how this might be optimized by supporting offline snapshots on disk (i.e. get the snapshot metadata out of the NN's heap), and performing the copying of all the INodes in parallel using several threads.

          This is what I am referring to. I fully support a design which implements O(1) snapshot creation time and O(1) memory usage, but the current proposed design does not describe such a thing. Instead of implementing an inefficient design and then optimizing it, we should come up with and implement an efficient design.

          Show
          Aaron T. Myers added a comment - Aaron, by O(N) where N=# of files + # of directories, I guess you mean O(N) snapshot creation time and O(N) memory usage at snapshot creation. Snapshot creation can be optimized by lazy INode creation. No INode is created at snapshot creation time. Only the INode modified after snapshot will be created. Then it becomes O(1) snapshot creation time and O(1) memory usage at snapshot creation. The design does not exclude this optimization. On page 7 the design document says the following: The memory usage is linear to the number of INode snapped because it copies all INodes when a snapshot is created. Then later on page 7 the design document goes on to discuss how this might be optimized by supporting offline snapshots on disk (i.e. get the snapshot metadata out of the NN's heap), and performing the copying of all the INodes in parallel using several threads. This is what I am referring to. I fully support a design which implements O(1) snapshot creation time and O(1) memory usage, but the current proposed design does not describe such a thing. Instead of implementing an inefficient design and then optimizing it, we should come up with and implement an efficient design.
          Hide
          Suresh Srinivas added a comment -

          Instead of implementing an inefficient design and then optimizing it, we should come up with and implement an efficient design.

          Why is starting with simple implementation and then optimizing it later not a choice?

          Show
          Suresh Srinivas added a comment - Instead of implementing an inefficient design and then optimizing it, we should come up with and implement an efficient design. Why is starting with simple implementation and then optimizing it later not a choice?
          Hide
          Tsz Wo Nicholas Sze added a comment -

          O(1) memory usage in general does not not seem possible since the original files/directories could be modified. So the best case is O(N) memory usage in general. However, it is possible to have O(1) memory usage at snapshot creation.

          For small subtrees, i.e. when N is small, it does not matter if it is O(1) or O(N). Such snapshot feature already benefits many applications. So we are going to implement O(N) snapshot creation in the first phase and then optimization it later. Then, we could have the snapshot feature out early instead of spending a long time to come up a complicated design and implementation. A complicated design also increases the risk of bugs in the implementation.

          Show
          Tsz Wo Nicholas Sze added a comment - O(1) memory usage in general does not not seem possible since the original files/directories could be modified. So the best case is O(N) memory usage in general. However, it is possible to have O(1) memory usage at snapshot creation. For small subtrees, i.e. when N is small, it does not matter if it is O(1) or O(N). Such snapshot feature already benefits many applications. So we are going to implement O(N) snapshot creation in the first phase and then optimization it later. Then, we could have the snapshot feature out early instead of spending a long time to come up a complicated design and implementation. A complicated design also increases the risk of bugs in the implementation.
          Hide
          Hari Mankude added a comment -

          Nicholas is right in that we do start off with O(1) memory usage. But depending on writes and updates on the base filesystem, memory usage for snapshot will increase. The worst case is when an application updates all the files in the snapshotted subtree. Even in this scenario, the snap inodes are minimized versions of the actual file inode and retain only the relevant information for snapshots. Additionally (in the prototype), if multiple snapshots are taken of the same subtree, then significant optimizations are done to reduce the memory footprint by representing more than one snapshot in a single snapINode.

          Show
          Hari Mankude added a comment - Nicholas is right in that we do start off with O(1) memory usage. But depending on writes and updates on the base filesystem, memory usage for snapshot will increase. The worst case is when an application updates all the files in the snapshotted subtree. Even in this scenario, the snap inodes are minimized versions of the actual file inode and retain only the relevant information for snapshots. Additionally (in the prototype), if multiple snapshots are taken of the same subtree, then significant optimizations are done to reduce the memory footprint by representing more than one snapshot in a single snapINode.
          Hide
          Aaron T. Myers added a comment -

          Why is starting with simple implementation and then optimizing it later not a choice?

          This sort of fundamental design decision is not something that can be easily improved incrementally. Copying huge portions of the working set, and then making that copying fast and space efficient, should not be the goal. The goal should be to entirely avoid copying huge portions of the working set.

          O(1) memory usage in general does not not seem possible since the original files/directories could be modified. So the best case is O(N) memory usage in general. However, it is possible to have O(1) memory usage at snapshot creation.

          I agree that it's not possible to have an O(1) solution in terms of the number of files/directories that are modified, but it is certainly possible to have an O(1) solution in terms of the number of files/directories that are not modified. That's the issue I'm concerned about, and as far as I can tell is not what is proposed by this design document.

          For small subtrees, i.e. when N is small, it does not matter if it is O(1) or O(N). Such snapshot feature already benefits many applications. So we are going to implement O(N) snapshot creation in the first phase and then optimization it later.

          What are the use cases for taking snapshots of small subtrees? An implementation that is suitable only for small subtrees is probably impractical to snapshot an HBase root directory, a Hive warehouse, or most /user directories that I'm aware of. You'd also presumably want to keep at least a handful (10s?) of snapshots available, so any small subtree that could be snapshotted must be multiplied by ~10 to consider its snapshot size. Note that the design document also explicitly states that it should be possible to take a snapshot of the root of the file system.

          Then, we could have the snapshot feature out early instead of spending a long time to come up a complicated design and implementation. A complicated design also increases the risk of bugs in the implementation.

          I'm all for a simple design, but the design must also meet the stated requirements. The design document states that it should be possible to create a snapshot of the root of the file system, but I don't think the proposed design can do such a thing.

          Suresh had previously said that "May be the design document is fairly early and might have misled you. That is not the goal. The goal is to have efficient implementation." If we're on the same page that the end goal is an O(1) implementation, in terms of the number of files that are not modified between snapshots, in this branch then we can move on.

          Show
          Aaron T. Myers added a comment - Why is starting with simple implementation and then optimizing it later not a choice? This sort of fundamental design decision is not something that can be easily improved incrementally. Copying huge portions of the working set, and then making that copying fast and space efficient, should not be the goal. The goal should be to entirely avoid copying huge portions of the working set. O(1) memory usage in general does not not seem possible since the original files/directories could be modified. So the best case is O(N) memory usage in general. However, it is possible to have O(1) memory usage at snapshot creation. I agree that it's not possible to have an O(1) solution in terms of the number of files/directories that are modified, but it is certainly possible to have an O(1) solution in terms of the number of files/directories that are not modified. That's the issue I'm concerned about, and as far as I can tell is not what is proposed by this design document. For small subtrees, i.e. when N is small, it does not matter if it is O(1) or O(N). Such snapshot feature already benefits many applications. So we are going to implement O(N) snapshot creation in the first phase and then optimization it later. What are the use cases for taking snapshots of small subtrees? An implementation that is suitable only for small subtrees is probably impractical to snapshot an HBase root directory, a Hive warehouse, or most /user directories that I'm aware of. You'd also presumably want to keep at least a handful (10s?) of snapshots available, so any small subtree that could be snapshotted must be multiplied by ~10 to consider its snapshot size. Note that the design document also explicitly states that it should be possible to take a snapshot of the root of the file system. Then, we could have the snapshot feature out early instead of spending a long time to come up a complicated design and implementation. A complicated design also increases the risk of bugs in the implementation. I'm all for a simple design, but the design must also meet the stated requirements. The design document states that it should be possible to create a snapshot of the root of the file system, but I don't think the proposed design can do such a thing. Suresh had previously said that "May be the design document is fairly early and might have misled you. That is not the goal. The goal is to have efficient implementation." If we're on the same page that the end goal is an O(1) implementation, in terms of the number of files that are not modified between snapshots, in this branch then we can move on.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          > ... but it is certainly possible to have an O(1) solution in terms of the number of files/directories that are not modified. ...

          I think the meaning of O(1) above is a very special use of the big-O notation. I would rather keep saying

          • O(N), where N = # files + # directories + # symlinks; and
          • O(M), where M = # modified files + # modified directories + # modified symlinks.

          I agree that O(M) implementation (i.e. include O(1) snapshot creation) is our end goal.

          Show
          Tsz Wo Nicholas Sze added a comment - > ... but it is certainly possible to have an O(1) solution in terms of the number of files/directories that are not modified. ... I think the meaning of O(1) above is a very special use of the big-O notation. I would rather keep saying O(N), where N = # files + # directories + # symlinks; and O(M), where M = # modified files + # modified directories + # modified symlinks. I agree that O(M) implementation (i.e. include O(1) snapshot creation) is our end goal.
          Hide
          Suresh Srinivas added a comment -

          This sort of fundamental design decision is not something that can be easily improved incrementally. Copying huge portions of the working set, and then making that copying fast and space efficient, should not be the goal. The goal should be to entirely avoid copying huge portions of the working set.

          That is the goal. Where I disagree is with the assertion that we cannot get to it starting with a simple implementation. I would like to get the simple implementation done ASAP with all the tests developed in parallel. Then we will change the implementation for better efficiency both in time and space. Tests will help ensure the correctness of optimizations.

          We already have a bunch of ideas on how to do it. Some of those ideas are in the early prototype that Hari has uploaded. HDFS-4103 has been created to allay these concerns that have been repeatedly brought up to convey that it is goal of HDFS-2802 to have optimized snapshots. Will update the design document as a part of that jira.

          Show
          Suresh Srinivas added a comment - This sort of fundamental design decision is not something that can be easily improved incrementally. Copying huge portions of the working set, and then making that copying fast and space efficient, should not be the goal. The goal should be to entirely avoid copying huge portions of the working set. That is the goal. Where I disagree is with the assertion that we cannot get to it starting with a simple implementation. I would like to get the simple implementation done ASAP with all the tests developed in parallel. Then we will change the implementation for better efficiency both in time and space. Tests will help ensure the correctness of optimizations. We already have a bunch of ideas on how to do it. Some of those ideas are in the early prototype that Hari has uploaded. HDFS-4103 has been created to allay these concerns that have been repeatedly brought up to convey that it is goal of HDFS-2802 to have optimized snapshots. Will update the design document as a part of that jira.
          Hide
          Aaron T. Myers added a comment -

          That is the goal.

          Great, I'm glad you agree. I'd be happy to personally work on a more time/space efficient snapshot solution.

          Where I disagree is with the assertion that we cannot get to it starting with a simple implementation. I would like to get the simple implementation done ASAP with all the tests developed in parallel. Then we will change the implementation for better efficiency both in time and space. Tests will help ensure the correctness of optimizations.

          I suspect you'll find that implementing the copying solution will not be much easier than a zero-copy/COW solution, and so seems to me like implementing the copying solution will result in a lot of wasted development/testing work.

          To be completely clear, if it makes sense to implement a copying solution as an interim development stage, then that's fine, but I don't think that a solution which copies all of the files/directories that are being snapshotted should be merged to trunk.

          Show
          Aaron T. Myers added a comment - That is the goal. Great, I'm glad you agree. I'd be happy to personally work on a more time/space efficient snapshot solution. Where I disagree is with the assertion that we cannot get to it starting with a simple implementation. I would like to get the simple implementation done ASAP with all the tests developed in parallel. Then we will change the implementation for better efficiency both in time and space. Tests will help ensure the correctness of optimizations. I suspect you'll find that implementing the copying solution will not be much easier than a zero-copy/COW solution, and so seems to me like implementing the copying solution will result in a lot of wasted development/testing work. To be completely clear, if it makes sense to implement a copying solution as an interim development stage, then that's fine, but I don't think that a solution which copies all of the files/directories that are being snapshotted should be merged to trunk.
          Hide
          Suresh Srinivas added a comment -

          Great, I'm glad you agree.

          I have agreed many times here and here. I am happy to get beyond that issue Thanks for the offer to help. I think for now we are on a path to get it done very soon.

          To be completely clear, if it makes sense to implement a copying solution as an interim development stage

          Sounds good. I misunderstood you to be saying opposite of that here

          I don't think that a solution which copies all of the files/directories that are being snapshotted should be merged to trunk.

          I disagree. But we may just end up doing optimization prior to merge.

          Show
          Suresh Srinivas added a comment - Great, I'm glad you agree. I have agreed many times here and here . I am happy to get beyond that issue Thanks for the offer to help. I think for now we are on a path to get it done very soon. To be completely clear, if it makes sense to implement a copying solution as an interim development stage Sounds good. I misunderstood you to be saying opposite of that here I don't think that a solution which copies all of the files/directories that are being snapshotted should be merged to trunk. I disagree. But we may just end up doing optimization prior to merge.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          > I don't think that a solution which copies all of the files/directories that are being snapshotted should be merged to trunk.

          I disagree. But we may just end up doing optimization prior to merge.

          I understand that O(1) snapshot creation is desirable but I don't think it is a requirement. Currently, the snapshot feature is missing in HDFS. Having snapshot with O(N) snapshot creation already opens the door of opportunity to many applications. Only the applications requiring O(1) snapshot creation should wait for the improvement but not everyone has to wait for it.

          Show
          Tsz Wo Nicholas Sze added a comment - > I don't think that a solution which copies all of the files/directories that are being snapshotted should be merged to trunk. I disagree. But we may just end up doing optimization prior to merge. I understand that O(1) snapshot creation is desirable but I don't think it is a requirement. Currently, the snapshot feature is missing in HDFS. Having snapshot with O(N) snapshot creation already opens the door of opportunity to many applications. Only the applications requiring O(1) snapshot creation should wait for the improvement but not everyone has to wait for it.
          Hide
          Aaron T. Myers added a comment -

          Hi all, attached please find a somewhat different design for implementing snapshot support in HDFS that myself and a few others have discussed. Please have a look at it.

          Though this design differs somewhat from the previous design posted by Nicholas, I don't think the two designs insurmountably far apart. Though I certainly don't expect to switch to this design wholesale, I would like to see if we can come up with a hybrid design which incorporates some aspects of both. Let me try to outline where I see the designs differing, and suggest ways we can move forward with a hybrid design.

          1. Efficiency of snapshot creation. In the design posted by Nicholas, creation of a snapshot is O {n}

            in terms of the number of files/directories captured by the snapshot, both in terms of time and space efficiency. The design proposed in this document would be O

            {1} at snapshot creation time, and then copy-on-write thereafter for files/directories which are modified after the snapshot is created. This is accomplished by assigning unique, increasing integer IDs to snapshots and giving each INode a start_snap and end_snap ID to denote which snapshots the INode should be a part of. I'm not wedded to the precise design described in this document, but it seems like a reasonable design to me, so I'd like to consider this for the design to implement HDFS-4103 (Support O{1}

            snapshot creation).

          2. Support for subdirectory snapshots. The design posted by Nicholas allows for individual subdirectories of an HDFS namespace to be snapshotted by introducing "snapshottable directories." The design proposed in this document would only support snapshots at the root level of the file system. I think an easy way to produce a hybrid between these two designs would be to stick with the "snapshottable directory" system described in the document posted by Nicholas, and store the snapshot ID info at that INodeDirectory, instead of globally for the whole file system as is described in the document I've just posted. Such a scheme will allow both for efficient snapshot creation and creation of snapshots of subdirectories of the file system.
          3. Support for non-super users to create snapshots. The design posted by Nicholas allows for non-super users to create snapshots. The scheme described in the document I've just posted would only allow super users to create snapshots, in instances where administrators want tight control over the snapshots in their system. I propose we stick with the design described in the document posted by Nicholas, but allow for user-initiated snapshot creation to be optionally disabled by the administrator, either globally or per-snapshottable directory. This should allow for both use cases simultaneously.
          4. Materialization of snapshots. The scheme described in the document posted by Nicholas allows for the state of the FS in a snapshot to only be accessed from the snapshot root, i.e. the snapshottable directory, and allows for snapshots to be created with arbitrary names. The scheme described by the document I've just posted would have the return value of ClientProtocol#getListing modified on the fly by the NameNode so that a ".snapshots" directory will appear to be present in every directory which has a snapshot available for it, with the available snapshots listed under this "directory" by their snapshot ID. This is similar to the user experience that users of WAFL file systems are familiar to, and so should be familiar to many users of FS snapshots. I'd like us to consider going with this scheme.

          Please consider this proposal. I'd love to discuss this further at the design meeting later this week as previously mentioned by Suresh. By the way, can we nail down the precise date/time for that meeting? Sanjay mentioned to me offline that it would probably be on Wednesday, but I haven't heard anything beyond that. I'd be happy to offer up space in the Cloudera office, if that would be helpful. Let me know.

          Thanks everyone.

          Show
          Aaron T. Myers added a comment - Hi all, attached please find a somewhat different design for implementing snapshot support in HDFS that myself and a few others have discussed. Please have a look at it. Though this design differs somewhat from the previous design posted by Nicholas, I don't think the two designs insurmountably far apart. Though I certainly don't expect to switch to this design wholesale, I would like to see if we can come up with a hybrid design which incorporates some aspects of both. Let me try to outline where I see the designs differing, and suggest ways we can move forward with a hybrid design. Efficiency of snapshot creation. In the design posted by Nicholas, creation of a snapshot is O {n} in terms of the number of files/directories captured by the snapshot, both in terms of time and space efficiency. The design proposed in this document would be O {1} at snapshot creation time, and then copy-on-write thereafter for files/directories which are modified after the snapshot is created. This is accomplished by assigning unique, increasing integer IDs to snapshots and giving each INode a start_snap and end_snap ID to denote which snapshots the INode should be a part of. I'm not wedded to the precise design described in this document, but it seems like a reasonable design to me, so I'd like to consider this for the design to implement HDFS-4103 (Support O{1} snapshot creation). Support for subdirectory snapshots. The design posted by Nicholas allows for individual subdirectories of an HDFS namespace to be snapshotted by introducing "snapshottable directories." The design proposed in this document would only support snapshots at the root level of the file system. I think an easy way to produce a hybrid between these two designs would be to stick with the "snapshottable directory" system described in the document posted by Nicholas, and store the snapshot ID info at that INodeDirectory, instead of globally for the whole file system as is described in the document I've just posted. Such a scheme will allow both for efficient snapshot creation and creation of snapshots of subdirectories of the file system. Support for non-super users to create snapshots. The design posted by Nicholas allows for non-super users to create snapshots. The scheme described in the document I've just posted would only allow super users to create snapshots, in instances where administrators want tight control over the snapshots in their system. I propose we stick with the design described in the document posted by Nicholas, but allow for user-initiated snapshot creation to be optionally disabled by the administrator, either globally or per-snapshottable directory. This should allow for both use cases simultaneously. Materialization of snapshots. The scheme described in the document posted by Nicholas allows for the state of the FS in a snapshot to only be accessed from the snapshot root, i.e. the snapshottable directory, and allows for snapshots to be created with arbitrary names. The scheme described by the document I've just posted would have the return value of ClientProtocol#getListing modified on the fly by the NameNode so that a ".snapshots" directory will appear to be present in every directory which has a snapshot available for it, with the available snapshots listed under this "directory" by their snapshot ID. This is similar to the user experience that users of WAFL file systems are familiar to, and so should be familiar to many users of FS snapshots. I'd like us to consider going with this scheme. Please consider this proposal. I'd love to discuss this further at the design meeting later this week as previously mentioned by Suresh. By the way, can we nail down the precise date/time for that meeting? Sanjay mentioned to me offline that it would probably be on Wednesday, but I haven't heard anything beyond that. I'd be happy to offer up space in the Cloudera office, if that would be helpful. Let me know. Thanks everyone.
          Hide
          Suresh Srinivas added a comment -

          Aaron, I have not looked at your design. Will do so when I get a chance.

          Efficient of snapshot and materialization of snapshots

          Many of the things you are repeating again, about space and creation efficiency etc. I have addressed them in many comments. Please do not comment as if the current work is not considering it. Our solutions will be based on well known solution in the literature for this problem. Please see on HDFS-4125.

          Some of the differences around administrative can be made configurable, if need be. These are surround aspect of the feature.

          Given repeated feedback about optimizations we are working on updating the implementation and design. Earlier the goal was to get the inefficient but correct first implementation with tests and have design review, followed by optimization. Given the feedback so far, we will post updated design next week. Possibly hold a design review in around two weeks.

          BTW if this design/implementation needs changes later we can have a discussion. At this point in time, I would like to focus on getting parts of the implementation we have had in our prototype out in this branch. I am fine if we need to throw away parts of it.

          Show
          Suresh Srinivas added a comment - Aaron, I have not looked at your design. Will do so when I get a chance. Efficient of snapshot and materialization of snapshots Many of the things you are repeating again, about space and creation efficiency etc. I have addressed them in many comments. Please do not comment as if the current work is not considering it. Our solutions will be based on well known solution in the literature for this problem. Please see on HDFS-4125 . Some of the differences around administrative can be made configurable, if need be. These are surround aspect of the feature. Given repeated feedback about optimizations we are working on updating the implementation and design. Earlier the goal was to get the inefficient but correct first implementation with tests and have design review, followed by optimization. Given the feedback so far, we will post updated design next week. Possibly hold a design review in around two weeks. BTW if this design/implementation needs changes later we can have a discussion. At this point in time, I would like to focus on getting parts of the implementation we have had in our prototype out in this branch. I am fine if we need to throw away parts of it.
          Hide
          Aaron T. Myers added a comment -

          Many of the things you are repeating again, about space and creation efficiency etc. I have addressed them in many comments. Please do not comment as if the current work is not considering it.

          I definitely didn't mean to beat a dead horse on this subject. My apologies that the comment came across that way. I was trying to enumerate the differences between the documents, and propose a specific solution to this problem. I certainly didn't intend to imply that the current work is not considering this problem, hence why I referenced HDFS-4125 in my comment.

          Our solutions will be based on well known solution in the literature for this problem. Please see on HDFS-4125.

          I have been following HDFS-4125, but the comment in HDFS-4125 regarding the solution you're referring to wasn't posted until after my most recent comment.

          Some of the differences around administrative can be made configurable, if need be. These are surround aspect of the feature.

          Sounds good.

          Given repeated feedback about optimizations we are working on updating the implementation and design. Earlier the goal was to get the inefficient but correct first implementation with tests and have design review, followed by optimization. Given the feedback so far, we will post updated design next week. Possibly hold a design review in around two weeks.

          Cool. I look forward to reading the updated design doc.

          BTW if this design/implementation needs changes later we can have a discussion. At this point in time, I would like to focus on getting parts of the implementation we have had in our prototype out in this branch. I am fine if we need to throw away parts of it.

          My preference would be to collaborate on and agree to a design upfront so that we can come up with something that will be amenable to everyone before we spend a lot of time implementing something and potentially discarding it.

          By the way, I forgot to mention that the design document I posted contains some discussion and specific design proposals regarding the consistency of NN/DN data in snapshots. Please take a look at that as well.

          Show
          Aaron T. Myers added a comment - Many of the things you are repeating again, about space and creation efficiency etc. I have addressed them in many comments. Please do not comment as if the current work is not considering it. I definitely didn't mean to beat a dead horse on this subject. My apologies that the comment came across that way. I was trying to enumerate the differences between the documents, and propose a specific solution to this problem. I certainly didn't intend to imply that the current work is not considering this problem, hence why I referenced HDFS-4125 in my comment. Our solutions will be based on well known solution in the literature for this problem. Please see on HDFS-4125 . I have been following HDFS-4125 , but the comment in HDFS-4125 regarding the solution you're referring to wasn't posted until after my most recent comment. Some of the differences around administrative can be made configurable, if need be. These are surround aspect of the feature. Sounds good. Given repeated feedback about optimizations we are working on updating the implementation and design. Earlier the goal was to get the inefficient but correct first implementation with tests and have design review, followed by optimization. Given the feedback so far, we will post updated design next week. Possibly hold a design review in around two weeks. Cool. I look forward to reading the updated design doc. BTW if this design/implementation needs changes later we can have a discussion. At this point in time, I would like to focus on getting parts of the implementation we have had in our prototype out in this branch. I am fine if we need to throw away parts of it. My preference would be to collaborate on and agree to a design upfront so that we can come up with something that will be amenable to everyone before we spend a lot of time implementing something and potentially discarding it. By the way, I forgot to mention that the design document I posted contains some discussion and specific design proposals regarding the consistency of NN/DN data in snapshots. Please take a look at that as well.
          Hide
          Eli Collins added a comment -

          Setting the target version to 3.0 since this will require a file system upgrade due to metadata changes (eg HDFS-4126, HDFS-4086, and HDFS-4082) and we're heading to GA soon.

          Show
          Eli Collins added a comment - Setting the target version to 3.0 since this will require a file system upgrade due to metadata changes (eg HDFS-4126 , HDFS-4086 , and HDFS-4082 ) and we're heading to GA soon.
          Hide
          Eli Collins added a comment -

          Will schedule a design review meeting to go over the requirements and design in detail with the folks who are interested. Especially to get feedback from other Hadoop projects. This meeting will be in the week of Oct 29, post Hadoop World.

          I went ahead and scheduled this, will host at the Cloudera office in Palo Alto. Since we've got interested people from Sunnyvale to SF, Palo Alto seems like a good middle ground. Also, I've confirmed with some HBase people that they will be able to attend so we can feedback from other projects. Here's the meetup page, I'll add Webex notes for remote people soon. Also, I'll take minutes and post them back here.

          http://www.meetup.com/Hadoop-Contributors/events/89110832/

          Show
          Eli Collins added a comment - Will schedule a design review meeting to go over the requirements and design in detail with the folks who are interested. Especially to get feedback from other Hadoop projects. This meeting will be in the week of Oct 29, post Hadoop World. I went ahead and scheduled this, will host at the Cloudera office in Palo Alto. Since we've got interested people from Sunnyvale to SF, Palo Alto seems like a good middle ground. Also, I've confirmed with some HBase people that they will be able to attend so we can feedback from other projects. Here's the meetup page, I'll add Webex notes for remote people soon. Also, I'll take minutes and post them back here. http://www.meetup.com/Hadoop-Contributors/events/89110832/
          Hide
          Suresh Srinivas added a comment -

          Given repeated feedback about optimizations we are working on updating the implementation and design. Earlier the goal was to get the inefficient but correct first implementation with tests and have design review, followed by optimization. Given the feedback so far, we will post updated design next week. Possibly hold a design review in around two weeks.

          So I prefer having this meeting later in the week of Nov 5th. Nov 1st and 2nd I am out of town.

          Show
          Suresh Srinivas added a comment - Given repeated feedback about optimizations we are working on updating the implementation and design. Earlier the goal was to get the inefficient but correct first implementation with tests and have design review, followed by optimization. Given the feedback so far, we will post updated design next week. Possibly hold a design review in around two weeks. So I prefer having this meeting later in the week of Nov 5th. Nov 1st and 2nd I am out of town.
          Hide
          Sanjay Radia added a comment -

          A couple of the key folks (suresh, nicholas) working on HDFS-2802 cannot attends because they are traveling. I am rescheduling for the following week.
          I have rescheduled the meeting for the following week.
          http://www.meetup.com/Hadoop-Contributors/events/89116912/

          Please sign up.

          Show
          Sanjay Radia added a comment - A couple of the key folks (suresh, nicholas) working on HDFS-2802 cannot attends because they are traveling. I am rescheduling for the following week. I have rescheduled the meeting for the following week. http://www.meetup.com/Hadoop-Contributors/events/89116912/ Please sign up.
          Hide
          Eli Collins added a comment -

          Let's have both on the calendar, will be useful to have multiple sessions anyway, and Sunnyvale is pretty far for SF people. Will be useful to hear from HBase people in the meantime. I'll post notes for people who can't make it (happy to have remote people dial in as well).

          Show
          Eli Collins added a comment - Let's have both on the calendar, will be useful to have multiple sessions anyway, and Sunnyvale is pretty far for SF people. Will be useful to hear from HBase people in the meantime. I'll post notes for people who can't make it (happy to have remote people dial in as well).
          Hide
          Konstantin Shvachko added a comment -

          What if we think about snapshots in terms of versioning.
          Suppose we have file versions, which can be rolled on demand. Then taking a snapshot is just rolling the version of a subtree. Would it be easier to deal with versions? I am talking about the abstraction rather than implementation.

          Show
          Konstantin Shvachko added a comment - What if we think about snapshots in terms of versioning. Suppose we have file versions, which can be rolled on demand. Then taking a snapshot is just rolling the version of a subtree. Would it be easier to deal with versions? I am talking about the abstraction rather than implementation.
          Hide
          Suresh Srinivas added a comment -

          Konstantin, not sure what you mean rolled on demand exactly. However since you brought of versioning, the current implementation is based on versioning. Instead of version for every change, the version is tracked at the snapshot boundaries. Look at HDFS-4125 and the proposal to use partial persistent data structure.

          Show
          Suresh Srinivas added a comment - Konstantin, not sure what you mean rolled on demand exactly. However since you brought of versioning, the current implementation is based on versioning. Instead of version for every change, the version is tracked at the snapshot boundaries. Look at HDFS-4125 and the proposal to use partial persistent data structure.
          Hide
          Aaron T. Myers added a comment -

          Hi Konstantin, if I correctly understand what you're suggesting, it sounds very similar to what is described in the "Representing Snapshots at the NameNode" section of the design proposal I posted. Under this scheme, each file and directory is tagged with its start and end version numbers. The start version number represents the point in the file system history when this file/directory came into existence, and the end version number represents the point in the file system history where the file/directory either was modified or was deleted. Taking a snapshot is, as you described, as simple as incrementing the current version number of the file system (or a subtree) as you described. Would you mind taking a look at that portion of the design document to see if it is inline with what you're thinking about?

          Show
          Aaron T. Myers added a comment - Hi Konstantin, if I correctly understand what you're suggesting, it sounds very similar to what is described in the "Representing Snapshots at the NameNode" section of the design proposal I posted. Under this scheme, each file and directory is tagged with its start and end version numbers. The start version number represents the point in the file system history when this file/directory came into existence, and the end version number represents the point in the file system history where the file/directory either was modified or was deleted. Taking a snapshot is, as you described, as simple as incrementing the current version number of the file system (or a subtree) as you described. Would you mind taking a look at that portion of the design document to see if it is inline with what you're thinking about?
          Hide
          Konstantin Shvachko added a comment -

          By rolling version on demand I just mean there is an API to create a new version of an object and keep the old one.
          My feeling is that thinking explicitly about versions rather than snapshots in the context of this design will simplify things and make APIs more understandable for users.
          Especially since the design is already based on versioning. And the document has an implementation proposal for that.

          Show
          Konstantin Shvachko added a comment - By rolling version on demand I just mean there is an API to create a new version of an object and keep the old one. My feeling is that thinking explicitly about versions rather than snapshots in the context of this design will simplify things and make APIs more understandable for users. Especially since the design is already based on versioning. And the document has an implementation proposal for that.
          Hide
          Suresh Srinivas added a comment -

          Attaching the updated design document. Hopefully it addresses the issues that have been rasied. This should give sufficient details about the implementation we are currently working on. To summarize:

          1. Snapshot allowed only at the root vs snapshot at the subdirectories - Ability to snapshot a sub-directory is very important requirement for many Hadoop users. Please see the requirements in the document posted for more details. The alternate proposal to allow snapshots only at the root is a non-starter in this regard.
          2. Efficiency of snapshot creation and management - The current design addresses the concerns raised. To summarize the creation of snapshot is O(1). The design uses copy-on-modify approach so that the cost of snapshot is zero when there is no modification and is proportional the modifications when they are made. Please provide feedback.
          3. Snapshot of being written files and consistency - Our document describes several design choices - some easy, some complicated. Please see the proposed choice in the document. We could continue this discussion in HDFS-3960.

          I took a look at the alternate proposal. It is too high level without sufficient details to evaluate. From my limited understanding of the alternate proposal, the design document we have posted here has several significant advantages over it:

          1. It supports sub-directory snapshots, an important use case for many Hadoop users.
          2. It supports on-demand and user managed snapshots.
          3. When snapshots are not created, there is no cost incurred in terms of memory. The alternate proposal has O(N) memory cost for storing tags.
          4. Our design can also be extended to do RW snapshots, if we feel a need for it.

          Hopefully this clarifies the design better. We would like continue focus on implementing it. Any feedback provided will be incorporated into the design and the implementation.

          Show
          Suresh Srinivas added a comment - Attaching the updated design document. Hopefully it addresses the issues that have been rasied. This should give sufficient details about the implementation we are currently working on. To summarize: Snapshot allowed only at the root vs snapshot at the subdirectories - Ability to snapshot a sub-directory is very important requirement for many Hadoop users. Please see the requirements in the document posted for more details. The alternate proposal to allow snapshots only at the root is a non-starter in this regard. Efficiency of snapshot creation and management - The current design addresses the concerns raised. To summarize the creation of snapshot is O(1). The design uses copy-on-modify approach so that the cost of snapshot is zero when there is no modification and is proportional the modifications when they are made. Please provide feedback. Snapshot of being written files and consistency - Our document describes several design choices - some easy, some complicated. Please see the proposed choice in the document. We could continue this discussion in HDFS-3960 . I took a look at the alternate proposal. It is too high level without sufficient details to evaluate. From my limited understanding of the alternate proposal, the design document we have posted here has several significant advantages over it: It supports sub-directory snapshots, an important use case for many Hadoop users. It supports on-demand and user managed snapshots. When snapshots are not created, there is no cost incurred in terms of memory. The alternate proposal has O(N) memory cost for storing tags. Our design can also be extended to do RW snapshots, if we feel a need for it. Hopefully this clarifies the design better. We would like continue focus on implementing it. Any feedback provided will be incorporated into the design and the implementation.
          Hide
          Aaron T. Myers added a comment -

          Hi Suresh, thanks a lot for posting an updated design doc. From my quick look, this seems to clarify much of the proposed design.

          I do have one question for you from my quick read of this updated doc: It seems that since every current file which is snapshotted is represented by an INodeFileWithLink, at some point all files which are to be snapshotted under a sub-directory would need to be converted from an INodeFile to an INodeFileWithLink. When would this conversion take place? Perhaps when the directory is marked as being snapshottable using the `dfsadmin -allowSnapshot' command? Or perhaps when a snapshot is first created of a snapshottable directory? (Not suggesting this is unacceptable - just trying to understand the design.)

          Show
          Aaron T. Myers added a comment - Hi Suresh, thanks a lot for posting an updated design doc. From my quick look, this seems to clarify much of the proposed design. I do have one question for you from my quick read of this updated doc: It seems that since every current file which is snapshotted is represented by an INodeFileWithLink, at some point all files which are to be snapshotted under a sub-directory would need to be converted from an INodeFile to an INodeFileWithLink. When would this conversion take place? Perhaps when the directory is marked as being snapshottable using the `dfsadmin -allowSnapshot' command? Or perhaps when a snapshot is first created of a snapshottable directory? (Not suggesting this is unacceptable - just trying to understand the design.)
          Hide
          Suresh Srinivas added a comment -

          From my quick look, this seems to clarify much of the proposed design

          I am happy to hear this.

          It seems that since every current file which is snapshotted is represented by an INodeFileWithLink, at some point all files which are to be snapshotted under a sub-directory would need to be converted from an INodeFile to an INodeFileWithLink.

          When a snapshotted file is modified, the conversion to INodeFileWithLink takes place. So if all the snapshotted files under a directory are modified post snapshot, they will need to be converted into INodeFileWithLink. BTW We have flexibility here and we can eliminate the need for this completely, if we decide snapshot of file does not preserve the replication factor at the time of snapshot - see replication factor related section for details.

          Show
          Suresh Srinivas added a comment - From my quick look, this seems to clarify much of the proposed design I am happy to hear this. It seems that since every current file which is snapshotted is represented by an INodeFileWithLink, at some point all files which are to be snapshotted under a sub-directory would need to be converted from an INodeFile to an INodeFileWithLink. When a snapshotted file is modified, the conversion to INodeFileWithLink takes place. So if all the snapshotted files under a directory are modified post snapshot, they will need to be converted into INodeFileWithLink. BTW We have flexibility here and we can eliminate the need for this completely, if we decide snapshot of file does not preserve the replication factor at the time of snapshot - see replication factor related section for details.
          Hide
          Konstantin Shvachko added a comment -

          I propose to divide this discussion into three categories: design goals, API and semantics, algorithms and implementation. If people can agree on one we can move to the next.

          1. I see three main design goals proposed: snapshots should be (a) read-only, (b) directory-level, (c) multiple.
            This should hopefully work for everybody.
          2. API. Seems to me the most important point now.
            HDFSSnapshotsDesign.pdf doesn't talk much about APIs except a reference to WAFL.
            Snapshots20121030.pdf has examples of shell commands, which look a bit convoluted. I mean using delimiter ".snapshot" to specify a snapshot means I cannot have entries with that name.
            Wouldn't it be better to control access to snapshots via -version option:
            rm -r -version 3 /user/shv/hbase/ remove snapshot with id 3.
            ls -version 2 /user/shv/hbase/ listing of the snapshot #2.
            ls -versions /user/shv/hbase/ listing of snapshot ids of the directory.
            Where non -versioned commands deal with "current" state of the file system as today.
            I like the idea of generating globally unique version ids, and assigning them to snapshots internally rather than letting people invent their own. One can always list available versions and read the desired one. So the -createSnapshot command does not need to pass <snapname>, but will instead get it in return.
          3. Algorithms. I agree the length of an under-construction file in the snapshot should come directly from the namespace. And we provide means to update it with hflush before the snapshot is taken.
            Creating duplicate INodes with a diff, this is sort of COW technique, right? Sounds hard.
            It is simpler for me to think of versioned files and directories in this case. Creating a snapshot assigns a new version to objects.
            Deleting a file should remove current version, but leave other versions unchanged. Can be implemented by marking the file "deleted" until all versions disappear, when it can be physically removed.

          My dumb question: can I create a snapshot of a subdirectory that is a part of a snapshot above it?

          Show
          Konstantin Shvachko added a comment - I propose to divide this discussion into three categories: design goals, API and semantics, algorithms and implementation. If people can agree on one we can move to the next. I see three main design goals proposed: snapshots should be (a) read-only, (b) directory-level, (c) multiple. This should hopefully work for everybody. API . Seems to me the most important point now. HDFSSnapshotsDesign.pdf doesn't talk much about APIs except a reference to WAFL. Snapshots20121030.pdf has examples of shell commands, which look a bit convoluted. I mean using delimiter ".snapshot" to specify a snapshot means I cannot have entries with that name. Wouldn't it be better to control access to snapshots via -version option: rm -r -version 3 /user/shv/hbase/ remove snapshot with id 3. ls -version 2 /user/shv/hbase/ listing of the snapshot #2. ls -versions /user/shv/hbase/ listing of snapshot ids of the directory. Where non -versioned commands deal with "current" state of the file system as today. I like the idea of generating globally unique version ids, and assigning them to snapshots internally rather than letting people invent their own. One can always list available versions and read the desired one. So the -createSnapshot command does not need to pass <snapname>, but will instead get it in return. Algorithms . I agree the length of an under-construction file in the snapshot should come directly from the namespace. And we provide means to update it with hflush before the snapshot is taken. Creating duplicate INodes with a diff, this is sort of COW technique, right? Sounds hard. It is simpler for me to think of versioned files and directories in this case. Creating a snapshot assigns a new version to objects. Deleting a file should remove current version, but leave other versions unchanged. Can be implemented by marking the file "deleted" until all versions disappear, when it can be physically removed. My dumb question: can I create a snapshot of a subdirectory that is a part of a snapshot above it?
          Hide
          Suresh Srinivas added a comment -

          Konstantin ShvachkoThanks for your comments. Some answers here:
          versions seem like an interesting capability to support. I want to think about it more. However snapshots have their own use cases as well. Ability to snapshot and attach a name to it comes handy for writing applications using this functionality.

          .snapshot is a convention that is used widely and requires no changes to many of the APIs such as rm, ls etc. In fact we had initially proposed using .snap_<snapshot_name>. After looking at many other file systems, where .snapshot convention is used to identify the snapshot, we decided to go with it.

          Creating duplicate INodes with a diff, this is sort of COW technique, right? Sounds hard.

          Hopefully this should not add too much code complexity. But this is being done to avoid taking too much memory for snapshots on the namenode.

          My dumb question: can I create a snapshot of a subdirectory that is a part of a snapshot above it?

          This is not a dumb question The design already supports nested snapshots. I will add an update to the document describing this.

          Show
          Suresh Srinivas added a comment - Konstantin Shvachko Thanks for your comments. Some answers here: versions seem like an interesting capability to support. I want to think about it more. However snapshots have their own use cases as well. Ability to snapshot and attach a name to it comes handy for writing applications using this functionality. .snapshot is a convention that is used widely and requires no changes to many of the APIs such as rm, ls etc. In fact we had initially proposed using .snap_<snapshot_name>. After looking at many other file systems, where .snapshot convention is used to identify the snapshot, we decided to go with it. Creating duplicate INodes with a diff, this is sort of COW technique, right? Sounds hard. Hopefully this should not add too much code complexity. But this is being done to avoid taking too much memory for snapshots on the namenode. My dumb question: can I create a snapshot of a subdirectory that is a part of a snapshot above it? This is not a dumb question The design already supports nested snapshots. I will add an update to the document describing this.
          Hide
          Suresh Srinivas added a comment -

          I went ahead and scheduled this...

          Eli CollinsI did not schedule it the week of Hadoop World even though it was convenient for me and Nicholas. I have shown sensitivity towards your team's travels. Unfortunately I and Nicholas are scheduled for a travel this week. Even though I had indicated that I will schedule a meeting, instead of giving me your feedback on time, you have gone ahead and scheduled this meeting without consulting the availability of the main authors of this jira. I am not sure what you are trying to achieve here.

          My suggestion is to cancel the first meeting and stick with the second meeting. If the time for the second meeting does not work, Sanjay could reschedule it.

          Show
          Suresh Srinivas added a comment - I went ahead and scheduled this... Eli Collins I did not schedule it the week of Hadoop World even though it was convenient for me and Nicholas. I have shown sensitivity towards your team's travels. Unfortunately I and Nicholas are scheduled for a travel this week. Even though I had indicated that I will schedule a meeting, instead of giving me your feedback on time, you have gone ahead and scheduled this meeting without consulting the availability of the main authors of this jira. I am not sure what you are trying to achieve here. My suggestion is to cancel the first meeting and stick with the second meeting. If the time for the second meeting does not work, Sanjay could reschedule it.
          Hide
          Eli Collins added a comment -

          Hey Suresh,

          This isn't the one and only discussion on snapshots, a lot of us plan to attend the following one as well. It's OK to have multiple discussions. A bunch of people are going to be around and discussing things on Thursday anyway, so I thought it better to open up the discussion to more people in the community. I'm sorry that you'll be out of town, I've setup a dial in and would love it if you and others that are out could attend. I'll post minutes here so that people who can not attend can see and participate.

          Show
          Eli Collins added a comment - Hey Suresh, This isn't the one and only discussion on snapshots, a lot of us plan to attend the following one as well. It's OK to have multiple discussions. A bunch of people are going to be around and discussing things on Thursday anyway, so I thought it better to open up the discussion to more people in the community. I'm sorry that you'll be out of town, I've setup a dial in and would love it if you and others that are out could attend. I'll post minutes here so that people who can not attend can see and participate.
          Hide
          Konstantin Shvachko added a comment -

          > After looking at many other file systems, where .snapshot convention

          Can you send a link?
          My concern is that you have a directory dr with two subdirectories sd2 and sd3. You create a snapshot of sd2 and sd3 under the same name. Then create a snapshot of dr with the same name. What happens?
          If the system controls ids this never happens because they are unique.

          Show
          Konstantin Shvachko added a comment - > After looking at many other file systems, where .snapshot convention Can you send a link? My concern is that you have a directory dr with two subdirectories sd2 and sd3. You create a snapshot of sd2 and sd3 under the same name. Then create a snapshot of dr with the same name. What happens? If the system controls ids this never happens because they are unique.
          Hide
          Suresh Srinivas added a comment -

          Can you send a link?

          NetApp uses this - search on google, you will find many other file systems such as HP X9000 File System etc.

          My concern is that you have a directory dr with two subdirectories sd2 and sd3. You create a snapshot of sd2 and sd3 under the same name. Then create a snapshot of dr with the same name. What happens?

          The design should work fine. Every snapshot has the following:

          • A directory where the snapshot is created
          • A name that is unique for a given directory where snapshot is taken
          • Internally snapshot dir + name is mapped to a unique snapshot ID and the diffs are maintained against that snapshot ID.

          When snapshot is accessed, the path used provides snapshotted directory and snapshot name. It gets mapped to a unique snapshot ID. For this we can get to snapshotted version.

          Show
          Suresh Srinivas added a comment - Can you send a link? NetApp uses this - search on google, you will find many other file systems such as HP X9000 File System etc. My concern is that you have a directory dr with two subdirectories sd2 and sd3. You create a snapshot of sd2 and sd3 under the same name. Then create a snapshot of dr with the same name. What happens? The design should work fine. Every snapshot has the following: A directory where the snapshot is created A name that is unique for a given directory where snapshot is taken Internally snapshot dir + name is mapped to a unique snapshot ID and the diffs are maintained against that snapshot ID. When snapshot is accessed, the path used provides snapshotted directory and snapshot name. It gets mapped to a unique snapshot ID. For this we can get to snapshotted version.
          Hide
          Aaron T. Myers added a comment -

          Hello all, attached please find an updated design proposal which attempts to merge the two designs as I previously described in this comment. This hybrid design merges the concepts of a snapshottable directory and user-initiated snapshots from the designs posted by Nicholas and Suresh, with the NN snapshot representation and client materialization of snapshots from the earlier design I posted. This document also expands upon the CLI commands/API that would be supported, as well as provide more details on how users would be able to access the snapshots available to them.

          Please have a look and review. I'd appreciate any feedback you have.

          I'm also uploading the .tex file used to produce this document and have removed the author listing. I hope this facilitates collaborating on a single design that we can eventually all agree to.

          Show
          Aaron T. Myers added a comment - Hello all, attached please find an updated design proposal which attempts to merge the two designs as I previously described in this comment . This hybrid design merges the concepts of a snapshottable directory and user-initiated snapshots from the designs posted by Nicholas and Suresh, with the NN snapshot representation and client materialization of snapshots from the earlier design I posted. This document also expands upon the CLI commands/API that would be supported, as well as provide more details on how users would be able to access the snapshots available to them. Please have a look and review. I'd appreciate any feedback you have. I'm also uploading the .tex file used to produce this document and have removed the author listing. I hope this facilitates collaborating on a single design that we can eventually all agree to.
          Hide
          Aaron T. Myers added a comment -

          Hi Konstantin,

          I like the idea of generating globally unique version ids, and assigning them to snapshots internally rather than letting people invent their own. One can always list available versions and read the desired one. So the -createSnapshot command does not need to pass <snapname>, but will instead get it in return.

          Please take a look at the updated design document I just uploaded. It goes into a lot more detail about how snapshots would be named and accessed by users. From my understanding of you're suggestion, I think it describes pretty much exactly what you propose with regard to this.

          Show
          Aaron T. Myers added a comment - Hi Konstantin, I like the idea of generating globally unique version ids, and assigning them to snapshots internally rather than letting people invent their own. One can always list available versions and read the desired one. So the -createSnapshot command does not need to pass <snapname>, but will instead get it in return. Please take a look at the updated design document I just uploaded. It goes into a lot more detail about how snapshots would be named and accessed by users. From my understanding of you're suggestion, I think it describes pretty much exactly what you propose with regard to this.
          Hide
          Aaron T. Myers added a comment -

          Hi Suresh, I've now had some more time to examine the document, and I have a few questions/comments:

          1. Regarding the ownership requirement for users to create snapshot - this answers my question above regarding what would happen when a user tries to create a snapshot of a subtree where the user doesn't have read access to all the files. But, why not decouple it completely from Unix permissions? They really have nothing to do with each other now. Perhaps we should just make a snapshottable directory have a list of users that are allowed to create snapshots of it?
          2. Related to the above, the document shows that deleting a snapshot consists of just using the FSShell to delete it. The document also says that even though a non super-user may create a snapshot for a given snapshottable directory, files under that directory that the user cannot read will continue to not be readable in the snapshot. Doesn't this imply that the user may not be able to delete a snapshot the user has just created? If so, I find the asymmetry between being able to create a snapshot, but not necessarily delete that snapshot, a little odd. I think it'd be better to have a deleteSnapshot administrative command to correspond to the createSnapshot command.
          3. Rename only within snapshottable directory - I don't see why this limitation is necessary or desirable. I see that the doc says "This can be revisited" but I'd really like to know why this restriction is proposed at all.
          4. Any other use for the linked list (INodeFileWithLink) besides determining the max replicaiton factor? The document says that this is one of the uses, but I don't see any other uses mentioned.
          5. You mentioned to Konstantin that the design supports nested snapshots, but I don't see any discussion of it in the document. My apologies if I missed this.
          6. Regarding open question #2 - I think that the values of the permission, atime, mtime, etc. attributes should definitely not change when the attributes of the original file changes. That should be part of the guarantee of a snapshot. In fact, I believe this open question is already answered by the #2 and #3 "Operations Supported on Snapshots". I do think it may make sense to treat the replication factor separately from this, however. Is this what you were referring to when you said "...if we decide snapshot of file does not preserve the replication factor at the time of snapshot"?
          Show
          Aaron T. Myers added a comment - Hi Suresh, I've now had some more time to examine the document, and I have a few questions/comments: Regarding the ownership requirement for users to create snapshot - this answers my question above regarding what would happen when a user tries to create a snapshot of a subtree where the user doesn't have read access to all the files. But, why not decouple it completely from Unix permissions? They really have nothing to do with each other now. Perhaps we should just make a snapshottable directory have a list of users that are allowed to create snapshots of it? Related to the above, the document shows that deleting a snapshot consists of just using the FSShell to delete it. The document also says that even though a non super-user may create a snapshot for a given snapshottable directory, files under that directory that the user cannot read will continue to not be readable in the snapshot. Doesn't this imply that the user may not be able to delete a snapshot the user has just created? If so, I find the asymmetry between being able to create a snapshot, but not necessarily delete that snapshot, a little odd. I think it'd be better to have a deleteSnapshot administrative command to correspond to the createSnapshot command. Rename only within snapshottable directory - I don't see why this limitation is necessary or desirable. I see that the doc says "This can be revisited" but I'd really like to know why this restriction is proposed at all. Any other use for the linked list (INodeFileWithLink) besides determining the max replicaiton factor? The document says that this is one of the uses, but I don't see any other uses mentioned. You mentioned to Konstantin that the design supports nested snapshots, but I don't see any discussion of it in the document. My apologies if I missed this. Regarding open question #2 - I think that the values of the permission, atime, mtime, etc. attributes should definitely not change when the attributes of the original file changes. That should be part of the guarantee of a snapshot. In fact, I believe this open question is already answered by the #2 and #3 "Operations Supported on Snapshots". I do think it may make sense to treat the replication factor separately from this, however. Is this what you were referring to when you said "...if we decide snapshot of file does not preserve the replication factor at the time of snapshot"?
          Hide
          Suresh Srinivas added a comment -

          Consolidated patch with early snapshot implementation and tests.

          Show
          Suresh Srinivas added a comment - Consolidated patch with early snapshot implementation and tests.
          Hide
          Sanjay Radia added a comment -

          Aaron
          wrt to your alternate proposal, folks can get confused about which of the two series of documents matched the patches in the Jira. Can you please rename yours as snapshot-alternate-design-proposal.

          {tex,pdf}

          and also reflect that in the title inside the document.

          Show
          Sanjay Radia added a comment - Aaron wrt to your alternate proposal, folks can get confused about which of the two series of documents matched the patches in the Jira. Can you please rename yours as snapshot-alternate-design-proposal. {tex,pdf} and also reflect that in the title inside the document.
          Hide
          Konstantin Shvachko added a comment -

          > .snapshot convention

          Agreed, this seems to be a recognized convention. This also helps to resolve name collision on subdirectory snapshots. Referring to my earlier example:

          dr/.snapshot/1/sd2/file.txt
          dr/sd2/.snapshot/1/file.txt
          

          The first points to file.txt under snapshot #1 for dr/. And the second refers to file.txt under snapshot #1 for sd2/.
          The API makes sense to me. I like the use of regular rm and ls commands dealing with snapshots.

          I would make the snapshot name optional in -createSnapshot (choosing names is such a hastle). Should be easy since you already have internal unique ids. Looking at Aaron's last design I see similar APIs, modular plural -allowSnapshots, and moving all snapshot operations under dfsadmin. The latter is not plausable, as users should be able to create and maintain snapshots as long as the administrator allowed it.

          Suresh, do I understand correctly that files under .snapshot are read-only, so rm -r will not remove it. You need to clarify how regular rm -r works in the design.
          Same with ls -r. I see Aaron proposes to skip .snapshot, which makes sense.
          Should we have something like ls -rs to list snapshotable directories in the tree?

          Aaron, I understand that you propose to store consequent ranges of snapshot versions in snapshoted files, and I see you adapted this "to support subtree snapshots".
          Not sure how you will handle the snapshot deletion in the middle of the range. Say you have snapshots 1 through 5, and then delete snapshot 3.
          The idea of monotonously increasing snapshot ids seems to be the only major diversion of your design from Suresh & Co. No doubt it should be discussed in details, but does that justify/qualify-for submitting an alternative design. You don't want to start Apache Hadoop design-document-wars here, do you?

          Show
          Konstantin Shvachko added a comment - > .snapshot convention Agreed, this seems to be a recognized convention. This also helps to resolve name collision on subdirectory snapshots. Referring to my earlier example: dr/.snapshot/1/sd2/file.txt dr/sd2/.snapshot/1/file.txt The first points to file.txt under snapshot #1 for dr/ . And the second refers to file.txt under snapshot #1 for sd2/ . The API makes sense to me. I like the use of regular rm and ls commands dealing with snapshots. I would make the snapshot name optional in -createSnapshot (choosing names is such a hastle). Should be easy since you already have internal unique ids. Looking at Aaron's last design I see similar APIs, modular plural -allowSnapshots , and moving all snapshot operations under dfsadmin. The latter is not plausable, as users should be able to create and maintain snapshots as long as the administrator allowed it. Suresh, do I understand correctly that files under .snapshot are read-only, so rm -r will not remove it. You need to clarify how regular rm -r works in the design. Same with ls -r . I see Aaron proposes to skip .snapshot, which makes sense. Should we have something like ls -rs to list snapshotable directories in the tree? Aaron, I understand that you propose to store consequent ranges of snapshot versions in snapshoted files, and I see you adapted this "to support subtree snapshots". Not sure how you will handle the snapshot deletion in the middle of the range. Say you have snapshots 1 through 5, and then delete snapshot 3. The idea of monotonously increasing snapshot ids seems to be the only major diversion of your design from Suresh & Co. No doubt it should be discussed in details, but does that justify/qualify-for submitting an alternative design. You don't want to start Apache Hadoop design-document-wars here, do you?
          Hide
          David S. Wang added a comment -

          Minutes from meeting on November 1, 2012.

          Show
          David S. Wang added a comment - Minutes from meeting on November 1, 2012.
          Hide
          Suresh Srinivas added a comment -

          Konstantin, thanks for the comments.

          I would make the snapshot name optional in -createSnapshot (choosing names is such a hastle). Should be easy since you already have internal unique ids.

          This is a good idea. Typically the snapshot names are date+timestamp. Should we choose that as the snapshot name when no name is provided?

          Suresh, do I understand correctly that files under .snapshot are read-only, so rm -r will not remove it. You need to clarify how regular rm -r works in the design.

          Yes all the contents below <path>/.snapshot/snapshotname is read-only. rm works only at the level <path>/.snapshot/snapshotname and not below it. I will add more details to the design doc.

          Show
          Suresh Srinivas added a comment - Konstantin, thanks for the comments. I would make the snapshot name optional in -createSnapshot (choosing names is such a hastle). Should be easy since you already have internal unique ids. This is a good idea. Typically the snapshot names are date+timestamp. Should we choose that as the snapshot name when no name is provided? Suresh, do I understand correctly that files under .snapshot are read-only, so rm -r will not remove it. You need to clarify how regular rm -r works in the design. Yes all the contents below <path>/.snapshot/snapshotname is read-only. rm works only at the level <path>/.snapshot/snapshotname and not below it. I will add more details to the design doc.
          Hide
          Suresh Srinivas added a comment -

          Konstantin, missed a few comments:

          Same with ls -r. I see Aaron proposes to skip .snapshot, which makes sense. Should we have something like ls -rs to list snapshotable directories in the tree?

          .snapshot is skipped when regular path is given. Let me go over some examples:

          1. <path> related operations that does not include .snapshot only operates on non snapshot related files.
            • ls -r <path> - will only list non snapshot files
            • rm -r <path> - as already stated in the document, if there are snapshots under that path, then deletion is not allowed, until all the snapshots are deleted.
          2. <path>/.snapshot applies to all the snapshots
            • ls <path>/.snapshot - lists all the snapshots under that path
            • ls -r <path>/.snapshot - I was thinking this would be same as ls <path>/.snapshot. I was thinking recursively listing all the snapshots could be expensive and may
              not be what a user wants. I want to get your thoughts on that.
            • rm <path>/.snapshot or rm -r <path>/.snapshot - removes all the snapshots. I want to get your thoughts on this as well.
          3. <path>/.snapshot/snapname
            • ls <path>/.snapshot/snapname - list files in the snapshot
            • ls -r <path>/.snapshot/snapname - recursively lists all the files in the snapshot name
            • rm <path>/.snapshot/snapname or rm -r <path>/.snapshot/snapname - deletes the snapshot
            • rm <path>/.snapshot/snapname/file - fails. snapshot is read-only.

          I will update the design document with these details.

          Show
          Suresh Srinivas added a comment - Konstantin, missed a few comments: Same with ls -r. I see Aaron proposes to skip .snapshot, which makes sense. Should we have something like ls -rs to list snapshotable directories in the tree? .snapshot is skipped when regular path is given. Let me go over some examples: <path> related operations that does not include .snapshot only operates on non snapshot related files. ls -r <path> - will only list non snapshot files rm -r <path> - as already stated in the document, if there are snapshots under that path, then deletion is not allowed, until all the snapshots are deleted. <path>/.snapshot applies to all the snapshots ls <path>/.snapshot - lists all the snapshots under that path ls -r <path>/.snapshot - I was thinking this would be same as ls <path>/.snapshot. I was thinking recursively listing all the snapshots could be expensive and may not be what a user wants. I want to get your thoughts on that. rm <path>/.snapshot or rm -r <path>/.snapshot - removes all the snapshots. I want to get your thoughts on this as well. <path>/.snapshot/snapname ls <path>/.snapshot/snapname - list files in the snapshot ls -r <path>/.snapshot/snapname - recursively lists all the files in the snapshot name rm <path>/.snapshot/snapname or rm -r <path>/.snapshot/snapname - deletes the snapshot rm <path>/.snapshot/snapname/file - fails. snapshot is read-only. I will update the design document with these details.
          Hide
          Aaron T. Myers added a comment -

          Sanjay Radia

          wrt to your alternate proposal, folks can get confused about which of the two series of documents matched the patches in the Jira. Can you please rename yours as snapshot-alternate-design-proposal.{tex,pdf} and also reflect that in the title inside the document.

          Sure, I'll name the files differently from now on, though my hope is to get us collaborating on a single design document, hence why I uploaded the .tex file as well. As Konst points out, the two design proposals are currently very close to each other.

          Konstantin Shvachko

          Not sure how you will handle the snapshot deletion in the middle of the range. Say you have snapshots 1 through 5, and then delete snapshot 3.

          I agree that this should be discussed more in the document. The idea is that to delete a snapshot from the perspective of the client, we'd only need to remove the info about that snapshot from the given snapshottable directory. That could be done in constant time, and clients would no longer be able to access that snapshot at that moment. Actual cleanup of files/directories which are no longer referenced by any snapshot could then be done synchronously or asynchronously at the NameNode.

          The idea of monotonously increasing snapshot ids seems to be the only major diversion of your design from Suresh & Co. No doubt it should be discussed in details, but does that justify/qualify-for submitting an alternative design.

          I agree that the latest posted design doc is very close to the ones posted by Suresh and Nicholas, but that was exactly the point - I was proposing a merged design that incorporated parts of both. I would've just posted an updated doc which modifies the previously posted proposal, as we often do with .patch files containing code, but that's difficult to do when only the .pdf is uploaded, hence why I posted an entirely different document.

          You don't want to start Apache Hadoop design-document-wars here, do you?

          Certainly not. I'm trying to collaborate on the design. The latest one I posted is an attempt at that.

          Show
          Aaron T. Myers added a comment - Sanjay Radia wrt to your alternate proposal, folks can get confused about which of the two series of documents matched the patches in the Jira. Can you please rename yours as snapshot-alternate-design-proposal.{tex,pdf} and also reflect that in the title inside the document. Sure, I'll name the files differently from now on, though my hope is to get us collaborating on a single design document, hence why I uploaded the .tex file as well. As Konst points out, the two design proposals are currently very close to each other. Konstantin Shvachko Not sure how you will handle the snapshot deletion in the middle of the range. Say you have snapshots 1 through 5, and then delete snapshot 3. I agree that this should be discussed more in the document. The idea is that to delete a snapshot from the perspective of the client, we'd only need to remove the info about that snapshot from the given snapshottable directory. That could be done in constant time, and clients would no longer be able to access that snapshot at that moment. Actual cleanup of files/directories which are no longer referenced by any snapshot could then be done synchronously or asynchronously at the NameNode. The idea of monotonously increasing snapshot ids seems to be the only major diversion of your design from Suresh & Co. No doubt it should be discussed in details, but does that justify/qualify-for submitting an alternative design. I agree that the latest posted design doc is very close to the ones posted by Suresh and Nicholas, but that was exactly the point - I was proposing a merged design that incorporated parts of both. I would've just posted an updated doc which modifies the previously posted proposal, as we often do with .patch files containing code, but that's difficult to do when only the .pdf is uploaded, hence why I posted an entirely different document. You don't want to start Apache Hadoop design-document-wars here, do you? Certainly not. I'm trying to collaborate on the design. The latest one I posted is an attempt at that.
          Hide
          Konstantin Shvachko added a comment -

          Sure date+timestamp sounds good as the default snapshot name.

          > rm -r <path> - as already stated in the document, if there are snapshots under that path, then deletion is not allowed, until all the snapshots are deleted.

          The document says you cannot delete the directory, which is the root of a snapshot. You should be able to remove -r a subdirectory if it is not a root of another snapshot, no?

          > ls -r <path>/.snapshot - I was thinking this would be same as ls <path>/.snapshot.

          Yeh, ls -r can be expensive on HDFS directories in general. I would just support it rather than deciding for users.

          > rm <path>/.snapshot or rm -r <path>/.snapshot - removes all the snapshots.
          > rm <path>/.snapshot/snapname or rm -r <path>/.snapshot/snapname - deletes the snapshot

          You know, I just remembered a 1 PB disaster that was a result of a bug in a script.
          I am afraid of a script that does ls -r, and then applies rm or rm-r to each entry.
          Let's make both rm and rm -r fail on anything that includes .snapshot. This ensures that snapshots are read-only.
          And let snapshots be deleted with a new option rm -rs.
          The .snapshot directory can disappear automatically when all snapshots are gone.

          What about ls -rs? How do you discover directories that are snapshotable?
          Should we introduce -rs option to list all directories that are snapshottable under the path.

          Show
          Konstantin Shvachko added a comment - Sure date+timestamp sounds good as the default snapshot name. > rm -r <path> - as already stated in the document, if there are snapshots under that path, then deletion is not allowed, until all the snapshots are deleted. The document says you cannot delete the directory, which is the root of a snapshot. You should be able to remove -r a subdirectory if it is not a root of another snapshot, no? > ls -r <path>/.snapshot - I was thinking this would be same as ls <path>/.snapshot. Yeh, ls -r can be expensive on HDFS directories in general. I would just support it rather than deciding for users. > rm <path>/.snapshot or rm -r <path>/.snapshot - removes all the snapshots. > rm <path>/.snapshot/snapname or rm -r <path>/.snapshot/snapname - deletes the snapshot You know, I just remembered a 1 PB disaster that was a result of a bug in a script. I am afraid of a script that does ls -r, and then applies rm or rm-r to each entry. Let's make both rm and rm -r fail on anything that includes .snapshot. This ensures that snapshots are read-only. And let snapshots be deleted with a new option rm -rs. The .snapshot directory can disappear automatically when all snapshots are gone. What about ls -rs? How do you discover directories that are snapshotable? Should we introduce -rs option to list all directories that are snapshottable under the path.
          Hide
          Daryn Sharp added a comment -

          I really need to catch up on this jira but the shell discussion caught my eye. W/o knowing all the details, I'm a bit uneasy about enforcing a policy on shapshots at the client level unless it's just a convenience. I'll read the doc this weekend to understand why standard permissions are insufficient.

          Anyway, I eventually plan to allow option bundling ala the unix shell, so choosing "s" or any existing ls option is ill advised. Maybe --snapshot to help ensure we don't collide with real ls flags.

          Show
          Daryn Sharp added a comment - I really need to catch up on this jira but the shell discussion caught my eye. W/o knowing all the details, I'm a bit uneasy about enforcing a policy on shapshots at the client level unless it's just a convenience. I'll read the doc this weekend to understand why standard permissions are insufficient. Anyway, I eventually plan to allow option bundling ala the unix shell, so choosing "s" or any existing ls option is ill advised. Maybe --snapshot to help ensure we don't collide with real ls flags.
          Hide
          Konstantin Shvachko added a comment -

          Agreed "s" is taken. Let's pick another name.
          My point is that rm should fail on snapshots because they are read-only. But they should be deletable with a different rm option.

          Show
          Konstantin Shvachko added a comment - Agreed "s" is taken. Let's pick another name. My point is that rm should fail on snapshots because they are read-only. But they should be deletable with a different rm option.
          Hide
          Konstantin Shvachko added a comment - - edited

          Thanks for posting meeting notes. Can anybody please elaborate on these topics.

          • ** Current design solves single-client consistency but not multiple-client consistency.
          • *** Would admins like to restrict based on users as well as directories for snapshots?
          • ** Current design adds extra overhead for every inode,
          Show
          Konstantin Shvachko added a comment - - edited Thanks for posting meeting notes. Can anybody please elaborate on these topics. ** Current design solves single-client consistency but not multiple-client consistency. *** Would admins like to restrict based on users as well as directories for snapshots? ** Current design adds extra overhead for every inode,
          Hide
          Aaron T. Myers added a comment -

          ** Current design solves single-client consistency but not multiple-client consistency.

          This is with respect to multiple distinct DFSClients which are simultaneously writing to open files at the time of snapshot creation. All of the designs discussed so far discuss single client ordering of namespace operations/data writes, but not multiple clients. The implication is that multi client consistency would likely require such far-reaching changes as to be unrealistic.

          *** Would admins like to restrict based on users as well as directories for snapshots?

          We discussed the possibility of adding a configurable white list of users who were allowed to create snapshots, and/or an ACL per snapshottable directory of users allowed to create snapshots of that directory.

          ** Current design adds extra overhead for every inode,

          The design for storing snapshots in the NN in the proposal I posted would add integer start_snap and end_snap fields to every INode in the FS in order to track which snapshots a given INode should be included in. This is probably the biggest difference between the design proposed by Suresh/Nicholas and the design I've proposed. That proposal adds a "diff" per INodeDirectory which has been snapshotted to keep track of changes made to that directory across different snapshots. This has the advantage of saving NN memory space for files/directories which have never had snapshots created of them.

          Show
          Aaron T. Myers added a comment - ** Current design solves single-client consistency but not multiple-client consistency. This is with respect to multiple distinct DFSClients which are simultaneously writing to open files at the time of snapshot creation. All of the designs discussed so far discuss single client ordering of namespace operations/data writes, but not multiple clients. The implication is that multi client consistency would likely require such far-reaching changes as to be unrealistic. *** Would admins like to restrict based on users as well as directories for snapshots? We discussed the possibility of adding a configurable white list of users who were allowed to create snapshots, and/or an ACL per snapshottable directory of users allowed to create snapshots of that directory. ** Current design adds extra overhead for every inode, The design for storing snapshots in the NN in the proposal I posted would add integer start_snap and end_snap fields to every INode in the FS in order to track which snapshots a given INode should be included in. This is probably the biggest difference between the design proposed by Suresh/Nicholas and the design I've proposed. That proposal adds a "diff" per INodeDirectory which has been snapshotted to keep track of changes made to that directory across different snapshots. This has the advantage of saving NN memory space for files/directories which have never had snapshots created of them.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          > You don't want to start Apache Hadoop design-document-wars here, do you?

          Certainly not. I'm trying to collaborate on the design. The latest one I posted is an attempt at that.

          Aaron, I guess that it probably is not you intention. However, it looks that you try to post some documents here by coping our design and then claim that you are one of the authors of the design.

          Show
          Tsz Wo Nicholas Sze added a comment - > You don't want to start Apache Hadoop design-document-wars here, do you? Certainly not. I'm trying to collaborate on the design. The latest one I posted is an attempt at that. Aaron, I guess that it probably is not you intention. However, it looks that you try to post some documents here by coping our design and then claim that you are one of the authors of the design.
          Hide
          Aaron T. Myers added a comment -

          However, it looks that you try to post some documents here by coping our design and then claim that you are one of the authors of the design.

          This certainly was not my intention. I specifically said I was attempting to merge the designs and deliberately removed the author tags from the document so that it would not appear that I was attempting to claim authorship of it.

          Show
          Aaron T. Myers added a comment - However, it looks that you try to post some documents here by coping our design and then claim that you are one of the authors of the design. This certainly was not my intention. I specifically said I was attempting to merge the designs and deliberately removed the author tags from the document so that it would not appear that I was attempting to claim authorship of it.
          Hide
          Jagane Sundar added a comment -

          First off, I think this is a critical feature for HDFS, and I would like to thank all of you folks working on this. In order to make snapshots usable to end users, there is more work to be done in the application software above HDFS and in the management software. So, getting this feature into HDFS expeditiously is very important.

          Having read the design by Suresh, Nicholas, Sanjay et. al, and the (counter?) proposal by Aaron, Colin, Todd and Eli - it occurs to me that both designs share a great deal, and that there are really clever ideas in both proposals. Acutely aware of the political sensitivity of this JIRA, I want to say only one thing, hopefully, without offending any of the parties: Considering the amount of work the layers above have to do, it may be in the best interest of the Hadoop community to get this feature into HDFS quickly. Given that Suresh, Nicholas et. al. seem to have working code, and seem much further ahead, it would serve the HDFS community well if all of us could put our weight behind that proposal.

          And now, on to my real feedback:
          1. Having the ability to snapshot sub-directories of the file system, and to have different snapshot policies for each 'snapshottable' directory is absolutely essential.
          2. In your design document, you state that the administrator needs to configure a directory as being snapshottable. It would be useful to allow users to be able to set directories in their own home dir as 'snapshottable'. For example, the administrator provisions a new user xyzzy and his home directory /user/xyzzy. xyzzy may then want to create a dir /user/xyzzy/hbase and want to set it as snapshottable. xyzzy should not have to call the administrator to make his /home/xyzzy/hbase directory snapshottable. I believe that this was discussed in the meetup of last week in terms of an ACL. I think this is important.

          Finally, a question:
          I still don't understand the 'single client consistency' versus 'multiple client consistency' issues discussed above. Regardless of how many HDFS client connections are active on the namenode, a snapshot will capture the HDFS state at that PIT, right?

          Show
          Jagane Sundar added a comment - First off, I think this is a critical feature for HDFS, and I would like to thank all of you folks working on this. In order to make snapshots usable to end users, there is more work to be done in the application software above HDFS and in the management software. So, getting this feature into HDFS expeditiously is very important. Having read the design by Suresh, Nicholas, Sanjay et. al, and the (counter?) proposal by Aaron, Colin, Todd and Eli - it occurs to me that both designs share a great deal, and that there are really clever ideas in both proposals. Acutely aware of the political sensitivity of this JIRA, I want to say only one thing, hopefully, without offending any of the parties: Considering the amount of work the layers above have to do, it may be in the best interest of the Hadoop community to get this feature into HDFS quickly. Given that Suresh, Nicholas et. al. seem to have working code, and seem much further ahead, it would serve the HDFS community well if all of us could put our weight behind that proposal. And now, on to my real feedback: 1. Having the ability to snapshot sub-directories of the file system, and to have different snapshot policies for each 'snapshottable' directory is absolutely essential. 2. In your design document, you state that the administrator needs to configure a directory as being snapshottable. It would be useful to allow users to be able to set directories in their own home dir as 'snapshottable'. For example, the administrator provisions a new user xyzzy and his home directory /user/xyzzy. xyzzy may then want to create a dir /user/xyzzy/hbase and want to set it as snapshottable. xyzzy should not have to call the administrator to make his /home/xyzzy/hbase directory snapshottable. I believe that this was discussed in the meetup of last week in terms of an ACL. I think this is important. Finally, a question: I still don't understand the 'single client consistency' versus 'multiple client consistency' issues discussed above. Regardless of how many HDFS client connections are active on the namenode, a snapshot will capture the HDFS state at that PIT, right?
          Hide
          Eli Collins added a comment -

          Hey Jagane,
          With regard to your first comment, the goal is definitely not to have dueling proposals, the doc ATM posted only attempts to flesh out things in the first proposal that were not covered or left as future work. Namely, (a) snapshot creation time and memory usage should be O(1), and (b) consistency semantics sufficient to implement HBase snapshots. We think these are fundamental requirements that should be addressed in the initial design vs fixed up later; which is why we started fleshing out a design that satisfies them, not to push on a separate proposal. If Suresh and Nicholas now agree, which I think they do, correct me if I'm wrong, that we should come up with a design that handles these requirements first, rather than go with the original approach and modify it later then I think we should merge these two design documents, which is what ATM was attempting to do.

          Show
          Eli Collins added a comment - Hey Jagane, With regard to your first comment, the goal is definitely not to have dueling proposals, the doc ATM posted only attempts to flesh out things in the first proposal that were not covered or left as future work. Namely, (a) snapshot creation time and memory usage should be O(1), and (b) consistency semantics sufficient to implement HBase snapshots. We think these are fundamental requirements that should be addressed in the initial design vs fixed up later; which is why we started fleshing out a design that satisfies them, not to push on a separate proposal. If Suresh and Nicholas now agree, which I think they do, correct me if I'm wrong, that we should come up with a design that handles these requirements first, rather than go with the original approach and modify it later then I think we should merge these two design documents, which is what ATM was attempting to do.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          > ... (a) snapshot creation time and memory usage should be O(1), ...

          Eli, the memory usage in ATM's proposal is not O(1) since it adds tags for every INode, therefore, it is O(N), where N is the number of INodes. It is much worse that the additional memory usage is required even if the snapshot feature is not used. You are right that ATM's design attempt to solve such problems but I believe the attempt fails.

          Show
          Tsz Wo Nicholas Sze added a comment - > ... (a) snapshot creation time and memory usage should be O(1), ... Eli, the memory usage in ATM's proposal is not O(1) since it adds tags for every INode, therefore, it is O(N), where N is the number of INodes. It is much worse that the additional memory usage is required even if the snapshot feature is not used. You are right that ATM's design attempt to solve such problems but I believe the attempt fails.
          Hide
          Suresh Srinivas added a comment -

          It would be useful to allow users to be able to set directories in their own home dir as 'snapshottable'.

          Jagane. This is really a good idea. This makes the administration simpler. Will add it to the next iteration of design doc.

          We think these are fundamental requirements that should be addressed in the initial design vs fixed up later;

          Eli, please read the comments. While this is how the discussions started, I disagree in this case we could not have incrementally improved it. Anyway discussing this is a moot point now, because we reorganized our work and have addressed the issues brought up and the efficient implementation is now almost done.

          If Suresh and Nicholas now agree, which I think they do, correct me if I'm wrong, that we should come up with a design that handles these requirements first, rather than go with the original approach and modify it later then I think we should merge these two design documents, which is what ATM was attempting to do.

          I am not sure if you are on top of all the discussions and design udpates. The design has already addressed the concerns here. Please read why this design is superior as well here

          Show
          Suresh Srinivas added a comment - It would be useful to allow users to be able to set directories in their own home dir as 'snapshottable'. Jagane. This is really a good idea. This makes the administration simpler. Will add it to the next iteration of design doc. We think these are fundamental requirements that should be addressed in the initial design vs fixed up later; Eli, please read the comments. While this is how the discussions started, I disagree in this case we could not have incrementally improved it. Anyway discussing this is a moot point now, because we reorganized our work and have addressed the issues brought up and the efficient implementation is now almost done. If Suresh and Nicholas now agree, which I think they do, correct me if I'm wrong, that we should come up with a design that handles these requirements first, rather than go with the original approach and modify it later then I think we should merge these two design documents, which is what ATM was attempting to do. I am not sure if you are on top of all the discussions and design udpates. The design has already addressed the concerns here. Please read why this design is superior as well here
          Hide
          Aaron T. Myers added a comment -

          Eli, the memory usage in ATM's proposal is not O(1) since it adds tags for every INode, therefore, it is O(N), where N is the number of INodes...

          I believe the distinction here is how much memory is used at snapshot creation time. Both of the current proposals are constant in terms of time/space at snapshot creation time, but you're right that the most recent proposal I posted would add some overhead per INode.

          In any case, I agree that the solution described in the most recent document posted by Suresh will be more space efficient than the latest proposal I posted. No disagreement there. This is what I was referring to when I said "This [the design posted by Suresh] has the advantage of saving NN memory space for files/directories which have never had snapshots created of them."

          Show
          Aaron T. Myers added a comment - Eli, the memory usage in ATM's proposal is not O(1) since it adds tags for every INode, therefore, it is O(N), where N is the number of INodes... I believe the distinction here is how much memory is used at snapshot creation time . Both of the current proposals are constant in terms of time/space at snapshot creation time, but you're right that the most recent proposal I posted would add some overhead per INode. In any case, I agree that the solution described in the most recent document posted by Suresh will be more space efficient than the latest proposal I posted. No disagreement there. This is what I was referring to when I said "This [the design posted by Suresh] has the advantage of saving NN memory space for files/directories which have never had snapshots created of them."
          Hide
          Aaron T. Myers added a comment -

          It would be useful to allow users to be able to set directories in their own home dir as 'snapshottable'.

          Not suggesting this is a bad idea, but if we do decide to go with this then I think we should relax the proposed restriction that renames of files/directories must only be within the subtree of a snapshottable root, as described in part 1a of "Operations Supported on Snapshots" section of the most recent design document posted by Suresh. I think users might be surprised to find that because they've marked two directories in their home dir as snapshottable that they cannot then rename files/dirs between those directories. (I realize the document already says "this can be revisited" - I'm just pointing out that this might be a good reason to revisit it.)

          Show
          Aaron T. Myers added a comment - It would be useful to allow users to be able to set directories in their own home dir as 'snapshottable'. Not suggesting this is a bad idea, but if we do decide to go with this then I think we should relax the proposed restriction that renames of files/directories must only be within the subtree of a snapshottable root, as described in part 1a of "Operations Supported on Snapshots" section of the most recent design document posted by Suresh. I think users might be surprised to find that because they've marked two directories in their home dir as snapshottable that they cannot then rename files/dirs between those directories. (I realize the document already says "this can be revisited" - I'm just pointing out that this might be a good reason to revisit it.)
          Hide
          Jagane Sundar added a comment -

          Adding the capability for a user to set his/her own directory as 'snapshottable' is a good enhancement for the future. It should not hold up progress right now.

          Show
          Jagane Sundar added a comment - Adding the capability for a user to set his/her own directory as 'snapshottable' is a good enhancement for the future. It should not hold up progress right now.
          Hide
          Konstantin Shvachko added a comment -

          We still need to solve 2 API problems, mentioned in my previous comment.

          1. rm (with or without -r) should be banned on snapshots, which will make them read-only.
            We should use a different option (rm -snapshot ?) that will actually delete the snapshot.
          2. We should have a way to discover snapshots in the namespace tree. Using ls -snapshot may be the answer.
          Show
          Konstantin Shvachko added a comment - We still need to solve 2 API problems, mentioned in my previous comment . rm (with or without -r) should be banned on snapshots, which will make them read-only. We should use a different option (rm -snapshot ?) that will actually delete the snapshot. We should have a way to discover snapshots in the namespace tree. Using ls -snapshot may be the answer.
          Hide
          Aaron T. Myers added a comment -

          Just FYI - I've set up a nighly Jenkins build of the HDFS-2802 branch here: https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-Hdfs-Snapshots-Branch-build/

          Currently I'm the only one who gets emailed about it. If you're a committer or the like you can just add your email to that job configuration if you want to get emails with the restuls. If you're not, let me or someone else with access know and we can add your email to the list of who gets emailed by that job. Last night a bunch of tests failed because of HDFS-4158, but if someone can review that soon I'll check it in tonight and hopefully get a better test run.

          Show
          Aaron T. Myers added a comment - Just FYI - I've set up a nighly Jenkins build of the HDFS-2802 branch here: https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-Hdfs-Snapshots-Branch-build/ Currently I'm the only one who gets emailed about it. If you're a committer or the like you can just add your email to that job configuration if you want to get emails with the restuls. If you're not, let me or someone else with access know and we can add your email to the list of who gets emailed by that job. Last night a bunch of tests failed because of HDFS-4158 , but if someone can review that soon I'll check it in tonight and hopefully get a better test run.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Hi Aaron, thanks for setting up the Jenkins build although it is a little bit too early. I have added Suresh, Jing and myself to the Jenkins email list.

          Show
          Tsz Wo Nicholas Sze added a comment - Hi Aaron, thanks for setting up the Jenkins build although it is a little bit too early. I have added Suresh, Jing and myself to the Jenkins email list.
          Hide
          Suresh Srinivas added a comment -

          Slides from yesterday's meetup.

          Show
          Suresh Srinivas added a comment - Slides from yesterday's meetup.
          Hide
          Sanjay Radia added a comment -

          Minutes of Meetup on Snapshots for HDFS
          Date: 8 Nov 2012, Location: Hortonworks Offices
          Attendance: approx 25 to 30 in person and about 7 calls into webex.
          Format: Suresh presented slides (attached in this Jira HDFS-2802) and a discussion followed along with the slides.

          Main areas of discussions

          1. CLI - most felt that we shouldn't use the fs command, introduce a new command instead.
          2. Scheduling snapshots - most folks felt it was best to leave it outside
          3. File attributes (permissions, replication factor etc)
            • Most folks felt that we should preserve the original attributes in the snapshots, however no access time to be recorded for snapshots.
            • # replicas is max of current and the snapshots
          4. Quotas
            • Everyone agreed that snapshot metadata should count into the regular snapshot quotas.
            • Most felt that max snapshot count per snapshottable directory is not needed. A system level snapshot count however may be necessary as system limit.
          5. Length of file open-for-writing: most agreed with the proposal to update length at hearbeats and for applications that care about a specific length to do a SuperFlush/SuperSync.
          6. HBase and snapshots: HBase snapshots will be able to use the proprosed solution.
          7. Exclude list - most (everyone?) disliked the idea of a "Snapshot exclude directory list"
          8. Restores
            An interesting messy example was raised: A subtree was renamed to outside of snapshoted subtree and then that snapshot is restored. What happens to the subtree that is outside - does it get deleted?
            Again this would be discussed further in Jiras
          9. Should we introduce the notion of volumes? This would incorporate
            • a) snapshot able-dir,
            • b) where quotas are specified,
            • c) not allow renames across volumes
              It was noted that c) would break apps.
              Allen W and Sanjay were in favor, Most were not in favor of the rename restriction. This topic to be discussed further in a Jira.
          Show
          Sanjay Radia added a comment - Minutes of Meetup on Snapshots for HDFS Date: 8 Nov 2012, Location: Hortonworks Offices Attendance: approx 25 to 30 in person and about 7 calls into webex. Format: Suresh presented slides (attached in this Jira HDFS-2802 ) and a discussion followed along with the slides. Main areas of discussions CLI - most felt that we shouldn't use the fs command, introduce a new command instead. Scheduling snapshots - most folks felt it was best to leave it outside File attributes (permissions, replication factor etc) Most folks felt that we should preserve the original attributes in the snapshots, however no access time to be recorded for snapshots. # replicas is max of current and the snapshots Quotas Everyone agreed that snapshot metadata should count into the regular snapshot quotas. Most felt that max snapshot count per snapshottable directory is not needed. A system level snapshot count however may be necessary as system limit. Length of file open-for-writing: most agreed with the proposal to update length at hearbeats and for applications that care about a specific length to do a SuperFlush/SuperSync. HBase and snapshots: HBase snapshots will be able to use the proprosed solution. Exclude list - most (everyone?) disliked the idea of a "Snapshot exclude directory list" Restores An interesting messy example was raised: A subtree was renamed to outside of snapshoted subtree and then that snapshot is restored. What happens to the subtree that is outside - does it get deleted? Again this would be discussed further in Jiras Should we introduce the notion of volumes? This would incorporate a) snapshot able-dir, b) where quotas are specified, c) not allow renames across volumes It was noted that c) would break apps. Allen W and Sanjay were in favor, Most were not in favor of the rename restriction. This topic to be discussed further in a Jira.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-Snapshots-Branch-build #18 (See https://builds.apache.org/job/Hadoop-Hdfs-Snapshots-Branch-build/18/)
          HDFS-4175. Add the jira to CHANGES.HDFS-2802.txt. (Revision 1411690)

          Result = FAILURE
          suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1411690
          Files :

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-Snapshots-Branch-build #18 (See https://builds.apache.org/job/Hadoop-Hdfs-Snapshots-Branch-build/18/ ) HDFS-4175 . Add the jira to CHANGES. HDFS-2802 .txt. (Revision 1411690) Result = FAILURE suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1411690 Files : /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/CHANGES. HDFS-2802 .txt
          Hide
          Konstantin Shvachko added a comment -

          > Should we introduce the notion of volumes? This would incorporate a) snapshot able-dir

          The design allows nested snapshot-able dirs. It is not intuitively clear what a subdirectory snapshot will be mapped to if snapshots are allowed only on volumes.

          Show
          Konstantin Shvachko added a comment - > Should we introduce the notion of volumes? This would incorporate a) snapshot able-dir The design allows nested snapshot-able dirs. It is not intuitively clear what a subdirectory snapshot will be mapped to if snapshots are allowed only on volumes.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-Snapshots-Branch-build #31 (See https://builds.apache.org/job/Hadoop-Hdfs-Snapshots-Branch-build/31/)
          Merging trunk to HDFS-2802 branch. (Revision 1416603)

          Result = FAILURE
          suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1416603
          Files :

          • /hadoop/common/branches/HDFS-2802
          • /hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/main/docs
          • /hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/main/java
          • /hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/MetricsSystem.java
          • /hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsSinkAdapter.java
          • /hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsSystemImpl.java
          • /hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/test/core
          • /hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/impl/TestGangliaMetrics.java
          • /hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/impl/TestMetricsSystemImpl.java
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupNode.java
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupState.java
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFileUnderConstruction.java
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/LeaseManager.java
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeJspHelper.java
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/INodeDirectorySnapshottable.java
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/INodeDirectoryWithSnapshot.java
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/INodeFileWithLink.java
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/Snapshot.java
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/SnapshotManager.java
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/native
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/secondary
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/test/hdfs
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestLease.java
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestBackupNode.java
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsLimits.java
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestINodeFile.java
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestMetaSave.java
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSaveNamespace.java
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSnapshotPathINodes.java
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestINodeDirectoryWithSnapshot.java
          • /hadoop/common/branches/HDFS-2802/hadoop-mapreduce-project
          • /hadoop/common/branches/HDFS-2802/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/branches/HDFS-2802/hadoop-mapreduce-project/conf
          • /hadoop/common/branches/HDFS-2802/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/dao/TaskInfo.java
          • /hadoop/common/branches/HDFS-2802/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/CHANGES.txt
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/CMgrCompletedContainersEvent.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeManager.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/MockNodeStatusUpdater.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeManagerShutdown.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-Snapshots-Branch-build #31 (See https://builds.apache.org/job/Hadoop-Hdfs-Snapshots-Branch-build/31/ ) Merging trunk to HDFS-2802 branch. (Revision 1416603) Result = FAILURE suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1416603 Files : /hadoop/common/branches/ HDFS-2802 /hadoop/common/branches/ HDFS-2802 /hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/ HDFS-2802 /hadoop-common-project/hadoop-common/src/main/docs /hadoop/common/branches/ HDFS-2802 /hadoop-common-project/hadoop-common/src/main/java /hadoop/common/branches/ HDFS-2802 /hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/MetricsSystem.java /hadoop/common/branches/ HDFS-2802 /hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsSinkAdapter.java /hadoop/common/branches/ HDFS-2802 /hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsSystemImpl.java /hadoop/common/branches/ HDFS-2802 /hadoop-common-project/hadoop-common/src/test/core /hadoop/common/branches/ HDFS-2802 /hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/impl/TestGangliaMetrics.java /hadoop/common/branches/ HDFS-2802 /hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/impl/TestMetricsSystemImpl.java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/main/java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupNode.java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupState.java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFileUnderConstruction.java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/LeaseManager.java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeJspHelper.java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/INodeDirectorySnapshottable.java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/INodeDirectoryWithSnapshot.java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/INodeFileWithLink.java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/Snapshot.java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/SnapshotManager.java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/main/native /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/secondary /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/test/hdfs /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestLease.java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestBackupNode.java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsLimits.java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestINodeFile.java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestMetaSave.java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSaveNamespace.java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSnapshotPathINodes.java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestINodeDirectoryWithSnapshot.java /hadoop/common/branches/ HDFS-2802 /hadoop-mapreduce-project /hadoop/common/branches/ HDFS-2802 /hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/ HDFS-2802 /hadoop-mapreduce-project/conf /hadoop/common/branches/ HDFS-2802 /hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/dao/TaskInfo.java /hadoop/common/branches/ HDFS-2802 /hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/CHANGES.txt /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/CMgrCompletedContainersEvent.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeManager.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/MockNodeStatusUpdater.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeManagerShutdown.java
          Hide
          robert chiu added a comment -

          test

          Show
          robert chiu added a comment - test
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-Snapshots-Branch-build #102 (See https://builds.apache.org/job/Hadoop-Hdfs-Snapshots-Branch-build/102/)
          Merge trunk to HDFS-2802 branch (Revision 1446507)

          Result = FAILURE
          suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1446507
          Files :

          • /hadoop/common/branches/HDFS-2802
          • /hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/dev-support/findbugsExcludeFile.xml
          • /hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/pom.xml
          • /hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/main/docs
          • /hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/main/java
          • /hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/AbstractMapWritable.java
          • /hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/SortedMapWritable.java
          • /hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/ProtobufRpcEngine.java
          • /hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java
          • /hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/GenericOptionsParser.java
          • /hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/site/apt/CommandsManual.apt.vm
          • /hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/test/core
          • /hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/test/ddl/buffer.jr
          • /hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/test/ddl/int.jr
          • /hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/test/ddl/string.jr
          • /hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/test/ddl/test.jr
          • /hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestSortedMapWritable.java
          • /hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/record/FromCpp.java
          • /hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/record/RecordBench.java
          • /hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/record/TestBuffer.java
          • /hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/record/TestRecordIO.java
          • /hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/record/TestRecordVersioning.java
          • /hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/record/ToCpp.java
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/pom.xml
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/pom.xml
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/native
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/secondary
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/test/hdfs
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestLease.java
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameEditsConfigs.java
          • /hadoop/common/branches/HDFS-2802/hadoop-mapreduce-project
          • /hadoop/common/branches/HDFS-2802/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/branches/HDFS-2802/hadoop-mapreduce-project/conf
          • /hadoop/common/branches/HDFS-2802/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskImpl.java
          • /hadoop/common/branches/HDFS-2802/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/recover/RecoveryService.java
          • /hadoop/common/branches/HDFS-2802/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/TaskPage.java
          • /hadoop/common/branches/HDFS-2802/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRecovery.java
          • /hadoop/common/branches/HDFS-2802/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskImpl.java
          • /hadoop/common/branches/HDFS-2802/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/pom.xml
          • /hadoop/common/branches/HDFS-2802/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalClientProtocolProvider.java
          • /hadoop/common/branches/HDFS-2802/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/pom.xml
          • /hadoop/common/branches/HDFS-2802/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryParser.java
          • /hadoop/common/branches/HDFS-2802/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          • /hadoop/common/branches/HDFS-2802/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsTaskPage.java
          • /hadoop/common/branches/HDFS-2802/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsTasksBlock.java
          • /hadoop/common/branches/HDFS-2802/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsTasksPage.java
          • /hadoop/common/branches/HDFS-2802/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestLineRecordReader.java
          • /hadoop/common/branches/HDFS-2802/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/lib/TestChainMapReduce.java
          • /hadoop/common/branches/HDFS-2802/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TestClientProtocolProviderImpls.java
          • /hadoop/common/branches/HDFS-2802/hadoop-maven-plugins/src/main/java/org/apache/hadoop/maven/plugin/protoc
          • /hadoop/common/branches/HDFS-2802/hadoop-maven-plugins/src/main/java/org/apache/hadoop/maven/plugin/protoc/ProtocMojo.java
          • /hadoop/common/branches/HDFS-2802/hadoop-project/src/site/site.xml
          • /hadoop/common/branches/HDFS-2802/hadoop-tools/hadoop-streaming/src/test/java/org/apache/hadoop/typedbytes/TestIO.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/CHANGES.txt
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/pom.xml
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/pom.xml
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/ResponseInfo.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/view/InfoBlock.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/pom.xml
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/pom.xml
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/CgroupsLCEResourcesHandler.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/DefaultLCEResourcesHandler.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/UserInfo.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/CapacitySchedulerPage.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/JAXBContextResolver.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerLeafQueueInfo.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerQueueInfo.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/ResourceInfo.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/UsersInfo.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMRestart.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesCapacitySched.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/ResourceManagerRest.apt.vm
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-Snapshots-Branch-build #102 (See https://builds.apache.org/job/Hadoop-Hdfs-Snapshots-Branch-build/102/ ) Merge trunk to HDFS-2802 branch (Revision 1446507) Result = FAILURE suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1446507 Files : /hadoop/common/branches/ HDFS-2802 /hadoop/common/branches/ HDFS-2802 /hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/ HDFS-2802 /hadoop-common-project/hadoop-common/dev-support/findbugsExcludeFile.xml /hadoop/common/branches/ HDFS-2802 /hadoop-common-project/hadoop-common/pom.xml /hadoop/common/branches/ HDFS-2802 /hadoop-common-project/hadoop-common/src/main/docs /hadoop/common/branches/ HDFS-2802 /hadoop-common-project/hadoop-common/src/main/java /hadoop/common/branches/ HDFS-2802 /hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/AbstractMapWritable.java /hadoop/common/branches/ HDFS-2802 /hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/SortedMapWritable.java /hadoop/common/branches/ HDFS-2802 /hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/ProtobufRpcEngine.java /hadoop/common/branches/ HDFS-2802 /hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java /hadoop/common/branches/ HDFS-2802 /hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/GenericOptionsParser.java /hadoop/common/branches/ HDFS-2802 /hadoop-common-project/hadoop-common/src/site/apt/CommandsManual.apt.vm /hadoop/common/branches/ HDFS-2802 /hadoop-common-project/hadoop-common/src/test/core /hadoop/common/branches/ HDFS-2802 /hadoop-common-project/hadoop-common/src/test/ddl/buffer.jr /hadoop/common/branches/ HDFS-2802 /hadoop-common-project/hadoop-common/src/test/ddl/int.jr /hadoop/common/branches/ HDFS-2802 /hadoop-common-project/hadoop-common/src/test/ddl/string.jr /hadoop/common/branches/ HDFS-2802 /hadoop-common-project/hadoop-common/src/test/ddl/test.jr /hadoop/common/branches/ HDFS-2802 /hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestSortedMapWritable.java /hadoop/common/branches/ HDFS-2802 /hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/record/FromCpp.java /hadoop/common/branches/ HDFS-2802 /hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/record/RecordBench.java /hadoop/common/branches/ HDFS-2802 /hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/record/TestBuffer.java /hadoop/common/branches/ HDFS-2802 /hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/record/TestRecordIO.java /hadoop/common/branches/ HDFS-2802 /hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/record/TestRecordVersioning.java /hadoop/common/branches/ HDFS-2802 /hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/record/ToCpp.java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/pom.xml /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/pom.xml /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/main/java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/main/native /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/secondary /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/test/hdfs /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestLease.java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameEditsConfigs.java /hadoop/common/branches/ HDFS-2802 /hadoop-mapreduce-project /hadoop/common/branches/ HDFS-2802 /hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/ HDFS-2802 /hadoop-mapreduce-project/conf /hadoop/common/branches/ HDFS-2802 /hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskImpl.java /hadoop/common/branches/ HDFS-2802 /hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/recover/RecoveryService.java /hadoop/common/branches/ HDFS-2802 /hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/TaskPage.java /hadoop/common/branches/ HDFS-2802 /hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRecovery.java /hadoop/common/branches/ HDFS-2802 /hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskImpl.java /hadoop/common/branches/ HDFS-2802 /hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/pom.xml /hadoop/common/branches/ HDFS-2802 /hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalClientProtocolProvider.java /hadoop/common/branches/ HDFS-2802 /hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/pom.xml /hadoop/common/branches/ HDFS-2802 /hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryParser.java /hadoop/common/branches/ HDFS-2802 /hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml /hadoop/common/branches/ HDFS-2802 /hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsTaskPage.java /hadoop/common/branches/ HDFS-2802 /hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsTasksBlock.java /hadoop/common/branches/ HDFS-2802 /hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsTasksPage.java /hadoop/common/branches/ HDFS-2802 /hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestLineRecordReader.java /hadoop/common/branches/ HDFS-2802 /hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/lib/TestChainMapReduce.java /hadoop/common/branches/ HDFS-2802 /hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TestClientProtocolProviderImpls.java /hadoop/common/branches/ HDFS-2802 /hadoop-maven-plugins/src/main/java/org/apache/hadoop/maven/plugin/protoc /hadoop/common/branches/ HDFS-2802 /hadoop-maven-plugins/src/main/java/org/apache/hadoop/maven/plugin/protoc/ProtocMojo.java /hadoop/common/branches/ HDFS-2802 /hadoop-project/src/site/site.xml /hadoop/common/branches/ HDFS-2802 /hadoop-tools/hadoop-streaming/src/test/java/org/apache/hadoop/typedbytes/TestIO.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/CHANGES.txt /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/pom.xml /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/pom.xml /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/ResponseInfo.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/view/InfoBlock.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/pom.xml /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/pom.xml /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/CgroupsLCEResourcesHandler.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/DefaultLCEResourcesHandler.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/UserInfo.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/CapacitySchedulerPage.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/JAXBContextResolver.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerLeafQueueInfo.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerQueueInfo.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/ResourceInfo.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/UsersInfo.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMRestart.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesCapacitySched.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/ResourceManagerRest.apt.vm
          Hide
          Suresh Srinivas added a comment -

          Early version of the merge patch.

          Show
          Suresh Srinivas added a comment - Early version of the merge patch.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12572699/2802.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 24 new or modified test files.

          -1 one of tests included doesn't have a timeout.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 javadoc. The javadoc tool appears to have generated 29 warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 4 new Findbugs (version 1.3.9) warnings.

          -1 release audit. The applied patch generated 1 release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.fs.TestFilterFileSystem
          org.apache.hadoop.hdfs.server.namenode.TestCheckpoint
          org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4058//testReport/
          Release audit warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/4058//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/4058//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4058//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12572699/2802.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 24 new or modified test files. -1 one of tests included doesn't have a timeout. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 javadoc . The javadoc tool appears to have generated 29 warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 4 new Findbugs (version 1.3.9) warnings. -1 release audit . The applied patch generated 1 release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.fs.TestFilterFileSystem org.apache.hadoop.hdfs.server.namenode.TestCheckpoint org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4058//testReport/ Release audit warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/4058//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/4058//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4058//console This message is automatically generated.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12573240/2802.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 24 new or modified test files.

          +1 tests included appear to have a timeout.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 javadoc. The javadoc tool appears to have generated 21 warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 4 new Findbugs (version 1.3.9) warnings.

          -1 release audit. The applied patch generated 1 release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.fs.TestFilterFileSystem
          org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup
          org.apache.hadoop.hdfs.server.datanode.TestDataDirs
          org.apache.hadoop.hdfs.server.namenode.TestCheckpoint
          org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4083//testReport/
          Release audit warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/4083//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/4083//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4083//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12573240/2802.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 24 new or modified test files. +1 tests included appear to have a timeout. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 javadoc . The javadoc tool appears to have generated 21 warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 4 new Findbugs (version 1.3.9) warnings. -1 release audit . The applied patch generated 1 release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.fs.TestFilterFileSystem org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup org.apache.hadoop.hdfs.server.datanode.TestDataDirs org.apache.hadoop.hdfs.server.namenode.TestCheckpoint org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4083//testReport/ Release audit warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/4083//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/4083//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4083//console This message is automatically generated.
          Hide
          Eli Collins added a comment -

          Hey guys,

          Are you hoping to try to get this into branch-2 for 2.0.4 beta? The target version is currently v3. Since this is an invasive change and requires a metadata upgrade it's IMO something that would be hard to put in post v2 GA, and since we're only planning one v2 beta it seems like we'd need to target that release if we want this in branch-2.

          Thanks,
          Eli

          Show
          Eli Collins added a comment - Hey guys, Are you hoping to try to get this into branch-2 for 2.0.4 beta? The target version is currently v3. Since this is an invasive change and requires a metadata upgrade it's IMO something that would be hard to put in post v2 GA, and since we're only planning one v2 beta it seems like we'd need to target that release if we want this in branch-2. Thanks, Eli
          Hide
          Suresh Srinivas added a comment - - edited

          Eli, currently the functionality is almost complete (renames have been pain). We are doing reviews of areas that need more unit tests and doing long running tests and testing at scale on a cluster. First goal is to get this to trunk and we hope we can get this done by the end of this month. Hopefully by that time we are done with our tests and can get it into 2.0.5 beta.

          Show
          Suresh Srinivas added a comment - - edited Eli, currently the functionality is almost complete (renames have been pain). We are doing reviews of areas that need more unit tests and doing long running tests and testing at scale on a cluster. First goal is to get this to trunk and we hope we can get this done by the end of this month. Hopefully by that time we are done with our tests and can get it into 2.0.5 beta.
          Hide
          Konstantin Shvachko added a comment -

          > Hopefully by that time we are done with our tests and can get it into 2.0.5 beta.

          I think this is not right. Because it will destabilize branch 2. Dot releases are for bug fixes, not new features.

          Show
          Konstantin Shvachko added a comment - > Hopefully by that time we are done with our tests and can get it into 2.0.5 beta. I think this is not right. Because it will destabilize branch 2. Dot releases are for bug fixes, not new features.
          Hide
          Suresh Srinivas added a comment - - edited

          I think this is not right. Because it will destabilize branch 2. Dot releases are for bug fixes, not new features.

          Other dot versions did add features. That is how we have significant functionality such quorum journal manager, Automatic failover and many other features. I would like to make snapshots feature available as soon as possible and stabilize a release.

          Show
          Suresh Srinivas added a comment - - edited I think this is not right. Because it will destabilize branch 2. Dot releases are for bug fixes, not new features. Other dot versions did add features. That is how we have significant functionality such quorum journal manager, Automatic failover and many other features. I would like to make snapshots feature available as soon as possible and stabilize a release.
          Hide
          Suresh Srinivas added a comment -

          Latest diff patch attached.

          Show
          Suresh Srinivas added a comment - Latest diff patch attached.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12575235/2802.diff
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 25 new or modified test files.

          -1 one of tests included doesn't have a timeout.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 javadoc. The javadoc tool appears to have generated 21 warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 4 new Findbugs (version 1.3.9) warnings.

          -1 release audit. The applied patch generated 1 release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.fs.TestFilterFileSystem
          org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4136//testReport/
          Release audit warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/4136//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/4136//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4136//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12575235/2802.diff against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 25 new or modified test files. -1 one of tests included doesn't have a timeout. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 javadoc . The javadoc tool appears to have generated 21 warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 4 new Findbugs (version 1.3.9) warnings. -1 release audit . The applied patch generated 1 release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.fs.TestFilterFileSystem org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4136//testReport/ Release audit warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/4136//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/4136//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4136//console This message is automatically generated.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-Snapshots-Branch-build #140 (See https://builds.apache.org/job/Hadoop-Hdfs-Snapshots-Branch-build/140/)
          Merging trunk to branch HDFS-2802 (Revision 1460410)

          Result = FAILURE
          suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1460410
          Files :

          • /hadoop/common/branches/HDFS-2802
          • /hadoop/common/branches/HDFS-2802/BUILDING.txt
          • /hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/KerberosName.java
          • /hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/pom.xml
          • /hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/main/docs
          • /hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/main/java
          • /hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/DF.java
          • /hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java
          • /hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/ProtobufRpcEngine.java
          • /hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java
          • /hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/CachedDNSToSwitchMapping.java
          • /hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/DNSToSwitchMapping.java
          • /hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetworkTopology.java
          • /hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NodeBase.java
          • /hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/ScriptBasedMapping.java
          • /hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/TableMapping.java
          • /hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/HadoopKerberosName.java
          • /hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java
          • /hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/test/core
          • /hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestDFVariations.java
          • /hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/StaticMapping.java
          • /hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/TestSwitchMapping.java
          • /hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/TestTableMapping.java
          • /hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestSecurityUtil.java
          • /hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestUserGroupInformation.java
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/native
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/secondary
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/test/hdfs
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientExcludedNodes.java
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/TestNNWithQJM.java
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestAuditLogs.java
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestGetConf.java
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/net/TestNetworkTopology.java
          • /hadoop/common/branches/HDFS-2802/hadoop-mapreduce-project
          • /hadoop/common/branches/HDFS-2802/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/branches/HDFS-2802/hadoop-mapreduce-project/conf
          • /hadoop/common/branches/HDFS-2802/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/launcher/ContainerLauncherImpl.java
          • /hadoop/common/branches/HDFS-2802/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/local/LocalContainerAllocator.java
          • /hadoop/common/branches/HDFS-2802/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java
          • /hadoop/common/branches/HDFS-2802/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerRequestor.java
          • /hadoop/common/branches/HDFS-2802/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestEvents.java
          • /hadoop/common/branches/HDFS-2802/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestJobHistoryEventHandler.java
          • /hadoop/common/branches/HDFS-2802/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRAppBenchmark.java
          • /hadoop/common/branches/HDFS-2802/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRAppMaster.java
          • /hadoop/common/branches/HDFS-2802/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java
          • /hadoop/common/branches/HDFS-2802/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          • /hadoop/common/branches/HDFS-2802/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestFetcher.java
          • /hadoop/common/branches/HDFS-2802/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestJobHistoryEntities.java
          • /hadoop/common/branches/HDFS-2802/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestJobHistoryEvents.java
          • /hadoop/common/branches/HDFS-2802/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestJobHistoryParsing.java
          • /hadoop/common/branches/HDFS-2802/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientClusterFactory.java
          • /hadoop/common/branches/HDFS-2802/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRCluster.java
          • /hadoop/common/branches/HDFS-2802/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java
          • /hadoop/common/branches/HDFS-2802/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/CopyListing.java
          • /hadoop/common/branches/HDFS-2802/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java
          • /hadoop/common/branches/HDFS-2802/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpConstants.java
          • /hadoop/common/branches/HDFS-2802/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/SimpleCopyListing.java
          • /hadoop/common/branches/HDFS-2802/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/RetriableFileCopyCommand.java
          • /hadoop/common/branches/HDFS-2802/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/util/ThrottledInputStream.java
          • /hadoop/common/branches/HDFS-2802/hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestCopyListing.java
          • /hadoop/common/branches/HDFS-2802/hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestIntegration.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/CHANGES.txt
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/AllocateResponse.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/AllocateResponsePBImpl.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetAllApplicationsResponsePBImpl.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetClusterNodesResponsePBImpl.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetQueueUserAclsInfoResponsePBImpl.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/AMResponse.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationAttemptId.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationId.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ContainerId.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/NodeId.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/Priority.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ResourceRequest.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/AMResponsePBImpl.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_service_protos.proto
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/Client.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop/yarn/applications/distributedshell/TestDistributedShell.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-unmanaged-am-launcher/src/test/java/org/apache/hadoop/yarn/applications/unmanagedamlauncher/TestUnmanagedAMLauncher.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/AMRMClientAsync.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/AMRMClientImpl.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/ApplicationCLI.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestAMRMClient.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestAMRMClientAsync.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestYarnCLI.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/AggregatedLogFormat.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/LogDumper.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/BuilderUtils.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/TestRecordFactory.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestProcfsBasedProcessTree.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestRackResolver.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ResourceView.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/NodePage.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/dao/NodeInfo.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/TestLogAggregationService.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/webapp/TestContainerLogsPage.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/webapp/TestNMWebServer.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/webapp/TestNMWebServices.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/webapp/TestNMWebServicesApps.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/webapp/TestNMWebServicesContainers.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockAM.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestFifoScheduler.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMRestart.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestAMRMRPCNodeUpdates.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestAMRMRPCResponseId.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/security/TestApplicationTokens.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/TestContainerManagerSecurity.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/WritingYarnApplications.apt.vm
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-Snapshots-Branch-build #140 (See https://builds.apache.org/job/Hadoop-Hdfs-Snapshots-Branch-build/140/ ) Merging trunk to branch HDFS-2802 (Revision 1460410) Result = FAILURE suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1460410 Files : /hadoop/common/branches/ HDFS-2802 /hadoop/common/branches/ HDFS-2802 /BUILDING.txt /hadoop/common/branches/ HDFS-2802 /hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/KerberosName.java /hadoop/common/branches/ HDFS-2802 /hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/ HDFS-2802 /hadoop-common-project/hadoop-common/pom.xml /hadoop/common/branches/ HDFS-2802 /hadoop-common-project/hadoop-common/src/main/docs /hadoop/common/branches/ HDFS-2802 /hadoop-common-project/hadoop-common/src/main/java /hadoop/common/branches/ HDFS-2802 /hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/DF.java /hadoop/common/branches/ HDFS-2802 /hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java /hadoop/common/branches/ HDFS-2802 /hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/ProtobufRpcEngine.java /hadoop/common/branches/ HDFS-2802 /hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java /hadoop/common/branches/ HDFS-2802 /hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/CachedDNSToSwitchMapping.java /hadoop/common/branches/ HDFS-2802 /hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/DNSToSwitchMapping.java /hadoop/common/branches/ HDFS-2802 /hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetworkTopology.java /hadoop/common/branches/ HDFS-2802 /hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NodeBase.java /hadoop/common/branches/ HDFS-2802 /hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/ScriptBasedMapping.java /hadoop/common/branches/ HDFS-2802 /hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/TableMapping.java /hadoop/common/branches/ HDFS-2802 /hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/HadoopKerberosName.java /hadoop/common/branches/ HDFS-2802 /hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java /hadoop/common/branches/ HDFS-2802 /hadoop-common-project/hadoop-common/src/test/core /hadoop/common/branches/ HDFS-2802 /hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestDFVariations.java /hadoop/common/branches/ HDFS-2802 /hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/StaticMapping.java /hadoop/common/branches/ HDFS-2802 /hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/TestSwitchMapping.java /hadoop/common/branches/ HDFS-2802 /hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/TestTableMapping.java /hadoop/common/branches/ HDFS-2802 /hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestSecurityUtil.java /hadoop/common/branches/ HDFS-2802 /hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestUserGroupInformation.java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/main/java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/main/native /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/secondary /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/test/hdfs /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientExcludedNodes.java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/TestNNWithQJM.java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestAuditLogs.java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestGetConf.java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/net/TestNetworkTopology.java /hadoop/common/branches/ HDFS-2802 /hadoop-mapreduce-project /hadoop/common/branches/ HDFS-2802 /hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/ HDFS-2802 /hadoop-mapreduce-project/conf /hadoop/common/branches/ HDFS-2802 /hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/launcher/ContainerLauncherImpl.java /hadoop/common/branches/ HDFS-2802 /hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/local/LocalContainerAllocator.java /hadoop/common/branches/ HDFS-2802 /hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java /hadoop/common/branches/ HDFS-2802 /hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerRequestor.java /hadoop/common/branches/ HDFS-2802 /hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestEvents.java /hadoop/common/branches/ HDFS-2802 /hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestJobHistoryEventHandler.java /hadoop/common/branches/ HDFS-2802 /hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRAppBenchmark.java /hadoop/common/branches/ HDFS-2802 /hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRAppMaster.java /hadoop/common/branches/ HDFS-2802 /hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java /hadoop/common/branches/ HDFS-2802 /hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml /hadoop/common/branches/ HDFS-2802 /hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestFetcher.java /hadoop/common/branches/ HDFS-2802 /hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestJobHistoryEntities.java /hadoop/common/branches/ HDFS-2802 /hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestJobHistoryEvents.java /hadoop/common/branches/ HDFS-2802 /hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestJobHistoryParsing.java /hadoop/common/branches/ HDFS-2802 /hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientClusterFactory.java /hadoop/common/branches/ HDFS-2802 /hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRCluster.java /hadoop/common/branches/ HDFS-2802 /hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java /hadoop/common/branches/ HDFS-2802 /hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/CopyListing.java /hadoop/common/branches/ HDFS-2802 /hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java /hadoop/common/branches/ HDFS-2802 /hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpConstants.java /hadoop/common/branches/ HDFS-2802 /hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/SimpleCopyListing.java /hadoop/common/branches/ HDFS-2802 /hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/RetriableFileCopyCommand.java /hadoop/common/branches/ HDFS-2802 /hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/util/ThrottledInputStream.java /hadoop/common/branches/ HDFS-2802 /hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestCopyListing.java /hadoop/common/branches/ HDFS-2802 /hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestIntegration.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/CHANGES.txt /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/AllocateResponse.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/AllocateResponsePBImpl.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetAllApplicationsResponsePBImpl.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetClusterNodesResponsePBImpl.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetQueueUserAclsInfoResponsePBImpl.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/AMResponse.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationAttemptId.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationId.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ContainerId.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/NodeId.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/Priority.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ResourceRequest.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/AMResponsePBImpl.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_service_protos.proto /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/Client.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop/yarn/applications/distributedshell/TestDistributedShell.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-unmanaged-am-launcher/src/test/java/org/apache/hadoop/yarn/applications/unmanagedamlauncher/TestUnmanagedAMLauncher.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/AMRMClientAsync.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/AMRMClientImpl.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/ApplicationCLI.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestAMRMClient.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestAMRMClientAsync.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestYarnCLI.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/AggregatedLogFormat.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/LogDumper.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/BuilderUtils.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/TestRecordFactory.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestProcfsBasedProcessTree.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestRackResolver.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ResourceView.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/NodePage.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/dao/NodeInfo.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/TestLogAggregationService.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/webapp/TestContainerLogsPage.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/webapp/TestNMWebServer.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/webapp/TestNMWebServices.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/webapp/TestNMWebServicesApps.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/webapp/TestNMWebServicesContainers.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockAM.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestFifoScheduler.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMRestart.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestAMRMRPCNodeUpdates.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestAMRMRPCResponseId.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/security/TestApplicationTokens.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/TestContainerManagerSecurity.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/WritingYarnApplications.apt.vm
          Hide
          Jing Zhao added a comment -

          Upload the current test plan.

          We followed this test plan and tested the existing snapshot functionalities (except rename and concat) in a small cluster with 5 nodes for a week. The tests we have done include 1) the basic operation test, 2) the long-running test, and 3) the stress test described in the test plan. Currently the tests run well and the snapshot functionality seems stable. Several bugs have been caught and we have filed HDFS-4616, HDFS-4627, HDFS-4636, and HDFS-4637 for fix. We will continue running the tests and fixing bugs.

          Show
          Jing Zhao added a comment - Upload the current test plan. We followed this test plan and tested the existing snapshot functionalities (except rename and concat) in a small cluster with 5 nodes for a week. The tests we have done include 1) the basic operation test, 2) the long-running test, and 3) the stress test described in the test plan. Currently the tests run well and the snapshot functionality seems stable. Several bugs have been caught and we have filed HDFS-4616 , HDFS-4627 , HDFS-4636 , and HDFS-4637 for fix. We will continue running the tests and fixing bugs.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12575442/snapshot-testplan.pdf
          against trunk revision .

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4145//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12575442/snapshot-testplan.pdf against trunk revision . -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4145//console This message is automatically generated.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          h2802_20130417.patch: from snapshot branch

          Show
          Tsz Wo Nicholas Sze added a comment - h2802_20130417.patch: from snapshot branch
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12579183/h2802_20130417.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 27 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 javadoc. The javadoc tool appears to have generated 15 warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 4 new Findbugs (version 1.3.9) warnings.

          -1 release audit. The applied patch generated 1 release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.fs.TestFilterFileSystem
          org.apache.hadoop.hdfs.server.namenode.TestNameEditsConfigs
          org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
          org.apache.hadoop.hdfs.server.namenode.TestSecondaryNameNodeUpgrade
          org.apache.hadoop.hdfs.server.namenode.TestCheckpoint

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4271//testReport/
          Release audit warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/4271//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/4271//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4271//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12579183/h2802_20130417.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 27 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 javadoc . The javadoc tool appears to have generated 15 warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 4 new Findbugs (version 1.3.9) warnings. -1 release audit . The applied patch generated 1 release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.fs.TestFilterFileSystem org.apache.hadoop.hdfs.server.namenode.TestNameEditsConfigs org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer org.apache.hadoop.hdfs.server.namenode.TestSecondaryNameNodeUpgrade org.apache.hadoop.hdfs.server.namenode.TestCheckpoint +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4271//testReport/ Release audit warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/4271//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/4271//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4271//console This message is automatically generated.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-Snapshots-Branch-build #164 (See https://builds.apache.org/job/Hadoop-Hdfs-Snapshots-Branch-build/164/)
          Merge trunk to HDFS-2802 branch. This involves fixing many conflict with HDFS-4434. (Revision 1470225)
          HDFS-4434. Reverting change r1470089 that merges trunk to HDFS-2802. (Revision 1470194)

          Result = FAILURE
          suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1470225
          Files :

          • /hadoop/common/branches/HDFS-2802
          • /hadoop/common/branches/HDFS-2802/hadoop-assemblies/src/main/resources/assemblies/hadoop-mapreduce-dist.xml
          • /hadoop/common/branches/HDFS-2802/hadoop-assemblies/src/main/resources/assemblies/hadoop-tools.xml
          • /hadoop/common/branches/HDFS-2802/hadoop-assemblies/src/main/resources/assemblies/hadoop-yarn-dist.xml
          • /hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/main/docs
          • /hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/main/java
          • /hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileUtil.java
          • /hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/Shell.java
          • /hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/main/winutils/task.c
          • /hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/test/core
          • /hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileUtil.java
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlocksMap.java
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeId.java
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/native
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/secondary
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/test/hdfs
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestINodeFile.java
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestPipelinesFailover.java
          • /hadoop/common/branches/HDFS-2802/hadoop-mapreduce-project
          • /hadoop/common/branches/HDFS-2802/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/branches/HDFS-2802/hadoop-mapreduce-project/conf
          • /hadoop/common/branches/HDFS-2802/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/pom.xml
          • /hadoop/common/branches/HDFS-2802/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/launcher/ContainerLauncherImpl.java
          • /hadoop/common/branches/HDFS-2802/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/launcher/TestContainerLauncherImpl.java
          • /hadoop/common/branches/HDFS-2802/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          • /hadoop/common/branches/HDFS-2802/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMRJobs.java
          • /hadoop/common/branches/HDFS-2802/hadoop-project/pom.xml
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/CHANGES.txt
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/AllocateRequest.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/StartContainerResponse.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/AllocateRequestPBImpl.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/StartContainerResponsePBImpl.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationSubmissionContext.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/BuilderUtils.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ContainerExecutor.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/LocalResourcesTracker.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/LocalResourcesTrackerImpl.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/LocalizedResource.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ResourceLocalizationService.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeManagerShutdown.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/BaseContainerManagerTest.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManager.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestLocalResourcesTrackerImpl.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestLocalizedResource.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestResourceLocalizationService.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMRestart.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/TestRMAppTransitions.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesApps.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/TestContainerManagerSecurity.java

          suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1470194
          Files :

          • /hadoop/common/branches/HDFS-2802
          • /hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/main/docs
          • /hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/main/java
          • /hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileUtil.java
          • /hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/Shell.java
          • /hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/main/winutils/task.c
          • /hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/test/core
          • /hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileUtil.java
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlocksMap.java
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeId.java
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/native
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/secondary
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/test/hdfs
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImageWithSnapshot.java
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestINodeFile.java
          • /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestPipelinesFailover.java
          • /hadoop/common/branches/HDFS-2802/hadoop-mapreduce-project
          • /hadoop/common/branches/HDFS-2802/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/branches/HDFS-2802/hadoop-mapreduce-project/conf
          • /hadoop/common/branches/HDFS-2802/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/pom.xml
          • /hadoop/common/branches/HDFS-2802/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/launcher/ContainerLauncherImpl.java
          • /hadoop/common/branches/HDFS-2802/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/launcher/TestContainerLauncherImpl.java
          • /hadoop/common/branches/HDFS-2802/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          • /hadoop/common/branches/HDFS-2802/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMRJobs.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/CHANGES.txt
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/AllocateRequest.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/StartContainerResponse.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/AllocateRequestPBImpl.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/StartContainerResponsePBImpl.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/BuilderUtils.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ContainerExecutor.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeManagerShutdown.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/BaseContainerManagerTest.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManager.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java
          • /hadoop/common/branches/HDFS-2802/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/TestContainerManagerSecurity.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-Snapshots-Branch-build #164 (See https://builds.apache.org/job/Hadoop-Hdfs-Snapshots-Branch-build/164/ ) Merge trunk to HDFS-2802 branch. This involves fixing many conflict with HDFS-4434 . (Revision 1470225) HDFS-4434 . Reverting change r1470089 that merges trunk to HDFS-2802 . (Revision 1470194) Result = FAILURE suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1470225 Files : /hadoop/common/branches/ HDFS-2802 /hadoop/common/branches/ HDFS-2802 /hadoop-assemblies/src/main/resources/assemblies/hadoop-mapreduce-dist.xml /hadoop/common/branches/ HDFS-2802 /hadoop-assemblies/src/main/resources/assemblies/hadoop-tools.xml /hadoop/common/branches/ HDFS-2802 /hadoop-assemblies/src/main/resources/assemblies/hadoop-yarn-dist.xml /hadoop/common/branches/ HDFS-2802 /hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/ HDFS-2802 /hadoop-common-project/hadoop-common/src/main/docs /hadoop/common/branches/ HDFS-2802 /hadoop-common-project/hadoop-common/src/main/java /hadoop/common/branches/ HDFS-2802 /hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileUtil.java /hadoop/common/branches/ HDFS-2802 /hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/Shell.java /hadoop/common/branches/ HDFS-2802 /hadoop-common-project/hadoop-common/src/main/winutils/task.c /hadoop/common/branches/ HDFS-2802 /hadoop-common-project/hadoop-common/src/test/core /hadoop/common/branches/ HDFS-2802 /hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileUtil.java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/main/java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlocksMap.java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeId.java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/main/native /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/secondary /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/test/hdfs /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestINodeFile.java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestPipelinesFailover.java /hadoop/common/branches/ HDFS-2802 /hadoop-mapreduce-project /hadoop/common/branches/ HDFS-2802 /hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/ HDFS-2802 /hadoop-mapreduce-project/conf /hadoop/common/branches/ HDFS-2802 /hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/pom.xml /hadoop/common/branches/ HDFS-2802 /hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/launcher/ContainerLauncherImpl.java /hadoop/common/branches/ HDFS-2802 /hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/launcher/TestContainerLauncherImpl.java /hadoop/common/branches/ HDFS-2802 /hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml /hadoop/common/branches/ HDFS-2802 /hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMRJobs.java /hadoop/common/branches/ HDFS-2802 /hadoop-project/pom.xml /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/CHANGES.txt /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/AllocateRequest.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/StartContainerResponse.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/AllocateRequestPBImpl.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/StartContainerResponsePBImpl.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationSubmissionContext.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/BuilderUtils.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ContainerExecutor.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/LocalResourcesTracker.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/LocalResourcesTrackerImpl.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/LocalizedResource.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ResourceLocalizationService.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeManagerShutdown.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/BaseContainerManagerTest.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManager.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestLocalResourcesTrackerImpl.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestLocalizedResource.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestResourceLocalizationService.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMRestart.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/TestRMAppTransitions.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesApps.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/TestContainerManagerSecurity.java suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1470194 Files : /hadoop/common/branches/ HDFS-2802 /hadoop/common/branches/ HDFS-2802 /hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/ HDFS-2802 /hadoop-common-project/hadoop-common/src/main/docs /hadoop/common/branches/ HDFS-2802 /hadoop-common-project/hadoop-common/src/main/java /hadoop/common/branches/ HDFS-2802 /hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileUtil.java /hadoop/common/branches/ HDFS-2802 /hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/Shell.java /hadoop/common/branches/ HDFS-2802 /hadoop-common-project/hadoop-common/src/main/winutils/task.c /hadoop/common/branches/ HDFS-2802 /hadoop-common-project/hadoop-common/src/test/core /hadoop/common/branches/ HDFS-2802 /hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileUtil.java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/main/java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlocksMap.java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeId.java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/main/native /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/secondary /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/test/hdfs /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImageWithSnapshot.java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestINodeFile.java /hadoop/common/branches/ HDFS-2802 /hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestPipelinesFailover.java /hadoop/common/branches/ HDFS-2802 /hadoop-mapreduce-project /hadoop/common/branches/ HDFS-2802 /hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/ HDFS-2802 /hadoop-mapreduce-project/conf /hadoop/common/branches/ HDFS-2802 /hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/pom.xml /hadoop/common/branches/ HDFS-2802 /hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/launcher/ContainerLauncherImpl.java /hadoop/common/branches/ HDFS-2802 /hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/launcher/TestContainerLauncherImpl.java /hadoop/common/branches/ HDFS-2802 /hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml /hadoop/common/branches/ HDFS-2802 /hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMRJobs.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/CHANGES.txt /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/AllocateRequest.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/StartContainerResponse.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/AllocateRequestPBImpl.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/StartContainerResponsePBImpl.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/BuilderUtils.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ContainerExecutor.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeManagerShutdown.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/BaseContainerManagerTest.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManager.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java /hadoop/common/branches/ HDFS-2802 /hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/TestContainerManagerSecurity.java
          Hide
          Tsz Wo Nicholas Sze added a comment -

          h2802_20130422.patch: merged with inode id map.

          Show
          Tsz Wo Nicholas Sze added a comment - h2802_20130422.patch: merged with inode id map.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12579943/h2802_20130422.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 28 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 javadoc. The javadoc tool appears to have generated 17 warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

          -1 release audit. The applied patch generated 1 release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer
          org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
          org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4288//testReport/
          Release audit warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/4288//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/4288//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4288//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12579943/h2802_20130422.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 28 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 javadoc . The javadoc tool appears to have generated 17 warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. -1 release audit . The applied patch generated 1 release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4288//testReport/ Release audit warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/4288//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/4288//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4288//console This message is automatically generated.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          h2802_20130423.patch: fixes javadoc and some other warnings.

          Show
          Tsz Wo Nicholas Sze added a comment - h2802_20130423.patch: fixes javadoc and some other warnings.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12580219/h2802_20130423.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 28 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer
          org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
          org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4304//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4304//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12580219/h2802_20130423.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 28 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4304//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4304//console This message is automatically generated.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          h2802_20130425.patch

          Show
          Tsz Wo Nicholas Sze added a comment - h2802_20130425.patch
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12580642/h2802_20130425.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 29 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.server.namenode.TestFsck
          org.apache.hadoop.hdfs.server.datanode.TestBlockRecovery
          org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
          org.apache.hadoop.hdfs.server.namenode.TestNameEditsConfigs
          org.apache.hadoop.hdfs.server.namenode.TestCheckpoint
          org.apache.hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer
          org.apache.hadoop.hdfs.server.namenode.TestSecondaryNameNodeUpgrade

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4320//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4320//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12580642/h2802_20130425.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 29 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.namenode.TestFsck org.apache.hadoop.hdfs.server.datanode.TestBlockRecovery org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer org.apache.hadoop.hdfs.server.namenode.TestNameEditsConfigs org.apache.hadoop.hdfs.server.namenode.TestCheckpoint org.apache.hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer org.apache.hadoop.hdfs.server.namenode.TestSecondaryNameNodeUpgrade +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4320//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4320//console This message is automatically generated.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          h2802_20130426.patch

          Show
          Tsz Wo Nicholas Sze added a comment - h2802_20130426.patch
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12580803/h2802_20130426.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 29 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer
          org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4331//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4331//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12580803/h2802_20130426.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 29 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4331//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4331//console This message is automatically generated.
          Hide
          Suresh Srinivas added a comment -

          Updated design document with the following changes:

          1. Added a section “Phase 1 feature set” to describe the feature set implemented in phase 1
          2. Removed configuration “dfs.namenode.home.snapshots.allowed” from design.
          3. Updated the design to indicate rename is allowed within and across snapshottable directory boundaries
          4. Added how quota applies to snapshots
          5. Added that permission and other attributes for a file/directory in snapshot cannot be changed.
          6. Updated CLI list
          7. Added information about rename design and implementation
          Show
          Suresh Srinivas added a comment - Updated design document with the following changes: Added a section “Phase 1 feature set” to describe the feature set implemented in phase 1 Removed configuration “dfs.namenode.home.snapshots.allowed” from design. Updated the design to indicate rename is allowed within and across snapshottable directory boundaries Added how quota applies to snapshots Added that permission and other attributes for a file/directory in snapshot cannot be changed. Updated CLI list Added information about rename design and implementation
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12581255/Snapshots20120429.pdf
          against trunk revision .

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4346//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12581255/Snapshots20120429.pdf against trunk revision . -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4346//console This message is automatically generated.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          h2802_20130430.patch

          Show
          Tsz Wo Nicholas Sze added a comment - h2802_20130430.patch