Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-5323

Trash documentation should describe its directory structure and configurations

    Details

    • Target Version/s:
    • Hadoop Flags:
      Reviewed
    • Tags:
      ygridqa

      Description

      Trash documentation should mention the significance of "Current" and "<time-stamp>" directories which get generated inside Trash directory. The documentation should also incorporate modifications done in HADOOP: 4970.

        Issue Links

          Activity

          Hide
          cheersyang Weiwei Yang added a comment -

          We surely should improve the document for trash, it is out-of-date and people would easily get confused. The document should cover issues that resolved in HADOOP-6761. I proposed to revise the doc to

          File Deletes and Undeletes

          When a file is deleted by a user or an application, it is not immediately removed from HDFS. Instead, HDFS moves it to a trash directory (each user has its own trash directory under `/user/<username>/.Trash`). Most recent deleted files are moved to the current trash directory (`/user/<username>/.Trash/Current`), and in a configurable interval, HDFS creates checkpoints (under `/user/<username>/.Trash/<date>`) for files in current trash directory and deletes old checkpoints when they are expired.

          Current default the trash feature is disabled (Delete files without storing in trash), user can enable this feature by setting a value greater than zero for parameter `fs.trash.interval` (in core-site.xml). This value tells the NameNode how long a checkpoint will be expired and removed from HDFS. In addition, user can configure an appropriate time to tell NameNode how often to create checkpoints in trash (the parameter stored as `fs.trash.checkpoint.interval` in core-site.xml), this value should be smaller or equal to fs.trash.interval.

          Show
          cheersyang Weiwei Yang added a comment - We surely should improve the document for trash, it is out-of-date and people would easily get confused. The document should cover issues that resolved in HADOOP-6761 . I proposed to revise the doc to File Deletes and Undeletes When a file is deleted by a user or an application, it is not immediately removed from HDFS. Instead, HDFS moves it to a trash directory (each user has its own trash directory under `/user/<username>/.Trash`). Most recent deleted files are moved to the current trash directory (`/user/<username>/.Trash/Current`), and in a configurable interval, HDFS creates checkpoints (under `/user/<username>/.Trash/<date>`) for files in current trash directory and deletes old checkpoints when they are expired. Current default the trash feature is disabled (Delete files without storing in trash), user can enable this feature by setting a value greater than zero for parameter `fs.trash.interval` (in core-site.xml). This value tells the NameNode how long a checkpoint will be expired and removed from HDFS. In addition, user can configure an appropriate time to tell NameNode how often to create checkpoints in trash (the parameter stored as `fs.trash.checkpoint.interval` in core-site.xml), this value should be smaller or equal to fs.trash.interval.
          Hide
          hadoopqa Hadoop QA added a comment -



          +1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 3m 11s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 release audit 0m 20s The applied patch does not increase the total number of release audit warnings.
          +1 site 3m 0s Site still builds.
          +1 whitespace 0m 1s The patch has no lines that end in whitespace.
              6m 35s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12753726/HDFS-5323.patch
          Optional Tests site
          git revision trunk / 095ab9a
          Java 1.7.0_55
          uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7578/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 3m 11s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 release audit 0m 20s The applied patch does not increase the total number of release audit warnings. +1 site 3m 0s Site still builds. +1 whitespace 0m 1s The patch has no lines that end in whitespace.     6m 35s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12753726/HDFS-5323.patch Optional Tests site git revision trunk / 095ab9a Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/7578/console This message was automatically generated.
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          +1, checking this in.

          Show
          ozawa Tsuyoshi Ozawa added a comment - +1, checking this in.
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          Committed this to trunk, branch-2, and branch-2.7. Thanks Weiwei Yang for your contribution!

          Show
          ozawa Tsuyoshi Ozawa added a comment - Committed this to trunk, branch-2, and branch-2.7. Thanks Weiwei Yang for your contribution!
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8395 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8395/)
          HADOOP-5323. Trash documentation should describe its directory structure and configurations. Contributed by Weiwei Yang. (ozawa: rev 9a87f81ce6839d1b5ab3d663c1e82c7e572bdeb8)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsDesign.md
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8395 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8395/ ) HADOOP-5323 . Trash documentation should describe its directory structure and configurations. Contributed by Weiwei Yang. (ozawa: rev 9a87f81ce6839d1b5ab3d663c1e82c7e572bdeb8) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsDesign.md
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #347 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/347/)
          HADOOP-5323. Trash documentation should describe its directory structure and configurations. Contributed by Weiwei Yang. (ozawa: rev 9a87f81ce6839d1b5ab3d663c1e82c7e572bdeb8)

          • hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsDesign.md
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #347 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/347/ ) HADOOP-5323 . Trash documentation should describe its directory structure and configurations. Contributed by Weiwei Yang. (ozawa: rev 9a87f81ce6839d1b5ab3d663c1e82c7e572bdeb8) hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsDesign.md hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #1077 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1077/)
          HADOOP-5323. Trash documentation should describe its directory structure and configurations. Contributed by Weiwei Yang. (ozawa: rev 9a87f81ce6839d1b5ab3d663c1e82c7e572bdeb8)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsDesign.md
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #1077 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1077/ ) HADOOP-5323 . Trash documentation should describe its directory structure and configurations. Contributed by Weiwei Yang. (ozawa: rev 9a87f81ce6839d1b5ab3d663c1e82c7e572bdeb8) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsDesign.md
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2289 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2289/)
          HADOOP-5323. Trash documentation should describe its directory structure and configurations. Contributed by Weiwei Yang. (ozawa: rev 9a87f81ce6839d1b5ab3d663c1e82c7e572bdeb8)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsDesign.md
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2289 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2289/ ) HADOOP-5323 . Trash documentation should describe its directory structure and configurations. Contributed by Weiwei Yang. (ozawa: rev 9a87f81ce6839d1b5ab3d663c1e82c7e572bdeb8) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsDesign.md
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #340 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/340/)
          HADOOP-5323. Trash documentation should describe its directory structure and configurations. Contributed by Weiwei Yang. (ozawa: rev 9a87f81ce6839d1b5ab3d663c1e82c7e572bdeb8)

          • hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsDesign.md
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #340 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/340/ ) HADOOP-5323 . Trash documentation should describe its directory structure and configurations. Contributed by Weiwei Yang. (ozawa: rev 9a87f81ce6839d1b5ab3d663c1e82c7e572bdeb8) hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsDesign.md hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2268 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2268/)
          HADOOP-5323. Trash documentation should describe its directory structure and configurations. Contributed by Weiwei Yang. (ozawa: rev 9a87f81ce6839d1b5ab3d663c1e82c7e572bdeb8)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsDesign.md
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2268 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2268/ ) HADOOP-5323 . Trash documentation should describe its directory structure and configurations. Contributed by Weiwei Yang. (ozawa: rev 9a87f81ce6839d1b5ab3d663c1e82c7e572bdeb8) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsDesign.md
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #330 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/330/)
          HADOOP-5323. Trash documentation should describe its directory structure and configurations. Contributed by Weiwei Yang. (ozawa: rev 9a87f81ce6839d1b5ab3d663c1e82c7e572bdeb8)

          • hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsDesign.md
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #330 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/330/ ) HADOOP-5323 . Trash documentation should describe its directory structure and configurations. Contributed by Weiwei Yang. (ozawa: rev 9a87f81ce6839d1b5ab3d663c1e82c7e572bdeb8) hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsDesign.md hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          cheersyang Weiwei Yang added a comment -

          Thanks Tsuyoshi

          Show
          cheersyang Weiwei Yang added a comment - Thanks Tsuyoshi
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          FYI, the CHANGES.txt entry got put into hdfs file instead of the common file, fixed it myself.

          Also, fixing fix-versions - no need to mention 3.0, 2.8.0 etc once we mark it against 2.7.2.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - FYI, the CHANGES.txt entry got put into hdfs file instead of the common file, fixed it myself. Also, fixing fix-versions - no need to mention 3.0, 2.8.0 etc once we mark it against 2.7.2.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #9102 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9102/)
          HADOOP-5323. Trash documentation should describe its directory structure (vinodkv: rev cb73c753dab836105247945392ad4e43695a50bc)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9102 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9102/ ) HADOOP-5323 . Trash documentation should describe its directory structure (vinodkv: rev cb73c753dab836105247945392ad4e43695a50bc) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-common-project/hadoop-common/CHANGES.txt

            People

            • Assignee:
              cheersyang Weiwei Yang
              Reporter:
              sumanshg Suman Sehgal
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development