Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-1869

access times of HDFS files

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.19.0
    • None
    • None
    • Incompatible change, Reviewed
    • Hide
      Added HDFS file access times. By default, access times will be precise to the most recent hour boundary. A configuration parameter dfs.access.time.precision (milliseconds) is used to control this precision. Setting a value of 0 will disable persisting access times for HDFS files.
      Show
      Added HDFS file access times. By default, access times will be precise to the most recent hour boundary. A configuration parameter dfs.access.time.precision (milliseconds) is used to control this precision. Setting a value of 0 will disable persisting access times for HDFS files.

    Description

      HDFS should support some type of statistics that allows an administrator to determine when a file was last accessed.

      Since HDFS does not have quotas yet, it is likely that users keep on accumulating files in their home directories without much regard to the amount of space they are occupying. This causes memory-related problems with the namenode.

      Access times are costly to maintain. AFS does not maintain access times. I thind DCE-DFS does maintain access times with a coarse granularity.

      One proposal for HDFS would be to implement something like an "access bit".
      1. This access-bit is set when a file is accessed. If the access bit is already set, then this call does not result in a transaction.
      2. A FileSystem.clearAccessBits() indicates that the access bits of all files need to be cleared.

      An administrator can effectively use the above mechanism (maybe a daily cron job) to determine files that are recently used.

      Attachments

        1. accessTime6.patch
          45 kB
          Dhruba Borthakur
        2. accessTime5.patch
          41 kB
          Dhruba Borthakur
        3. accessTime4.patch
          41 kB
          Dhruba Borthakur
        4. accessTime1.patch
          37 kB
          Dhruba Borthakur

        Issue Links

          blocks

          Bug - A problem which impairs or prevents the functions of the product. HDFS-430 create posix-like (as far as we can) layer for Linux on top of libhdfs

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          is depended upon by

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-4077 Access permissions for setting access times and modification times for files

          • Blocker - Blocks development and/or testing work, production could not run
          • Closed

          New Feature - A new feature of the product, which has yet to be developed. HDFS-220 Transparent archival and restore of files from HDFS

          • Major - Major loss of function.
          • Resolved
          is related to

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-4099 HFTP interface compatibility with older releases broken

          • Blocker - Blocks development and/or testing work, production could not run
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-4986 FSNamesystem.getBlockLocations sets access time without holding the namespace locks

          • Blocker - Blocks development and/or testing work, production could not run
          • Closed

          New Feature - A new feature of the product, which has yet to be developed. HADOOP-3336 Direct a subset of namenode RPC events for audit logging

          • Major - Major loss of function.
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. HDFS-2712 setTimes should support only for files and move the atime field down to iNodeFile.

          • Major - Major loss of function.
          • Patch Available

          Task - A task that needs to be done. HDFS-258 Confirm that all block history events are available in logs

          • Major - Major loss of function.
          • Open

          Activity

            People

              dhruba Dhruba Borthakur
              dhruba Dhruba Borthakur
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: