Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-2744

Extend FSDataInputStream to allow fadvise

    Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: hdfs-client
    • Labels:

      Description

      Now that we have direct reads from local HDFS block files (HDFS-2246), it might make sense to make FSDataInputStream support fadvise calls. I have an application (HBase) that would like to tell the OS that it should not buffer data in the OS buffer cache.

        Issue Links

          Activity

          Hide
          dhruba borthakur added a comment -
            /**
             * Tells the system to not buffer data in the OS buffer cache if
             * appropriate.
             *
             * @param mode random, sequential, normal, dontneed
             * @param offset the starting file offset of the file
             * @param size the size of data that need not be cached
             */
            public void fadvise(Enum mode, long position, int length)
              throws IOException
          
          
          Show
          dhruba borthakur added a comment - /** * Tells the system to not buffer data in the OS buffer cache if * appropriate. * * @param mode random, sequential, normal, dontneed * @param offset the starting file offset of the file * @param size the size of data that need not be cached */ public void fadvise(Enum mode, long position, int length) throws IOException
          Hide
          Suresh Srinivas added a comment -

          +1.

          Should we add some interface called FAdvisable and then provide default implementation in FSInputStream and FSOutputStream (that is advise is ignored). The real implementation could be provided in DFS classes.

          Show
          Suresh Srinivas added a comment - +1. Should we add some interface called FAdvisable and then provide default implementation in FSInputStream and FSOutputStream (that is advise is ignored). The real implementation could be provided in DFS classes.
          Hide
          Steve Loughran added a comment -

          HADOOP-13203 is implementing an fadvise-like mechanism, but in the absence of an API, it's going to be a configuration option to set the policy for opened streams.

          Ideally I'd like: fadvise(policy), or just the ability to set a policy when you open a stream i.e Filesystem.open(path, fadvise-policy). The latter would avoid having to deal with the situation of a policy changing during the life of an input stream.

          Show
          Steve Loughran added a comment - HADOOP-13203 is implementing an fadvise-like mechanism, but in the absence of an API, it's going to be a configuration option to set the policy for opened streams. Ideally I'd like: fadvise(policy) , or just the ability to set a policy when you open a stream i.e Filesystem.open(path, fadvise-policy) . The latter would avoid having to deal with the situation of a policy changing during the life of an input stream.

            People

            • Assignee:
              dhruba borthakur
              Reporter:
              dhruba borthakur
            • Votes:
              0 Vote for this issue
              Watchers:
              14 Start watching this issue

              Dates

              • Created:
                Updated:

                Development