Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-14111

hdfsOpenFile on HDFS causes unnecessary IO from file offset 0

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.2.0
    • 3.3.0
    • hdfs-client, libhdfs
    • None

    Description

      hdfsOpenFile() calls readDirect() with a 0-length argument in order to check whether the underlying stream supports bytebuffer reads. With DFSInputStream, the read(0) isn't short circuited, and results in the DFSClient opening a block reader. In the case of a remote block, the block reader will actually issue a read of the whole block, causing the datanode to perform unnecessary IO and network transfers in order to fill up the client's TCP buffers. This causes performance degradation.

      Attachments

        1. HDFS-14111.001.patch
          5 kB
          Sahil Takiar
        2. HDFS-14111.002.patch
          5 kB
          Sahil Takiar
        3. HDFS-14111.003.patch
          6 kB
          Sahil Takiar

        Issue Links

          Activity

            People

              stakiar Sahil Takiar
              tlipcon Todd Lipcon
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: