Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-14111

hdfsOpenFile on HDFS causes unnecessary IO from file offset 0

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.2.0
    • Fix Version/s: 3.3.0
    • Component/s: hdfs-client, libhdfs
    • Labels:
      None

      Description

      hdfsOpenFile() calls readDirect() with a 0-length argument in order to check whether the underlying stream supports bytebuffer reads. With DFSInputStream, the read(0) isn't short circuited, and results in the DFSClient opening a block reader. In the case of a remote block, the block reader will actually issue a read of the whole block, causing the datanode to perform unnecessary IO and network transfers in order to fill up the client's TCP buffers. This causes performance degradation.

        Attachments

        1. HDFS-14111.001.patch
          5 kB
          Sahil Takiar
        2. HDFS-14111.002.patch
          5 kB
          Sahil Takiar
        3. HDFS-14111.003.patch
          6 kB
          Sahil Takiar

          Issue Links

            Activity

              People

              • Assignee:
                stakiar Sahil Takiar
                Reporter:
                tlipcon Todd Lipcon
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: