Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-14111

hdfsOpenFile on HDFS causes unnecessary IO from file offset 0

VotersStop watchingWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.2.0
    • Fix Version/s: 3.3.0
    • Component/s: hdfs-client, libhdfs
    • Labels:
      None

      Description

      hdfsOpenFile() calls readDirect() with a 0-length argument in order to check whether the underlying stream supports bytebuffer reads. With DFSInputStream, the read(0) isn't short circuited, and results in the DFSClient opening a block reader. In the case of a remote block, the block reader will actually issue a read of the whole block, causing the datanode to perform unnecessary IO and network transfers in order to fill up the client's TCP buffers. This causes performance degradation.

        Attachments

        1. HDFS-14111.001.patch
          5 kB
          Sahil Takiar
        2. HDFS-14111.002.patch
          5 kB
          Sahil Takiar
        3. HDFS-14111.003.patch
          6 kB
          Sahil Takiar

        Issue Links

          Activity

            People

            • Assignee:
              stakiar Sahil Takiar
              Reporter:
              tlipcon Todd Lipcon

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment