Hadoop Common
  1. Hadoop Common
  2. HADOOP-3205

Read multiple chunks directly from FSInputChecker subclass into user buffers

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.22.0
    • Fix Version/s: 0.21.0
    • Component/s: fs
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Implementations of FSInputChecker and FSOutputSummer like DFS do not have access to full user buffer. At any time DFS can access only up to 512 bytes even though user usually reads with a much larger buffer (often controlled by io.file.buffer.size). This requires implementations to double buffer data if an implementation wants to read or write larger chunks of data from underlying storage.

      We could separate changes for FSInputChecker and FSOutputSummer into two separate jiras.

      1. hadoop-3205.txt
        19 kB
        Todd Lipcon
      2. hadoop-3205.txt
        11 kB
        Todd Lipcon
      3. hadoop-3205.txt
        11 kB
        Todd Lipcon
      4. hadoop-3205.txt
        9 kB
        Todd Lipcon
      5. hadoop-3205.txt
        8 kB
        Todd Lipcon

        Issue Links

          Activity

          Tom White made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Tom White made changes -
          Fix Version/s 0.21.0 [ 12313563 ]
          Fix Version/s 0.22.0 [ 12314296 ]
          Tom White made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags [Reviewed]
          Fix Version/s 0.22.0 [ 12314296 ]
          Resolution Fixed [ 1 ]
          Todd Lipcon made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Affects Version/s 0.22.0 [ 12314296 ]
          Todd Lipcon made changes -
          Attachment hadoop-3205.txt [ 12426818 ]
          Todd Lipcon made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Todd Lipcon made changes -
          Link This issue blocks HDFS-755 [ HDFS-755 ]
          Todd Lipcon made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Todd Lipcon made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Todd Lipcon made changes -
          Attachment hadoop-3205.txt [ 12424246 ]
          Todd Lipcon made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Todd Lipcon made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Todd Lipcon made changes -
          Attachment hadoop-3205.txt [ 12424169 ]
          Todd Lipcon made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Todd Lipcon made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Todd Lipcon made changes -
          Attachment hadoop-3205.txt [ 12424165 ]
          Todd Lipcon made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Todd Lipcon made changes -
          Summary FSInputChecker and FSOutputSummer should allow better access to user buffer Read multiple chunks directly from FSInputChecker subclass into user buffers
          Assignee Raghu Angadi [ rangadi ] Todd Lipcon [ tlipcon ]
          Description
          Implementations of FSInputChecker and FSOutputSummer like DFS do not have access to full user buffer. At any time DFS can access only up to 512 bytes even though user usually reads with a much larger buffer (often controlled by io.file.buffer.size). This requires implementations to double buffer data if an implementation wants to read or write larger chunks of data from underlying storage.

          We could separate changes for FSInputChecker and FSOutputSummer into two separate jiras.

          Implementations of FSInputChecker and FSOutputSummer like DFS do not have access to full user buffer. At any time DFS can access only up to 512 bytes even though user usually reads with a much larger buffer (often controlled by io.file.buffer.size). This requires implementations to double buffer data if an implementation wants to read or write larger chunks of data from underlying storage.

          We could separate changes for FSInputChecker and FSOutputSummer into two separate jiras.

          Todd Lipcon made changes -
          Attachment hadoop-3205.txt [ 12424076 ]
          Todd Lipcon made changes -
          Field Original Value New Value
          Link This issue is related to HDFS-347 [ HDFS-347 ]
          Raghu Angadi created issue -

            People

            • Assignee:
              Todd Lipcon
              Reporter:
              Raghu Angadi
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development