Details

    • Type: Sub-task
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.23.1
    • Component/s: hdfs-client, performance
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      BlockReader has been reimplemented to use direct byte buffers. If you use a custom socket factory, it must generate sockets that have associated Channels.

      Description

      BlockReader is currently quite complicated since it has to conform to the FSInputChecker inheritance structure. It would be much simpler to implement it standalone. Benchmarking indicates it's slightly faster, as well.

        Attachments

        1. seq-read-1gb-bench.png
          39 kB
          Todd Lipcon
        2. hdfs-2129-benchmark.png
          124 kB
          Todd Lipcon
        3. hdfs-2129.txt
          32 kB
          Todd Lipcon
        4. hdfs-2129.txt
          33 kB
          Todd Lipcon
        5. hdfs-2129.txt
          44 kB
          Todd Lipcon
        6. hdfs-2129.txt
          48 kB
          Todd Lipcon
        7. hdfs-2129.txt
          52 kB
          Todd Lipcon
        8. hdfs-2129.txt
          63 kB
          Todd Lipcon
        9. hdfs-2129.txt
          63 kB
          Todd Lipcon
        10. hdfs-2129.txt
          63 kB
          Todd Lipcon

          Issue Links

            Activity

              People

              • Assignee:
                tlipcon Todd Lipcon
                Reporter:
                tlipcon Todd Lipcon
              • Votes:
                0 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: