Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-8425 [umbrella] Performance tuning, investigation and optimization for erasure coding
  3. HDFS-8287

DFSStripedOutputStream.writeChunk should not wait for writing parity

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Patch Available
    • Major
    • Resolution: Unresolved
    • None
    • None
    • erasure-coding, hdfs-client
    • None

    Description

      When a stripping cell is full, writeChunk computes and generates parity packets. It sequentially calls waitAndQueuePacket so that user client cannot continue to write data until it finishes.

      We should allow user client to continue writing instead but not blocking it when writing parity.

      Attachments

        1. jstack-dump.txt
          25 kB
          Kai Sasaki
        2. HDFS-8287-performance-report.pdf
          180 kB
          Kai Sasaki
        3. HDFS-8287-HDFS-7285.WIP.patch
          11 kB
          Kai Sasaki
        4. HDFS-8287-HDFS-7285.11.patch
          8 kB
          Kai Sasaki
        5. HDFS-8287-HDFS-7285.10.patch
          8 kB
          Kai Sasaki
        6. HDFS-8287-HDFS-7285.09.patch
          12 kB
          Kai Sasaki
        7. HDFS-8287-HDFS-7285.08.patch
          8 kB
          Kai Sasaki
        8. HDFS-8287-HDFS-7285.07.patch
          8 kB
          Kai Sasaki
        9. HDFS-8287-HDFS-7285.06.patch
          8 kB
          Kai Sasaki
        10. HDFS-8287-HDFS-7285.05.patch
          9 kB
          Kai Sasaki
        11. HDFS-8287-HDFS-7285.04.patch
          8 kB
          Kai Sasaki
        12. HDFS-8287-HDFS-7285.03.patch
          8 kB
          Kai Sasaki
        13. HDFS-8287-HDFS-7285.02.patch
          8 kB
          Kai Sasaki
        14. HDFS-8287-HDFS-7285.01.patch
          7 kB
          Kai Sasaki
        15. HDFS-8287-HDFS-7285.00.patch
          7 kB
          Kai Sasaki
        16. HDFS-8287.15.patch
          9 kB
          Kai
        17. HDFS-8287.14.patch
          9 kB
          Kai
        18. HDFS-8287.13.patch
          9 kB
          Kai
        19. HDFS-8287.12.patch
          8 kB
          Kai
        20. h8287_20150911.patch
          11 kB
          Tsz-wo Sze

        Issue Links

          is duplicated by

          Sub-task - The sub-task of the issue HDFS-8528 Erasure Coding: optimize client writing by making the writing of data and parity concurrently

          • Major - Major loss of function.
          • Resolved
          relates to

          Bug - A problem which impairs or prevents the functions of the product. HDFS-8931 Erasure Coding: Notify exception to client side from ParityGenerator.

          • Major - Major loss of function.
          • Open

          Activity

            People

              lewuathe Kai
              szetszwo Tsz-wo Sze
              Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

                Created:
                Updated: