Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-7276

Limit the number of byte arrays used by DFSOutputStream

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 2.6.0
    • hdfs-client
    • None
    • Reviewed
    • Hide
      The following configuration properties are added.

      - dfs.client.write.byte-array-manager.enabled:
      for enabling/disabling byte array manger. Default is false.

      - dfs.client.write.byte-array-manager.count-threshold:
      The count threshold for each array length so that a manager is created only after the allocation count exceeds the threshold. In other words, the particular array length is not managed until the allocation count exceeds the threshold. Default is 128.

      - dfs.client.write.byte-array-manager.count-limit:
      The maximum number of arrays allowed for each array length. Default is 2048.

      - dfs.client.write.byte-array-manager.count-reset-time-period-ms:
      The time period in milliseconds that the allocation count for each array length is reset to zero if there is no increment. Default is 10,000ms, i.e. 10 seconds.
      Show
      The following configuration properties are added. - dfs.client.write.byte-array-manager.enabled: for enabling/disabling byte array manger. Default is false. - dfs.client.write.byte-array-manager.count-threshold: The count threshold for each array length so that a manager is created only after the allocation count exceeds the threshold. In other words, the particular array length is not managed until the allocation count exceeds the threshold. Default is 128. - dfs.client.write.byte-array-manager.count-limit: The maximum number of arrays allowed for each array length. Default is 2048. - dfs.client.write.byte-array-manager.count-reset-time-period-ms: The time period in milliseconds that the allocation count for each array length is reset to zero if there is no increment. Default is 10,000ms, i.e. 10 seconds.

    Description

      When there are a lot of DFSOutputStream's writing concurrently, the number of outstanding packets could be large. The byte arrays created by those packets could occupy a lot of memory.

      Attachments

        1. h7276_20141031.patch
          50 kB
          Tsz-wo Sze
        2. h7276_20141030.patch
          49 kB
          Tsz-wo Sze
        3. h7276_20141029b.patch
          47 kB
          Tsz-wo Sze
        4. h7276_20141029.patch
          48 kB
          Tsz-wo Sze
        5. h7276_20141028.patch
          43 kB
          Tsz-wo Sze
        6. h7276_20141027b.patch
          43 kB
          Tsz-wo Sze
        7. h7276_20141027.patch
          35 kB
          Tsz-wo Sze
        8. h7276_20141024.patch
          28 kB
          Tsz-wo Sze
        9. h7276_20141023.patch
          28 kB
          Tsz-wo Sze
        10. h7276_20141022.patch
          23 kB
          Tsz-wo Sze
        11. h7276_20141021.patch
          10 kB
          Tsz-wo Sze

        Issue Links

          Activity

            People

              szetszwo Tsz-wo Sze
              szetszwo Tsz-wo Sze
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: