Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-7276

Limit the number of byte arrays used by DFSOutputStream

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.6.0
    • Component/s: hdfs-client
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      The following configuration properties are added.

      - dfs.client.write.byte-array-manager.enabled:
      for enabling/disabling byte array manger. Default is false.

      - dfs.client.write.byte-array-manager.count-threshold:
      The count threshold for each array length so that a manager is created only after the allocation count exceeds the threshold. In other words, the particular array length is not managed until the allocation count exceeds the threshold. Default is 128.

      - dfs.client.write.byte-array-manager.count-limit:
      The maximum number of arrays allowed for each array length. Default is 2048.

      - dfs.client.write.byte-array-manager.count-reset-time-period-ms:
      The time period in milliseconds that the allocation count for each array length is reset to zero if there is no increment. Default is 10,000ms, i.e. 10 seconds.
      Show
      The following configuration properties are added. - dfs.client.write.byte-array-manager.enabled: for enabling/disabling byte array manger. Default is false. - dfs.client.write.byte-array-manager.count-threshold: The count threshold for each array length so that a manager is created only after the allocation count exceeds the threshold. In other words, the particular array length is not managed until the allocation count exceeds the threshold. Default is 128. - dfs.client.write.byte-array-manager.count-limit: The maximum number of arrays allowed for each array length. Default is 2048. - dfs.client.write.byte-array-manager.count-reset-time-period-ms: The time period in milliseconds that the allocation count for each array length is reset to zero if there is no increment. Default is 10,000ms, i.e. 10 seconds.

      Description

      When there are a lot of DFSOutputStream's writing concurrently, the number of outstanding packets could be large. The byte arrays created by those packets could occupy a lot of memory.

        Attachments

        1. h7276_20141031.patch
          50 kB
          Tsz-wo Sze
        2. h7276_20141030.patch
          49 kB
          Tsz-wo Sze
        3. h7276_20141029b.patch
          47 kB
          Tsz-wo Sze
        4. h7276_20141029.patch
          48 kB
          Tsz-wo Sze
        5. h7276_20141028.patch
          43 kB
          Tsz-wo Sze
        6. h7276_20141027b.patch
          43 kB
          Tsz-wo Sze
        7. h7276_20141027.patch
          35 kB
          Tsz-wo Sze
        8. h7276_20141024.patch
          28 kB
          Tsz-wo Sze
        9. h7276_20141023.patch
          28 kB
          Tsz-wo Sze
        10. h7276_20141022.patch
          23 kB
          Tsz-wo Sze
        11. h7276_20141021.patch
          10 kB
          Tsz-wo Sze

          Issue Links

            Activity

              People

              • Assignee:
                szetszwo Tsz-wo Sze
                Reporter:
                szetszwo Tsz-wo Sze
              • Votes:
                0 Vote for this issue
                Watchers:
                10 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: