Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-6581 Write to single replica in memory
  3. HDFS-7129

Metrics to track usage of memory for writes

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: HDFS-6581
    • Fix Version/s: 2.6.0
    • Component/s: datanode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      A few metrics to evaluate feature usage and suggest improvements. Thanks to Suresh Srinivas for some of these suggestions.

      1. Number of times a block in memory was read (before being ejected)
      2. Average block size for data written to memory tier
      3. Time the block was in memory before being ejected
      4. Number of blocks written to memory
      5. Number of memory writes requested but not satisfied (failed-over to disk)
      6. Number of blocks evicted without ever being read from memory
      7. Average delay between memory write and disk write (window where a node restart could cause data loss).
      8. Replicas written to disk by lazy writer
      9. Bytes written to disk by lazy writer
      10. Replicas deleted by application before being persisted to disk

        Attachments

        1. HDFS-7129.0.patch
          11 kB
          Xiaoyu Yao
        2. HDFS-7129.1.patch
          20 kB
          Xiaoyu Yao
        3. HDFS-7129.2.patch
          25 kB
          Xiaoyu Yao
        4. HDFS-7129.3.patch
          23 kB
          Xiaoyu Yao

          Issue Links

            Activity

              People

              • Assignee:
                xyao Xiaoyu Yao
                Reporter:
                arp Arpit Agarwal
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: