Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-16947

Stale record should be remove when MutableRollingAverages generating aggregate data.

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.3.1, 3.4.0, 3.1.5, 3.2.3
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      SlowPeersReport is generated by the SampleStat between tow dn, so it can present on nn's jmx like this:

      "SlowPeersReport" :[{"SlowNode":"dn2","ReportingNodes":["dn1"]}]
      

      In each period, MutableRollingAverages will do a rollOverAvgs(), it will generate a SumAndCount object which is based on SampleStat, and store it in a LinkedBlockingDeque<SumAndCount>, the deque will be used to generate  SlowPeersReport. And the old member of deque won't be removed until the queue is full. However, if dn1 don't send any packet to dn2 in the last of 36*300_000 ms, the deque will be filled with an old member, because the number of last SampleStat never change.I think these old SampleStats should be considered as expired message and ignore them when generating a new SlowPeersReport.

        Attachments

        1. HADOOP-16947-001.patch
          5 kB
          Haibin Huang
        2. HADOOP-16947-002.patch
          5 kB
          Haibin Huang
        3. HADOOP-16947-003.patch
          6 kB
          Haibin Huang
        4. HADOOP-16947-004.patch
          6 kB
          Haibin Huang
        5. HADOOP-16947-005.patch
          6 kB
          Haibin Huang
        6. HADOOP-16947-006.patch
          7 kB
          Haibin Huang
        7. HADOOP-16947-007.patch
          6 kB
          Haibin Huang
        8. HADOOP-16947-008.patch
          8 kB
          Haibin Huang
        9. HADOOP-16947-009.patch
          8 kB
          Haibin Huang
        10. HADOOP-16947-010.patch
          12 kB
          Haibin Huang
        11. HADOOP-16947-011.patch
          12 kB
          Haibin Huang
        12. HADOOP-16947-012.patch
          12 kB
          Haibin Huang
        13. HADOOP-16947-branch-3.1.001.patch
          12 kB
          Haibin Huang
        14. HADOOP-16947-branch-3.2.001.patch
          12 kB
          Haibin Huang
        15. HDFS-14783-001.patch
          8 kB
          Haibin Huang
        16. HDFS-14783-002.patch
          8 kB
          Haibin Huang
        17. HDFS-14783-003.patch
          8 kB
          Haibin Huang
        18. HDFS-14783-004.patch
          5 kB
          Haibin Huang
        19. HDFS-14783-005.patch
          5 kB
          Haibin Huang

          Issue Links

            Activity

              People

              • Assignee:
                huanghaibin Haibin Huang
                Reporter:
                huanghaibin Haibin Huang
              • Votes:
                0 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: