Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-34845

ProcfsMetricsGetter.computeAllMetrics may return partial metrics when some of child pids metrics are missing

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0, 3.0.1, 3.0.2, 3.1.0, 3.1.1
    • Fix Version/s: 3.0.3, 3.1.2, 3.2.0
    • Component/s: Spark Core
    • Labels:
      None

      Description

      When the procfs metrics of some child pids are unavailable, ProcfsMetricsGetter.computeAllMetrics() may return partial metrics (the sum of a subset of child pids), instead of an all 0 result. This can be misleading and is undesired per the current code comments in https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/executor/ProcfsMetricsGetter.scala#L214.

      How to reproduce it?

      This unit test is kind of self-explanatory:

          val p = new ProcfsMetricsGetter(getTestResourcePath("ProcfsMetrics"))
          val mockedP = spy(p)
      
          // proc file of pid 22764 doesn't exist, so partial metrics shouldn't be returned
          var ptree = Set(26109, 22764, 22763)
          when(mockedP.computeProcessTree).thenReturn(ptree)
          var r = mockedP.computeAllMetrics
          assert(r.jvmVmemTotal == 0)
          assert(r.jvmRSSTotal == 0)
          assert(r.pythonVmemTotal == 0)
          assert(r.pythonRSSTotal == 0)
      

      In the current implementation, computeAllMetrics will reset the allMetrics to 0 when processing 22764 because 22764's proc file doesn't exist, but then it will continue processing pid 22763, and update allMetrics to procfs metrics of pid 22763.

      Also, a side effect of this bug is that it can lead to a verbose warning log if many pids' stat files are missing. An early terminating can make the warning logs more concise.

      How to solve it?

      The issue can be fixed by throwing IOException to computeAllMetrics(), in that case, computeAllMetrics can aware that at lease one child pid's procfs metrics is missing and then terminate the metrics reporting.

        Attachments

          Activity

            People

            • Assignee:
              Baohe Zhang Baohe Zhang
              Reporter:
              Baohe Zhang Baohe Zhang
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: