Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-34845

ProcfsMetricsGetter.computeAllMetrics may return partial metrics when some of child pids metrics are missing

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.0.0, 3.0.1, 3.0.2, 3.1.0, 3.1.1
    • 3.0.3, 3.1.2, 3.2.0
    • Spark Core
    • None

    Description

      When the procfs metrics of some child pids are unavailable, ProcfsMetricsGetter.computeAllMetrics() may return partial metrics (the sum of a subset of child pids), instead of an all 0 result. This can be misleading and is undesired per the current code comments in https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/executor/ProcfsMetricsGetter.scala#L214.

      How to reproduce it?

      This unit test is kind of self-explanatory:

          val p = new ProcfsMetricsGetter(getTestResourcePath("ProcfsMetrics"))
          val mockedP = spy(p)
      
          // proc file of pid 22764 doesn't exist, so partial metrics shouldn't be returned
          var ptree = Set(26109, 22764, 22763)
          when(mockedP.computeProcessTree).thenReturn(ptree)
          var r = mockedP.computeAllMetrics
          assert(r.jvmVmemTotal == 0)
          assert(r.jvmRSSTotal == 0)
          assert(r.pythonVmemTotal == 0)
          assert(r.pythonRSSTotal == 0)
      

      In the current implementation, computeAllMetrics will reset the allMetrics to 0 when processing 22764 because 22764's proc file doesn't exist, but then it will continue processing pid 22763, and update allMetrics to procfs metrics of pid 22763.

      Also, a side effect of this bug is that it can lead to a verbose warning log if many pids' stat files are missing. An early terminating can make the warning logs more concise.

      How to solve it?

      The issue can be fixed by throwing IOException to computeAllMetrics(), in that case, computeAllMetrics can aware that at lease one child pid's procfs metrics is missing and then terminate the metrics reporting.

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Baohe Zhang Baohe Zhang
            Baohe Zhang Baohe Zhang
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment