Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-28352

Spark UI shows RDD blocks as 0 in some executor metrics

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Minor
    • Resolution: Incomplete
    • 2.4.1
    • None
    • Web UI

    Description

      We have recently migrated our application from 2.2.1 to 2.4.1. We have realized that Spark UI is missing some critical information or shows wrong ones.

      We have a big cluster, with 80+ executors. It is Spark Streaming with customer receivers. On Spark UI in storage tab, we can see memory blocks and their size. This page shows correct information that we can see on Ganglia and host level monitors. However, in executor tab, `RDD blocks` shows `0` while we have thousands of them in memory. Moreover, `Storage Memory` shows incorrect numbers.
       

      Attachments

        1. executor_page.png
          26 kB
          Burak KÖSE
        2. executor_summary.png
          22 kB
          Burak KÖSE
        3. storage_page.png
          51 kB
          Burak KÖSE

        Activity
          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            whisper Burak KÖSE
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment