Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-20391

Properly rename the memory related fields in ExecutorSummary REST API

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Blocker
    • Resolution: Fixed
    • 2.2.0
    • 2.2.0
    • Spark Core
    • None

    Description

      Currently in Spark we could get executor summary through REST API /api/v1/applications/<app-id>/executors. The format of executor summary is:

      class ExecutorSummary private[spark](
          val id: String,
          val hostPort: String,
          val isActive: Boolean,
          val rddBlocks: Int,
          val memoryUsed: Long,
          val diskUsed: Long,
          val totalCores: Int,
          val maxTasks: Int,
          val activeTasks: Int,
          val failedTasks: Int,
          val completedTasks: Int,
          val totalTasks: Int,
          val totalDuration: Long,
          val totalGCTime: Long,
          val totalInputBytes: Long,
          val totalShuffleRead: Long,
          val totalShuffleWrite: Long,
          val isBlacklisted: Boolean,
          val maxMemory: Long,
          val executorLogs: Map[String, String],
          val onHeapMemoryUsed: Option[Long],
          val offHeapMemoryUsed: Option[Long],
          val maxOnHeapMemory: Option[Long],
          val maxOffHeapMemory: Option[Long])
      

      Here are 6 memory related fields: memoryUsed, maxMemory, onHeapMemoryUsed, offHeapMemoryUsed, maxOnHeapMemory, maxOffHeapMemory.

      These all 6 fields reflects the storage memory usage in Spark, but from the name of this 6 fields, user doesn't really know it is referring to storage memory or the total memory (storage memory + execution memory). This will be misleading.

      So I think we should properly rename these fields to reflect their real meanings. Or we should will document it.

      Attachments

        Activity

          People

            jerryshao Saisai Shao
            jerryshao Saisai Shao
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: