Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-23206 Additional Memory Tuning Metrics
  3. SPARK-26399

Define query parameters to support various filtering conditions in REST API for overall stages

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.1.0
    • 3.2.0
    • Spark Core
    • None

    Description

      angerszhuuu and ron8hu discussed a generic and consistent way for overall stages, i.e. endpoint /application/{app-id}/stages.  It can be:

      /application/{app-id}/stages?details=[true|false]&status=[ACTIVE|COMPLETE|FAILED|PENDING|SKIPPED]&withSummaries=[true|false]&taskStatus=[RUNNING|SUCCESS|FAILED|KILLED|PENDING]

      where

      • query parameter details=true is to show the detailed task information within each stage.  The default value is details=false;
      • query parameter status can select those stages with the specified status.  When status parameter is not specified, a list of all stages are generated.  
      • query parameter withSummaries=true is to show both task metrics summary information in percentile distribution (see example of a single stage in taskMetricsDistributions.json ) and executor metrics summary information in percentile distribution (see example of a single stage in executorMetricsDistributions.json  ).  The default value is withSummaries=false.
      • query parameter taskStatus is to show only those tasks with the specified status within their corresponding stages.  This parameter can be set when details=true (i.e. this parameter will be ignored when details=false).

      The output is an aggregate of all stages meeting the filtering conditions for a given application.   

      Attachments

        1. executorMetricsDistributions.json
          2 kB
          Ron Hu
        2. executorMetricsSummary.json
          2 kB
          Ron Hu
        3. lispark230_restapi_ex2_stages_failedTasks.json
          19 kB
          Ron Hu
        4. lispark230_restapi_ex2_stages_withSummaries.json
          806 kB
          Ron Hu
        5. stage_executorSummary_image1.png
          164 kB
          Ron Hu
        6. taskMetricsDistributions.json
          2 kB
          Ron Hu
        7. taskMetricsDistributions.json
          2 kB
          Ron Hu

        Activity

          People

            angerszhuuu angerszhu
            elu Edward Lu
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: