[SPARK-25504] [Spark Job History]If the number of tasks exceeds spark.ui.retainedTask value, then stage page of UI should display upto the value set in the retainedTask.

• Type: Documentation
• Status: Resolved
• Priority: Minor
• Resolution: Not A Problem
• Affects Version/s: 2.3.1
• Component/s: Web UI
• Assignee: Unassigned
• Reporter: ABHISHEK KUMAR GUPTA
• Created: 21/Sep/18 10:32
• Updated: 25/Nov/18 19:08
• Resolved: 25/Nov/18 19:08

Description

Steps
1. spark.ui.retainedTask =60000( spark-default.conf of Job History)
2. Run a job for creating 100 task as below from Spark-shell
val rdd = sc.parallelize(1 to 49000, 49000)
rdd.count
3. Launch Job History Page
4. Click on corresponding spark-shell Application ID link in Job History and Launch Job Page for the application
5. Click on the Job ID description and Launch Stage Page
6. Click on the Description of Stage ID to Launch Stage Page of specific Stage ID

Atcual Result
Showing Total number of task above Pagination Panel as Tasks(49000)

It is correct as number of task is 49000 created.

Now Execute as below

1. Run a job for creating 100010 task as below from Spark-shell
val rdd = sc.parallelize(1 to 100100, 100100))
rdd.count
2. Launch Job History Page
3. Click on corresponding spark-shell Application ID link in Job History and Launch Job Page for the application
4. Click on the Job ID description and Launch Stage Page
5. Click on the Description of Stage ID to Launch Stage Page of specific Stage ID

Atcual Result
Showing Total number of task above Pagination Panel as Tasks(58100, showing 100100)

Expected:
spark.ui.retainedTask =60000
So it should show as Tasks( 60000, Showing 100100) as spark.ui.retainedTask =60000

Issue Links

• links to: [Github] Pull Request #23134 (shahidki31)
• links to: [Github] Pull Request #23134 (shahidki31)