Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Won't Fix
-
2.3.1
-
None
-
None
Description
- spark.ui.retainedJobs =200( spark-default.conf of Job History)
2. Launch spark-shell --master yarn --conf spark.ui.retainedJobs=100
3. Execute Below command for creating 1000 jobs
val rdd = sc.parallelize(1 to 5, 5)
for(i <- 1 to 1000) { rdd.count }4. Launch Job History Page
5. Click on corresponding spark-shell Application ID link in Job History and Launch Job Page for the application
6. Go to Environment Page check the value against spark.ui.retainedJobs. It shows spark.ui.retainedJobs = 100
7. Check the Total number of Jobs Summery Display in Job Page. It shows Completed only 200 which is configured in spark-default.conf file of Job History
Actual Result:
Environment Page showing spark.ui.retainedJobs value which is set from command prompt but Job Page displaying number of Jobs showing
based on value set in spark-default.conf of Job History
Expected Result:
Value of spark.ui.retainedJobs in Environment Page should display the value which is configured in spark-default.conf of Job History