The progress API introduced in
SPARK-2321 provides a new way for user to monitor job progress. However the information exposed in the API is relatively limited. It'll be much more useful if we can enhance the API to expose more data.
Some improvement for example may include but not limited to:
1. Stage submission and completion time.
2. Task metrics.
The requirement is initially identified for the hive on spark project(
HIVE-7292), other application should benefit as well.