Description
The Instrumentation class (which is an internal private class) is some what limited by it's current APIs. The class requires an estimator and dataset be passed to the constructor which limits how it can be used. Furthermore, the current APIs make it hard to intercept failures and record anything related to those failures.
The following changes could make the instrumentation class easier to work with. All these changes are for private APIs and should not be visible to users.
// New no-argument constructor: Instrumentation() // New api to log previous constructor arguments. logTrainingContext(estimator: Estimator[_], dataset: Dataset[_]) logFailure(e: Throwable): Unit // Log success with no arguments logSuccess(): Unit // Log result model explicitly instead of passing to logSuccess logModel(model: Model[_]): Unit // On Companion object Instrumentation.instrumented[T](body: (Instrumentation => T)): T // The above API will allow us to write instrumented methods more clearly and handle logging success and failure automatically: def someMethod(...): T = instrumented { instr => instr.logNamedValue(name, value) // more code here instr.logModel(model) }
Attachments
Issue Links
- relates to
-
SPARK-24852 Have spark.ml training use updated `Instrumentation` APIs.
-
- Resolved
-
- links to
