I am not sure if this is a bug report or a feature request. The code is is the same in current versions of Spark and maybe this ticket saves someone some time for debugging.
We migrated some older code to Spark 2.4.0, and suddently the integration tests on our build machine were much slower than expected.
On local machines it was running perfectly.
At the end it turned out, that Spark was wasting CPU Cycles during DataFrame analyzing in the following functions
- AnalysisHelper.assertNotAnalysisRule calling
Utils.isTesting is traversing all environment variables.
The offending build machine was a Kubernetes Pod which automatically exposed all services as environment variables, so it had more than 3000 environment variables.
As Utils.isTesting is called very often throgh AnalysisHelper.assertNotAnalysisRule (via AnalysisHelper.transformDown, transformUp).
Of course we will restrict the number of environment variables, on the other side Utils.isTesting could also use a lazy val for
to not make it that expensive.