Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-33005 Kubernetes GA Preparation
  3. SPARK-34115

Long runtime on many environment variables

Attach filesAttach ScreenshotVotersWatch issueWatchersLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.4.0, 2.4.7, 3.0.1
    • Fix Version/s: 3.0.2, 3.1.1
    • Component/s: Spark Core, SQL
    • Labels:
      None
    • Environment:

      Spark 2.4.0 local[2] on a Kubernetes Pod

      Description

      I am not sure if this is a bug report or a feature request. The code is is the same in current versions of Spark and maybe this ticket saves someone some time for debugging.

      We migrated some older code to Spark 2.4.0, and suddently the integration tests on our build machine were much slower than expected.

      On local machines it was running perfectly.

      At the end it turned out, that Spark was wasting CPU Cycles during DataFrame analyzing in the following functions

      • AnalysisHelper.assertNotAnalysisRule calling
      • Utils.isTesting

      Utils.isTesting is traversing all environment variables.

      The offending build machine was a Kubernetes Pod which automatically exposed all services as environment variables, so it had more than 3000 environment variables.

      As Utils.isTesting is called very often throgh AnalysisHelper.assertNotAnalysisRule (via AnalysisHelper.transformDown, transformUp).

       

      Of course we will restrict the number of environment variables, on the other side Utils.isTesting could also use a lazy val for

       

      sys.env.contains("SPARK_TESTING") 

       

      to not make it that expensive.

        Attachments

          Activity

            People

            • Assignee:
              nob13 Norbert Schultz
              Reporter:
              nob13 Norbert Schultz

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment