Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-1146

Decrease spark runner startup overhead

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • P2
    • Resolution: Fixed
    • None
    • 0.5.0
    • runner-spark
    • None

    Description

      BEAM-921 introduced a lazy singleton instantiated once in each JVM (driver & executors) which utilizes reflection to find all subclasses of Source and Coder
      While this is beneficial in its own right, the change added about one minute of overhead in spark runner startup time (which cause the first job/stage to take up to a minute).
      The change is in class BeamSparkRunnerRegistrator.

      Attachments

        Activity

          People

            aviemzur Aviem Zur
            aviemzur Aviem Zur
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: