Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-8384

Spark runner is not respecting spark.default.parallelism user defined configuration

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • P3
    • Resolution: Unresolved
    • 2.16.0
    • None
    • runner-spark
    • None

    Description

      It was reported in the mailing list that the Spark runner is not respecting the user defined Spark default parallelism configuration. We should investigate and if it is the case ensure that a user defined configuration is always respected. Runner optimizations should apply only for default (unconfigured) values otherwise we will confuse users and limit them from parametrizing Spark for their best convenience.

      Attachments

        Issue Links

          is related to

          Bug - A problem which impairs or prevents the functions of the product. BEAM-8191 Multiple Flatten.pCollections() transforms generate an overwhelming number of tasks

          • P3 - Non-urgent bugs, features, and improvements
          • Resolved

          Activity

            People

              Unassigned Unassigned
              iemejia Ismaël Mejía
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated: