Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-8384

Spark runner is not respecting spark.default.parallelism user defined configuration

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 2.16.0
    • Fix Version/s: None
    • Component/s: runner-spark
    • Labels:
      None

      Description

      It was reported in the mailing list that the Spark runner is not respecting the user defined Spark default parallelism configuration. We should investigate and if it is the case ensure that a user defined configuration is always respected. Runner optimizations should apply only for default (unconfigured) values otherwise we will confuse users and limit them from parametrizing Spark for their best convenience.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                iemejia Ismaël Mejía
                Reporter:
                iemejia Ismaël Mejía
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated: