Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-26192

MesosClusterScheduler reads options from dispatcher conf instead of submission conf

Details

    • Improvement
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 2.3.0, 2.3.1, 2.3.2, 2.4.0
    • 2.4.4, 3.0.0
    • Mesos
    • None

    Description

      There is at least one option accessed in MesosClusterScheduler that should come from the submission's configuration instead of the dispatcher's:

      spark.mesos.fetcherCache.enable

      Coincidentally, the spark.mesos.fetcherCache.enable option was previously misnamed, as referenced in the linked JIRA.

      Attachments

        Issue Links

          Activity

            dongjoon Dongjoon Hyun added a comment - This is resolved via https://github.com/apache/spark/pull/23924 .
            dongjoon Dongjoon Hyun added a comment - - edited

            mwlon. Thank you for reporting and making a PR. However, please don't set 'Fix Versions`. `Fixed Version` and `Target Version` are used in the different way. Please refer the contribution guide.

            For me, this is an improvement for Spark 3.0.

            dongjoon Dongjoon Hyun added a comment - - edited mwlon . Thank you for reporting and making a PR. However, please don't set 'Fix Versions`. `Fixed Version` and `Target Version` are used in the different way. Please refer the contribution guide. https://spark.apache.org/contributing.html For me, this is an improvement for Spark 3.0.
            mwlon Martin Loncaric added a comment - - edited

            dongjoon Thanks, I will pay more attention to those fields.

            However, I believe this is a bug. It violates behavior specified in https://spark.apache.org/docs/latest/running-on-mesos.html#configuration. Can we merge into at least 2.4.1 as well?

            mwlon Martin Loncaric added a comment - - edited dongjoon Thanks, I will pay more attention to those fields. However, I believe this is a bug. It violates behavior specified in https://spark.apache.org/docs/latest/running-on-mesos.html#configuration . Can we merge into at least 2.4.1 as well?
            dongjoon Dongjoon Hyun added a comment -

            Sorry for being late, mwlon. Could you make a PR against `branch-2.4`?

            dongjoon Dongjoon Hyun added a comment - Sorry for being late, mwlon . Could you make a PR against `branch-2.4`?
            mwlon Martin Loncaric added a comment - dongjoon I made a PR: https://github.com/apache/spark/pull/24713
            apachespark Apache Spark added a comment -

            User 'mwlon' has created a pull request for this issue:
            https://github.com/apache/spark/pull/24713

            apachespark Apache Spark added a comment - User 'mwlon' has created a pull request for this issue: https://github.com/apache/spark/pull/24713

            People

              mwlon Martin Loncaric
              mwlon Martin Loncaric
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: