Uploaded image for project: 'Bigtop'
  1. Bigtop
  2. BIGTOP-2821

expose extra config options for spark

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 1.2.0
    • Fix Version/s: 1.2.1
    • Component/s: None
    • Labels:
      None

      Description

      It would be nice to allow users to configure extra libraries for the spark driver and executor. For example, this would automatically configure spark to be aware of cuda libs when installed on a gpu-enabled machine.

        Issue Links

          Activity

          Hide
          kwmonroe Kevin W Monroe added a comment -

          While we're here, let's go nuts! Expose driver/executor mem options in addition to the extraLibraryPath.

          Show
          kwmonroe Kevin W Monroe added a comment - While we're here, let's go nuts! Expose driver/executor mem options in addition to the extraLibraryPath.
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user kwmonroe opened a pull request:

          https://github.com/apache/bigtop/pull/238

          BIGTOP-2821: expose extra config options for spark

          Let users configure extra library paths and driver/executor memory by exposing these options in puppet and using them in `spark-defaults.conf`.

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/juju-solutions/bigtop feature/BIGTOP-2821/spark-expose-config

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/bigtop/pull/238.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #238


          commit f18f523f44b5cc959c2ff978a4f45c53417b498b
          Author: Kevin W Monroe <kevin.monroe@canonical.com>
          Date: 2017-06-23T16:41:54Z

          BIGTOP-2821: expose extra config options for spark


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user kwmonroe opened a pull request: https://github.com/apache/bigtop/pull/238 BIGTOP-2821 : expose extra config options for spark Let users configure extra library paths and driver/executor memory by exposing these options in puppet and using them in `spark-defaults.conf`. You can merge this pull request into a Git repository by running: $ git pull https://github.com/juju-solutions/bigtop feature/ BIGTOP-2821 /spark-expose-config Alternatively you can review and apply these changes as the patch at: https://github.com/apache/bigtop/pull/238.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #238 commit f18f523f44b5cc959c2ff978a4f45c53417b498b Author: Kevin W Monroe <kevin.monroe@canonical.com> Date: 2017-06-23T16:41:54Z BIGTOP-2821 : expose extra config options for spark
          Hide
          evans_ye Evans Ye added a comment -

          LGTM, +1.

          Show
          evans_ye Evans Ye added a comment - LGTM, +1.
          Hide
          oflebbe Olaf Flebbe added a comment -

          LGTM, +1

          Show
          oflebbe Olaf Flebbe added a comment - LGTM, +1
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/bigtop/pull/238

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/bigtop/pull/238
          Hide
          kwmonroe Kevin W Monroe added a comment -

          Thanks for the reviews! This is now committed to master.

          Show
          kwmonroe Kevin W Monroe added a comment - Thanks for the reviews! This is now committed to master.

            People

            • Assignee:
              kwmonroe Kevin W Monroe
              Reporter:
              kwmonroe Kevin W Monroe
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development