Uploaded image for project: 'Bigtop'
  1. Bigtop
  2. BIGTOP-2822

spark charm: leverage puppet config, gpu enablement

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 1.2.0
    • Fix Version/s: 1.2.1
    • Component/s: deployment
    • Labels:
      None

      Description

      With BIGTOP-2821, we can easily configure memory and lib paths for spark. Leverage this capability in the spark charm.

        Issue Links

          Activity

          Hide
          kwmonroe Kevin W Monroe added a comment -

          This is now available in the charm store (rev 49 or later contain this commit):

          https://jujucharms.com/spark/49

          Show
          kwmonroe Kevin W Monroe added a comment - This is now available in the charm store (rev 49 or later contain this commit): https://jujucharms.com/spark/49
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/bigtop/pull/239

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/bigtop/pull/239
          Hide
          kwmonroe Kevin W Monroe added a comment -

          This is working nicely.

          juju deploy ~/charms/xenial/spark sgpu --constraints "instance-type=p2.xlarge root-disk=16G"

          On the unit, spark-defaults looks as expected:
          $ cat /etc/spark/conf/spark-defaults.conf
          ...
          spark.driver.extraLibraryPath /usr/lib/hadoop/lib/native
          spark.executor.extraLibraryPath /usr/lib/hadoop/lib/native
          spark.driver.memory 1g
          spark.executor.memory 1g
          ...

          Now reconfigure spark:
          juju config sgpu install-cuda=True driver_memory=2g executor_memory=70%

          And check the spark unit again:
          $ cat /etc/spark/conf/spark-defaults.conf
          ...
          spark.driver.extraLibraryPath /usr/lib/hadoop/lib/native:/usr/local/cuda/lib64
          spark.executor.extraLibraryPath /usr/lib/hadoop/lib/native:/usr/local/cuda/lib64
          spark.driver.memory 2g
          spark.executor.memory 42984m

          Show
          kwmonroe Kevin W Monroe added a comment - This is working nicely. juju deploy ~/charms/xenial/spark sgpu --constraints "instance-type=p2.xlarge root-disk=16G" On the unit, spark-defaults looks as expected: $ cat /etc/spark/conf/spark-defaults.conf ... spark.driver.extraLibraryPath /usr/lib/hadoop/lib/native spark.executor.extraLibraryPath /usr/lib/hadoop/lib/native spark.driver.memory 1g spark.executor.memory 1g ... Now reconfigure spark: juju config sgpu install-cuda=True driver_memory=2g executor_memory=70% And check the spark unit again: $ cat /etc/spark/conf/spark-defaults.conf ... spark.driver.extraLibraryPath /usr/lib/hadoop/lib/native:/usr/local/cuda/lib64 spark.executor.extraLibraryPath /usr/lib/hadoop/lib/native:/usr/local/cuda/lib64 spark.driver.memory 2g spark.executor.memory 42984m

            People

            • Assignee:
              kwmonroe Kevin W Monroe
              Reporter:
              kwmonroe Kevin W Monroe
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development