Uploaded image for project: 'Bigtop'
  1. Bigtop
  2. BIGTOP-2737

Spark charm doesn't handle HA or examples well

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 1.2.0
    • Fix Version/s: 1.2.1, 1.3.0
    • Component/s: deployment
    • Labels:
      None

      Description

      The Spark charm can be reconfigured at runtime to work in local, standalone, standalone-HA (with zookeeper), or yarn modes. In any of these modes, the user can run things like SparkPI to verify functionality. There are a few problems with this in the current charm:

      • transitioning out of HA mode does not reset the zk connection string correctly
      • transitioning into HA mode before the zk ensemble has settled can cause the spark master to fetch invalid data from zk; ensure zk is settled before starting spark master
      • master/worker services are always started; ensure we start services relevant for the execution mode [0]
      • spark-examples.jar location has changed in spark2
      • the add-on benchmark suite (sparkbench from ibm) is not compatible with spark 2.1; include a pagerank benchmark that works across 1.5 and 2.1

      To ease future maintenance, the spark charm needs a good refactoring to simplify the reactive logic. It's currently too hard to follow what is happening when the deployment topology changes.

      [0] - this may belong in the spark puppet recipe, but it's easy enough to fix in the charm while investigating a better place.

        Issue Links

          Activity

          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/bigtop/pull/194

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/bigtop/pull/194
          Hide
          kwmonroe Kevin W Monroe added a comment -

          New spark charm looks good in standalone mode across clouds:

          http://bigtop.charm.qa/cwr_bundle_spark_processing/36/report.html

          As well as yarn mode:

          http://bigtop.charm.qa/cwr_bundle_hadoop_spark/5/report.html

          Show
          kwmonroe Kevin W Monroe added a comment - New spark charm looks good in standalone mode across clouds: http://bigtop.charm.qa/cwr_bundle_spark_processing/36/report.html As well as yarn mode: http://bigtop.charm.qa/cwr_bundle_hadoop_spark/5/report.html

            People

            • Assignee:
              kwmonroe Kevin W Monroe
              Reporter:
              kwmonroe Kevin W Monroe
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development