Uploaded image for project: 'Accumulo'
  1. Accumulo
  2. ACCUMULO-3167

Decouple MiniAccumuloCluster from integration test base class

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 1.6.2, 1.7.0
    • mini, test
    • None

    Description

      (apologies if I already had a ticket for this somewhere, I couldn't find it)

      Our integration tests are very nice and automated at the moment because we can use MiniAccumuloCluster to "provision" an Accumulo instance (or used a shared instance), and run a test against it. For the most part, this works well and provides an accurate test harness.

      Thus, to run integration tests, you need a sufficiently beefy machine since the same host will be running all of Accumulo as well as performing any client work. When resources are available to use, it would be nice to leverage them – whether these are yarn, mesos, a vanila installation, etc.

      In addition to the additional computational power from using extra hardware, it also encourages us to use the public API as much as possible instead of relying on "hidden" impl methods in MiniAccumuloCluster.

      I propose making changes to the IT test base (AbstractMacIT, SimpleMacIT, ConfigurableMacIT) to add an extra step between them an test classes to allow "injection" of a more generic Accumulo "cluster" that is not associated with MAC.

      Attachments

        Issue Links

          Activity

            elserj Josh Elser added a comment -

            My initial thoughts on how to implement this is to introduce an extra shim between integration tests and the underlying cluster implementation (e.g. SimpleMacIT). Thus, our ITs that need a running instance of Accumulo will extend this new class that provides a "deployment agnostic" set of functions to interact with that instance.

            Some complexity issues arise around this because some deployments (notably, the plain-jane Accumulo instance) cannot guarantee the breadth of functionality that some tests expect: the inability to restart Accumulo processes, control site configuration files (HDFS or Accumulo), and alter JVM level properties. All other configuration should be capable of modification via use of ZooKeeper through the normal API. MAC and Slider (Accumulo on YARN) both solve these problems because they can dynamically start a new instance with the requested configuration – I've been referring to this as "managed" clusters.

            My initial intent is to support running tests against MAC as currently happens (the default won't change), provide some configuration to allow use an existing vanilla Accumulo instance, ability to skip tests which require "managed" clusters, and, if not full support initially, changes which will help Slider integration down the road.

            elserj Josh Elser added a comment - My initial thoughts on how to implement this is to introduce an extra shim between integration tests and the underlying cluster implementation (e.g. SimpleMacIT). Thus, our ITs that need a running instance of Accumulo will extend this new class that provides a "deployment agnostic" set of functions to interact with that instance. Some complexity issues arise around this because some deployments (notably, the plain-jane Accumulo instance) cannot guarantee the breadth of functionality that some tests expect: the inability to restart Accumulo processes, control site configuration files (HDFS or Accumulo), and alter JVM level properties. All other configuration should be capable of modification via use of ZooKeeper through the normal API. MAC and Slider (Accumulo on YARN) both solve these problems because they can dynamically start a new instance with the requested configuration – I've been referring to this as "managed" clusters. My initial intent is to support running tests against MAC as currently happens (the default won't change), provide some configuration to allow use an existing vanilla Accumulo instance, ability to skip tests which require "managed" clusters, and, if not full support initially, changes which will help Slider integration down the road.
            elserj Josh Elser added a comment -

            I did a little bit of injustice. SimpleMacIT does have some intent for its shared internal instance to be something other than a MiniAccumuloCluster, but I think it kind of misses the mark a bit. I hope to take some lessons learned from it.

            elserj Josh Elser added a comment - I did a little bit of injustice. SimpleMacIT does have some intent for its shared internal instance to be something other than a MiniAccumuloCluster, but I think it kind of misses the mark a bit. I hope to take some lessons learned from it.
            elserj Josh Elser added a comment -

            I have a partially reworking implementation that decouples the ITs from MAC and lets most of "basic" ITs run against an existing cluster up at https://github.com/joshelser/accumulo/tree/clusters

            At the moment, tests that will run are ones that stick to the public API, don't try to manipulate the processes or set any special configuration (the low-hanging fruit). Used some basic java properties to configure how to run the tests. Needs a bunch more polishing, but it's a pretty cool start.

            elserj Josh Elser added a comment - I have a partially reworking implementation that decouples the ITs from MAC and lets most of "basic" ITs run against an existing cluster up at https://github.com/joshelser/accumulo/tree/clusters At the moment, tests that will run are ones that stick to the public API, don't try to manipulate the processes or set any special configuration (the low-hanging fruit). Used some basic java properties to configure how to run the tests. Needs a bunch more polishing, but it's a pretty cool start.
            elserj Josh Elser added a comment -

            Attaching reviewboard. Initial patch that switches a number of ITs over to being able to run against a standalone cluster or miniaccumulo cluster with no test code changes.

            elserj Josh Elser added a comment - Attaching reviewboard. Initial patch that switches a number of ITs over to being able to run against a standalone cluster or miniaccumulo cluster with no test code changes.
            elserj Josh Elser added a comment -

            Submitted – a couple of follow on tasks already identified.

            elserj Josh Elser added a comment - Submitted – a couple of follow on tasks already identified.

            People

              elserj Josh Elser
              elserj Josh Elser
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 2h 10m
                  2h 10m