Accumulo
  1. Accumulo
  2. ACCUMULO-2984

Enable running MAC using an existing instance

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.6.2, 1.7.0
    • Component/s: mini
    • Labels:
      None

      Description

      Currently MAC expects to run it's own zookeeper and do initialization of the instance for the given site.xml. I believe with a few minor changes, it could also be used to point against a live instance. Mainly, this requires using the existing accumulo-site.xml file in lieu of the pseudo-provided/generated one and not starting it's own ZK.

        Issue Links

          Activity

          John Vines created issue -
          Hide
          John Vines added a comment - - edited

          I would like this in 1.6.1 (or 1.6.x in general) and I'm going to attempt to get it to land there unless it means API alterations.

          Show
          John Vines added a comment - - edited I would like this in 1.6.1 (or 1.6.x in general) and I'm going to attempt to get it to land there unless it means API alterations.
          John Vines made changes -
          Field Original Value New Value
          Fix Version/s 1.6.1 [ 12325441 ]
          Hide
          Sean Busbey added a comment -

          is the plan for this to be another implementation of the MiniAccumuloCluster class ala Josh's work to have a cluster that spins up in yarn? or is the plan for it to be alterations on MiniAccumuloClusterImpl?

          I'd prefer the former if you're looking to run against an Accumulo config rather than e.g. just running against an existing HDFS/ZK.

          Show
          Sean Busbey added a comment - is the plan for this to be another implementation of the MiniAccumuloCluster class ala Josh's work to have a cluster that spins up in yarn? or is the plan for it to be alterations on MiniAccumuloClusterImpl? I'd prefer the former if you're looking to run against an Accumulo config rather than e.g. just running against an existing HDFS/ZK.
          Hide
          John Vines added a comment -

          The latter. This is exactly like existing MAC, but just a case of A. providing an external ZK B. (may be optional) providing an external site.xml file and C. telling it not to try to initialize.

          Why would you prefer the former?

          Show
          John Vines added a comment - The latter. This is exactly like existing MAC, but just a case of A. providing an external ZK B. (may be optional) providing an external site.xml file and C. telling it not to try to initialize. Why would you prefer the former?
          Hide
          Sean Busbey added a comment -

          Because running on multiple machines is a substantially different behavior. I'd like to minimize the risk around a partial implementation delaying 1.6.1.

          FWIW, I really want something like this so I can run the ITs against a real cluster. When just trying to get MAC to use an external HDFS / MR we ran into ACCUMULO-2855. That'll probably have to be fixed as a part of this.

          Show
          Sean Busbey added a comment - Because running on multiple machines is a substantially different behavior. I'd like to minimize the risk around a partial implementation delaying 1.6.1. FWIW, I really want something like this so I can run the ITs against a real cluster. When just trying to get MAC to use an external HDFS / MR we ran into ACCUMULO-2855 . That'll probably have to be fixed as a part of this.
          Hide
          John Vines added a comment -

          This ticket isn't about running ITs against a real cluster, that sounds like something related to Josh's work you referenced before. Work that is orthogonal to this. This ticket is about running an fake cluster in memory against a real instance, not about running a real cluster against a fake instance.

          As for ACCUMULO-2855, that's a non-issue because this is about NOT using a generated site.xml, but rather using a provided one.

          Show
          John Vines added a comment - This ticket isn't about running ITs against a real cluster, that sounds like something related to Josh's work you referenced before. Work that is orthogonal to this. This ticket is about running an fake cluster in memory against a real instance, not about running a real cluster against a fake instance. As for ACCUMULO-2855 , that's a non-issue because this is about NOT using a generated site.xml, but rather using a provided one.
          Hide
          Sean Busbey added a comment -

          so is this essentially just allowing for the creation of a MiniAccumuloConfig based on a provided accumulo-site.xml?

          Show
          Sean Busbey added a comment - so is this essentially just allowing for the creation of a MiniAccumuloConfig based on a provided accumulo-site.xml?
          Hide
          John Vines added a comment -

          That + not starting ZK (because it's already running) + not initializing it (because it's already initialized)

          Show
          John Vines added a comment - That + not starting ZK (because it's already running) + not initializing it (because it's already initialized)
          Hide
          Sean Busbey added a comment -

          that addresses my concerns then.

          Show
          Sean Busbey added a comment - that addresses my concerns then.
          Hide
          John Vines added a comment -

          Beautiful

          Show
          John Vines added a comment - Beautiful
          John Vines made changes -
          Status Open [ 1 ] In Progress [ 3 ]
          John Vines made changes -
          Link This issue is broken by ACCUMULO-2985 [ ACCUMULO-2985 ]
          John Vines made changes -
          Remote Link This issue links to "review board link (Web Link)" [ 15832 ]
          Corey J. Nolet made changes -
          Fix Version/s 1.6.2 [ 12328644 ]
          Fix Version/s 1.6.1 [ 12325441 ]
          ASF subversion and git services logged work - 21/Oct/14 20:41
          • Time Spent:
            10m
             
            Commit e3a743cb445723a3d5664a4bf1ebf37833152aae in accumulo's branch refs/heads/1.6 from [~vines]
            [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=e3a743c ]

            ACCUMULO-2984 adding ability to run MAC against a permanent accumulo instance

            Addressing NPEs with existingInstance

            ACCUMULO-2984 add test for existing mac using existing instance

            Fixing my NPE fix and integrating Keith's test

            Adding ability to clean out singleton stuff

            Now no longer doing weird things with statics

            Warning cleanup

            Removing added features no longer needed

            Cleaning up minor ticket items

            Fixing another Configuration plumbing issue and adding some info statements
          ASF subversion and git services made changes -
          Worklog Id 18334 [ 18334 ]
          Remaining Estimate 0h [ 0 ]
          Time Spent 10m [ 600 ]
          ASF subversion and git services logged work - 21/Oct/14 20:41
          • Time Spent:
            10m
             
            Commit e3a743cb445723a3d5664a4bf1ebf37833152aae in accumulo's branch refs/heads/1.6 from [~vines]
            [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=e3a743c ]

            ACCUMULO-2984 adding ability to run MAC against a permanent accumulo instance

            Addressing NPEs with existingInstance

            ACCUMULO-2984 add test for existing mac using existing instance

            Fixing my NPE fix and integrating Keith's test

            Adding ability to clean out singleton stuff

            Now no longer doing weird things with statics

            Warning cleanup

            Removing added features no longer needed

            Cleaning up minor ticket items

            Fixing another Configuration plumbing issue and adding some info statements
          ASF subversion and git services made changes -
          Time Spent 10m [ 600 ] 20m [ 1200 ]
          Worklog Id 18335 [ 18335 ]
          ASF subversion and git services logged work - 21/Oct/14 20:47
          • Time Spent:
            10m
             
            Commit e3a743cb445723a3d5664a4bf1ebf37833152aae in accumulo's branch refs/heads/master from [~vines]
            [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=e3a743c ]

            ACCUMULO-2984 adding ability to run MAC against a permanent accumulo instance

            Addressing NPEs with existingInstance

            ACCUMULO-2984 add test for existing mac using existing instance

            Fixing my NPE fix and integrating Keith's test

            Adding ability to clean out singleton stuff

            Now no longer doing weird things with statics

            Warning cleanup

            Removing added features no longer needed

            Cleaning up minor ticket items

            Fixing another Configuration plumbing issue and adding some info statements
          ASF subversion and git services made changes -
          Time Spent 20m [ 1200 ] 0.5h [ 1800 ]
          Worklog Id 18336 [ 18336 ]
          ASF subversion and git services logged work - 21/Oct/14 20:47
          • Time Spent:
            10m
             
            Commit e3a743cb445723a3d5664a4bf1ebf37833152aae in accumulo's branch refs/heads/master from [~vines]
            [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=e3a743c ]

            ACCUMULO-2984 adding ability to run MAC against a permanent accumulo instance

            Addressing NPEs with existingInstance

            ACCUMULO-2984 add test for existing mac using existing instance

            Fixing my NPE fix and integrating Keith's test

            Adding ability to clean out singleton stuff

            Now no longer doing weird things with statics

            Warning cleanup

            Removing added features no longer needed

            Cleaning up minor ticket items

            Fixing another Configuration plumbing issue and adding some info statements
          ASF subversion and git services made changes -
          Time Spent 0.5h [ 1800 ] 40m [ 2400 ]
          Worklog Id 18337 [ 18337 ]
          John Vines made changes -
          Status In Progress [ 3 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          ASF subversion and git services logged work - 27/Oct/14 21:29
          ASF subversion and git services made changes -
          Time Spent 40m [ 2400 ] 50m [ 3000 ]
          Worklog Id 18363 [ 18363 ]
          ASF subversion and git services logged work - 27/Oct/14 21:29
          ASF subversion and git services made changes -
          Time Spent 50m [ 3000 ] 1h [ 3600 ]
          Worklog Id 18364 [ 18364 ]
          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open In Progress In Progress
          1h 47m 1 John Vines 08/Jul/14 21:53
          In Progress In Progress Resolved Resolved
          104d 22h 54m 1 John Vines 21/Oct/14 20:47

            People

            • Assignee:
              John Vines
              Reporter:
              John Vines
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 1h
                1h

                  Development