Uploaded image for project: 'Accumulo'
  1. Accumulo
  2. ACCUMULO-3842

[UMBRELLA] Remove non-transient data from ZooKeeper

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: client, tserver
    • Labels:
      None

      Description

      Wanted to start brainstorming about this.

      We store a lot of persistent data in ZooKeeper that would better stored in something backed by HDFS. ZooKeeper can be a very convenient place to store persisted data so that it's available to all nodes, but it comes at a price and often must be asynchronously accessed to achieve good performance.

      • Table/Namespace configuration
      • Users/Authorizations
      • Problem reports (maybe?)
      • System configuration overrides (maybe?)

      Some benefits we'd see from this:

      • Loss of ZooKeeper doesn't lose table configuration and users.
      • Greatly reduce zookeeper watchers (assume watchers=50*num_tables*num_tservers)
      • Consistent updates of table constraints and all other table properties

      The last note is the most important one IMO. The number of test issues alone that we've had with constraints not being seen on all servers are bound to affect users.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                elserj Josh Elser
              • Votes:
                1 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated: