[HBASE-24800] Enhance ACL region initialization

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: acl, MTTR
• Assignee: Pankaj Kumar
• Reporter: Pankaj Kumar
• Created: 30/Jul/20 12:24
• Updated: 03/Aug/20 03:48

Description

RegionServer persist ACL table entries into Zookeeper during ACL region open,

      private void initialize(RegionCoprocessorEnvironment e) throws IOException {
    final Region region = e.getRegion();
    Configuration conf = e.getConfiguration();
    Map<byte[], ListMultimap<String, UserPermission>> tables = PermissionStorage.loadAll(region);
    // For each table, write out the table's permissions to the respective
    // znode for that table.
    for (Map.Entry<byte[], ListMultimap<String, UserPermission>> t:
      tables.entrySet()) {
      byte[] entry = t.getKey();
      ListMultimap<String, UserPermission> perms = t.getValue();
      byte[] serialized = PermissionStorage.writePermissionsAsBytes(perms, conf);
      zkPermissionWatcher.writeToZookeeper(entry, serialized);
    }
    initialized = true;
  }

Currently RegionServer send 2 RPC (one to create the table path and another to set the data) for each table sequentially.

 try {
      ZKUtil.createWithParents(watcher, zkNode);
      ZKUtil.updateExistingNodeData(watcher, zkNode, permsData, -1);
    } catch (KeeperException e) {
      LOG.error("Failed updating permissions for entry '" +
          entryName + "'", e);
      watcher.abort("Failed writing node "+zkNode+" to zookeeper", e);
    }

If a cluster have huge number of tables then ACL region open will take time. Example, it took ~9 min to write 60k tables ACL into ZK. 

We should send ZK Ops in a single multi() to enhance this.