Details
-
Sub-task
-
Status: Open
-
Major
-
Resolution: Unresolved
-
None
-
None
-
None
-
None
Description
We want persistent configuration and not an ad-hoc pin API because the latter is both unreliable (e.g. node might go down and restart, so someone has to keep track of global state anyway - might as well have it done explicitly), and also dangerous because errant pin calls can fill cache with garbage and make it unusable.
So, the idea is that we can allow to
1) Give some tables or partitions (tags, see HIVE-16992) a priority boost at cache time.
2) Exclude some tables or partitions from being cached.
Ideally we also want this changeable without LLAP restart (and managed by HS2). So the choice here is either make this a config setting, like many Hadoop settings that have user/group/etc lists, and implement ZK-based/HS2-based configuration changes without restart, that we want to do anyway at some point;
or make this a separate thing, probably managed and stored as policies in metastore (kind of like workload management) and update via ZK.
It's possible to go with config settings that do require restart for the first cut.
Separately, we can add a workload management policy that would disable cache for certain query pools (i.e. users/groups/apps).