Description
Currently, the spark-security module is tightly coupled with "HiveTableRelation" and "MetastoreRelation" for authorization of non-transactional hive tables.
This does not work for Hive transactional (ACID enabled) table. Since Hive has changed the way data/metadata is stored for transactional tables when compared to non-transactional tables. Therefore, Spark can not read Hive transactional tables directly.
So even if security module may enforce security on such tables, spark can't actually read anything.
Reading hive's transactional tables in spark, needs Spark-Acid, implemented as Spark Datasource. Since security module is tightly coupled with "HiveTableRelation" and "MetastoreRelation", it does not provides authorization support for any datasource.
The idea is to support Spark datasource authorization (which has db/table/column/partitions analogous to hive).
We can create generic interfaces for datasource, which each datasource can implement and then it can be authorized using the existing codebase.
Attachments
Issue Links
- is related to
-
SUBMARINE-409 Submarine spark security
-
- Resolved
-