[RANGER-3923] Dataset policies

• Type: New Feature
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Component/s: Ranger
• Assignee: Madhan Neethiraj
• Reporter: Madhan Neethiraj
• Created: 20/Sep/22 21:06
• Updated: 20/Oct/23 12:44

Description

Given the primary business value of Apache Ranger is to enable sharing of resources, it will help if Apache Ranger provides an abstraction that enables a set of resources/data across services, a dataset, to be the unit of sharing instead of one or more resources in each service. This has several benefits, like:

1. A single policy to manage access to data in multiple services - like HBase, Hive, Snowflake, Kafka, Google BigQuery, AWS S3, AWS Redshift, ADLS-Gen2. This enables authorization to be centered around a purpose, like:
   • Marketing Campaign 2022 dataset
   • Sales 2021 dataset
   • CA Claims 2021 dataset
2. Enables different set of users to manage sharing data into a dataset and manage access to the data in a dataset:
   • Data owners share data into a dataset, with necessary masking,  row-filters and schedules; they can update the share details, including stop sharing into a dataset.
   • Dataset admins manage who has access to the data in the dataset. This relieves data owners from having to micromanage access to the shared data, for example when a user needs access to the data in multiple services to participate in a project.

Attached document has more details on this new abstraction, including a number of questions & answers that to help understand various aspects of this feature. Please read and add your comments/suggestions.

Sub-Tasks

• update ServicePolicies to include GDS policies relevant to a service: Resolved
• REST APIs for plugins to receive GDS data relevant to a service: Resolved
• Link resource-based services with GDS service: Resolved
• Increment policy-version of linked services on change to GDS policy: Resolved
• GDS context-enricher to find datasets relevant to accessed resource: Resolved
• Policy engine enhancements to evaluate GDS policies: Resolved
• Audit log to capture datasets and projects: Resolved
• UI for GDS: dashboard, datasets, projects, data-shares: In Progress
• GDS Download API with Cache: Resolved
• Java patches to support GDS changes: Resolved
• Dataset lookup is failing in GDS policy: Resolved
• API to get DataShare id,name,description List: Resolved
• Need API to get list of Datashare/Dataset request for current logged in user: Open
• Support for delta download of GDS data to plugin: Open
• Need a new API to get DataShareInDataset request details to be displayed in UI: Open
• Need additional filters in GET DataShare,Dataset API: Resolved
• Fix pagination for GDS APIs: Resolved
• Need a new API to create multiple datashare requests for a dataset: Resolved
• Need a new API to get datashares to be dispalyed on pop up for Add Request: Resolved
• Add default policy for new dataset: Resolved
• Add support for filter by service and zone name in get datashare API: Closed
• Need a new API to get history of operations on dataset/datashare requests: Open
• Add field approver in dataset request entities: Closed
• Need a new API to get list of datasets for a datashare with request status: Resolved
• Need a new API to get list of zones mapped to a service: Resolved
• Create datshare and sharedResource in cascade: Resolved
• Need additional filters for GET sharedResources API: Resolved
• Need a new API for service lookup: Resolved
• Implement ACL for get API for datashare by id: Open
• Need a new API to get list of requests for the current user: Resolved
• Support cascading delete for datashare: Closed
• Support cascading delete for dataset: Resolved
• Add loggedIn user as the admin in dataset/datshare ACL: Closed
• Implement request state transition evaluation for DataShareinDataset and DatasetInProject: Open
• When security-zone is deleted with force, trigger cascade delete of datashare: Resolved
• GDS APIs to manage policies: Resolved
• Need an API to return dataset summary: Resolved
• Need a new API to get security-zone summary: Closed
• GDS CRUD APIs should return proper response and message in case of failure: Resolved
• Dataset with ACL for "public" group List permission are not listed if the user is not added to public: Closed
• Need a new API to get datasets for a resource name like db or column name: Open
• Need additional filter on GET sharedResource API: Resolved
• GDS validation - service/zone admins not allowed to update/delete dataShares: Resolved
• PrincipalCount not getting updated in DatasetSummary: Resolved
• Need a new API to get dataShare summary: Resolved
• Update approver when request status is updated: Resolved
• Dataset summary API throws 403: Resolved
• GDS data structure updates to capture masking order and sub-resource type: Resolved
• Sort param sortType not considered if sortBy not passed: Resolved
• Generate trx-logs for dataset/project/data-share/shared-resource CRUD: Resolved
• support for policies to refer access-types using category, like Create/Read/Update/Delete/Manage: Resolved
• Dataset/Datashare/Shared-resource/project name should not be created with trailing spaces: Resolved
• Need new API for DataShare in Dataset Summary: Resolved
• Need new API for DataShare in Dataset Summary: Resolved
• update service-defs to assign category for access-types: Resolved
• GDS --> newly created user with admin role doesn't get permission for GDS module automatically: Resolved
• NPE in RangerGdsInfoRefresher.run(): Resolved
• GDS update APIs fail when guid is not included in payload: Resolved
• update GdsPolicyEngine with methods to find datasets/projects for principals: Resolved
• GET dataset API should return public:LIST in ACL if available: Resolved
• Audit UI updates to support datasets and projects: Resolved
• upgrade audit schema ,in Solr and Elasticsearch, to support datasets and projects: Open
• Plugin status should track GDS info download details as well: Resolved
• Plugin status UI should render GDS download details: Resolved
• Need api to get count of all request list according to their status and the total count of requests.: Resolved
• Java patch to assign category to existing accessTypeDefs in Ranger database: Resolved
• API to support retrieval of datasets/projects shared with the caller: Resolved
• Need to add query param createdBy for dataset GET API: Resolved
• Add java patch to update changes in db: Resolved
• Delete GDS objects on security-zone update/delete: Resolved
• [Ranger UI] Clicking on the policy Id in the access audits (audits related to gds) leads to an error: Resolved
• Add Unit tests for GDS related implementation files: Open
• [Ranger UI] Text overflowing in the datashares table listing in the dataset -> datashares page: Resolved
• Return proper error message if GDS entities name is longer than 512 characters and assertions in the UI form as well: Resolved
• Value for datasets and projects field in the access audits must be JSON serializable: Resolved
• [Ranger React UI] Handle Dataset and Datashare creation errors gracefully: Resolved
• Hive: USE fails even though user has access in the database via GDS policies: Resolved
• Restrict the use of special characters (other than _ and - ) in GDS entity names: Open
• update GDS Python client to support forceDelete flag: Resolved
• Ranger Reports doesn't show the created policy for the "gds" service type: In Progress
• support scheduling of a dataset to expire at a specific time: Resolved