Uploaded image for project: 'Ranger'
  1. Ranger
  2. RANGER-3923

Dataset policies

Attach filesAttach ScreenshotAdd voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • New Feature
    • Status: In Progress
    • Major
    • Resolution: Unresolved
    • None
    • None
    • Ranger
    • None

    Description

      Given the primary business value of Apache Ranger is to enable sharing of resources, it will help if Apache Ranger provides an abstraction that enables a set of resources/data across services, a dataset, to be the unit of sharing instead of one or more resources in each service. This has several benefits, like:

      1. A single policy to manage access to data in multiple services - like HBase, Hive, Snowflake, Kafka, Google BigQuery, AWS S3, AWS Redshift, ADLS-Gen2. This enables authorization to be centered around a purpose, like:
        • Marketing Campaign 2022 dataset
        • Sales 2021 dataset
        • CA Claims 2021 dataset
      2. Enables different set of users to manage sharing data into a dataset and manage access to the data in a dataset:
        • Data owners share data into a dataset, with necessary masking,  row-filters and schedules; they can update the share details, including stop sharing into a dataset.
        • Dataset admins manage who has access to the data in the dataset. This relieves data owners from having to micromanage access to the shared data, for example when a user needs access to the data in multiple services to participate in a project.

      Attached document has more details on this new abstraction, including a number of questions & answers that to help understand various aspects of this feature. Please read and add your comments/suggestions.

      Attachments

        1.
        update ServicePolicies to include GDS policies relevant to a service Sub-task Resolved Madhan Neethiraj Actions
        2.
        REST APIs for plugins to receive GDS data relevant to a service Sub-task Resolved Madhan Neethiraj Actions
        3.
        Link resource-based services with GDS service Sub-task Resolved Madhan Neethiraj Actions
        4.
        Increment policy-version of linked services on change to GDS policy Sub-task Resolved Madhan Neethiraj Actions
        5.
        GDS context-enricher to find datasets relevant to accessed resource Sub-task Resolved Madhan Neethiraj Actions
        6.
        Policy engine enhancements to evaluate GDS policies Sub-task Resolved Madhan Neethiraj Actions
        7.
        Audit log to capture datasets and projects Sub-task Resolved Madhan Neethiraj Actions
        8.
        UI for GDS: dashboard, datasets, projects, data-shares Sub-task In Progress Anand Nadar Actions
        9.
        GDS Download API with Cache Sub-task Resolved Subhrat Chaudhary Actions
        10.
        Java patches to support GDS changes Sub-task Resolved Monika kachhadiya Actions
        11.
        Dataset lookup is failing in GDS policy Sub-task Resolved Unassigned Actions
        12.
        API to get DataShare id,name,description List Sub-task Resolved Prashant Satam Actions
        13.
        Need API to get list of Datashare/Dataset request for current logged in user Sub-task Open Subhrat Chaudhary Actions
        14.
        Support for delta download of GDS data to plugin Sub-task Open Unassigned Actions
        15.
        Need a new API to get DataShareInDataset request details to be displayed in UI Sub-task Open Subhrat Chaudhary Actions
        16.
        Need additional filters in GET DataShare,Dataset API Sub-task Resolved Prashant Satam Actions
        17.
        Fix pagination for GDS APIs Sub-task Resolved Subhrat Chaudhary Actions
        18.
        Need a new API to create multiple datashare requests for a dataset Sub-task Resolved Subhrat Chaudhary Actions
        19.
        Need a new API to get datashares to be dispalyed on pop up for Add Request Sub-task Resolved Unassigned Actions
        20.
        Add default policy for new dataset Sub-task Resolved Prashant Satam Actions
        21.
        Add support for filter by service and zone name in get datashare API Sub-task Closed Subhrat Chaudhary Actions
        22.
        Need a new API to get history of operations on dataset/datashare requests Sub-task Open Abhishek Actions
        23.
        Add field approver in dataset request entities Sub-task Closed Subhrat Chaudhary Actions
        24.
        Need a new API to get list of datasets for a datashare with request status Sub-task Resolved Prashant Satam Actions
        25.
        Need a new API to get list of zones mapped to a service Sub-task Resolved Prashant Satam Actions
        26.
        Create datshare and sharedResource in cascade Sub-task Resolved Unassigned Actions
        27.
        Need additional filters for GET sharedResources API Sub-task Resolved Unassigned Actions
        28.
        Need a new API for service lookup Sub-task Resolved Subhrat Chaudhary Actions
        29.
        Implement ACL for get API for datashare by id Sub-task Open Unassigned Actions
        30.
        Need a new API to get list of requests for the current user Sub-task Resolved Unassigned Actions
        31.
        Support cascading delete for datashare Sub-task Closed Subhrat Chaudhary Actions
        32.
        Support cascading delete for dataset Sub-task Resolved Prashant Satam Actions
        33.
        Add loggedIn user as the admin in dataset/datshare ACL Sub-task Closed Prashant Satam Actions
        34.
        Implement request state transition evaluation for DataShareinDataset and DatasetInProject Sub-task Open Unassigned Actions
        35.
        When security-zone is deleted with force, trigger cascade delete of datashare Sub-task Resolved Madhan Neethiraj Actions
        36.
        GDS APIs to manage policies Sub-task Resolved Madhan Neethiraj Actions
        37.
        Need an API to return dataset summary Sub-task Resolved Subhrat Chaudhary Actions
        38.
        Need a new API to get security-zone summary Sub-task Closed Subhrat Chaudhary Actions
        39.
        GDS CRUD APIs should return proper response and message in case of failure Sub-task Resolved Prashant Satam Actions
        40.
        Dataset with ACL for "public" group List permission are not listed if the user is not added to public Sub-task Closed Prashant Satam Actions
        41.
        Need a new API to get datasets for a resource name like db or column name Sub-task Open Abhishek Actions
        42.
        Need additional filter on GET sharedResource API Sub-task Resolved Subhrat Chaudhary Actions
        43.
        GDS validation - service/zone admins not allowed to update/delete dataShares Sub-task Resolved Madhan Neethiraj Actions
        44.
        PrincipalCount not getting updated in DatasetSummary Sub-task Resolved Subhrat Chaudhary Actions
        45.
        Need a new API to get dataShare summary Sub-task Resolved Subhrat Chaudhary Actions
        46.
        Update approver when request status is updated Sub-task Resolved Subhrat Chaudhary Actions
        47.
        Dataset summary API throws 403 Sub-task Resolved Subhrat Chaudhary Actions
        48.
        GDS data structure updates to capture masking order and sub-resource type Sub-task Resolved Madhan Neethiraj Actions
        49.
        Sort param sortType not considered if sortBy not passed Sub-task Resolved Subhrat Chaudhary Actions
        50.
        Generate trx-logs for dataset/project/data-share/shared-resource CRUD Sub-task Resolved Madhan Neethiraj Actions
        51.
        support for policies to refer access-types using category, like Create/Read/Update/Delete/Manage Sub-task Resolved Madhan Neethiraj Actions
        52.
        Dataset/Datashare/Shared-resource/project name should not be created with trailing spaces Sub-task Resolved Prashant Satam Actions
        53.
        Need new API for DataShare in Dataset Summary Sub-task Resolved Prashant Satam Actions
        54.
        Need new API for DataShare in Dataset Summary Sub-task Resolved Unassigned Actions
        55.
        update service-defs to assign category for access-types Sub-task Resolved Madhan Neethiraj Actions
        56.
        GDS --> newly created user with admin role doesn't get permission for GDS module automatically Sub-task Resolved Abhishek Actions
        57.
        NPE in RangerGdsInfoRefresher.run() Sub-task Resolved Madhan Neethiraj Actions
        58.
        GDS update APIs fail when guid is not included in payload Sub-task Resolved Madhan Neethiraj Actions
        59.
        update GdsPolicyEngine with methods to find datasets/projects for principals Sub-task Resolved Madhan Neethiraj Actions
        60.
        GET dataset API should return public:LIST in ACL if available Sub-task Resolved Subhrat Chaudhary Actions
        61.
        Audit UI updates to support datasets and projects Sub-task Resolved Abhishek Actions
        62.
        upgrade audit schema ,in Solr and Elasticsearch, to support datasets and projects Sub-task Open Unassigned Actions
        63.
        Plugin status should track GDS info download details as well Sub-task Resolved Madhan Neethiraj Actions
        64.
        Plugin status UI should render GDS download details Sub-task Resolved Brijesh Bhalala Actions
        65.
        Need api to get count of all request list according to their status and the total count of requests. Sub-task Resolved Unassigned Actions
        66.
        Java patch to assign category to existing accessTypeDefs in Ranger database Sub-task Resolved Madhan Neethiraj Actions
        67.
        API to support retrieval of datasets/projects shared with the caller Sub-task Resolved Madhan Neethiraj Actions
        68.
        Need to add query param createdBy for dataset GET API Sub-task Resolved Prashant Satam Actions
        69.
        Add java patch to update changes in db Sub-task Resolved Monika kachhadiya Actions
        70.
        Delete GDS objects on security-zone update/delete Sub-task Resolved Madhan Neethiraj Actions
        71.
        [Ranger UI] Clicking on the policy Id in the access audits (audits related to gds) leads to an error Sub-task Resolved Abhishek Actions
        72.
        Add Unit tests for GDS related implementation files Sub-task Open Abhishek Actions
        73.
        [Ranger UI] Text overflowing in the datashares table listing in the dataset -> datashares page Sub-task Resolved Abhishek Actions
        74.
        Return proper error message if GDS entities name is longer than 512 characters and assertions in the UI form as well Sub-task Resolved Abhishek Actions
        75.
        Value for datasets and projects field in the access audits must be JSON serializable Sub-task Resolved Abhishek Actions
        76.
        [Ranger React UI] Handle Dataset and Datashare creation errors gracefully Sub-task Resolved Abhishek Actions
        77.
        Hive: USE fails even though user has access in the database via GDS policies Sub-task Resolved Madhan Neethiraj Actions
        78.
        Restrict the use of special characters (other than _ and - ) in GDS entity names Sub-task Open Abhishek Actions
        79.
        update GDS Python client to support forceDelete flag Sub-task Resolved Madhan Neethiraj Actions

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            madhan Madhan Neethiraj
            madhan Madhan Neethiraj

            Dates

              Created:
              Updated:

              Slack

                Issue deployment