Uploaded image for project: 'Atlas'
  1. Atlas
  2. ATLAS-3416

Import API: delete non-exported hive_table entities for table level replication

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • trunk
    • None
    • None

    Description

      This will delete hive_table entities which are present in the hive_db but are not the part of the incremental export request.
      This enhances functionality provided by ATLAS-3324.

      Background:
      Requirement for import API is that if request contains "replicatedFrom" option, in this case import API should find & delete tables which are not imported but are part of database entity.
      E.g. Consider t1...t5 under db1 replicated from source to destination using table level incremental export from source.
      Now suppose subsequent import request at destination contains only t1...t3, in such case t4 & t5 must be deleted at destination.

      Solution:

      • Check if import request contains "replicatedFrom" option.
      • Extract exportRequest from imported zip.
      • Identify table entities which are imported using exportRequest.
      • Identify database entity using imported table entities.
      • Identify table entities which are part of database entity obtained in previous step but are not imported in current import request.
      • Delete tables.

      Attachments

        1. repl_exp_2.zip
          11 kB
          Nikhil P Bonte
        2. repl_exp_1.zip
          14 kB
          Nikhil P Bonte
        3. Import-API-delete-non-exported-hive_table-for-table-v11.patch
          31 kB
          Nikhil P Bonte
        4. Import-API-delete-non-exported-hive_table-for-table-v2-branch-0.8.patch
          30 kB
          Nikhil P Bonte

        Activity

          People

            nbonte Nikhil P Bonte
            nbonte Nikhil P Bonte
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: