Uploaded image for project: 'Atlas'
  1. Atlas
  2. ATLAS-1665

Export API: Improve Generated ZIP File Using AtlasEntityWithExtInfo

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 1.0.0
    • 0.8.1, 1.0.0
    • atlas-core
    • None

    Description

      Background

      Existing implementation of Export API w.r.t ZIP file generation adds 1 .json file per entity. This makes ZIP file creation inefficient. The ZIP files are 75% larger in size than what could be possible with fewer .json file entries.

      Solution

      The implementation uses the new v2 API AtlasEntityWithExtInfo representation instead of AtlasEntity. This format combines an entity with related entities as one. E.g. hive_table will contain all the hive_columns that it is made up of. (See example section below.)

      This results in significant reduction of generated JSON files. This impacts reduction in generated ZIP file.

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            amestry Ashutosh Mestry
            amestry Ashutosh Mestry
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment