[SENTRY-2401] FullUpdateInitializer should do a single fetch from HMS instead of 1 per task

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Not A Bug
• Component/s: Sentry
• Assignee: Arjun Mishra
• Reporter: Arjun Mishra
• Created: 13/Sep/18 00:06
• Updated: 09/Jan/19 05:15
• Resolved: 09/Jan/19 05:15

Description

This is an optimization ticket. Currently the way we do fetch is

1. Get all database names from HMS. Create a Db Task for each database name. HMS Request = 1.
2. For each database task
   1. Get all database objects from HMS
   2. Get all table names, for each database
   3. Create a table task for "maxTablesPerCall" tables
3. For each table task
   1. For each table task get all table objects
   2. For each "maxTablesPerCall" tables fetched, get partition names
   3. For each ""maxTablesPerCall" tables fetched, created partition tasks for "maxPartitionsPerCall" partitions
4. For each partition task
   1. Get all partition objects for, database, table, and list of partition names

Instead of splitting connecting to HMS and getting bits of information, it is better to get them all in a single request