[ATLAS-510] High availability of Atlas

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Done
• Assignee: Unassigned
• Reporter: Hemanth Yamijala
• Created: 23/Feb/16 06:52
• Updated: 18/May/20 04:07
• Resolved: 18/May/20 04:07

Description

There has been some work done at various levels to improve availability of Atlas. Architecturally, using highly available dependent components like HBase, Solr and Kafka allows Atlas to rely on the HA capabilities of these components (like replicas) to ensure it is not impacted by single machine failures. Integrating components (like Hive, Sqoop, Falcon, Storm) use Kafka to integrate with Atlas. In principle, this allows the integrating components to not be impacted due to loss of connectivity with Atlas while ensuring no loss of data.

However, there are still some important changes that the system requires for improving availability guarantees of Atlas. This JIRA is a master JIRA to track this work. I expect to open several child JIRAs to elaborate on the tasks and make it more manageable from a development / review perspective.

Issue Links

• is related to: ATLAS-488 Atlas service scalability

Sub-Tasks

• Ability to run multiple instances of Atlas Server with automatic failover to one active server: Resolved
• Decouple currently integrating components from availability of Atlas service for raising metadata events: Resolved
• Admin support for HA: Resolved
• Monitoring and alerts for HA: Open
• Ability to initialize Kafka topics with more than 1 replica: Resolved
• Ability to run multiple Atlas dev instances on the same host for dev testing of HA: Resolved
• Ability to cascade failover in an multiple instance HA cluster: Resolved
• Modify Atlas client for necessary changes in context of HA: Resolved
• Handle secure instance of Zookeeper for leader election.: Resolved
• Document High Availability of Atlas: Resolved