[FALCON-1627] Provider integration with Azure Data Factory pipelines

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.9
• Fix Version/s: 0.10
• Assignee: Ying Zheng
• Reporter: Venkat Ranganathan
• Created: 22/Nov/15 00:46
• Updated: 06/Jun/16 21:35
• Resolved: 19/Feb/16 04:40

Description

Micrsosoft Azure Data Factory is Data processing service available on Microsoft Azure environment which allows users to compose, orchestrate and manage data pipelines somewhat similar to Falcon on the azure environment.

The integration will allow ADF to invoke a Falcon activity (replication, hive or pig proessing work) from the Azure Data Factory data pipeline so that Azure cloud and Hadoop onprem processing can be realized.

The integration work has a few steps and will also be enhanced when ADF V3 protocol is introduced

Sub-Tasks

• Create ADFService for listening to Azure Service Bus messages: Resolved
• Define ADF interop message formats: Resolved
• Create Job Management functionality for ADF messages: Resolved
• Integrate with Proxy user support: Resolved
• Implement the notification service: Resolved
• Tutorial documentation: Resolved
• ADFProviderService should be optional as default setting: Resolved