[FALCON-327] Simplify message passing framework

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.5
• Fix Version/s: 0.6
• Component/s: messaging
• Labels: messaging
• Assignee: Venkatesh Seetharam
• Reporter: Venkatesh Seetharam
• Created: 02/Mar/14 03:33
• Updated: 03/Sep/14 18:11
• Resolved: 08/Aug/14 20:34

Description

Issues with the current implementation:

• hard to evolve the schema
  If I need to add one attribute, requires change in various places
• Too many enum classes for the same variable
  Confuses the heck out of me. Some small, some caps
• FalconPostProcessing gets args, parses the args into CLI and converts 'em back into args repeatedly
  Too much redundant processing
• Timestamp should be long as opposed to a String - minor?
  I need to compare dates and thought long is easier instead of constructing expensive SimpleDateFormats
• Hard dependency on JMS.

Suggest the following:

• Have the payload in a Map serialized as JSON

• wonder how to pass this from oozie

• Have one central Enum class for the keys in the payload
• Each class now depends on this Enum and takes what it needs from the Map

We also could rethink about the current messaging which falcon relies on (had started a discuss thread but did not get any response):

• Continue to use JMS
• Switch to FS Polling
• Use both

Thoughts?

Issue Links

• blocks: FALCON-222 Capturing Oozie logs post workflow completion needs to move into Falcon daemon
• blocks: FALCON-476 Introduce database to persist falcon materialized workflow status in workflow engine
• is blocked by: FALCON-287 Record lineage information in post processing
• is related to: FALCON-489 Consume ProcessSubscriberService into WorkflowJobEndNotificationService
• is related to: FALCON-490 Create a new Polling Subscriber to poll hdfs for job end events in WorkflowJobEndNotificationService
• is related to: FALCON-491 Add Rerun context in Rerun framework
• is related to: FALCON-665 Handle message consumption failures in JMSMessageConsumer

Sub-Tasks

• Refactor workflow related code into context, listener and Notification: Resolved
• Simplify JMS Message Sender/Consumer and use Workflow Context: Resolved
• Introduce Workflow Context in Post Processing: Resolved
• Introduce Workflow Context in Rerun framework: Resolved
• Introduce Workflow Context in Lineage Service: Resolved
• Refactor workflow listener implementation init config driven: Resolved