[REEF-1164] Refactor bridge as IPC between Java and .NET

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: REEF Bridge, REEF Driver, REEF.NET Driver
• Assignee: Unassigned
• Reporter: Anupam
• Created: 27/Jan/16 19:44
• Updated: 24/Jun/16 16:07

Description

REEF driver logic is mainly implemented in Java. It is undesirable to implement driver core logic again and again as it is bug-prone and tricky. Further, YARN and MESOS provide Java based APIs.

For implementing driver in languages other than Java we propose an IPC based driver could be implemented in Java for which multiple remote drivers could be authored in users’ language of choice.

• This would clean up the API, make it less error prone.
• Make it easy to add driver API for a new language.
• Remove the tricky to debug .NET/Java surface area.
• Help us get Linux support for REEF .NET.

Potential break down for the work items:

• API specification for driver protocol in language agnostic manner. Avro is the choice of technology currently in REEF
• Communicate between .NET driver and Java remote driver using above API
• Split .NET driver out of process

Issue Links

• is related to: REEF-334 Make sure REEF.NET builds and runs on Linux