[REEF-2020] Refactor IJobSubmissionResult Interface and JobSubmissionResult implementation

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: REEF.NET Client
• Assignee: Unassigned
• Reporter: Chenxi Zhao
• Created: 21/May/18 22:05
• Updated: 21/May/18 22:54

Description

There are several defects in Org.Apache.REEF.Client.Common.IJobSubmissionResult needs to be improved:

1. It contains fields work for YARN only - FinalState, AppId, which does not serve the purpose of abstraction of different runtimes - YARN, Local, Azure Batch.
2. In the runtime of Azure Batch, there is no capability of connecting to driver directly. Abstraction of DriverUrl as string needs to be improved.

 

In its implementation JobSubmissionResult, several problems noticed:

1. Retry handling that serves for all runtime is using YARN retry policy.
2. IREEFClient instance passed in to get YARN state.
3. Several unused methods needs to be cleaned up.
4. WaitForDriverToFinish() will generate WebException if Driver task is completed and HTTP server is closed.