[SQOOP-1897] Sqoop2: Submission Engine API change for better performance

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.99.5
• Assignee: Veena Basavaraj
• Reporter: Veena Basavaraj
• Created: 14/Dec/14 02:36
• Updated: 16/Dec/14 00:55
• Resolved: 16/Dec/14 00:37

Description

/**
   * Return status of given submission.
   *
   * @param externalJobId Submission external job id.
   * @return Current submission status.
   */
  public abstract SubmissionStatus status(String externalJobId);

  /**
   * Return failure info if the job status is FAILED
   *
   * @param submissionId Submission internal id.
   * @return Current failure info
   */
  public abstract SubmissionError error(String externalJobId);

  /**
   * Return submission progress.
   *
   * Expected is number from interval <0, 1> denoting how far the processing
   * has gone or -1 in case that this submission engine do not supports
   * progress reporting.
   *
   * @param externalJobId Submission external job id.
   * @return {-1} union <0, 1>
   */
  public double progress(String externalJobId) {
    return -1;
  }

  /**
   * Return statistics for given submission id.
   *
   * Sqoop will call counters only for submission in state SUCCEEDED,
   * it's consider exceptional state to call this method for other states.
   *
   * @param externalJobId Submission external job id.
   * @return Submission statistics
   */
  public Counters counters(String externalJobId) {
    return null;
  }

  /**
   * Return link to external web page with given submission.
   *
   * @param externalJobId Submission external job id.
   * @return Null in case that external page is not supported or available or
   *  HTTP link to given submission.
   */
  public String externalLink(String externalJobId) {
    return null;
  }

can we collapse all this to one API

  public SubmissionIfo submissionInfo (String externalJobId) {
    return null;
  }

We are making 5-6 calls to the MR engine every time we poll for an update in the updateThread, because we dont hold the instance of RunningJob.

And with spark it may not be the same interface, so this means using externalJobId makes sense

Issue Links

• is a clone of: SQOOP-1879 Sqoop2: Submission Engine does not set all details on SubmissionRecord in Local mode
• links to: RB