[SQOOP-1603] Sqoop2: Explicit support for Merge in the Sqoop Job lifecyle in the MR engine

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Fix Version/s: 2.0.0
• Assignee: Unassigned
• Reporter: Veena Basavaraj
• Created: 22/Oct/14 17:21
• Updated: 06/Mar/15 16:14

Description

The Destroyer api and its javadoc

/**
 * This allows connector to define work to complete execution, for example,
 * resource cleaning.
 */
public abstract class Destroyer<LinkConfiguration, JobConfiguration> {

  /**
   * Callback to clean up after job execution.
   *
   * @param context Destroyer context
   * @param linkConfiguration link configuration object
   * @param jobConfiguration job configuration object for the FROM and TO
   *        In case of the FROM initializer this will represent the FROM job configuration
   *        In case of the TO initializer this will represent the TO job configuration
   */
  public abstract void destroy(DestroyerContext context,
                               LinkConfiguration linkConfiguration,
                               JobConfiguration jobConfiguration);

}

This ticket was created while reviewing the Kite Connector use case where the destroyer does the actual temp data set merge
https://reviews.apache.org/r/26963/diff/# stanleyxu2005

public void destroy(DestroyerContext context, LinkConfiguration link,
      ToJobConfiguration job) {
    LOG.info("Running Kite connector destroyer");
    // Every loader instance creates a temporary dataset. If the MR job is
    // successful, all temporary dataset should be merged as one dataset,
    // otherwise they should be deleted all.
    String[] uris = KiteDatasetExecutor.listTemporaryDatasetUris(
        job.toDataset.uri);
    if (context.isSuccess()) {
      KiteDatasetExecutor executor = new KiteDatasetExecutor(job.toDataset.uri,
          context.getSchema(), link.link.fileFormat);
      for (String uri : uris) {
        executor.mergeDataset(uri);
        LOG.info(String.format("Temporary dataset %s merged", uri));
      }
    } else {
      for (String uri : uris) {
        KiteDatasetExecutor.deleteDataset(uri);
        LOG.info(String.format("Temporary dataset %s deleted", uri));
      }
    }
  }

Wondering if such things should be its own phase rather than in destroyers. The responsibility of destroyer is more to clean up/ closing/ daat sources for both FROM/TO data sources to be more precise .. should such operations that modify records / merge/ munge be its own step ?.

Issue Links

• is a clone of: SQOOP-1602 Sqoop2: Fix the current balancing across Loaders internal to Sqoop
• is related to: SQOOP-1601 Sqoop2: To part of the Connector API to support balancing/ re-partioning step