Uploaded image for project: 'Sqoop'
  1. Sqoop
  2. SQOOP-1168 Sqoop2: Delta Fetch/ Merge ( formerly called Incremental Import )
  3. SQOOP-1804

Repository changes: Add editable and override attribute to inputs

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.99.5
    • Component/s: None
    • Labels:
      None

      Description

      Details of this proposal are in the wiki.
      https://cwiki.apache.org/confluence/display/SQOOP/Delta+Fetch+And+Merge+Design#DeltaFetchAndMergeDesign-Wheretostoretheoutputinsqoop?
      Update: The above highlights the pros and cons of each approach.

      #4 is chosen, since it is less intrusive, more clean and allows U/Edit per value in the output easily.

      Will use this ticket for more detailed discussion on storage options for the output from connectors
      1.

      // will have FK to submission
       public static final String QUERY_CREATE_TABLE_SQ_JOB_OUTPUT_SUBMISSION =
           "CREATE TABLE " + TABLE_SQ_JOB_OUTPUT + " ("
           + COLUMN_SQ_JOB_OUT_ID + " BIGINT GENERATED ALWAYS AS IDENTITY (START WITH 1, INCREMENT BY 1), "
           + COLUMN_SQ_JOB_OUT_KEY + " VARCHAR(32), "
           + COLUMN_SQ_JOB_OUT_VALUE + " LONG VARCHAR,"
           + COLUMN_SQ_JOB_OUT_TYPE + " VARCHAR(32),"
           + COLUMN_SQD_ID + " VARCHAR(32)," // FK to the direction table, since this allows to distinguish output from FROM/ TO part of the job
         + COLUMN_SQRS_SUBMISSION + " BIGINT, "
         + "CONSTRAINT " + CONSTRAINT_SQRS_SQS + " "
           + "FOREIGN KEY (" + COLUMN_SQRS_SUBMISSION + ") "
             + "REFERENCES " + TABLE_SQ_SUBMISSION + "(" + COLUMN_SQS_ID + ") ON DELETE CASCADE "
      
      
      

      2.
      At the code level, we will define MOutputType, one of the types can be BLOB as well, if a connector decides to store the value as a BLOB

      class JobOutput {
      
      String key;
      Object value;
      MOutputType type;
      
      }
      
      

      3.
      At the repository API, add a new API to get job output for a particular submission Id and allow updates on values.

        Attachments

        1. SQOOP-1804.patch
          111 kB
          Veena Basavaraj
        2. SQOOP-1804-v1.patch
          127 kB
          Veena Basavaraj
        3. SQOOP-1804-v4.patch
          133 kB
          Veena Basavaraj
        4. SQOOP-1804-v5.patch
          133 kB
          Veena Basavaraj
        5. SQOOP-1804-v5-1.patch
          133 kB
          Veena Basavaraj
        6. SQOOP-1804-v6.patch
          134 kB
          Veena Basavaraj
        7. SQOOP-1804-v7.patch
          153 kB
          Veena Basavaraj

          Issue Links

            Activity

              People

              • Assignee:
                vybs Veena Basavaraj
                Reporter:
                vybs Veena Basavaraj
              • Votes:
                0 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: