Details
-
Sub-task
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
-
None
-
None
Description
Details of this proposal are in the wiki.
https://cwiki.apache.org/confluence/display/SQOOP/Delta+Fetch+And+Merge+Design#DeltaFetchAndMergeDesign-Wheretostoretheoutputinsqoop?
Update: The above highlights the pros and cons of each approach.
#4 is chosen, since it is less intrusive, more clean and allows U/Edit per value in the output easily.
Will use this ticket for more detailed discussion on storage options for the output from connectors
1.
// will have FK to submission public static final String QUERY_CREATE_TABLE_SQ_JOB_OUTPUT_SUBMISSION = "CREATE TABLE " + TABLE_SQ_JOB_OUTPUT + " (" + COLUMN_SQ_JOB_OUT_ID + " BIGINT GENERATED ALWAYS AS IDENTITY (START WITH 1, INCREMENT BY 1), " + COLUMN_SQ_JOB_OUT_KEY + " VARCHAR(32), " + COLUMN_SQ_JOB_OUT_VALUE + " LONG VARCHAR," + COLUMN_SQ_JOB_OUT_TYPE + " VARCHAR(32)," + COLUMN_SQD_ID + " VARCHAR(32)," // FK to the direction table, since this allows to distinguish output from FROM/ TO part of the job + COLUMN_SQRS_SUBMISSION + " BIGINT, " + "CONSTRAINT " + CONSTRAINT_SQRS_SQS + " " + "FOREIGN KEY (" + COLUMN_SQRS_SUBMISSION + ") " + "REFERENCES " + TABLE_SQ_SUBMISSION + "(" + COLUMN_SQS_ID + ") ON DELETE CASCADE "
2.
At the code level, we will define MOutputType, one of the types can be BLOB as well, if a connector decides to store the value as a BLOB
class JobOutput { String key; Object value; MOutputType type; }
3.
At the repository API, add a new API to get job output for a particular submission Id and allow updates on values.
Attachments
Attachments
Issue Links
- links to