[SQOOP-1656] SQOOP2: Support for "merge" with different changing dimension types

• Type: Wish
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Fix Version/s: 2.0.0
• Assignee: Unassigned
• Reporter: Gwen Shapira
• Created: 02/Nov/14 19:29
• Updated: 15/Dec/14 18:15

Description

Our current "incremental" design is for "append" only.

However, we do plan on adding "merge" capabilities somewhere in the future. Maybe.

Sqoop1 merges by overwriting existing rows with their newer versions.

But for DWH dimensions, there are other ways to merge:
http://en.wikipedia.org/wiki/Slowly_changing_dimension

For example, preserving both versions and adding "start date" and "end date" for each.

ETL tools can handle these situations. Will be cool if Sqoop2 will be able to do it too.