Details

      Description

      The details are in the design wiki, as the implementation happens more discussions can happen here.

      https://cwiki.apache.org/confluence/display/SQOOP/Delta+Fetch+And+Merge+Design#DeltaFetchAndMergeDesign-Howtogetoutputfromconnectortosqoop?

      The goal is to dynamically inject a IncrementalConfig instance into the FromJobConfiguration. The current MFromConfig and MToConfig can already hold a list of configs, and a strong sentiment was expressed to keep it as a list, why not for the first time actually make use of it and group the incremental related configs in one config object

      This task will prepare the FromJobConfiguration from the job config data, ExtractorContext with the relevant values from the prev job run

      This task will prepare the ToJobConfiguration from the job config data, LoaderContext with the relevant values from the prev job run if any

      We will use DistributedCache to get State information from the Extractor and Loader out and finally persist it into the sqoop repository depending on SQOOP-1804 once the outputcommitter commit is called

        Attachments

        1. SQOOP-1803.patch
          9 kB
          Jarek Jarcec Cecho
        2. SQOOP-1803-POC-2.patch
          23 kB
          Veena Basavaraj
        3. SQOOP-1803-POC.patch
          14 kB
          Jarek Jarcec Cecho

          Issue Links

            Activity

              People

              • Assignee:
                jarcec Jarek Jarcec Cecho
                Reporter:
                vybs Veena Basavaraj
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: