Uploaded image for project: 'Sqoop'
  1. Sqoop
  2. SQOOP-1341

Sqoop Export Upsert for MySQL lacks batch support

    XMLWordPrintableJSON

    Details

      Description

      MySQL export upserts are limited to one row per statement. This wastes bandwidth and makes the exports unusably slow.

      I wrote a patch to support multiple rows per statement at (https://github.com/cloudera/sqoop/pull/22/files). Now the records.per.statement parameter actually works, and I get fast exports (60k rows/sec) with 1000 records per statement, 10 statements per transactions, and 5 mappers.

        Attachments

        1. SQOOP-1341.patch
          2 kB
          Andy Skelton

          Activity

            People

            • Assignee:
              skeltoac Andy Skelton
              Reporter:
              skeltoac Andy Skelton
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - 1h
                1h
                Remaining:
                Remaining Estimate - 1h
                1h
                Logged:
                Time Spent - Not Specified
                Not Specified