Sqoop
  1. Sqoop
  2. SQOOP-604 Easy throttling feature for MySQL exports
  3. SQOOP-683

Documenting sqoop.mysql.export.sleep.ms - easy throttling feature for direct MySQL exports

    Details

    • Type: Sub-task Sub-task
    • Status: Resolved
    • Priority: Trivial Trivial
    • Resolution: Fixed
    • Affects Version/s: 1.4.2
    • Fix Version/s: 1.4.3
    • Component/s: connectors/mysql, docs
    • Labels:
      None

      Description

      Documenting feature added in parent task.

      1. SQOOP-683_v4.patch
        2 kB
        Zoltan Toth-Czifra

        Issue Links

          Activity

          Jarek Jarcec Cecho made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Hide
          Hudson added a comment -

          Integrated in Sqoop-ant-jdk-1.6-hadoop23 #434 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop23/434/)
          SQOOP-683: Documenting sqoop.mysql.export.sleep.ms - easy throttling feature for direct MySQL exports (Revision da16aa5dc0fd6f980288564b1e5160ca89b3a8eb)

          Result = SUCCESS
          jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=da16aa5dc0fd6f980288564b1e5160ca89b3a8eb
          Files :

          • src/docs/user/compatibility.txt
          Show
          Hudson added a comment - Integrated in Sqoop-ant-jdk-1.6-hadoop23 #434 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop23/434/ ) SQOOP-683 : Documenting sqoop.mysql.export.sleep.ms - easy throttling feature for direct MySQL exports (Revision da16aa5dc0fd6f980288564b1e5160ca89b3a8eb) Result = SUCCESS jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=da16aa5dc0fd6f980288564b1e5160ca89b3a8eb Files : src/docs/user/compatibility.txt
          Hide
          Hudson added a comment -

          Integrated in Sqoop-ant-jdk-1.6-hadoop20 #280 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop20/280/)
          SQOOP-683: Documenting sqoop.mysql.export.sleep.ms - easy throttling feature for direct MySQL exports (Revision da16aa5dc0fd6f980288564b1e5160ca89b3a8eb)

          Result = SUCCESS
          jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=da16aa5dc0fd6f980288564b1e5160ca89b3a8eb
          Files :

          • src/docs/user/compatibility.txt
          Show
          Hudson added a comment - Integrated in Sqoop-ant-jdk-1.6-hadoop20 #280 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop20/280/ ) SQOOP-683 : Documenting sqoop.mysql.export.sleep.ms - easy throttling feature for direct MySQL exports (Revision da16aa5dc0fd6f980288564b1e5160ca89b3a8eb) Result = SUCCESS jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=da16aa5dc0fd6f980288564b1e5160ca89b3a8eb Files : src/docs/user/compatibility.txt
          Hide
          Hudson added a comment -

          Integrated in Sqoop-ant-jdk-1.6-hadoop200 #280 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop200/280/)
          SQOOP-683: Documenting sqoop.mysql.export.sleep.ms - easy throttling feature for direct MySQL exports (Revision da16aa5dc0fd6f980288564b1e5160ca89b3a8eb)

          Result = SUCCESS
          jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=da16aa5dc0fd6f980288564b1e5160ca89b3a8eb
          Files :

          • src/docs/user/compatibility.txt
          Show
          Hudson added a comment - Integrated in Sqoop-ant-jdk-1.6-hadoop200 #280 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop200/280/ ) SQOOP-683 : Documenting sqoop.mysql.export.sleep.ms - easy throttling feature for direct MySQL exports (Revision da16aa5dc0fd6f980288564b1e5160ca89b3a8eb) Result = SUCCESS jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=da16aa5dc0fd6f980288564b1e5160ca89b3a8eb Files : src/docs/user/compatibility.txt
          Hide
          Hudson added a comment -

          Integrated in Sqoop-ant-jdk-1.6-hadoop100 #268 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop100/268/)
          SQOOP-683: Documenting sqoop.mysql.export.sleep.ms - easy throttling feature for direct MySQL exports (Revision da16aa5dc0fd6f980288564b1e5160ca89b3a8eb)

          Result = SUCCESS
          jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=da16aa5dc0fd6f980288564b1e5160ca89b3a8eb
          Files :

          • src/docs/user/compatibility.txt
          Show
          Hudson added a comment - Integrated in Sqoop-ant-jdk-1.6-hadoop100 #268 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop100/268/ ) SQOOP-683 : Documenting sqoop.mysql.export.sleep.ms - easy throttling feature for direct MySQL exports (Revision da16aa5dc0fd6f980288564b1e5160ca89b3a8eb) Result = SUCCESS jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=da16aa5dc0fd6f980288564b1e5160ca89b3a8eb Files : src/docs/user/compatibility.txt
          Hide
          Jarek Jarcec Cecho added a comment -
          Show
          Jarek Jarcec Cecho added a comment - Committed: https://git-wip-us.apache.org/repos/asf?p=sqoop.git;a=commit;h=da16aa5dc0fd6f980288564b1e5160ca89b3a8eb Thank you for your contribution Zoltan! Jarcec
          Jarek Jarcec Cecho made changes -
          Remote Link This issue links to "Review board (Web Link)" [ 11425 ]
          Zoltan Toth-Czifra made changes -
          Attachment SQOOP-683_v4.patch [ 12552835 ]
          Hide
          Zoltan Toth-Czifra added a comment -

          I attach the final patch that has passed the review.

          Thank you guys!

          Show
          Zoltan Toth-Czifra added a comment - I attach the final patch that has passed the review. Thank you guys!
          Hide
          Zoltan Toth-Czifra added a comment -

          Sure, not a problem: https://reviews.apache.org/r/7880

          Show
          Zoltan Toth-Czifra added a comment - Sure, not a problem: https://reviews.apache.org/r/7880
          Hide
          Jarek Jarcec Cecho added a comment -

          Hi Zoltan,
          thank you very much for your time and effort. Would you mind uploading your patch as a file and and the same time uploading it to Review board?

          Jarcec

          Show
          Jarek Jarcec Cecho added a comment - Hi Zoltan, thank you very much for your time and effort. Would you mind uploading your patch as a file and and the same time uploading it to Review board? Jarcec
          Zoltan Toth-Czifra made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Zoltan Toth-Czifra added a comment - - edited
          diff --git a/src/docs/user/compatibility.txt b/src/docs/user/compatibility.txt
          index 3576fd7..e8218d6 100644
          --- a/src/docs/user/compatibility.txt
          +++ b/src/docs/user/compatibility.txt
          @@ -138,9 +138,30 @@ bytes. Set _size_ to 0 to disable intermediate checkpoints,
           but individual files being exported will continue to be committed
           independently of one another.
           
          +Sometimes you need to export large data with Sqoop to a live MySQL cluster that
          +is under a high load serving random queries from the users of our product.
          +While data consistency issues during the export can be easily solved with a
          +staging table, there is still a problem: the performance impact caused by the
          +heavy export.
          +
          +First off, the resources of MySQL dedicated to the import process can affect
          +the performance of the live product, both on the master and on the slaves.
          +Second, even if the servers can handle the import with no significant
          +performance impact (mysqlimport should be relatively "cheap"), importing big
          +tables can cause serious replication lag in the cluster risking data
          +inconsistency.
          +
          +With +-D sqoop.mysql.export.sleep.ms=time+, where _time_ is a value in
          +milliseconds, you can let the server relax between checkpoints and the replicas
          +catch up by pausing the export process after transferring the number of bytes
          +specified in +sqoop.mysql.export.checkpoint.bytes+. Experiment with different
          +settings of these two parameters to archieve an export pace that doesn't
          +endanger the stability of your MySQL cluster.
          +
           IMPORTANT: Note that any arguments to Sqoop that are of the form +-D
           parameter=value+ are Hadoop _generic arguments_ and must appear before
           any tool-specific arguments (for example, +\--connect+, +\--table+, etc).
          +Don't forget that these parameters only work with the +\--direct+ flag set.
           
           PostgreSQL
           ~~~~~~~~~~
          
          Show
          Zoltan Toth-Czifra added a comment - - edited diff --git a/src/docs/user/compatibility.txt b/src/docs/user/compatibility.txt index 3576fd7..e8218d6 100644 --- a/src/docs/user/compatibility.txt +++ b/src/docs/user/compatibility.txt @@ -138,9 +138,30 @@ bytes. Set _size_ to 0 to disable intermediate checkpoints, but individual files being exported will continue to be committed independently of one another. +Sometimes you need to export large data with Sqoop to a live MySQL cluster that +is under a high load serving random queries from the users of our product. +While data consistency issues during the export can be easily solved with a +staging table, there is still a problem: the performance impact caused by the +heavy export. + +First off, the resources of MySQL dedicated to the import process can affect +the performance of the live product, both on the master and on the slaves. +Second, even if the servers can handle the import with no significant +performance impact (mysqlimport should be relatively "cheap" ), importing big +tables can cause serious replication lag in the cluster risking data +inconsistency. + +With +-D sqoop.mysql.export.sleep.ms=time+, where _time_ is a value in +milliseconds, you can let the server relax between checkpoints and the replicas + catch up by pausing the export process after transferring the number of bytes +specified in +sqoop.mysql.export.checkpoint.bytes+. Experiment with different +settings of these two parameters to archieve an export pace that doesn't +endanger the stability of your MySQL cluster. + IMPORTANT: Note that any arguments to Sqoop that are of the form +-D parameter=value+ are Hadoop _generic arguments_ and must appear before any tool-specific arguments ( for example, +\--connect+, +\--table+, etc). +Don't forget that these parameters only work with the +\--direct+ flag set. PostgreSQL ~~~~~~~~~~
          Zoltan Toth-Czifra made changes -
          Field Original Value New Value
          Fix Version/s 1.4.3 [ 12322644 ]
          Zoltan Toth-Czifra created issue -

            People

            • Assignee:
              Zoltan Toth-Czifra
              Reporter:
              Zoltan Toth-Czifra
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development