Sqoop
  1. Sqoop
  2. SQOOP-418

MySQLAuthTest hangs when run against MySQL 5.5.3 or higher version

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.4.0-incubating
    • Fix Version/s: 1.4.1-incubating
    • Component/s: None
    • Labels:
      None

      Description

      When running MySQLAuthTest against MySQL version 5.5.3 or higher, the test will hang and eventually fail due to timeout.

      1. SQOOP-418.patch
        2 kB
        Arvind Prabhakar

        Issue Links

          Activity

          Hide
          Arvind Prabhakar added a comment -

          This is because in recent version of MySQL 5.5.x the concept of metadata locking was introduced which causes DDL operations to timeout if the table in question has an unclosed transaction that ran a DML operation.

          The implementation of DBRecordReader in Sqoop is such that if a failure occurs, it will raise the exception without closing the associated statement/connection it was working with. This condition is triggered by the MySQLAuthTest which then hangs until timeout.

          Show
          Arvind Prabhakar added a comment - This is because in recent version of MySQL 5.5.x the concept of metadata locking was introduced which causes DDL operations to timeout if the table in question has an unclosed transaction that ran a DML operation. The implementation of DBRecordReader in Sqoop is such that if a failure occurs, it will raise the exception without closing the associated statement/connection it was working with. This condition is triggered by the MySQLAuthTest which then hangs until timeout.
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/3316/
          -----------------------------------------------------------

          Review request for Sqoop and Jarek Cecho.

          Summary
          -------

          Added clean-up logic to the DBRecordReader so that in case of failure the appropriate statement/connection is closed gracefully.

          Also added a couple of build properties which are needed to suppress Kerberos warnings when running on Mac OS. For more information see https://issues.apache.org/jira/browse/HADOOP-7489

          This addresses bug SQOOP-418.
          https://issues.apache.org/jira/browse/SQOOP-418

          Diffs


          /build.xml 1222526
          /src/java/org/apache/sqoop/mapreduce/db/DBRecordReader.java 1222526

          Diff: https://reviews.apache.org/r/3316/diff

          Testing
          -------

          Ran all unit and third party tests.

          Thanks,

          Arvind

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/3316/ ----------------------------------------------------------- Review request for Sqoop and Jarek Cecho. Summary ------- Added clean-up logic to the DBRecordReader so that in case of failure the appropriate statement/connection is closed gracefully. Also added a couple of build properties which are needed to suppress Kerberos warnings when running on Mac OS. For more information see https://issues.apache.org/jira/browse/HADOOP-7489 This addresses bug SQOOP-418 . https://issues.apache.org/jira/browse/SQOOP-418 Diffs /build.xml 1222526 /src/java/org/apache/sqoop/mapreduce/db/DBRecordReader.java 1222526 Diff: https://reviews.apache.org/r/3316/diff Testing ------- Ran all unit and third party tests. Thanks, Arvind
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/3316/#review4132
          -----------------------------------------------------------

          Ship it!

          Hi Arvind,
          changes looks good, thank you very much for finding and fixing that bug! I've noticed that you've already attached the patch to corresponding JIRA, so I'll commit it shortly.

          Jarcec

          • Jarek

          On 2011-12-28 05:13:32, Arvind Prabhakar wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/3316/

          -----------------------------------------------------------

          (Updated 2011-12-28 05:13:32)

          Review request for Sqoop and Jarek Cecho.

          Summary

          -------

          Added clean-up logic to the DBRecordReader so that in case of failure the appropriate statement/connection is closed gracefully.

          Also added a couple of build properties which are needed to suppress Kerberos warnings when running on Mac OS. For more information see https://issues.apache.org/jira/browse/HADOOP-7489

          This addresses bug SQOOP-418.

          https://issues.apache.org/jira/browse/SQOOP-418

          Diffs

          -----

          /build.xml 1222526

          /src/java/org/apache/sqoop/mapreduce/db/DBRecordReader.java 1222526

          Diff: https://reviews.apache.org/r/3316/diff

          Testing

          -------

          Ran all unit and third party tests.

          Thanks,

          Arvind

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/3316/#review4132 ----------------------------------------------------------- Ship it! Hi Arvind, changes looks good, thank you very much for finding and fixing that bug! I've noticed that you've already attached the patch to corresponding JIRA, so I'll commit it shortly. Jarcec Jarek On 2011-12-28 05:13:32, Arvind Prabhakar wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/3316/ ----------------------------------------------------------- (Updated 2011-12-28 05:13:32) Review request for Sqoop and Jarek Cecho. Summary ------- Added clean-up logic to the DBRecordReader so that in case of failure the appropriate statement/connection is closed gracefully. Also added a couple of build properties which are needed to suppress Kerberos warnings when running on Mac OS. For more information see https://issues.apache.org/jira/browse/HADOOP-7489 This addresses bug SQOOP-418 . https://issues.apache.org/jira/browse/SQOOP-418 Diffs ----- /build.xml 1222526 /src/java/org/apache/sqoop/mapreduce/db/DBRecordReader.java 1222526 Diff: https://reviews.apache.org/r/3316/diff Testing ------- Ran all unit and third party tests. Thanks, Arvind
          Hide
          Jarek Jarcec Cecho added a comment -

          Patch committed, thank you very much Arvind!

          Jarcec

          Show
          Jarek Jarcec Cecho added a comment - Patch committed, thank you very much Arvind! Jarcec
          Hide
          Hudson added a comment -

          Integrated in Sqoop-ant-jdk-1.6 #72 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6/72/)
          SQOOP-418. MySQLAuthTest hangs when run against MySQL 5.5.3 or higher version

          (Arvind Prabhakar via Jarek Jarcec Cecho)

          jarcec : http://svn.apache.org/viewvc/?view=rev&rev=1225123
          Files :

          • /incubator/sqoop/trunk/build.xml
          • /incubator/sqoop/trunk/src/java/org/apache/sqoop/mapreduce/db/DBRecordReader.java
          Show
          Hudson added a comment - Integrated in Sqoop-ant-jdk-1.6 #72 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6/72/ ) SQOOP-418 . MySQLAuthTest hangs when run against MySQL 5.5.3 or higher version (Arvind Prabhakar via Jarek Jarcec Cecho) jarcec : http://svn.apache.org/viewvc/?view=rev&rev=1225123 Files : /incubator/sqoop/trunk/build.xml /incubator/sqoop/trunk/src/java/org/apache/sqoop/mapreduce/db/DBRecordReader.java
          Hide
          Guido Serra aka Zeph added a comment -

          hey, MySQLAuthTest is not being executed on "ant test", is that on purpose?

          Jarek Jarcec Cecho shall I open a ticket and fix it? cause it is failing...

          Show
          Guido Serra aka Zeph added a comment - hey, MySQLAuthTest is not being executed on "ant test", is that on purpose? Jarek Jarcec Cecho shall I open a ticket and fix it? cause it is failing...
          Hide
          Guido Serra aka Zeph added a comment -

          k, created... SQOOP-794

          Show
          Guido Serra aka Zeph added a comment - k, created... SQOOP-794
          Hide
          Jarek Jarcec Cecho added a comment -

          Yes MySQL tests are not executed with "ant test" on purpose. The reasoning is simple - they have external dependencies and can't work out-of-the-box.

          Show
          Jarek Jarcec Cecho added a comment - Yes MySQL tests are not executed with "ant test" on purpose. The reasoning is simple - they have external dependencies and can't work out-of-the-box.

            People

            • Assignee:
              Arvind Prabhakar
              Reporter:
              Arvind Prabhakar
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development