Sqoop
  1. Sqoop
  2. SQOOP-354

SQOOP needs to be made compatible with Hadoop .23 release

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.4.0-incubating
    • Component/s: None
    • Labels:
    1. SQOOP-354-ivy.patch.txt
      9 kB
      Roman Shaposhnik
    2. SQOOP-354-0.23.0-SNAPSHOT.patch
      0.9 kB
      Tom White
    3. SQOOP-354.patch
      7 kB
      Tom White
    4. SQOOP-354.patch
      2 kB
      Tom White

      Issue Links

        Activity

        Hide
        Tom White added a comment -

        Closing this since it was committed in 1.4.0. Fixing tests is covered in SQOOP-397.

        Show
        Tom White added a comment - Closing this since it was committed in 1.4.0. Fixing tests is covered in SQOOP-397 .
        Hide
        Hudson added a comment -

        Integrated in Sqoop-jdk-1.6 #62 (See https://builds.apache.org/job/Sqoop-jdk-1.6/62/)
        SQOOP-354 SQOOP needs to be made compatible with Hadoop .23 release

        blee : http://svn.apache.org/viewvc/?view=rev&rev=1199149
        Files :

        • /incubator/sqoop/trunk/src/test/com/cloudera/sqoop/lib/TestLargeObjectLoader.java
        • /incubator/sqoop/trunk/src/test/com/cloudera/sqoop/mapreduce/db/TestDataDrivenDBInputFormat.java
        • /incubator/sqoop/trunk/src/test/com/cloudera/sqoop/testutil/MockObjectFactory.java
        Show
        Hudson added a comment - Integrated in Sqoop-jdk-1.6 #62 (See https://builds.apache.org/job/Sqoop-jdk-1.6/62/ ) SQOOP-354 SQOOP needs to be made compatible with Hadoop .23 release blee : http://svn.apache.org/viewvc/?view=rev&rev=1199149 Files : /incubator/sqoop/trunk/src/test/com/cloudera/sqoop/lib/TestLargeObjectLoader.java /incubator/sqoop/trunk/src/test/com/cloudera/sqoop/mapreduce/db/TestDataDrivenDBInputFormat.java /incubator/sqoop/trunk/src/test/com/cloudera/sqoop/testutil/MockObjectFactory.java
        Hide
        Roman Shaposhnik added a comment -

        Here's an update to the latest patch that Tom posted. This makes Sqoop build. If you want to keep up with the latest versions of this patch check it out from Bigtop.

        Show
        Roman Shaposhnik added a comment - Here's an update to the latest patch that Tom posted. This makes Sqoop build. If you want to keep up with the latest versions of this patch check it out from Bigtop.
        Hide
        Tom White added a comment -

        Apply SQOOP-354-0.23.0-SNAPSHOT.patch in addition to SQOOP-354.patch if you want to run against 0.23. (Not for commit.)

        Show
        Tom White added a comment - Apply SQOOP-354 -0.23.0-SNAPSHOT.patch in addition to SQOOP-354 .patch if you want to run against 0.23. (Not for commit.)
        Hide
        Tom White added a comment -
        Show
        Tom White added a comment - Reviewboard: https://reviews.apache.org/r/2758/
        Hide
        Tom White added a comment -

        It turns out that Mockito isn't needed, since MockObjectFactory was only being used by TestLargeObjectLoader, and then only to configure the output committer. I've simplified the test so that it doesn't use an output committer (since that isn't a part of the behaviour being tested), and now MockObjectFactory is no longer needed.

        Also, HadoopTestCase wasn't really being used, so I've removed that dependency too. The tests that I've changed work on both 0.20 and 0.23, so I think this change can be committed. Please review.

        Having a switch to build on either 0.20 or 0.23 should be tackled in another issue.

        Show
        Tom White added a comment - It turns out that Mockito isn't needed, since MockObjectFactory was only being used by TestLargeObjectLoader, and then only to configure the output committer. I've simplified the test so that it doesn't use an output committer (since that isn't a part of the behaviour being tested), and now MockObjectFactory is no longer needed. Also, HadoopTestCase wasn't really being used, so I've removed that dependency too. The tests that I've changed work on both 0.20 and 0.23, so I think this change can be committed. Please review. Having a switch to build on either 0.20 or 0.23 should be tackled in another issue.
        Hide
        Tom White added a comment -

        More precisely, the Hadoop test JARs are being published, but the HadoopTestCase class has not been moved to the Maven part of the build, so is missing from the test JARs. See HADOOP-7590.

        Show
        Tom White added a comment - More precisely, the Hadoop test JARs are being published, but the HadoopTestCase class has not been moved to the Maven part of the build, so is missing from the test JARs. See HADOOP-7590 .
        Hide
        Tom White added a comment -

        Here's a patch to demonstrate the problem. Changing the Hadoop dependency to 0.23 necessitates an incompatible change to MockObjectFactory. This is similar to the changes covered in BIGTOP-162, and which are handled either via shims (an interface whose implementation is conditionally compiled according to the Hadoop target version) or via mock object libraries. Since the changes to Sqoop are only in the tests, using a mock object library is probably most appropriate. Are folks open to adding Mockito (say) as a Sqoop dependency?

        The other problem, not fixed in the patch, is that the Hadoop test classes (in particular HadoopTestCase) are not yet published to a Maven repository, so it's not possible to compile TestDataDrivenDBInputFormat.

        Show
        Tom White added a comment - Here's a patch to demonstrate the problem. Changing the Hadoop dependency to 0.23 necessitates an incompatible change to MockObjectFactory. This is similar to the changes covered in BIGTOP-162 , and which are handled either via shims (an interface whose implementation is conditionally compiled according to the Hadoop target version) or via mock object libraries. Since the changes to Sqoop are only in the tests, using a mock object library is probably most appropriate. Are folks open to adding Mockito (say) as a Sqoop dependency? The other problem, not fixed in the patch, is that the Hadoop test classes (in particular HadoopTestCase) are not yet published to a Maven repository, so it's not possible to compile TestDataDrivenDBInputFormat.
        Hide
        Tom White added a comment -

        Roman: have you tried excluding the extra transitive dependencies?

        Show
        Tom White added a comment - Roman: have you tried excluding the extra transitive dependencies?
        Hide
        Roman Shaposhnik added a comment -

        This is .23 related. The root cause as far as I can see is the Mavenization of Hadoop. Ivy gets confused because of extra transitive dependencies that it fetches from the .23 pom files.

        Show
        Roman Shaposhnik added a comment - This is .23 related. The root cause as far as I can see is the Mavenization of Hadoop. Ivy gets confused because of extra transitive dependencies that it fetches from the .23 pom files.
        Hide
        Konstantin Boudnik added a comment -

        Is this for 0.23 or 0.22?

        Show
        Konstantin Boudnik added a comment - Is this for 0.23 or 0.22?
        Hide
        Konstantin Boudnik added a comment -

        A dep. problem?

        [ivy:resolve] 	==== maven2: tried
        [ivy:resolve] 	  http://repo1.maven.org/maven2/org/apache/velocity/velocity/1.7/velocity-1.7.pom
        [ivy:resolve] 		::::::::::::::::::::::::::::::::::::::::::::::
        [ivy:resolve] 		::          UNRESOLVED DEPENDENCIES         ::
        [ivy:resolve] 		::::::::::::::::::::::::::::::::::::::::::::::
        [ivy:resolve] 		:: org.apache.velocity#velocity;1.7: not found
        [ivy:resolve] 		::::::::::::::::::::::::::::::::::::::::::::::
        
        Show
        Konstantin Boudnik added a comment - A dep. problem? [ivy:resolve] ==== maven2: tried [ivy:resolve] http://repo1.maven.org/maven2/org/apache/velocity/velocity/1.7/velocity-1.7.pom [ivy:resolve] :::::::::::::::::::::::::::::::::::::::::::::: [ivy:resolve] :: UNRESOLVED DEPENDENCIES :: [ivy:resolve] :::::::::::::::::::::::::::::::::::::::::::::: [ivy:resolve] :: org.apache.velocity#velocity;1.7: not found [ivy:resolve] ::::::::::::::::::::::::::::::::::::::::::::::

          People

          • Assignee:
            Tom White
            Reporter:
            Roman Shaposhnik
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development