Uploaded image for project: 'Sqoop'
  1. Sqoop
  2. SQOOP-354

SQOOP needs to be made compatible with Hadoop .23 release

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.4.0-incubating
    • Component/s: None
    • Labels:
    1. SQOOP-354.patch
      7 kB
      Tom White
    2. SQOOP-354.patch
      2 kB
      Tom White
    3. SQOOP-354-0.23.0-SNAPSHOT.patch
      0.9 kB
      Tom White
    4. SQOOP-354-ivy.patch.txt
      9 kB
      Roman Shaposhnik

      Issue Links

        Activity

        Hide
        cos Konstantin Boudnik added a comment -

        A dep. problem?

        [ivy:resolve] 	==== maven2: tried
        [ivy:resolve] 	  http://repo1.maven.org/maven2/org/apache/velocity/velocity/1.7/velocity-1.7.pom
        [ivy:resolve] 		::::::::::::::::::::::::::::::::::::::::::::::
        [ivy:resolve] 		::          UNRESOLVED DEPENDENCIES         ::
        [ivy:resolve] 		::::::::::::::::::::::::::::::::::::::::::::::
        [ivy:resolve] 		:: org.apache.velocity#velocity;1.7: not found
        [ivy:resolve] 		::::::::::::::::::::::::::::::::::::::::::::::
        
        Show
        cos Konstantin Boudnik added a comment - A dep. problem? [ivy:resolve] ==== maven2: tried [ivy:resolve] http://repo1.maven.org/maven2/org/apache/velocity/velocity/1.7/velocity-1.7.pom [ivy:resolve] :::::::::::::::::::::::::::::::::::::::::::::: [ivy:resolve] :: UNRESOLVED DEPENDENCIES :: [ivy:resolve] :::::::::::::::::::::::::::::::::::::::::::::: [ivy:resolve] :: org.apache.velocity#velocity;1.7: not found [ivy:resolve] ::::::::::::::::::::::::::::::::::::::::::::::
        Hide
        cos Konstantin Boudnik added a comment -

        Is this for 0.23 or 0.22?

        Show
        cos Konstantin Boudnik added a comment - Is this for 0.23 or 0.22?
        Hide
        rvs Roman Shaposhnik added a comment -

        This is .23 related. The root cause as far as I can see is the Mavenization of Hadoop. Ivy gets confused because of extra transitive dependencies that it fetches from the .23 pom files.

        Show
        rvs Roman Shaposhnik added a comment - This is .23 related. The root cause as far as I can see is the Mavenization of Hadoop. Ivy gets confused because of extra transitive dependencies that it fetches from the .23 pom files.
        Hide
        tomwhite Tom White added a comment -

        Roman: have you tried excluding the extra transitive dependencies?

        Show
        tomwhite Tom White added a comment - Roman: have you tried excluding the extra transitive dependencies?
        Hide
        tomwhite Tom White added a comment -

        Here's a patch to demonstrate the problem. Changing the Hadoop dependency to 0.23 necessitates an incompatible change to MockObjectFactory. This is similar to the changes covered in BIGTOP-162, and which are handled either via shims (an interface whose implementation is conditionally compiled according to the Hadoop target version) or via mock object libraries. Since the changes to Sqoop are only in the tests, using a mock object library is probably most appropriate. Are folks open to adding Mockito (say) as a Sqoop dependency?

        The other problem, not fixed in the patch, is that the Hadoop test classes (in particular HadoopTestCase) are not yet published to a Maven repository, so it's not possible to compile TestDataDrivenDBInputFormat.

        Show
        tomwhite Tom White added a comment - Here's a patch to demonstrate the problem. Changing the Hadoop dependency to 0.23 necessitates an incompatible change to MockObjectFactory. This is similar to the changes covered in BIGTOP-162 , and which are handled either via shims (an interface whose implementation is conditionally compiled according to the Hadoop target version) or via mock object libraries. Since the changes to Sqoop are only in the tests, using a mock object library is probably most appropriate. Are folks open to adding Mockito (say) as a Sqoop dependency? The other problem, not fixed in the patch, is that the Hadoop test classes (in particular HadoopTestCase) are not yet published to a Maven repository, so it's not possible to compile TestDataDrivenDBInputFormat.
        Hide
        tomwhite Tom White added a comment -

        More precisely, the Hadoop test JARs are being published, but the HadoopTestCase class has not been moved to the Maven part of the build, so is missing from the test JARs. See HADOOP-7590.

        Show
        tomwhite Tom White added a comment - More precisely, the Hadoop test JARs are being published, but the HadoopTestCase class has not been moved to the Maven part of the build, so is missing from the test JARs. See HADOOP-7590 .
        Hide
        tomwhite Tom White added a comment -

        It turns out that Mockito isn't needed, since MockObjectFactory was only being used by TestLargeObjectLoader, and then only to configure the output committer. I've simplified the test so that it doesn't use an output committer (since that isn't a part of the behaviour being tested), and now MockObjectFactory is no longer needed.

        Also, HadoopTestCase wasn't really being used, so I've removed that dependency too. The tests that I've changed work on both 0.20 and 0.23, so I think this change can be committed. Please review.

        Having a switch to build on either 0.20 or 0.23 should be tackled in another issue.

        Show
        tomwhite Tom White added a comment - It turns out that Mockito isn't needed, since MockObjectFactory was only being used by TestLargeObjectLoader, and then only to configure the output committer. I've simplified the test so that it doesn't use an output committer (since that isn't a part of the behaviour being tested), and now MockObjectFactory is no longer needed. Also, HadoopTestCase wasn't really being used, so I've removed that dependency too. The tests that I've changed work on both 0.20 and 0.23, so I think this change can be committed. Please review. Having a switch to build on either 0.20 or 0.23 should be tackled in another issue.
        Hide
        tomwhite Tom White added a comment -
        Show
        tomwhite Tom White added a comment - Reviewboard: https://reviews.apache.org/r/2758/
        Hide
        tomwhite Tom White added a comment -

        Apply SQOOP-354-0.23.0-SNAPSHOT.patch in addition to SQOOP-354.patch if you want to run against 0.23. (Not for commit.)

        Show
        tomwhite Tom White added a comment - Apply SQOOP-354 -0.23.0-SNAPSHOT.patch in addition to SQOOP-354 .patch if you want to run against 0.23. (Not for commit.)
        Hide
        rvs Roman Shaposhnik added a comment -

        Here's an update to the latest patch that Tom posted. This makes Sqoop build. If you want to keep up with the latest versions of this patch check it out from Bigtop.

        Show
        rvs Roman Shaposhnik added a comment - Here's an update to the latest patch that Tom posted. This makes Sqoop build. If you want to keep up with the latest versions of this patch check it out from Bigtop.
        Hide
        hudson Hudson added a comment -

        Integrated in Sqoop-jdk-1.6 #62 (See https://builds.apache.org/job/Sqoop-jdk-1.6/62/)
        SQOOP-354 SQOOP needs to be made compatible with Hadoop .23 release

        blee : http://svn.apache.org/viewvc/?view=rev&rev=1199149
        Files :

        • /incubator/sqoop/trunk/src/test/com/cloudera/sqoop/lib/TestLargeObjectLoader.java
        • /incubator/sqoop/trunk/src/test/com/cloudera/sqoop/mapreduce/db/TestDataDrivenDBInputFormat.java
        • /incubator/sqoop/trunk/src/test/com/cloudera/sqoop/testutil/MockObjectFactory.java
        Show
        hudson Hudson added a comment - Integrated in Sqoop-jdk-1.6 #62 (See https://builds.apache.org/job/Sqoop-jdk-1.6/62/ ) SQOOP-354 SQOOP needs to be made compatible with Hadoop .23 release blee : http://svn.apache.org/viewvc/?view=rev&rev=1199149 Files : /incubator/sqoop/trunk/src/test/com/cloudera/sqoop/lib/TestLargeObjectLoader.java /incubator/sqoop/trunk/src/test/com/cloudera/sqoop/mapreduce/db/TestDataDrivenDBInputFormat.java /incubator/sqoop/trunk/src/test/com/cloudera/sqoop/testutil/MockObjectFactory.java
        Hide
        tomwhite Tom White added a comment -

        Closing this since it was committed in 1.4.0. Fixing tests is covered in SQOOP-397.

        Show
        tomwhite Tom White added a comment - Closing this since it was committed in 1.4.0. Fixing tests is covered in SQOOP-397 .

          People

          • Assignee:
            tomwhite Tom White
            Reporter:
            rvs Roman Shaposhnik
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development