Uploaded image for project: 'Sqoop'
  1. Sqoop
  2. SQOOP-1197

Enable Sqoop to build against Hadoop-2.1.0-beta jar files

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.4.3
    • Fix Version/s: 1.4.5
    • Component/s: build
    • Labels:
      None
    • Environment:

      RHEL 6.3 x86_64

      Description

      To update build.xml and ivy.xml to enable Sqoop to build against Hadoop-2.1.0-beta jar files

      1. SQOOP-1197-trunk-1.patch
        3 kB
        sam liu
      2. SQOOP-1197-trunk.patch
        3 kB
        sam liu
      3. SQOOP-1197.patch
        3 kB
        sam liu

        Activity

        Hide
        sam liu sam liu added a comment -

        Updated build.xml and ivy.xml

        Show
        sam liu sam liu added a comment - Updated build.xml and ivy.xml
        Hide
        jarcec Jarek Jarcec Cecho added a comment -

        Hi sam liu,
        thank you very much for working on this, greatly appreciated! I'm having difficulties to apply the patch on current head of the trunk branch. Would you mind rebasing your patch?

        $ git apply SQOOP-1197.patch
        error: patch failed: build.xml:83
        error: build.xml: patch does not apply
        error: patch failed: ivy.xml:41
        error: ivy.xml: patch does not apply
        
        Show
        jarcec Jarek Jarcec Cecho added a comment - Hi sam liu , thank you very much for working on this, greatly appreciated! I'm having difficulties to apply the patch on current head of the trunk branch. Would you mind rebasing your patch? $ git apply SQOOP-1197.patch error: patch failed: build.xml:83 error: build.xml: patch does not apply error: patch failed: ivy.xml:41 error: ivy.xml: patch does not apply
        Hide
        sam liu sam liu added a comment -

        Add a new patch for trunk

        Show
        sam liu sam liu added a comment - Add a new patch for trunk
        Hide
        jarcec Jarek Jarcec Cecho added a comment -

        sam liu, I've tried the patch, but it do not seem to be working:

        $ ant clean test -Dhadoopversion=210
        Buildfile: /home/jarcec/apache/repos/sqoop/build.xml
             [echo] Use Hadoop 210
             [echo] Use HBase profile 0.94 by default
             [echo] Use built-in SQL server connector by default
        
        clean:
           [delete] Deleting directory /home/jarcec/apache/repos/sqoop/build
        
        init:
        
        gen-version:
        
        ivy-download:
              [get] Getting: http://repo2.maven.org/maven2/org/apache/ivy/ivy/2.1.0/ivy-2.1.0.jar
              [get] To: /home/jarcec/apache/repos/sqoop/lib/ivy-2.1.0.jar
              [get] Not modified - so not downloaded
        
        ivy-probe-antlib:
        
        ivy-init-antlib:
        
        ivy-init:
        [ivy:configure] :: Ivy 2.1.0 - 20090925235825 :: http://ant.apache.org/ivy/ ::
        [ivy:configure] :: loading settings :: file = /home/jarcec/apache/repos/sqoop/ivy/ivysettings.xml
        
        ivy-resolve-hadoop:
        [ivy:resolve] unknown configuration 'hbasecompat${hbasecompatprofile}'. It is extended by hbase95 in file:/home/jarcec/apache/repos/sqoop/ivy.xml
        
        BUILD FAILED
        /home/jarcec/apache/repos/sqoop/build.xml:1343: syntax errors in ivy file: java.text.ParseException: unknown configuration 'hbasecompat${hbasecompatprofile}'. It is extended by hbase95 in file:/home/jarcec/apache/repos/sqoop/ivy.xml
                at org.apache.ivy.plugins.parser.xml.XmlModuleDescriptorParser$Parser.parse(XmlModuleDescriptorParser.java:263)
                at org.apache.ivy.plugins.parser.xml.XmlModuleDescriptorParser.parseDescriptor(XmlModuleDescriptorParser.java:102)
                at org.apache.ivy.plugins.parser.AbstractModuleDescriptorParser.parseDescriptor(AbstractModuleDescriptorParser.java:48)
                at org.apache.ivy.core.resolve.ResolveEngine.resolve(ResolveEngine.java:183)
                at org.apache.ivy.Ivy.resolve(Ivy.java:502)
                at org.apache.ivy.ant.IvyResolve.doExecute(IvyResolve.java:244)
                at org.apache.ivy.ant.IvyTask.execute(IvyTask.java:277)
                at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:291)
                at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                at java.lang.reflect.Method.invoke(Method.java:597)
                at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)
                at org.apache.tools.ant.Task.perform(Task.java:348)
                at org.apache.tools.ant.Target.execute(Target.java:390)
                at org.apache.tools.ant.Target.performTasks(Target.java:411)
                at org.apache.tools.ant.Project.executeSortedTargets(Project.java:1399)
                at org.apache.tools.ant.Project.executeTarget(Project.java:1368)
                at org.apache.tools.ant.helper.DefaultExecutor.executeTargets(DefaultExecutor.java:41)
                at org.apache.tools.ant.Project.executeTargets(Project.java:1251)
                at org.apache.tools.ant.Main.runBuild(Main.java:809)
                at org.apache.tools.ant.Main.startAnt(Main.java:217)
                at org.apache.tools.ant.launch.Launcher.run(Launcher.java:280)
                at org.apache.tools.ant.launch.Launcher.main(Launcher.java:109)
        Caused by: java.lang.IllegalStateException: unknown configuration 'hbasecompat${hbasecompatprofile}'. It is extended by hbase95
                at org.apache.ivy.core.module.descriptor.DefaultModuleDescriptor.checkConf(DefaultModuleDescriptor.java:658)
                at org.apache.ivy.core.module.descriptor.DefaultModuleDescriptor.check(DefaultModuleDescriptor.java:639)
                at org.apache.ivy.plugins.parser.xml.XmlModuleDescriptorParser$Parser.parse(XmlModuleDescriptorParser.java:255)
                ... 22 more
        
        Total time: 2 seconds
        
        

        I think that the problem is that if-else block in build.xml for 210 is not defining variable hbasecompatprofile as other branches of the if statement do.

        Show
        jarcec Jarek Jarcec Cecho added a comment - sam liu , I've tried the patch, but it do not seem to be working: $ ant clean test -Dhadoopversion=210 Buildfile: /home/jarcec/apache/repos/sqoop/build.xml [echo] Use Hadoop 210 [echo] Use HBase profile 0.94 by default [echo] Use built-in SQL server connector by default clean: [delete] Deleting directory /home/jarcec/apache/repos/sqoop/build init: gen-version: ivy-download: [get] Getting: http: //repo2.maven.org/maven2/org/apache/ivy/ivy/2.1.0/ivy-2.1.0.jar [get] To: /home/jarcec/apache/repos/sqoop/lib/ivy-2.1.0.jar [get] Not modified - so not downloaded ivy-probe-antlib: ivy-init-antlib: ivy-init: [ivy:configure] :: Ivy 2.1.0 - 20090925235825 :: http: //ant.apache.org/ivy/ :: [ivy:configure] :: loading settings :: file = /home/jarcec/apache/repos/sqoop/ivy/ivysettings.xml ivy-resolve-hadoop: [ivy:resolve] unknown configuration 'hbasecompat${hbasecompatprofile}'. It is extended by hbase95 in file:/home/jarcec/apache/repos/sqoop/ivy.xml BUILD FAILED /home/jarcec/apache/repos/sqoop/build.xml:1343: syntax errors in ivy file: java.text.ParseException: unknown configuration 'hbasecompat${hbasecompatprofile}'. It is extended by hbase95 in file:/home/jarcec/apache/repos/sqoop/ivy.xml at org.apache.ivy.plugins.parser.xml.XmlModuleDescriptorParser$Parser.parse(XmlModuleDescriptorParser.java:263) at org.apache.ivy.plugins.parser.xml.XmlModuleDescriptorParser.parseDescriptor(XmlModuleDescriptorParser.java:102) at org.apache.ivy.plugins.parser.AbstractModuleDescriptorParser.parseDescriptor(AbstractModuleDescriptorParser.java:48) at org.apache.ivy.core.resolve.ResolveEngine.resolve(ResolveEngine.java:183) at org.apache.ivy.Ivy.resolve(Ivy.java:502) at org.apache.ivy.ant.IvyResolve.doExecute(IvyResolve.java:244) at org.apache.ivy.ant.IvyTask.execute(IvyTask.java:277) at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:291) at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106) at org.apache.tools.ant.Task.perform(Task.java:348) at org.apache.tools.ant.Target.execute(Target.java:390) at org.apache.tools.ant.Target.performTasks(Target.java:411) at org.apache.tools.ant.Project.executeSortedTargets(Project.java:1399) at org.apache.tools.ant.Project.executeTarget(Project.java:1368) at org.apache.tools.ant.helper.DefaultExecutor.executeTargets(DefaultExecutor.java:41) at org.apache.tools.ant.Project.executeTargets(Project.java:1251) at org.apache.tools.ant.Main.runBuild(Main.java:809) at org.apache.tools.ant.Main.startAnt(Main.java:217) at org.apache.tools.ant.launch.Launcher.run(Launcher.java:280) at org.apache.tools.ant.launch.Launcher.main(Launcher.java:109) Caused by: java.lang.IllegalStateException: unknown configuration 'hbasecompat${hbasecompatprofile}'. It is extended by hbase95 at org.apache.ivy.core.module.descriptor.DefaultModuleDescriptor.checkConf(DefaultModuleDescriptor.java:658) at org.apache.ivy.core.module.descriptor.DefaultModuleDescriptor.check(DefaultModuleDescriptor.java:639) at org.apache.ivy.plugins.parser.xml.XmlModuleDescriptorParser$Parser.parse(XmlModuleDescriptorParser.java:255) ... 22 more Total time: 2 seconds I think that the problem is that if-else block in build.xml for 210 is not defining variable hbasecompatprofile as other branches of the if statement do.
        Hide
        sam liu sam liu added a comment -

        Jarek,

        Sorry, it's my fault that I did not mention such differences in trunk branch.

        I attached new patch SQOOP-1197-trunk-1.patch, and I can run 'ant package -Dhadoopversion=210' and 'ant clean test -Dhadoopversion=210' on trunk now.

        Please let me know if any further issues.

        Thanks!

        Show
        sam liu sam liu added a comment - Jarek, Sorry, it's my fault that I did not mention such differences in trunk branch. I attached new patch SQOOP-1197 -trunk-1.patch, and I can run 'ant package -Dhadoopversion=210' and 'ant clean test -Dhadoopversion=210' on trunk now. Please let me know if any further issues. Thanks!
        Hide
        jarcec Jarek Jarcec Cecho added a comment -

        +1

        Show
        jarcec Jarek Jarcec Cecho added a comment - +1
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit f2bf861042fe9bcbdbb15c8953e1d21db44b8310 in branch refs/heads/trunk from Jarek Jarcec Cecho
        [ https://git-wip-us.apache.org/repos/asf?p=sqoop.git;h=f2bf861 ]

        SQOOP-1197: Enable Sqoop to build against Hadoop-2.1.0-beta jar files

        (sam liu via Jarek Jarcec Cecho)

        Show
        jira-bot ASF subversion and git services added a comment - Commit f2bf861042fe9bcbdbb15c8953e1d21db44b8310 in branch refs/heads/trunk from Jarek Jarcec Cecho [ https://git-wip-us.apache.org/repos/asf?p=sqoop.git;h=f2bf861 ] SQOOP-1197 : Enable Sqoop to build against Hadoop-2.1.0-beta jar files (sam liu via Jarek Jarcec Cecho)
        Hide
        jarcec Jarek Jarcec Cecho added a comment -

        Thank you sam liu for your contribution!

        Show
        jarcec Jarek Jarcec Cecho added a comment - Thank you sam liu for your contribution!
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Sqoop-ant-jdk-1.6-hadoop100 #808 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop100/808/)
        SQOOP-1197: Enable Sqoop to build against Hadoop-2.1.0-beta jar files (jarcec: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=f2bf861042fe9bcbdbb15c8953e1d21db44b8310)

        • ivy.xml
        • build.xml
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Sqoop-ant-jdk-1.6-hadoop100 #808 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop100/808/ ) SQOOP-1197 : Enable Sqoop to build against Hadoop-2.1.0-beta jar files (jarcec: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=f2bf861042fe9bcbdbb15c8953e1d21db44b8310 ) ivy.xml build.xml
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Sqoop-ant-jdk-1.6-hadoop200 #852 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop200/852/)
        SQOOP-1197: Enable Sqoop to build against Hadoop-2.1.0-beta jar files (jarcec: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=f2bf861042fe9bcbdbb15c8953e1d21db44b8310)

        • ivy.xml
        • build.xml
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Sqoop-ant-jdk-1.6-hadoop200 #852 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop200/852/ ) SQOOP-1197 : Enable Sqoop to build against Hadoop-2.1.0-beta jar files (jarcec: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=f2bf861042fe9bcbdbb15c8953e1d21db44b8310 ) ivy.xml build.xml
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Sqoop-ant-jdk-1.6-hadoop20 #848 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop20/848/)
        SQOOP-1197: Enable Sqoop to build against Hadoop-2.1.0-beta jar files (jarcec: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=f2bf861042fe9bcbdbb15c8953e1d21db44b8310)

        • ivy.xml
        • build.xml
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Sqoop-ant-jdk-1.6-hadoop20 #848 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop20/848/ ) SQOOP-1197 : Enable Sqoop to build against Hadoop-2.1.0-beta jar files (jarcec: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=f2bf861042fe9bcbdbb15c8953e1d21db44b8310 ) ivy.xml build.xml
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Sqoop-ant-jdk-1.6-hadoop23 #1047 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop23/1047/)
        SQOOP-1197: Enable Sqoop to build against Hadoop-2.1.0-beta jar files (jarcec: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=f2bf861042fe9bcbdbb15c8953e1d21db44b8310)

        • ivy.xml
        • build.xml
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Sqoop-ant-jdk-1.6-hadoop23 #1047 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop23/1047/ ) SQOOP-1197 : Enable Sqoop to build against Hadoop-2.1.0-beta jar files (jarcec: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=f2bf861042fe9bcbdbb15c8953e1d21db44b8310 ) ivy.xml build.xml

          People

          • Assignee:
            sam liu sam liu
            Reporter:
            sam liu sam liu
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development