Sqoop
  1. Sqoop
  2. SQOOP-397

Make Sqoop work with Hadoop 0.23

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.4.1-incubating
    • Component/s: None
    • Labels:
      None

      Description

      SQOOP-354 made changes to get Sqoop to compile against 0.23; this issue is to get Sqoop running against 0.23, including passing tests.

      1. SQOOP-397.dualpath.patch
        14 kB
        Bilung Lee
      2. SQOOP-397.patch
        3 kB
        Tom White

        Issue Links

          Activity

          Hide
          Tom White added a comment -

          With this patch the only test failures are in TestHBaseImport and TestHBaseQueryImport (since a 0.23-compatible version of HBase is not being used), and TestDataDrivenDBInputFormat (which is due to MAPREDUCE-3433).

          This is not for commit, since it changes the default version of Hadoop. We should have a way of specifying the version via a command line property. (Does anyone know how to achieve this with Ant?)

          Show
          Tom White added a comment - With this patch the only test failures are in TestHBaseImport and TestHBaseQueryImport (since a 0.23-compatible version of HBase is not being used), and TestDataDrivenDBInputFormat (which is due to MAPREDUCE-3433 ). This is not for commit, since it changes the default version of Hadoop. We should have a way of specifying the version via a command line property. (Does anyone know how to achieve this with Ant?)
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/2983/
          -----------------------------------------------------------

          Review request for Sqoop, Arvind Prabhakar and Tom White.

          Summary
          -------

          Make Sqoop work with Hadoop 0.23

          This addresses bug SQOOP-397.
          https://issues.apache.org/jira/browse/SQOOP-397

          Diffs


          ./COMPILING.txt 1197706
          ./build.xml 1208854
          ./ivy.xml 1208716
          ./ivy/libraries.properties 1208716
          ./src/ivy/ivy-hdp20.xml PRE-CREATION
          ./src/ivy/ivy-hdp23.xml PRE-CREATION
          ./src/ivy/libraries-hdp20.properties PRE-CREATION
          ./src/ivy/libraries-hdp23.properties PRE-CREATION
          ./src/test/com/cloudera/sqoop/TestCompression.java 1208716
          ./src/test/com/cloudera/sqoop/hbase/HBaseTestCase.java 1208054
          ./src/test/com/cloudera/sqoop/hbase/TestHBaseImport.java 1208054
          ./src/test/com/cloudera/sqoop/hbase/TestHBaseQueryImport.java 1208054
          ./src/test/com/cloudera/sqoop/mapreduce/db/TestDataDrivenDBInputFormat.java 1208054

          Diff: https://reviews.apache.org/r/2983/diff

          Testing
          -------

          Thanks,

          Bilung

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2983/ ----------------------------------------------------------- Review request for Sqoop, Arvind Prabhakar and Tom White. Summary ------- Make Sqoop work with Hadoop 0.23 This addresses bug SQOOP-397 . https://issues.apache.org/jira/browse/SQOOP-397 Diffs ./COMPILING.txt 1197706 ./build.xml 1208854 ./ivy.xml 1208716 ./ivy/libraries.properties 1208716 ./src/ivy/ivy-hdp20.xml PRE-CREATION ./src/ivy/ivy-hdp23.xml PRE-CREATION ./src/ivy/libraries-hdp20.properties PRE-CREATION ./src/ivy/libraries-hdp23.properties PRE-CREATION ./src/test/com/cloudera/sqoop/TestCompression.java 1208716 ./src/test/com/cloudera/sqoop/hbase/HBaseTestCase.java 1208054 ./src/test/com/cloudera/sqoop/hbase/TestHBaseImport.java 1208054 ./src/test/com/cloudera/sqoop/hbase/TestHBaseQueryImport.java 1208054 ./src/test/com/cloudera/sqoop/mapreduce/db/TestDataDrivenDBInputFormat.java 1208054 Diff: https://reviews.apache.org/r/2983/diff Testing ------- Thanks, Bilung
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/2983/#review3599
          -----------------------------------------------------------

          Thanks for the patch Bilung. I have a few general comments:

          1. Since the new mechanism for build switch between 20 and 23 overwrites the ivy.xml file which is inside source control, it can possibly cause confusion when a build is done. One way to fix this would be to remove top level ivy.xml from source control, put top level ivy.xml in svn:ignore property, and the always copy the appropriate ivy-hdpXX.xml file during build.

          2. It would be great if we can use the hadoop version property to selectively exclude tests from running when the 23 version is used. Doing this will make sure that the tests continue to run when the build is done against 20.

          • Arvind

          On 2011-12-01 18:59:33, Bilung Lee wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/2983/

          -----------------------------------------------------------

          (Updated 2011-12-01 18:59:33)

          Review request for Sqoop, Arvind Prabhakar and Tom White.

          Summary

          -------

          Make Sqoop work with Hadoop 0.23

          This addresses bug SQOOP-397.

          https://issues.apache.org/jira/browse/SQOOP-397

          Diffs

          -----

          ./COMPILING.txt 1197706

          ./build.xml 1208854

          ./ivy.xml 1208716

          ./ivy/libraries.properties 1208716

          ./src/ivy/ivy-hdp20.xml PRE-CREATION

          ./src/ivy/ivy-hdp23.xml PRE-CREATION

          ./src/ivy/libraries-hdp20.properties PRE-CREATION

          ./src/ivy/libraries-hdp23.properties PRE-CREATION

          ./src/test/com/cloudera/sqoop/TestCompression.java 1208716

          ./src/test/com/cloudera/sqoop/hbase/HBaseTestCase.java 1208054

          ./src/test/com/cloudera/sqoop/hbase/TestHBaseImport.java 1208054

          ./src/test/com/cloudera/sqoop/hbase/TestHBaseQueryImport.java 1208054

          ./src/test/com/cloudera/sqoop/mapreduce/db/TestDataDrivenDBInputFormat.java 1208054

          Diff: https://reviews.apache.org/r/2983/diff

          Testing

          -------

          Thanks,

          Bilung

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2983/#review3599 ----------------------------------------------------------- Thanks for the patch Bilung. I have a few general comments: 1. Since the new mechanism for build switch between 20 and 23 overwrites the ivy.xml file which is inside source control, it can possibly cause confusion when a build is done. One way to fix this would be to remove top level ivy.xml from source control, put top level ivy.xml in svn:ignore property, and the always copy the appropriate ivy-hdpXX.xml file during build. 2. It would be great if we can use the hadoop version property to selectively exclude tests from running when the 23 version is used. Doing this will make sure that the tests continue to run when the build is done against 20. Arvind On 2011-12-01 18:59:33, Bilung Lee wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2983/ ----------------------------------------------------------- (Updated 2011-12-01 18:59:33) Review request for Sqoop, Arvind Prabhakar and Tom White. Summary ------- Make Sqoop work with Hadoop 0.23 This addresses bug SQOOP-397 . https://issues.apache.org/jira/browse/SQOOP-397 Diffs ----- ./COMPILING.txt 1197706 ./build.xml 1208854 ./ivy.xml 1208716 ./ivy/libraries.properties 1208716 ./src/ivy/ivy-hdp20.xml PRE-CREATION ./src/ivy/ivy-hdp23.xml PRE-CREATION ./src/ivy/libraries-hdp20.properties PRE-CREATION ./src/ivy/libraries-hdp23.properties PRE-CREATION ./src/test/com/cloudera/sqoop/TestCompression.java 1208716 ./src/test/com/cloudera/sqoop/hbase/HBaseTestCase.java 1208054 ./src/test/com/cloudera/sqoop/hbase/TestHBaseImport.java 1208054 ./src/test/com/cloudera/sqoop/hbase/TestHBaseQueryImport.java 1208054 ./src/test/com/cloudera/sqoop/mapreduce/db/TestDataDrivenDBInputFormat.java 1208054 Diff: https://reviews.apache.org/r/2983/diff Testing ------- Thanks, Bilung
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/2983/#review3600
          -----------------------------------------------------------

          ./COMPILING.txt
          <https://reviews.apache.org/r/2983/#comment8025>

          I think 0.20.x should continue to be the default until 0.23.x has stabilized.

          ./build.xml
          <https://reviews.apache.org/r/2983/#comment8026>

          Rather than using conditional logic in the build, it would be simpler to use the approach that Pig takes and drive the choice by a property.

          Search for "hadoopversion" in http://svn.apache.org/repos/asf/pig/trunk/build.xml and http://svn.apache.org/repos/asf/pig/trunk/ivy.xml (note the way the configuration name is parameterized in the latter).

          ./ivy/libraries.properties
          <https://reviews.apache.org/r/2983/#comment8027>

          We should parameterize HBase at the same time so we can switch to 0.92.0 when it comes out, since it supports Hadoop 0.23.

          ./ivy/libraries.properties
          <https://reviews.apache.org/r/2983/#comment8028>

          ZooKeeper too?

          ./src/test/com/cloudera/sqoop/hbase/HBaseTestCase.java
          <https://reviews.apache.org/r/2983/#comment8029>

          Rather than commenting out, why not just add an @Ignore annotation? Also open a JIRA and link to it so it doesn't get forgotten.

          • Tom

          On 2011-12-01 18:59:33, Bilung Lee wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/2983/

          -----------------------------------------------------------

          (Updated 2011-12-01 18:59:33)

          Review request for Sqoop, Arvind Prabhakar and Tom White.

          Summary

          -------

          Make Sqoop work with Hadoop 0.23

          This addresses bug SQOOP-397.

          https://issues.apache.org/jira/browse/SQOOP-397

          Diffs

          -----

          ./COMPILING.txt 1197706

          ./build.xml 1208854

          ./ivy.xml 1208716

          ./ivy/libraries.properties 1208716

          ./src/ivy/ivy-hdp20.xml PRE-CREATION

          ./src/ivy/ivy-hdp23.xml PRE-CREATION

          ./src/ivy/libraries-hdp20.properties PRE-CREATION

          ./src/ivy/libraries-hdp23.properties PRE-CREATION

          ./src/test/com/cloudera/sqoop/TestCompression.java 1208716

          ./src/test/com/cloudera/sqoop/hbase/HBaseTestCase.java 1208054

          ./src/test/com/cloudera/sqoop/hbase/TestHBaseImport.java 1208054

          ./src/test/com/cloudera/sqoop/hbase/TestHBaseQueryImport.java 1208054

          ./src/test/com/cloudera/sqoop/mapreduce/db/TestDataDrivenDBInputFormat.java 1208054

          Diff: https://reviews.apache.org/r/2983/diff

          Testing

          -------

          Thanks,

          Bilung

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2983/#review3600 ----------------------------------------------------------- ./COMPILING.txt < https://reviews.apache.org/r/2983/#comment8025 > I think 0.20.x should continue to be the default until 0.23.x has stabilized. ./build.xml < https://reviews.apache.org/r/2983/#comment8026 > Rather than using conditional logic in the build, it would be simpler to use the approach that Pig takes and drive the choice by a property. Search for "hadoopversion" in http://svn.apache.org/repos/asf/pig/trunk/build.xml and http://svn.apache.org/repos/asf/pig/trunk/ivy.xml (note the way the configuration name is parameterized in the latter). ./ivy/libraries.properties < https://reviews.apache.org/r/2983/#comment8027 > We should parameterize HBase at the same time so we can switch to 0.92.0 when it comes out, since it supports Hadoop 0.23. ./ivy/libraries.properties < https://reviews.apache.org/r/2983/#comment8028 > ZooKeeper too? ./src/test/com/cloudera/sqoop/hbase/HBaseTestCase.java < https://reviews.apache.org/r/2983/#comment8029 > Rather than commenting out, why not just add an @Ignore annotation? Also open a JIRA and link to it so it doesn't get forgotten. Tom On 2011-12-01 18:59:33, Bilung Lee wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2983/ ----------------------------------------------------------- (Updated 2011-12-01 18:59:33) Review request for Sqoop, Arvind Prabhakar and Tom White. Summary ------- Make Sqoop work with Hadoop 0.23 This addresses bug SQOOP-397 . https://issues.apache.org/jira/browse/SQOOP-397 Diffs ----- ./COMPILING.txt 1197706 ./build.xml 1208854 ./ivy.xml 1208716 ./ivy/libraries.properties 1208716 ./src/ivy/ivy-hdp20.xml PRE-CREATION ./src/ivy/ivy-hdp23.xml PRE-CREATION ./src/ivy/libraries-hdp20.properties PRE-CREATION ./src/ivy/libraries-hdp23.properties PRE-CREATION ./src/test/com/cloudera/sqoop/TestCompression.java 1208716 ./src/test/com/cloudera/sqoop/hbase/HBaseTestCase.java 1208054 ./src/test/com/cloudera/sqoop/hbase/TestHBaseImport.java 1208054 ./src/test/com/cloudera/sqoop/hbase/TestHBaseQueryImport.java 1208054 ./src/test/com/cloudera/sqoop/mapreduce/db/TestDataDrivenDBInputFormat.java 1208054 Diff: https://reviews.apache.org/r/2983/diff Testing ------- Thanks, Bilung
          Hide
          jiraposter@reviews.apache.org added a comment -

          On 2011-12-02 04:40:12, Tom White wrote:

          > ./COMPILING.txt, line 280

          > <https://reviews.apache.org/r/2983/diff/2/?file=61445#file61445line280>

          >

          > I think 0.20.x should continue to be the default until 0.23.x has stabilized.

          I hadn't seen https://issues.apache.org/jira/browse/SQOOP-384 which makes this more complex than I had thought, since Sqoop depends on features not in Hadoop 0.20.20x. So making a dependency on 0.23.x in trunk is probably acceptable.

          • Tom

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/2983/#review3600
          -----------------------------------------------------------

          On 2011-12-01 18:59:33, Bilung Lee wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/2983/

          -----------------------------------------------------------

          (Updated 2011-12-01 18:59:33)

          Review request for Sqoop, Arvind Prabhakar and Tom White.

          Summary

          -------

          Make Sqoop work with Hadoop 0.23

          This addresses bug SQOOP-397.

          https://issues.apache.org/jira/browse/SQOOP-397

          Diffs

          -----

          ./COMPILING.txt 1197706

          ./build.xml 1208854

          ./ivy.xml 1208716

          ./ivy/libraries.properties 1208716

          ./src/ivy/ivy-hdp20.xml PRE-CREATION

          ./src/ivy/ivy-hdp23.xml PRE-CREATION

          ./src/ivy/libraries-hdp20.properties PRE-CREATION

          ./src/ivy/libraries-hdp23.properties PRE-CREATION

          ./src/test/com/cloudera/sqoop/TestCompression.java 1208716

          ./src/test/com/cloudera/sqoop/hbase/HBaseTestCase.java 1208054

          ./src/test/com/cloudera/sqoop/hbase/TestHBaseImport.java 1208054

          ./src/test/com/cloudera/sqoop/hbase/TestHBaseQueryImport.java 1208054

          ./src/test/com/cloudera/sqoop/mapreduce/db/TestDataDrivenDBInputFormat.java 1208054

          Diff: https://reviews.apache.org/r/2983/diff

          Testing

          -------

          Thanks,

          Bilung

          Show
          jiraposter@reviews.apache.org added a comment - On 2011-12-02 04:40:12, Tom White wrote: > ./COMPILING.txt, line 280 > < https://reviews.apache.org/r/2983/diff/2/?file=61445#file61445line280 > > > I think 0.20.x should continue to be the default until 0.23.x has stabilized. I hadn't seen https://issues.apache.org/jira/browse/SQOOP-384 which makes this more complex than I had thought, since Sqoop depends on features not in Hadoop 0.20.20x. So making a dependency on 0.23.x in trunk is probably acceptable. Tom ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2983/#review3600 ----------------------------------------------------------- On 2011-12-01 18:59:33, Bilung Lee wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2983/ ----------------------------------------------------------- (Updated 2011-12-01 18:59:33) Review request for Sqoop, Arvind Prabhakar and Tom White. Summary ------- Make Sqoop work with Hadoop 0.23 This addresses bug SQOOP-397 . https://issues.apache.org/jira/browse/SQOOP-397 Diffs ----- ./COMPILING.txt 1197706 ./build.xml 1208854 ./ivy.xml 1208716 ./ivy/libraries.properties 1208716 ./src/ivy/ivy-hdp20.xml PRE-CREATION ./src/ivy/ivy-hdp23.xml PRE-CREATION ./src/ivy/libraries-hdp20.properties PRE-CREATION ./src/ivy/libraries-hdp23.properties PRE-CREATION ./src/test/com/cloudera/sqoop/TestCompression.java 1208716 ./src/test/com/cloudera/sqoop/hbase/HBaseTestCase.java 1208054 ./src/test/com/cloudera/sqoop/hbase/TestHBaseImport.java 1208054 ./src/test/com/cloudera/sqoop/hbase/TestHBaseQueryImport.java 1208054 ./src/test/com/cloudera/sqoop/mapreduce/db/TestDataDrivenDBInputFormat.java 1208054 Diff: https://reviews.apache.org/r/2983/diff Testing ------- Thanks, Bilung
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/2983/
          -----------------------------------------------------------

          (Updated 2011-12-06 19:12:36.179918)

          Review request for Sqoop, Arvind Prabhakar and Tom White.

          Changes
          -------

          Thanks for the feedback!

          A new patch is uploaded:
          1. To provide a property (hadoopversion) for the choice of Hadoop version.
          2. To exclude those test cases conditionally based on Hadoop version.

          Summary
          -------

          Make Sqoop work with Hadoop 0.23

          This addresses bug SQOOP-397.
          https://issues.apache.org/jira/browse/SQOOP-397

          Diffs (updated)


          ./COMPILING.txt 1197706
          ./build.xml 1210580
          ./ivy.xml 1210686
          ./ivy/libraries.properties 1210579
          ./src/test/com/cloudera/sqoop/TestCompression.java 1208716
          ./src/test/com/cloudera/sqoop/hbase/HBaseTestCase.java 1209661
          ./src/test/com/cloudera/sqoop/hbase/TestHBaseImport.java 1209661
          ./src/test/com/cloudera/sqoop/hbase/TestHBaseQueryImport.java 1209661
          ./src/test/com/cloudera/sqoop/mapreduce/db/TestDataDrivenDBInputFormat.java 1209644

          Diff: https://reviews.apache.org/r/2983/diff

          Testing
          -------

          Thanks,

          Bilung

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2983/ ----------------------------------------------------------- (Updated 2011-12-06 19:12:36.179918) Review request for Sqoop, Arvind Prabhakar and Tom White. Changes ------- Thanks for the feedback! A new patch is uploaded: 1. To provide a property (hadoopversion) for the choice of Hadoop version. 2. To exclude those test cases conditionally based on Hadoop version. Summary ------- Make Sqoop work with Hadoop 0.23 This addresses bug SQOOP-397 . https://issues.apache.org/jira/browse/SQOOP-397 Diffs (updated) ./COMPILING.txt 1197706 ./build.xml 1210580 ./ivy.xml 1210686 ./ivy/libraries.properties 1210579 ./src/test/com/cloudera/sqoop/TestCompression.java 1208716 ./src/test/com/cloudera/sqoop/hbase/HBaseTestCase.java 1209661 ./src/test/com/cloudera/sqoop/hbase/TestHBaseImport.java 1209661 ./src/test/com/cloudera/sqoop/hbase/TestHBaseQueryImport.java 1209661 ./src/test/com/cloudera/sqoop/mapreduce/db/TestDataDrivenDBInputFormat.java 1209644 Diff: https://reviews.apache.org/r/2983/diff Testing ------- Thanks, Bilung
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/2983/
          -----------------------------------------------------------

          (Updated 2011-12-06 19:14:02.584283)

          Review request for Sqoop, Ahmed Radwan, Arvind Prabhakar, and Tom White.

          Summary
          -------

          Make Sqoop work with Hadoop 0.23

          This addresses bug SQOOP-397.
          https://issues.apache.org/jira/browse/SQOOP-397

          Diffs


          ./COMPILING.txt 1197706
          ./build.xml 1210580
          ./ivy.xml 1210686
          ./ivy/libraries.properties 1210579
          ./src/test/com/cloudera/sqoop/TestCompression.java 1208716
          ./src/test/com/cloudera/sqoop/hbase/HBaseTestCase.java 1209661
          ./src/test/com/cloudera/sqoop/hbase/TestHBaseImport.java 1209661
          ./src/test/com/cloudera/sqoop/hbase/TestHBaseQueryImport.java 1209661
          ./src/test/com/cloudera/sqoop/mapreduce/db/TestDataDrivenDBInputFormat.java 1209644

          Diff: https://reviews.apache.org/r/2983/diff

          Testing
          -------

          Thanks,

          Bilung

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2983/ ----------------------------------------------------------- (Updated 2011-12-06 19:14:02.584283) Review request for Sqoop, Ahmed Radwan, Arvind Prabhakar, and Tom White. Summary ------- Make Sqoop work with Hadoop 0.23 This addresses bug SQOOP-397 . https://issues.apache.org/jira/browse/SQOOP-397 Diffs ./COMPILING.txt 1197706 ./build.xml 1210580 ./ivy.xml 1210686 ./ivy/libraries.properties 1210579 ./src/test/com/cloudera/sqoop/TestCompression.java 1208716 ./src/test/com/cloudera/sqoop/hbase/HBaseTestCase.java 1209661 ./src/test/com/cloudera/sqoop/hbase/TestHBaseImport.java 1209661 ./src/test/com/cloudera/sqoop/hbase/TestHBaseQueryImport.java 1209661 ./src/test/com/cloudera/sqoop/mapreduce/db/TestDataDrivenDBInputFormat.java 1209644 Diff: https://reviews.apache.org/r/2983/diff Testing ------- Thanks, Bilung
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/2983/#review3672
          -----------------------------------------------------------

          Ship it!

          This looks good to me. Are you planning on doing the HBase and ZooKeeper version parameterization in another JIRA?

          • Tom

          On 2011-12-06 19:14:02, Bilung Lee wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/2983/

          -----------------------------------------------------------

          (Updated 2011-12-06 19:14:02)

          Review request for Sqoop, Ahmed Radwan, Arvind Prabhakar, and Tom White.

          Summary

          -------

          Make Sqoop work with Hadoop 0.23

          This addresses bug SQOOP-397.

          https://issues.apache.org/jira/browse/SQOOP-397

          Diffs

          -----

          ./COMPILING.txt 1197706

          ./build.xml 1210580

          ./ivy.xml 1210686

          ./ivy/libraries.properties 1210579

          ./src/test/com/cloudera/sqoop/TestCompression.java 1208716

          ./src/test/com/cloudera/sqoop/hbase/HBaseTestCase.java 1209661

          ./src/test/com/cloudera/sqoop/hbase/TestHBaseImport.java 1209661

          ./src/test/com/cloudera/sqoop/hbase/TestHBaseQueryImport.java 1209661

          ./src/test/com/cloudera/sqoop/mapreduce/db/TestDataDrivenDBInputFormat.java 1209644

          Diff: https://reviews.apache.org/r/2983/diff

          Testing

          -------

          Thanks,

          Bilung

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2983/#review3672 ----------------------------------------------------------- Ship it! This looks good to me. Are you planning on doing the HBase and ZooKeeper version parameterization in another JIRA? Tom On 2011-12-06 19:14:02, Bilung Lee wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2983/ ----------------------------------------------------------- (Updated 2011-12-06 19:14:02) Review request for Sqoop, Ahmed Radwan, Arvind Prabhakar, and Tom White. Summary ------- Make Sqoop work with Hadoop 0.23 This addresses bug SQOOP-397 . https://issues.apache.org/jira/browse/SQOOP-397 Diffs ----- ./COMPILING.txt 1197706 ./build.xml 1210580 ./ivy.xml 1210686 ./ivy/libraries.properties 1210579 ./src/test/com/cloudera/sqoop/TestCompression.java 1208716 ./src/test/com/cloudera/sqoop/hbase/HBaseTestCase.java 1209661 ./src/test/com/cloudera/sqoop/hbase/TestHBaseImport.java 1209661 ./src/test/com/cloudera/sqoop/hbase/TestHBaseQueryImport.java 1209661 ./src/test/com/cloudera/sqoop/mapreduce/db/TestDataDrivenDBInputFormat.java 1209644 Diff: https://reviews.apache.org/r/2983/diff Testing ------- Thanks, Bilung
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/2983/#review3673
          -----------------------------------------------------------

          Ship it!

          Thanks! Built a release tar ball with the new patch and did basic testing against a real 4-node hadoop 0.23 cluster. One issue I saw is the record count in the import use-case is showing 0, but the records are physically imported, and the job is successfull. Seems to be a Sqoop problem with mapreduce counters in 0.23. I think this can be addressed in a separate jira.

          • Ahmed

          On 2011-12-06 19:14:02, Bilung Lee wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/2983/

          -----------------------------------------------------------

          (Updated 2011-12-06 19:14:02)

          Review request for Sqoop, Ahmed Radwan, Arvind Prabhakar, and Tom White.

          Summary

          -------

          Make Sqoop work with Hadoop 0.23

          This addresses bug SQOOP-397.

          https://issues.apache.org/jira/browse/SQOOP-397

          Diffs

          -----

          ./COMPILING.txt 1197706

          ./build.xml 1210580

          ./ivy.xml 1210686

          ./ivy/libraries.properties 1210579

          ./src/test/com/cloudera/sqoop/TestCompression.java 1208716

          ./src/test/com/cloudera/sqoop/hbase/HBaseTestCase.java 1209661

          ./src/test/com/cloudera/sqoop/hbase/TestHBaseImport.java 1209661

          ./src/test/com/cloudera/sqoop/hbase/TestHBaseQueryImport.java 1209661

          ./src/test/com/cloudera/sqoop/mapreduce/db/TestDataDrivenDBInputFormat.java 1209644

          Diff: https://reviews.apache.org/r/2983/diff

          Testing

          -------

          Thanks,

          Bilung

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2983/#review3673 ----------------------------------------------------------- Ship it! Thanks! Built a release tar ball with the new patch and did basic testing against a real 4-node hadoop 0.23 cluster. One issue I saw is the record count in the import use-case is showing 0, but the records are physically imported, and the job is successfull. Seems to be a Sqoop problem with mapreduce counters in 0.23. I think this can be addressed in a separate jira. Ahmed On 2011-12-06 19:14:02, Bilung Lee wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2983/ ----------------------------------------------------------- (Updated 2011-12-06 19:14:02) Review request for Sqoop, Ahmed Radwan, Arvind Prabhakar, and Tom White. Summary ------- Make Sqoop work with Hadoop 0.23 This addresses bug SQOOP-397 . https://issues.apache.org/jira/browse/SQOOP-397 Diffs ----- ./COMPILING.txt 1197706 ./build.xml 1210580 ./ivy.xml 1210686 ./ivy/libraries.properties 1210579 ./src/test/com/cloudera/sqoop/TestCompression.java 1208716 ./src/test/com/cloudera/sqoop/hbase/HBaseTestCase.java 1209661 ./src/test/com/cloudera/sqoop/hbase/TestHBaseImport.java 1209661 ./src/test/com/cloudera/sqoop/hbase/TestHBaseQueryImport.java 1209661 ./src/test/com/cloudera/sqoop/mapreduce/db/TestDataDrivenDBInputFormat.java 1209644 Diff: https://reviews.apache.org/r/2983/diff Testing ------- Thanks, Bilung
          Hide
          jiraposter@reviews.apache.org added a comment -

          On 2011-12-06 21:12:02, Ahmed Radwan wrote:

          > Thanks! Built a release tar ball with the new patch and did basic testing against a real 4-node hadoop 0.23 cluster. One issue I saw is the record count in the import use-case is showing 0, but the records are physically imported, and the job is successfull. Seems to be a Sqoop problem with mapreduce counters in 0.23. I think this can be addressed in a separate jira.

          I wonder if the counter issue is https://issues.apache.org/jira/browse/MAPREDUCE-3433 (fixed in 0.23.1).

          • Tom

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/2983/#review3673
          -----------------------------------------------------------

          On 2011-12-06 19:14:02, Bilung Lee wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/2983/

          -----------------------------------------------------------

          (Updated 2011-12-06 19:14:02)

          Review request for Sqoop, Ahmed Radwan, Arvind Prabhakar, and Tom White.

          Summary

          -------

          Make Sqoop work with Hadoop 0.23

          This addresses bug SQOOP-397.

          https://issues.apache.org/jira/browse/SQOOP-397

          Diffs

          -----

          ./COMPILING.txt 1197706

          ./build.xml 1210580

          ./ivy.xml 1210686

          ./ivy/libraries.properties 1210579

          ./src/test/com/cloudera/sqoop/TestCompression.java 1208716

          ./src/test/com/cloudera/sqoop/hbase/HBaseTestCase.java 1209661

          ./src/test/com/cloudera/sqoop/hbase/TestHBaseImport.java 1209661

          ./src/test/com/cloudera/sqoop/hbase/TestHBaseQueryImport.java 1209661

          ./src/test/com/cloudera/sqoop/mapreduce/db/TestDataDrivenDBInputFormat.java 1209644

          Diff: https://reviews.apache.org/r/2983/diff

          Testing

          -------

          Thanks,

          Bilung

          Show
          jiraposter@reviews.apache.org added a comment - On 2011-12-06 21:12:02, Ahmed Radwan wrote: > Thanks! Built a release tar ball with the new patch and did basic testing against a real 4-node hadoop 0.23 cluster. One issue I saw is the record count in the import use-case is showing 0, but the records are physically imported, and the job is successfull. Seems to be a Sqoop problem with mapreduce counters in 0.23. I think this can be addressed in a separate jira. I wonder if the counter issue is https://issues.apache.org/jira/browse/MAPREDUCE-3433 (fixed in 0.23.1). Tom ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2983/#review3673 ----------------------------------------------------------- On 2011-12-06 19:14:02, Bilung Lee wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2983/ ----------------------------------------------------------- (Updated 2011-12-06 19:14:02) Review request for Sqoop, Ahmed Radwan, Arvind Prabhakar, and Tom White. Summary ------- Make Sqoop work with Hadoop 0.23 This addresses bug SQOOP-397 . https://issues.apache.org/jira/browse/SQOOP-397 Diffs ----- ./COMPILING.txt 1197706 ./build.xml 1210580 ./ivy.xml 1210686 ./ivy/libraries.properties 1210579 ./src/test/com/cloudera/sqoop/TestCompression.java 1208716 ./src/test/com/cloudera/sqoop/hbase/HBaseTestCase.java 1209661 ./src/test/com/cloudera/sqoop/hbase/TestHBaseImport.java 1209661 ./src/test/com/cloudera/sqoop/hbase/TestHBaseQueryImport.java 1209661 ./src/test/com/cloudera/sqoop/mapreduce/db/TestDataDrivenDBInputFormat.java 1209644 Diff: https://reviews.apache.org/r/2983/diff Testing ------- Thanks, Bilung
          Hide
          jiraposter@reviews.apache.org added a comment -

          On 2011-12-06 21:12:02, Ahmed Radwan wrote:

          > Thanks! Built a release tar ball with the new patch and did basic testing against a real 4-node hadoop 0.23 cluster. One issue I saw is the record count in the import use-case is showing 0, but the records are physically imported, and the job is successfull. Seems to be a Sqoop problem with mapreduce counters in 0.23. I think this can be addressed in a separate jira.

          Tom White wrote:

          I wonder if the counter issue is https://issues.apache.org/jira/browse/MAPREDUCE-3433 (fixed in 0.23.1).

          Good point, I'll retry using the MAPREDUCE-3433 patch to confirm. Thanks!

          • Ahmed

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/2983/#review3673
          -----------------------------------------------------------

          On 2011-12-06 19:14:02, Bilung Lee wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/2983/

          -----------------------------------------------------------

          (Updated 2011-12-06 19:14:02)

          Review request for Sqoop, Ahmed Radwan, Arvind Prabhakar, and Tom White.

          Summary

          -------

          Make Sqoop work with Hadoop 0.23

          This addresses bug SQOOP-397.

          https://issues.apache.org/jira/browse/SQOOP-397

          Diffs

          -----

          ./COMPILING.txt 1197706

          ./build.xml 1210580

          ./ivy.xml 1210686

          ./ivy/libraries.properties 1210579

          ./src/test/com/cloudera/sqoop/TestCompression.java 1208716

          ./src/test/com/cloudera/sqoop/hbase/HBaseTestCase.java 1209661

          ./src/test/com/cloudera/sqoop/hbase/TestHBaseImport.java 1209661

          ./src/test/com/cloudera/sqoop/hbase/TestHBaseQueryImport.java 1209661

          ./src/test/com/cloudera/sqoop/mapreduce/db/TestDataDrivenDBInputFormat.java 1209644

          Diff: https://reviews.apache.org/r/2983/diff

          Testing

          -------

          Thanks,

          Bilung

          Show
          jiraposter@reviews.apache.org added a comment - On 2011-12-06 21:12:02, Ahmed Radwan wrote: > Thanks! Built a release tar ball with the new patch and did basic testing against a real 4-node hadoop 0.23 cluster. One issue I saw is the record count in the import use-case is showing 0, but the records are physically imported, and the job is successfull. Seems to be a Sqoop problem with mapreduce counters in 0.23. I think this can be addressed in a separate jira. Tom White wrote: I wonder if the counter issue is https://issues.apache.org/jira/browse/MAPREDUCE-3433 (fixed in 0.23.1). Good point, I'll retry using the MAPREDUCE-3433 patch to confirm. Thanks! Ahmed ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2983/#review3673 ----------------------------------------------------------- On 2011-12-06 19:14:02, Bilung Lee wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2983/ ----------------------------------------------------------- (Updated 2011-12-06 19:14:02) Review request for Sqoop, Ahmed Radwan, Arvind Prabhakar, and Tom White. Summary ------- Make Sqoop work with Hadoop 0.23 This addresses bug SQOOP-397 . https://issues.apache.org/jira/browse/SQOOP-397 Diffs ----- ./COMPILING.txt 1197706 ./build.xml 1210580 ./ivy.xml 1210686 ./ivy/libraries.properties 1210579 ./src/test/com/cloudera/sqoop/TestCompression.java 1208716 ./src/test/com/cloudera/sqoop/hbase/HBaseTestCase.java 1209661 ./src/test/com/cloudera/sqoop/hbase/TestHBaseImport.java 1209661 ./src/test/com/cloudera/sqoop/hbase/TestHBaseQueryImport.java 1209661 ./src/test/com/cloudera/sqoop/mapreduce/db/TestDataDrivenDBInputFormat.java 1209644 Diff: https://reviews.apache.org/r/2983/diff Testing ------- Thanks, Bilung
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/2983/#review3676
          -----------------------------------------------------------

          Ship it!

          +1

          • Arvind

          On 2011-12-06 19:14:02, Bilung Lee wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/2983/

          -----------------------------------------------------------

          (Updated 2011-12-06 19:14:02)

          Review request for Sqoop, Ahmed Radwan, Arvind Prabhakar, and Tom White.

          Summary

          -------

          Make Sqoop work with Hadoop 0.23

          This addresses bug SQOOP-397.

          https://issues.apache.org/jira/browse/SQOOP-397

          Diffs

          -----

          ./COMPILING.txt 1197706

          ./build.xml 1210580

          ./ivy.xml 1210686

          ./ivy/libraries.properties 1210579

          ./src/test/com/cloudera/sqoop/TestCompression.java 1208716

          ./src/test/com/cloudera/sqoop/hbase/HBaseTestCase.java 1209661

          ./src/test/com/cloudera/sqoop/hbase/TestHBaseImport.java 1209661

          ./src/test/com/cloudera/sqoop/hbase/TestHBaseQueryImport.java 1209661

          ./src/test/com/cloudera/sqoop/mapreduce/db/TestDataDrivenDBInputFormat.java 1209644

          Diff: https://reviews.apache.org/r/2983/diff

          Testing

          -------

          Thanks,

          Bilung

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2983/#review3676 ----------------------------------------------------------- Ship it! +1 Arvind On 2011-12-06 19:14:02, Bilung Lee wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2983/ ----------------------------------------------------------- (Updated 2011-12-06 19:14:02) Review request for Sqoop, Ahmed Radwan, Arvind Prabhakar, and Tom White. Summary ------- Make Sqoop work with Hadoop 0.23 This addresses bug SQOOP-397 . https://issues.apache.org/jira/browse/SQOOP-397 Diffs ----- ./COMPILING.txt 1197706 ./build.xml 1210580 ./ivy.xml 1210686 ./ivy/libraries.properties 1210579 ./src/test/com/cloudera/sqoop/TestCompression.java 1208716 ./src/test/com/cloudera/sqoop/hbase/HBaseTestCase.java 1209661 ./src/test/com/cloudera/sqoop/hbase/TestHBaseImport.java 1209661 ./src/test/com/cloudera/sqoop/hbase/TestHBaseQueryImport.java 1209661 ./src/test/com/cloudera/sqoop/mapreduce/db/TestDataDrivenDBInputFormat.java 1209644 Diff: https://reviews.apache.org/r/2983/diff Testing ------- Thanks, Bilung
          Hide
          jiraposter@reviews.apache.org added a comment -

          On 2011-12-06 21:04:57, Tom White wrote:

          > This looks good to me. Are you planning on doing the HBase and ZooKeeper version parameterization in another JIRA?

          Thanks! Yes, it would be better to wait until it is clear about what versions to use.

          • Bilung

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/2983/#review3672
          -----------------------------------------------------------

          On 2011-12-06 19:14:02, Bilung Lee wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/2983/

          -----------------------------------------------------------

          (Updated 2011-12-06 19:14:02)

          Review request for Sqoop, Ahmed Radwan, Arvind Prabhakar, and Tom White.

          Summary

          -------

          Make Sqoop work with Hadoop 0.23

          This addresses bug SQOOP-397.

          https://issues.apache.org/jira/browse/SQOOP-397

          Diffs

          -----

          ./COMPILING.txt 1197706

          ./build.xml 1210580

          ./ivy.xml 1210686

          ./ivy/libraries.properties 1210579

          ./src/test/com/cloudera/sqoop/TestCompression.java 1208716

          ./src/test/com/cloudera/sqoop/hbase/HBaseTestCase.java 1209661

          ./src/test/com/cloudera/sqoop/hbase/TestHBaseImport.java 1209661

          ./src/test/com/cloudera/sqoop/hbase/TestHBaseQueryImport.java 1209661

          ./src/test/com/cloudera/sqoop/mapreduce/db/TestDataDrivenDBInputFormat.java 1209644

          Diff: https://reviews.apache.org/r/2983/diff

          Testing

          -------

          Thanks,

          Bilung

          Show
          jiraposter@reviews.apache.org added a comment - On 2011-12-06 21:04:57, Tom White wrote: > This looks good to me. Are you planning on doing the HBase and ZooKeeper version parameterization in another JIRA? Thanks! Yes, it would be better to wait until it is clear about what versions to use. Bilung ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2983/#review3672 ----------------------------------------------------------- On 2011-12-06 19:14:02, Bilung Lee wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2983/ ----------------------------------------------------------- (Updated 2011-12-06 19:14:02) Review request for Sqoop, Ahmed Radwan, Arvind Prabhakar, and Tom White. Summary ------- Make Sqoop work with Hadoop 0.23 This addresses bug SQOOP-397 . https://issues.apache.org/jira/browse/SQOOP-397 Diffs ----- ./COMPILING.txt 1197706 ./build.xml 1210580 ./ivy.xml 1210686 ./ivy/libraries.properties 1210579 ./src/test/com/cloudera/sqoop/TestCompression.java 1208716 ./src/test/com/cloudera/sqoop/hbase/HBaseTestCase.java 1209661 ./src/test/com/cloudera/sqoop/hbase/TestHBaseImport.java 1209661 ./src/test/com/cloudera/sqoop/hbase/TestHBaseQueryImport.java 1209661 ./src/test/com/cloudera/sqoop/mapreduce/db/TestDataDrivenDBInputFormat.java 1209644 Diff: https://reviews.apache.org/r/2983/diff Testing ------- Thanks, Bilung
          Hide
          jiraposter@reviews.apache.org added a comment -

          On 2011-12-06 21:04:57, Tom White wrote:

          > This looks good to me. Are you planning on doing the HBase and ZooKeeper version parameterization in another JIRA?

          Bilung Lee wrote:

          Thanks! Yes, it would be better to wait until it is clear about what versions to use.

          Agreed. The HBase 0.92 RC (available now?) and ZooKeeper 3.4.0 (already out) are the ones to try with.

          • Tom

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/2983/#review3672
          -----------------------------------------------------------

          On 2011-12-06 19:14:02, Bilung Lee wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/2983/

          -----------------------------------------------------------

          (Updated 2011-12-06 19:14:02)

          Review request for Sqoop, Ahmed Radwan, Arvind Prabhakar, and Tom White.

          Summary

          -------

          Make Sqoop work with Hadoop 0.23

          This addresses bug SQOOP-397.

          https://issues.apache.org/jira/browse/SQOOP-397

          Diffs

          -----

          ./COMPILING.txt 1197706

          ./build.xml 1210580

          ./ivy.xml 1210686

          ./ivy/libraries.properties 1210579

          ./src/test/com/cloudera/sqoop/TestCompression.java 1208716

          ./src/test/com/cloudera/sqoop/hbase/HBaseTestCase.java 1209661

          ./src/test/com/cloudera/sqoop/hbase/TestHBaseImport.java 1209661

          ./src/test/com/cloudera/sqoop/hbase/TestHBaseQueryImport.java 1209661

          ./src/test/com/cloudera/sqoop/mapreduce/db/TestDataDrivenDBInputFormat.java 1209644

          Diff: https://reviews.apache.org/r/2983/diff

          Testing

          -------

          Thanks,

          Bilung

          Show
          jiraposter@reviews.apache.org added a comment - On 2011-12-06 21:04:57, Tom White wrote: > This looks good to me. Are you planning on doing the HBase and ZooKeeper version parameterization in another JIRA? Bilung Lee wrote: Thanks! Yes, it would be better to wait until it is clear about what versions to use. Agreed. The HBase 0.92 RC (available now?) and ZooKeeper 3.4.0 (already out) are the ones to try with. Tom ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2983/#review3672 ----------------------------------------------------------- On 2011-12-06 19:14:02, Bilung Lee wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2983/ ----------------------------------------------------------- (Updated 2011-12-06 19:14:02) Review request for Sqoop, Ahmed Radwan, Arvind Prabhakar, and Tom White. Summary ------- Make Sqoop work with Hadoop 0.23 This addresses bug SQOOP-397 . https://issues.apache.org/jira/browse/SQOOP-397 Diffs ----- ./COMPILING.txt 1197706 ./build.xml 1210580 ./ivy.xml 1210686 ./ivy/libraries.properties 1210579 ./src/test/com/cloudera/sqoop/TestCompression.java 1208716 ./src/test/com/cloudera/sqoop/hbase/HBaseTestCase.java 1209661 ./src/test/com/cloudera/sqoop/hbase/TestHBaseImport.java 1209661 ./src/test/com/cloudera/sqoop/hbase/TestHBaseQueryImport.java 1209661 ./src/test/com/cloudera/sqoop/mapreduce/db/TestDataDrivenDBInputFormat.java 1209644 Diff: https://reviews.apache.org/r/2983/diff Testing ------- Thanks, Bilung
          Hide
          Arvind Prabhakar added a comment -

          Patch committed. Thanks Bilung!

          Show
          Arvind Prabhakar added a comment - Patch committed. Thanks Bilung!
          Hide
          Hudson added a comment -

          Integrated in Sqoop-ant-jdk-1.6 #66 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6/66/)
          SQOOP-397. Make Sqoop work with Hadoop 0.23

          (Bilung Lee via Arvind Prabhakar)

          arvind : http://svn.apache.org/viewvc/?view=rev&rev=1211583
          Files :

          • /incubator/sqoop/trunk/COMPILING.txt
          • /incubator/sqoop/trunk/build.xml
          • /incubator/sqoop/trunk/ivy.xml
          • /incubator/sqoop/trunk/ivy/libraries.properties
          • /incubator/sqoop/trunk/src/test/com/cloudera/sqoop/TestCompression.java
          • /incubator/sqoop/trunk/src/test/com/cloudera/sqoop/hbase/HBaseTestCase.java
          • /incubator/sqoop/trunk/src/test/com/cloudera/sqoop/hbase/TestHBaseImport.java
          • /incubator/sqoop/trunk/src/test/com/cloudera/sqoop/hbase/TestHBaseQueryImport.java
          • /incubator/sqoop/trunk/src/test/com/cloudera/sqoop/mapreduce/db/TestDataDrivenDBInputFormat.java
          Show
          Hudson added a comment - Integrated in Sqoop-ant-jdk-1.6 #66 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6/66/ ) SQOOP-397 . Make Sqoop work with Hadoop 0.23 (Bilung Lee via Arvind Prabhakar) arvind : http://svn.apache.org/viewvc/?view=rev&rev=1211583 Files : /incubator/sqoop/trunk/COMPILING.txt /incubator/sqoop/trunk/build.xml /incubator/sqoop/trunk/ivy.xml /incubator/sqoop/trunk/ivy/libraries.properties /incubator/sqoop/trunk/src/test/com/cloudera/sqoop/TestCompression.java /incubator/sqoop/trunk/src/test/com/cloudera/sqoop/hbase/HBaseTestCase.java /incubator/sqoop/trunk/src/test/com/cloudera/sqoop/hbase/TestHBaseImport.java /incubator/sqoop/trunk/src/test/com/cloudera/sqoop/hbase/TestHBaseQueryImport.java /incubator/sqoop/trunk/src/test/com/cloudera/sqoop/mapreduce/db/TestDataDrivenDBInputFormat.java
          Hide
          jiraposter@reviews.apache.org added a comment -

          On 2011-12-06 21:12:02, Ahmed Radwan wrote:

          > Thanks! Built a release tar ball with the new patch and did basic testing against a real 4-node hadoop 0.23 cluster. One issue I saw is the record count in the import use-case is showing 0, but the records are physically imported, and the job is successfull. Seems to be a Sqoop problem with mapreduce counters in 0.23. I think this can be addressed in a separate jira.

          Tom White wrote:

          I wonder if the counter issue is https://issues.apache.org/jira/browse/MAPREDUCE-3433 (fixed in 0.23.1).

          Ahmed Radwan wrote:

          Good point, I'll retry using the MAPREDUCE-3433 patch to confirm. Thanks!

          Using a hadoop build having the MAPREDUCE-3433 patch resolves the counters problem described above. Thanks Tom for pointing that out. This patch is checked to trunk and 0.23.1 so this counters problem should only exist in 0.23.0, and not any of the following releases.

          • Ahmed

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/2983/#review3673
          -----------------------------------------------------------

          On 2011-12-06 19:14:02, Bilung Lee wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/2983/

          -----------------------------------------------------------

          (Updated 2011-12-06 19:14:02)

          Review request for Sqoop, Ahmed Radwan, Arvind Prabhakar, and Tom White.

          Summary

          -------

          Make Sqoop work with Hadoop 0.23

          This addresses bug SQOOP-397.

          https://issues.apache.org/jira/browse/SQOOP-397

          Diffs

          -----

          ./COMPILING.txt 1197706

          ./build.xml 1210580

          ./ivy.xml 1210686

          ./ivy/libraries.properties 1210579

          ./src/test/com/cloudera/sqoop/TestCompression.java 1208716

          ./src/test/com/cloudera/sqoop/hbase/HBaseTestCase.java 1209661

          ./src/test/com/cloudera/sqoop/hbase/TestHBaseImport.java 1209661

          ./src/test/com/cloudera/sqoop/hbase/TestHBaseQueryImport.java 1209661

          ./src/test/com/cloudera/sqoop/mapreduce/db/TestDataDrivenDBInputFormat.java 1209644

          Diff: https://reviews.apache.org/r/2983/diff

          Testing

          -------

          Thanks,

          Bilung

          Show
          jiraposter@reviews.apache.org added a comment - On 2011-12-06 21:12:02, Ahmed Radwan wrote: > Thanks! Built a release tar ball with the new patch and did basic testing against a real 4-node hadoop 0.23 cluster. One issue I saw is the record count in the import use-case is showing 0, but the records are physically imported, and the job is successfull. Seems to be a Sqoop problem with mapreduce counters in 0.23. I think this can be addressed in a separate jira. Tom White wrote: I wonder if the counter issue is https://issues.apache.org/jira/browse/MAPREDUCE-3433 (fixed in 0.23.1). Ahmed Radwan wrote: Good point, I'll retry using the MAPREDUCE-3433 patch to confirm. Thanks! Using a hadoop build having the MAPREDUCE-3433 patch resolves the counters problem described above. Thanks Tom for pointing that out. This patch is checked to trunk and 0.23.1 so this counters problem should only exist in 0.23.0, and not any of the following releases. Ahmed ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2983/#review3673 ----------------------------------------------------------- On 2011-12-06 19:14:02, Bilung Lee wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2983/ ----------------------------------------------------------- (Updated 2011-12-06 19:14:02) Review request for Sqoop, Ahmed Radwan, Arvind Prabhakar, and Tom White. Summary ------- Make Sqoop work with Hadoop 0.23 This addresses bug SQOOP-397 . https://issues.apache.org/jira/browse/SQOOP-397 Diffs ----- ./COMPILING.txt 1197706 ./build.xml 1210580 ./ivy.xml 1210686 ./ivy/libraries.properties 1210579 ./src/test/com/cloudera/sqoop/TestCompression.java 1208716 ./src/test/com/cloudera/sqoop/hbase/HBaseTestCase.java 1209661 ./src/test/com/cloudera/sqoop/hbase/TestHBaseImport.java 1209661 ./src/test/com/cloudera/sqoop/hbase/TestHBaseQueryImport.java 1209661 ./src/test/com/cloudera/sqoop/mapreduce/db/TestDataDrivenDBInputFormat.java 1209644 Diff: https://reviews.apache.org/r/2983/diff Testing ------- Thanks, Bilung

            People

            • Assignee:
              Bilung Lee
              Reporter:
              Tom White
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development