Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.8.0
    • Fix Version/s: 1.0.0
    • Component/s: general
    • Labels:
      None

      Description

      Bigtop 0.9 may include sqoop1:

      Sqoop2 is not intended for production deployment(http://sqoop.apache.org/)

      Latest stable release is 1.4.5 (download, documentation). Latest cut of Sqoop2 is 1.99.3 (download, documentation). Note that 1.99.3 is not compatible with 1.4.5 and not feature complete, it is not intended for production deployment.

      How to do?

      1. port sqoop1 from bigtop 0.5

      2. separate sqoop1 and sqoop2
      a) sqoop2 keeps same directory
      b) sqoop1 use sqoop1 directory as:

      ./bigtop-packages/src/common/sqoop1
      ./bigtop-packages/src/deb/sqoop1
      ./bigtop-tests/smoke-tests/sqoop1/build.gradle

      3. sqoop1 and sqoop2 use same RPM name

      sqoop1: sqoop-1.4.4-1.noarch.rpm
      sqoop2: sqoop-1.99.3-1.noarch.rpm

        Issue Links

          Activity

          Hide
          warwithin YoungWoo Kim added a comment -

          Thanks Evans Ye. I filed BIGTOP-1721 and attached a patch.

          Show
          warwithin YoungWoo Kim added a comment - Thanks Evans Ye . I filed BIGTOP-1721 and attached a patch.
          Hide
          evans_ye Evans Ye added a comment - - edited

          Committed. Thanks Sean Mackrory.
          YoungWoo Kim now you can do the clean up in another JIRA

          Show
          evans_ye Evans Ye added a comment - - edited Committed. Thanks Sean Mackrory . YoungWoo Kim now you can do the clean up in another JIRA
          Hide
          evans_ye Evans Ye added a comment -

          The latest patch is good enough. I think we can take YoungWoo Kim's idea to get this in and do the clean up in another patch. +1 to this patch. If no one stands for objection, I'll commit this later.

          Show
          evans_ye Evans Ye added a comment - The latest patch is good enough. I think we can take YoungWoo Kim 's idea to get this in and do the clean up in another patch. +1 to this patch. If no one stands for objection, I'll commit this later.
          Hide
          mackrorysd Sean Mackrory added a comment -

          I would like to get rid of the .cmd and .gitignore files, but I'm extremely pressed for time right now if anyone else would like to take a stab, or file a follow-up JIRA for me to fix later. I don't think those are serious enough to hold up a release if people want Sqoop 1 back in the stack...

          Show
          mackrorysd Sean Mackrory added a comment - I would like to get rid of the .cmd and .gitignore files, but I'm extremely pressed for time right now if anyone else would like to take a stab, or file a follow-up JIRA for me to fix later. I don't think those are serious enough to hold up a release if people want Sqoop 1 back in the stack...
          Hide
          cos Konstantin Boudnik added a comment -

          If something needs a review - it's better be in Patch Available state.
          Sean Mackrory are you planning to make any more iterations on this one?

          Show
          cos Konstantin Boudnik added a comment - If something needs a review - it's better be in Patch Available state. Sean Mackrory are you planning to make any more iterations on this one?
          Hide
          warwithin YoungWoo Kim added a comment -

          Is anyone reviewing this one? I think, It's not too hard to remove the unnecessary files from another jira. and once the patch is committed, I would like to upload a patch for the clean up

          Show
          warwithin YoungWoo Kim added a comment - Is anyone reviewing this one? I think, It's not too hard to remove the unnecessary files from another jira. and once the patch is committed, I would like to upload a patch for the clean up
          Hide
          warwithin YoungWoo Kim added a comment -

          With Sqoop2, I did pretty much the same thing:

          • Building rpms and install Sqoop2
          • Install MySQL JDBC
          • Start Sqoop2 server
          • On Sqoop2 client shell, create links(HDFS and Generic JDBC), create a Job(imports data from MySQL to HDFS) and Start the job!
          • Verified the imported result on HDFS

          Sean Mackrory, Sqoop2 works fine so far. Thanks!

          Show
          warwithin YoungWoo Kim added a comment - With Sqoop2, I did pretty much the same thing: Building rpms and install Sqoop2 Install MySQL JDBC Start Sqoop2 server On Sqoop2 client shell, create links(HDFS and Generic JDBC), create a Job(imports data from MySQL to HDFS) and Start the job! Verified the imported result on HDFS Sean Mackrory , Sqoop2 works fine so far. Thanks!
          Hide
          warwithin YoungWoo Kim added a comment - - edited

          Sean Mackrory Thanks for your patch! I'm looking into Sqoop(1.4.5).

          Minor issues:

          • $SQOOP_HOME/bin/*.cmd should be removed.
            8 -rwxr-xr-x. 1 root root 6329 2015-02-26 11:48 configure-sqoop.cmd
            ...
            4 -rwxr-xr-x. 1 root root 1026 2015-02-26 11:48 sqoop.cmd
            
          • "$SQOOP_HOME/conf/ {.gitignore,sqoop-env-template.cmd}

            ", files should be removed.

          So far, Simple tests with MySQL work fine for me! and then Sqoop2...

          Show
          warwithin YoungWoo Kim added a comment - - edited Sean Mackrory Thanks for your patch! I'm looking into Sqoop(1.4.5). Minor issues: $SQOOP_HOME/bin/*.cmd should be removed. 8 -rwxr-xr-x. 1 root root 6329 2015-02-26 11:48 configure-sqoop.cmd ... 4 -rwxr-xr-x. 1 root root 1026 2015-02-26 11:48 sqoop.cmd "$SQOOP_HOME/conf/ {.gitignore,sqoop-env-template.cmd} ", files should be removed. So far, Simple tests with MySQL work fine for me! and then Sqoop2...
          Hide
          evans_ye Evans Ye added a comment -

          Yup, that one still built though . Tested the new patch and all the packages are successfully built. I give my unprofessional +1 to the patch. Maybe someone like Roman Shaposhnik or Konstantin Boudnik with expertise can have a glance on the patch for the final review?
          Regarding to asciidoc, we can't just revert BIGTOP-1438 to add it back since there're changes to the related files during the time. A JIRA with a simple patch can address this.

          Show
          evans_ye Evans Ye added a comment - Yup, that one still built though . Tested the new patch and all the packages are successfully built. I give my unprofessional +1 to the patch. Maybe someone like Roman Shaposhnik or Konstantin Boudnik with expertise can have a glance on the patch for the final review? Regarding to asciidoc, we can't just revert BIGTOP-1438 to add it back since there're changes to the related files during the time. A JIRA with a simple patch can address this.
          Hide
          mackrorysd Sean Mackrory added a comment -

          I'm actally surprised you were able to build Sqoop 2 DEBs. I left some irrelevant variables in there... Attaching a new patch that I just tested (build-time only) on Ubuntu 12.04.

          Show
          mackrorysd Sean Mackrory added a comment - I'm actally surprised you were able to build Sqoop 2 DEBs. I left some irrelevant variables in there... Attaching a new patch that I just tested (build-time only) on Ubuntu 12.04.
          Hide
          evans_ye Evans Ye added a comment -

          I can have sqoop1 and sqoop2 rpms and sqoop2 debs successfully built. Tested the sqoop1 rpm in a real cluster as well, it works.
          However, I got an error when building sqoop1 debs:

          + install -d -m 0755 debian/sqoop//usr/lib/sqoop
          + install -d -m 0755 debian/sqoop//usr/lib/sqoop
          + cp 'build/sqoop-/sqoop*.jar' debian/sqoop//usr/lib/sqoop
          cp: cannot stat 'build/sqoop-/sqoop*.jar': No such file or directory
          make[1]: *** [override_dh_auto_install] Error 1
          make[1]: Leaving directory `/host/bigtop-test2/output/sqoop/sqoop-1.4.5'
          make: *** [binary] Error 2
          dpkg-buildpackage: error: fakeroot debian/rules binary gave error exit status 2
          debuild: fatal error at line 1364:
          dpkg-buildpackage -rfakeroot -D -us -uc -b
           failed
          :sqoop-deb FAILED
          
          FAILURE: Build failed with an exception.
          
          Show
          evans_ye Evans Ye added a comment - I can have sqoop1 and sqoop2 rpms and sqoop2 debs successfully built. Tested the sqoop1 rpm in a real cluster as well, it works. However, I got an error when building sqoop1 debs: + install -d -m 0755 debian/sqoop //usr/lib/sqoop + install -d -m 0755 debian/sqoop //usr/lib/sqoop + cp 'build/sqoop-/sqoop*.jar' debian/sqoop //usr/lib/sqoop cp: cannot stat 'build/sqoop-/sqoop*.jar': No such file or directory make[1]: *** [override_dh_auto_install] Error 1 make[1]: Leaving directory `/host/bigtop-test2/output/sqoop/sqoop-1.4.5' make: *** [binary] Error 2 dpkg-buildpackage: error: fakeroot debian/rules binary gave error exit status 2 debuild: fatal error at line 1364: dpkg-buildpackage -rfakeroot -D -us -uc -b failed :sqoop-deb FAILED FAILURE: Build failed with an exception.
          Hide
          evans_ye Evans Ye added a comment -

          I tested the patch and generally it's good. Here're some minor comments though:

          • I got some whitespace errors when applying the patch
            warning: squelched 24 whitespace errors
            warning: 29 lines add whitespace errors.
            
          • Some of the packages are missing during sqoop-rpm build
            error: Failed build dependencies:
                    asciidoc is needed by sqoop-1.4.5-1.el6.noarch
                    xmlto is needed by sqoop-1.4.5-1.el6.noarch
            Installing /host/bigtop-test2/output/sqoop/sqoop-1.4.5-1.el6.src.rpm
            
            :sqoop-rpm FAILED
            

            After missing packages are installed on build environment, both sqoop1 and sqoop2 rpms are successfully built. I remember we used to require asciidoc in order to build bigtop, but removed in bigtop 0.8, now we might need to add it back.

          Just a quick feedback, I'll do the testing more thoroughly

          Show
          evans_ye Evans Ye added a comment - I tested the patch and generally it's good. Here're some minor comments though: I got some whitespace errors when applying the patch warning: squelched 24 whitespace errors warning: 29 lines add whitespace errors. Some of the packages are missing during sqoop-rpm build error: Failed build dependencies: asciidoc is needed by sqoop-1.4.5-1.el6.noarch xmlto is needed by sqoop-1.4.5-1.el6.noarch Installing /host/bigtop-test2/output/sqoop/sqoop-1.4.5-1.el6.src.rpm :sqoop-rpm FAILED After missing packages are installed on build environment, both sqoop1 and sqoop2 rpms are successfully built. I remember we used to require asciidoc in order to build bigtop, but removed in bigtop 0.8, now we might need to add it back. Just a quick feedback, I'll do the testing more thoroughly
          Hide
          mackrorysd Sean Mackrory added a comment -

          Removing an unnecessary and seemingly vendor-specific file from the patch...

          Show
          mackrorysd Sean Mackrory added a comment - Removing an unnecessary and seemingly vendor-specific file from the patch...
          Hide
          mackrorysd Sean Mackrory added a comment -

          Attaching a patch. A few notes:

          • Tested by building / installing and starting on Fedora 20 and CentOS 6. Packages do not collide, Sqoop 1.99.x is always referred to as "sqoop2" where there's ambiguity.
          • Compared with the last Sqoop 1.4.x packaging in Bigtop - only differences were things that we've changed across Bigtop since then, like standardizing on bash instead of relying on /bin/sh, consolidating locations of bigtop-utils, etc.
          • I can't compile Sqoop 1.4.5, as the test code relies on the API in commons-lang 2.5+, but they list 2.4 as a dependency. Tried fighting Ivy and upgrading the dependency, but eventually settled for the sed in do-component-build for now. I tried Sqoop 1.4.4 and just had other compile issues.
          Show
          mackrorysd Sean Mackrory added a comment - Attaching a patch. A few notes: Tested by building / installing and starting on Fedora 20 and CentOS 6. Packages do not collide, Sqoop 1.99.x is always referred to as "sqoop2" where there's ambiguity. Compared with the last Sqoop 1.4.x packaging in Bigtop - only differences were things that we've changed across Bigtop since then, like standardizing on bash instead of relying on /bin/sh, consolidating locations of bigtop-utils, etc. I can't compile Sqoop 1.4.5, as the test code relies on the API in commons-lang 2.5+, but they list 2.4 as a dependency. Tried fighting Ivy and upgrading the dependency, but eventually settled for the sed in do-component-build for now. I tried Sqoop 1.4.4 and just had other compile issues.
          Hide
          evans_ye Evans Ye added a comment -

          +1 to Sean Mackrory's proposal and patch. I'm glad to do the testing.

          Show
          evans_ye Evans Ye added a comment - +1 to Sean Mackrory 's proposal and patch. I'm glad to do the testing.
          Hide
          mackrorysd Sean Mackrory added a comment -

          Assuming you meant attaching a patch, I will try do that tomorrow morning. I may have enough time to do some limited testing after all as well...

          Show
          mackrorysd Sean Mackrory added a comment - Assuming you meant attaching a patch, I will try do that tomorrow morning. I may have enough time to do some limited testing after all as well...
          Hide
          rvs Roman Shaposhnik added a comment -

          +1 to what Sean Mackrory said!

          Show
          rvs Roman Shaposhnik added a comment - +1 to what Sean Mackrory said!
          Hide
          mackrorysd Sean Mackrory added a comment -

          Go ahead - I started on the aforementioned ticket but I do not have bandwidth right now to do much testing and we never settled on how exactly we wanted to include both side-by-side. I'm +1 to include sqoop 1 as 'sqoop' and sqoop 2 as 'sqoop2'. In fact, that's exactly what I did in a Certain Distribution of Hadoop - I'd be happy to just post all that as a patch as long as someone would be able to test that everything deploys fine amongst the rest of the Bigtop stack, etc..

          Show
          mackrorysd Sean Mackrory added a comment - Go ahead - I started on the aforementioned ticket but I do not have bandwidth right now to do much testing and we never settled on how exactly we wanted to include both side-by-side. I'm +1 to include sqoop 1 as 'sqoop' and sqoop 2 as 'sqoop2'. In fact, that's exactly what I did in a Certain Distribution of Hadoop - I'd be happy to just post all that as a patch as long as someone would be able to test that everything deploys fine amongst the rest of the Bigtop stack, etc..
          Hide
          evans_ye Evans Ye added a comment -

          Is there anyone already working on this? Otherwise I'll take a look on it and trying to provide a patch

          Show
          evans_ye Evans Ye added a comment - Is there anyone already working on this? Otherwise I'll take a look on it and trying to provide a patch
          Hide
          warwithin YoungWoo Kim added a comment -

          From Apache Sqoop site:

          Latest stable release is 1.4.5 (download, documentation). Latest cut of Sqoop2 is 1.99.4 (download, documentation). Note that 1.99.4 is not compatible with 1.4.5 and not feature complete, it is not intended for production deployment.

          so Bigtop should include packages 'sqoop' as Sqoop 1.4.x and 'sqoop2' as Sqoop 1.99.x.

          Show
          warwithin YoungWoo Kim added a comment - From Apache Sqoop site: Latest stable release is 1.4.5 (download, documentation). Latest cut of Sqoop2 is 1.99.4 (download, documentation). Note that 1.99.4 is not compatible with 1.4.5 and not feature complete, it is not intended for production deployment. so Bigtop should include packages 'sqoop' as Sqoop 1.4.x and 'sqoop2' as Sqoop 1.99.x.
          Hide
          rvs Roman Shaposhnik added a comment -

          Guo Ruijing I think its fair to make it a subtask. Simplifies tracking and all.

          Show
          rvs Roman Shaposhnik added a comment - Guo Ruijing I think its fair to make it a subtask. Simplifies tracking and all.
          Hide
          cos Konstantin Boudnik added a comment -

          we have an ongoing discussion about next release BOM on the dev@ list - let's finish it first.

          Show
          cos Konstantin Boudnik added a comment - we have an ongoing discussion about next release BOM on the dev@ list - let's finish it first.
          Hide
          rguo Guo Ruijing added a comment -

          can we move it to subtask of BIGTOP-1480?

          Show
          rguo Guo Ruijing added a comment - can we move it to subtask of BIGTOP-1480 ?
          Hide
          mackrorysd Sean Mackrory added a comment -

          Also, the ticket referred to is this: BIGTOP-1016. There the idea was to add it back as "sqoop1" - but given that it's been more than a year - do we feel any differently about the relationship between Sqoop 1 and Sqoop 2 now?

          Show
          mackrorysd Sean Mackrory added a comment - Also, the ticket referred to is this: BIGTOP-1016 . There the idea was to add it back as "sqoop1" - but given that it's been more than a year - do we feel any differently about the relationship between Sqoop 1 and Sqoop 2 now?
          Hide
          mackrorysd Sean Mackrory added a comment -

          Yeah - as you point out CDH kept all the Sqoop 1 binaries and directories as 'sqoop', added Sqoop 2 ones as 'sqoop2', and the two users shared the 'sqoop' group. Do we want to go with that same layout? Or the one Guo suggests? I really don't have much of an opinion on how it should be.

          Show
          mackrorysd Sean Mackrory added a comment - Yeah - as you point out CDH kept all the Sqoop 1 binaries and directories as 'sqoop', added Sqoop 2 ones as 'sqoop2', and the two users shared the 'sqoop' group. Do we want to go with that same layout? Or the one Guo suggests? I really don't have much of an opinion on how it should be.
          Hide
          rvs Roman Shaposhnik added a comment -

          Yup. I remember the same. I also remember that CDH ended up using sqoop and sqoop2 as package names. Sean Mackrory if we decide to start including both in Bigtop, would it be possible for you guys to submit the patches?

          Show
          rvs Roman Shaposhnik added a comment - Yup. I remember the same. I also remember that CDH ended up using sqoop and sqoop2 as package names. Sean Mackrory if we decide to start including both in Bigtop, would it be possible for you guys to submit the patches?
          Hide
          cos Konstantin Boudnik added a comment -

          I recall a similar conversation in the past. There might be even a ticket about this.

          Show
          cos Konstantin Boudnik added a comment - I recall a similar conversation in the past. There might be even a ticket about this.

            People

            • Assignee:
              mackrorysd Sean Mackrory
              Reporter:
              rguo Guo Ruijing
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development