Details

    • Type: Task Task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.6.0
    • Component/s: None
    • Labels:
      None

      Description

      Sqoop community has spend last couple of months working on Sqoop 2 that differs from original Sqoop 1 in almost every aspect of the project. We've recently started process of releasing "first cut" to get our users chance to touch first actual bits. As BigTop is doing extremely good job in integrating all components together, I would like to propose to also include Sqoop 2 (in addition to Sqoop 1).

      I'll be more then happy to help, but I'll need guidance to do so.

      1. BIGTOP-805.1.patch
        67 kB
        Sean Mackrory
      2. BIGTOP-805.2.patch
        67 kB
        Sean Mackrory
      3. BIGTOP-805.3.patch
        62 kB
        Sean Mackrory
      4. BIGTOP-805.5.patch
        127 kB
        Sean Mackrory
      5. BIGTOP-805.6.patch
        63 kB
        Sean Mackrory
      6. BIGTOP-805.7.patch
        63 kB
        Sean Mackrory
      7. BIGTOP-805.8.patch
        67 kB
        Sean Mackrory

        Activity

        Hide
        Roman Shaposhnik added a comment -

        +1 and committed!

        Show
        Roman Shaposhnik added a comment - +1 and committed!
        Hide
        Sean Mackrory added a comment -

        Reformatting patch

        Show
        Sean Mackrory added a comment - Reformatting patch
        Hide
        Sean Mackrory added a comment -

        Updating based on some feedback from Roman on Reviewboard

        Show
        Sean Mackrory added a comment - Updating based on some feedback from Roman on Reviewboard
        Hide
        Sean Mackrory added a comment -

        Updating based on some feedback from Jarcec on Reviewboard (only removing an unnecessary file and tweaking the package descriptions).

        Show
        Sean Mackrory added a comment - Updating based on some feedback from Jarcec on Reviewboard (only removing an unnecessary file and tweaking the package descriptions).
        Hide
        Sean Mackrory added a comment -

        I'm attaching an improved patch that's been through more testing and has received some feedback offline. The new patch:

        • Cleans up the environment and configuration a lot and starts using the /etc/defaults/sqoop-server file for common settings.
        • Adds a /usr/bin/sqoop-server wrapper that is used by the init script and the CLI to facilitate manual invocation, ZooKeeper-style.
        • Splits the classpath into a client-only directory (client-lib) and server-only directory (server-lib). The "common" jar is already bundled in the war, so the server directory is empty, but is intended for user-provided add-ons like the MySQL connector.
        Show
        Sean Mackrory added a comment - I'm attaching an improved patch that's been through more testing and has received some feedback offline. The new patch: Cleans up the environment and configuration a lot and starts using the /etc/defaults/sqoop-server file for common settings. Adds a /usr/bin/sqoop-server wrapper that is used by the init script and the CLI to facilitate manual invocation, ZooKeeper-style. Splits the classpath into a client-only directory (client-lib) and server-only directory (server-lib). The "common" jar is already bundled in the war, so the server directory is empty, but is intended for user-provided add-ons like the MySQL connector.
        Hide
        Sean Mackrory added a comment -

        Yup - done.

        Show
        Sean Mackrory added a comment - Yup - done.
        Hide
        Mark Grover added a comment -

        Thanks Sean. Could you please update the review as well?

        Show
        Mark Grover added a comment - Thanks Sean. Could you please update the review as well?
        Hide
        Sean Mackrory added a comment -

        Previously uploaded the wrong patch - this is the one that incorporates all the feedback from ReviewBoard and has been tested a lot more.

        Show
        Sean Mackrory added a comment - Previously uploaded the wrong patch - this is the one that incorporates all the feedback from ReviewBoard and has been tested a lot more.
        Hide
        Sean Mackrory added a comment -

        Removing trailing whitespace from patch...

        Show
        Sean Mackrory added a comment - Removing trailing whitespace from patch...
        Hide
        Sean Mackrory added a comment -

        Implementing some code review feedback

        Show
        Sean Mackrory added a comment - Implementing some code review feedback
        Hide
        Sean Mackrory added a comment -
        Show
        Sean Mackrory added a comment - Mark Grover Absolutely! https://reviews.apache.org/r/9264/
        Hide
        Jarek Jarcec Cecho added a comment -

        Sean Mackrory, thank you for putting this together!

        Show
        Jarek Jarcec Cecho added a comment - Sean Mackrory , thank you for putting this together!
        Hide
        Mark Grover added a comment -

        Sean, thanks for this! This is a fairly large patch. Would you mind uploading it to reviews.apache.org as well?

        Show
        Mark Grover added a comment - Sean, thanks for this! This is a fairly large patch. Would you mind uploading it to reviews.apache.org as well?
        Hide
        Sean Mackrory added a comment -

        I've attached a patch that works for both DEB and RPM packages. In addition to upgrading from Sqoop 1.x to Sqoop 2 (specifically, Sqoop 1.99.1 at this point), this also takes the init script template used for the Hadoop services and provides it for all components. I'm not sure what we'll be doing in the future for upstart and systemd, but for future init scripts this should save a lot of hassle and help us be more consistent.

        Show
        Sean Mackrory added a comment - I've attached a patch that works for both DEB and RPM packages. In addition to upgrading from Sqoop 1.x to Sqoop 2 (specifically, Sqoop 1.99.1 at this point), this also takes the init script template used for the Hadoop services and provides it for all components. I'm not sure what we'll be doing in the future for upstart and systemd, but for future init scripts this should save a lot of hassle and help us be more consistent.
        Hide
        Jarek Jarcec Cecho added a comment -

        Fine with me

        Show
        Jarek Jarcec Cecho added a comment - Fine with me
        Hide
        Sean Mackrory added a comment -

        Given the timing, it looks like this will probably end up on the proposed BOM for Bigtop 0.6.0.

        Show
        Sean Mackrory added a comment - Given the timing, it looks like this will probably end up on the proposed BOM for Bigtop 0.6.0.
        Hide
        Jarek Jarcec Cecho added a comment -

        Hi Sean,
        thank you very much for your feedback, I greatly appreciate the detailed response. Right now we're planning to do Sqoop 2 "first cut" this year - we've already kicked off the release procedure. However this is indeed a first cut of development branch that is more targeted for early adopters and alpha users, not for general audience. Having that goal in mind we've decided to change the version to 1.99.1 instead of "2.0.0". We believe that with this version scheme it will be obvious that it's very far from last stable release (1.4.2) and very near to 2.0, but it's not 2.0 quite yet. Having said that, I would personal prefer to include both Sqoop versions in BigTop, but at the same time it would be acceptable to upgrade only to Sqoop 2 in case that this is desirable by the BigTop project.

        Jarcec

        Show
        Jarek Jarcec Cecho added a comment - Hi Sean, thank you very much for your feedback, I greatly appreciate the detailed response. Right now we're planning to do Sqoop 2 "first cut" this year - we've already kicked off the release procedure. However this is indeed a first cut of development branch that is more targeted for early adopters and alpha users, not for general audience. Having that goal in mind we've decided to change the version to 1.99.1 instead of "2.0.0". We believe that with this version scheme it will be obvious that it's very far from last stable release (1.4.2) and very near to 2.0, but it's not 2.0 quite yet. Having said that, I would personal prefer to include both Sqoop versions in BigTop, but at the same time it would be acceptable to upgrade only to Sqoop 2 in case that this is desirable by the BigTop project. Jarcec
        Hide
        Sean Mackrory added a comment -

        I can do the work of putting this in. We probably need a bit of a discussion around including both Sqoop 1 and Sqoop 2 concurrently though. We don't currently (and I don't believe we have previously) included multiple versions of the same component in a release of Bigtop.

        Jarek Jarcec Cecho: Do you know when Sqoop 2.0 is getting released?

        Show
        Sean Mackrory added a comment - I can do the work of putting this in. We probably need a bit of a discussion around including both Sqoop 1 and Sqoop 2 concurrently though. We don't currently (and I don't believe we have previously) included multiple versions of the same component in a release of Bigtop. Jarek Jarcec Cecho : Do you know when Sqoop 2.0 is getting released?

          People

          • Assignee:
            Sean Mackrory
            Reporter:
            Jarek Jarcec Cecho
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development