Details

    • Type: Sub-task Sub-task
    • Status: Reopened
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.6.0
    • Fix Version/s: 0.6.0
    • Component/s: Debian, General, RPM
    • Labels:
      None

      Description

      I would like us to consider bumping version of Sqoop in Bigtop trunk (and Bigtop 0.6) from 1.99.1 to 1.99.2. Here's why:

      As a part of BIGTOP-949, I started playing around with Sqoop in Bigtop trunk. During this process, I found out that even simple commands like show connector --all fail with an error like:

      Exception has occurred during processing command 
      Exception: com.sun.jersey.api.client.UniformInterfaceException Message: GET http://localhost:8080/sqoop/v1/connector/all returned a response status of 400 Bad Request
      

      It turns out that Sqoop's tomcat is trying to bind to port 8080 but MR2 uses the same port 8080 for its shuffle manager (via a property called mapreduce.shuffle.port), hence the error.

      This makes Sqoop unusable on Bigtop trunk and there not much point in writing integration tests for it. The upstream Sqoop JIRA (SQOOP-881) resolves this issue and changes the default Tomcat port to 12000. However, the first release of Sqoop that contains this fix is 1.99.2. Morever, 1.99.2 contains other many other improvements related to Sqoop2.

      Given the usability of current Sqoop version in trunk, I would suggest we bump up the sqoop version from 1.99.1 to 1.99.2 for Bigtop 0.6. Sorry for the late timing of this request. I would love to hear your thoughts or concerns here, so please share!

        Activity

        made changes -
        Status Closed [ 6 ] Reopened [ 4 ]
        Roman Shaposhnik made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Roman Shaposhnik made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Hide
        Roman Shaposhnik added a comment -

        +1 and committed!

        Show
        Roman Shaposhnik added a comment - +1 and committed!
        Hide
        Mark Grover added a comment -

        +1 (non-committer)

        Show
        Mark Grover added a comment - +1 (non-committer)
        Sean Mackrory made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Sean Mackrory made changes -
        Hide
        Sean Mackrory added a comment -

        This patch bumps the version to 1.99.2 and fixes a few other problems. Sqoop 1.99.2 introduced a change in setenv.sh, underscoring the importance of utilizing the upstream template and tweaking as needed (done with sed), and also preserving any user-added changes to this file (done with the symlink to /etc/conf/sqoop). I also found that debhelper was causing 2 copies of the default file to get installed (only 1 is used - causing confusion), a file permissions issue, and a typo in the defaults file (I don't recall why that file sets SQOOP_CONFIG in the first place, but it appears to have no effect so I will consider removing it in a subsequent patch).

        Show
        Sean Mackrory added a comment - This patch bumps the version to 1.99.2 and fixes a few other problems. Sqoop 1.99.2 introduced a change in setenv.sh, underscoring the importance of utilizing the upstream template and tweaking as needed (done with sed), and also preserving any user-added changes to this file (done with the symlink to /etc/conf/sqoop). I also found that debhelper was causing 2 copies of the default file to get installed (only 1 is used - causing confusion), a file permissions issue, and a typo in the defaults file (I don't recall why that file sets SQOOP_CONFIG in the first place, but it appears to have no effect so I will consider removing it in a subsequent patch).
        Sean Mackrory made changes -
        Assignee Sean Mackrory [ mackrorysd ]
        Hide
        Sean Mackrory added a comment -

        +1 (non-committer)

        I've already done some work in anticipation of 1.99.2 - will post a patch soon.

        Show
        Sean Mackrory added a comment - +1 (non-committer) I've already done some work in anticipation of 1.99.2 - will post a patch soon.
        Roman Shaposhnik made changes -
        Field Original Value New Value
        Parent BIGTOP-816 [ 12625123 ]
        Issue Type Bug [ 1 ] Sub-task [ 7 ]
        Hide
        Roman Shaposhnik added a comment -

        This is a bit late in the game but looks worth it. I'd be +1 on the version bump given the benefits in testability.

        Show
        Roman Shaposhnik added a comment - This is a bit late in the game but looks worth it. I'd be +1 on the version bump given the benefits in testability.
        Mark Grover created issue -

          People

          • Assignee:
            Sean Mackrory
            Reporter:
            Mark Grover
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development