Details

    • Type: Improvement
    • Status: Patch Available
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: connectors
    • Labels:
      None

      Description

      Sqoop currently does not have a connector for Kudu. We should add the functionality to allow Sqoop to ingest data directly into Kudu.

      1. SQOOP-2903.1.patch
        68 kB
        Sameer Abhyankar
      2. SQOOP-2903.2.patch
        68 kB
        Sameer Abhyankar
      3. SQOOP-2903.patch
        68 kB
        Sameer Abhyankar

        Issue Links

          Activity

          Hide
          sabhyankar Sameer Abhyankar added a comment -

          Jarek Jarcec Cecho Can you assign this Jira to me? I have the functionality working locally and am in the process of doing some more testing. Once fully baked, I will upload a patch and add additional details.

          Show
          sabhyankar Sameer Abhyankar added a comment - Jarek Jarcec Cecho Can you assign this Jira to me? I have the functionality working locally and am in the process of doing some more testing. Once fully baked, I will upload a patch and add additional details.
          Hide
          jarcec Jarek Jarcec Cecho added a comment -

          Assigning to you Sameer Abhyankar

          Show
          jarcec Jarek Jarcec Cecho added a comment - Assigning to you Sameer Abhyankar
          Hide
          sabhyankar Sameer Abhyankar added a comment -

          Jarek Jarcec Cecho I have uploaded a patch for the Kudu connector. Let me know once you get a chance to review it!

          Show
          sabhyankar Sameer Abhyankar added a comment - Jarek Jarcec Cecho I have uploaded a patch for the Kudu connector. Let me know once you get a chance to review it!
          Hide
          sabhyankar Sameer Abhyankar added a comment -

          Note that the connector has runtime dependencies on kudu-client-0.7.1 and async-1.4.1. I am currently adding these two deps to the sqoop/lib manually. If someone can point me to a better way to automatically package it via ant then ill be happy to add that in!

          Show
          sabhyankar Sameer Abhyankar added a comment - Note that the connector has runtime dependencies on kudu-client-0.7.1 and async-1.4.1. I am currently adding these two deps to the sqoop/lib manually. If someone can point me to a better way to automatically package it via ant then ill be happy to add that in!
          Hide
          sabhyankar Sameer Abhyankar added a comment -

          I think I figured out how the dependencies for Kudu client (kudu-client and async) can be pulled in automatically into the lib folder. Attached SQOOP-2903.1.patch with the updated ivy deps for Kudu client.

          Show
          sabhyankar Sameer Abhyankar added a comment - I think I figured out how the dependencies for Kudu client (kudu-client and async) can be pulled in automatically into the lib folder. Attached SQOOP-2903 .1.patch with the updated ivy deps for Kudu client.
          Hide
          sabhyankar Sameer Abhyankar added a comment -

          Hi Jarek Jarcec Cecho - I just wanted to see if you have had a chance to review this patch! Thanks!

          Show
          sabhyankar Sameer Abhyankar added a comment - Hi Jarek Jarcec Cecho - I just wanted to see if you have had a chance to review this patch! Thanks!
          Hide
          jarcec Jarek Jarcec Cecho added a comment -

          Can you upload the patch to review board? It's quite sizable so it would be useful to have it there.

          Show
          jarcec Jarek Jarcec Cecho added a comment - Can you upload the patch to review board ? It's quite sizable so it would be useful to have it there.
          Hide
          maugli Attila Szabo added a comment -

          Hi Jarek Jarcec Cecho, Sameer Abhyankar,

          I would be more than happy to get involved into this connector, and also I offer my help to review the changes on my side too. I find it very exciting to have a Kudu + Sqoop connector.

          Cheers,
          Attila

          Show
          maugli Attila Szabo added a comment - Hi Jarek Jarcec Cecho , Sameer Abhyankar , I would be more than happy to get involved into this connector, and also I offer my help to review the changes on my side too. I find it very exciting to have a Kudu + Sqoop connector. Cheers, Attila
          Hide
          sabhyankar Sameer Abhyankar added a comment -

          [Jarek Jarcec Cecho Thanks!
          That would be awesome Attila Szabo . I will bump up the Kudu-client version to the latest supported (0.8.0) and put up a new patch for review!

          Show
          sabhyankar Sameer Abhyankar added a comment - [ Jarek Jarcec Cecho Thanks! That would be awesome Attila Szabo . I will bump up the Kudu-client version to the latest supported (0.8.0) and put up a new patch for review!
          Hide
          sabhyankar Sameer Abhyankar added a comment -

          Bumped up Kudu-client support to version 0.8.

          Show
          sabhyankar Sameer Abhyankar added a comment - Bumped up Kudu-client support to version 0.8.
          Hide
          sabhyankar Sameer Abhyankar added a comment -

          Hi Jarek Jarcec Cecho and Attila Szabo - I have uploaded the latest patch to review board. Can you guys please review and provide feedback?

          Thanks!

          Show
          sabhyankar Sameer Abhyankar added a comment - Hi Jarek Jarcec Cecho and Attila Szabo - I have uploaded the latest patch to review board. Can you guys please review and provide feedback? Thanks!
          Hide
          sabhyankar Sameer Abhyankar added a comment -

          Here is the review link:
          https://reviews.apache.org/r/47575/

          Show
          sabhyankar Sameer Abhyankar added a comment - Here is the review link: https://reviews.apache.org/r/47575/
          Hide
          jansechong jansechong added a comment - - edited

          HI, Sameer Abhyankar ,apply to which version of sqoop?THANKS.

          Show
          jansechong jansechong added a comment - - edited HI, Sameer Abhyankar ,apply to which version of sqoop?THANKS.
          Hide
          cam.mach.inspur Cam Quoc Mach added a comment -

          Hi Sameer Abhyankar Jarek Jarcec Cecho, I want to use this Kudu connector, but don't know how to apply the patch, and which sqoop version/branch should I use the patch? Can you guys give instructions or create a pull request or build for this patch? Please help!
          THanks

          Show
          cam.mach.inspur Cam Quoc Mach added a comment - Hi Sameer Abhyankar Jarek Jarcec Cecho , I want to use this Kudu connector, but don't know how to apply the patch, and which sqoop version/branch should I use the patch? Can you guys give instructions or create a pull request or build for this patch? Please help! THanks
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user cammachusa opened a pull request:

          https://github.com/apache/sqoop/pull/37

          SQOOP-2903 - Add Kudu connector for Sqoop

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/InspurUSA/sqoop SQOOP-2903

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/sqoop/pull/37.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #37


          commit c9c07b2e8eabfb2066187c42a4d533e29ceded6c
          Author: cam <cammach@inspur.com>
          Date: 2017-07-04T00:04:41Z

          SQOOP-2903 - Add Kudu connector for Sqoop


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user cammachusa opened a pull request: https://github.com/apache/sqoop/pull/37 SQOOP-2903 - Add Kudu connector for Sqoop You can merge this pull request into a Git repository by running: $ git pull https://github.com/InspurUSA/sqoop SQOOP-2903 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/sqoop/pull/37.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #37 commit c9c07b2e8eabfb2066187c42a4d533e29ceded6c Author: cam <cammach@inspur.com> Date: 2017-07-04T00:04:41Z SQOOP-2903 - Add Kudu connector for Sqoop
          Hide
          vasas Szabolcs Vasas added a comment -

          Hi Cam Quoc Mach,

          This patch was submitted a long time ago so most probably it has to be rebased to the latest trunk before it can applied. Is that what you have done in your change in the pull request or have you added something else too?

          Regards,
          Szabolcs

          Show
          vasas Szabolcs Vasas added a comment - Hi Cam Quoc Mach , This patch was submitted a long time ago so most probably it has to be rebased to the latest trunk before it can applied. Is that what you have done in your change in the pull request or have you added something else too? Regards, Szabolcs
          Hide
          cam.mach.inspur Cam Quoc Mach added a comment -

          Hi Szabolcs Vasas,
          No, I don't have anything new added, it's just the above patch in my pull request since I couldn't find this kudu connector in the latest trunk. If you can find it, pls to point me to where it is in the trunk? If not, can you please review the pull request and approve it, so others can use it?

          Thanks,
          Cam

          Show
          cam.mach.inspur Cam Quoc Mach added a comment - Hi Szabolcs Vasas , No, I don't have anything new added, it's just the above patch in my pull request since I couldn't find this kudu connector in the latest trunk. If you can find it, pls to point me to where it is in the trunk? If not, can you please review the pull request and approve it, so others can use it? Thanks, Cam
          Hide
          sabhyankar Sameer Abhyankar added a comment -

          Szabolcs Vasas Cam Quoc Mach If there is still interest in this patch, then I can update it to the latest trunk for both Kudu and Sqoop. Let me know.

          Show
          sabhyankar Sameer Abhyankar added a comment - Szabolcs Vasas Cam Quoc Mach If there is still interest in this patch, then I can update it to the latest trunk for both Kudu and Sqoop. Let me know.
          Hide
          cam.mach.inspur Cam Quoc Mach added a comment -

          Hi Sameer Abhyankar,

          Of course, we do, as least the whole of my team. So please help to update it the latest trunk.

          Thanks,
          Cam

          Show
          cam.mach.inspur Cam Quoc Mach added a comment - Hi Sameer Abhyankar , Of course, we do, as least the whole of my team. So please help to update it the latest trunk. Thanks, Cam
          Hide
          vasas Szabolcs Vasas added a comment -

          Hi Sameer Abhyankar,

          Yes, users definitely want this feature in Sqoop I think one of the reasons it has not yet committed is that there were no test cases added to the patch. Does Kudu provide a Java friendly minicluster now? If yes, can you please add test cases to your patch?

          Thanks,
          Szabolcs

          Show
          vasas Szabolcs Vasas added a comment - Hi Sameer Abhyankar , Yes, users definitely want this feature in Sqoop I think one of the reasons it has not yet committed is that there were no test cases added to the patch. Does Kudu provide a Java friendly minicluster now? If yes, can you please add test cases to your patch? Thanks, Szabolcs

            People

            • Assignee:
              sabhyankar Sameer Abhyankar
              Reporter:
              sabhyankar Sameer Abhyankar
            • Votes:
              1 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:

                Development