Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 1.5.0
    • Storage - Other
    • None

    Description

      Merge the work done here into Drill master so others can utilize the plugin: https://github.com/dremio/drill-storage-kudu

      Attachments

        Activity

          githubbot ASF GitHub Bot added a comment -

          GitHub user jacques-n opened a pull request:

          https://github.com/apache/drill/pull/314

          DRILL-4241: Add Kudu reader

          Implements an experimental Kudu reader & writer.

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/jacques-n/drill DRILL-4241

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/drill/pull/314.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #314


          commit bc9da49f93b90507c94a6eab3f83187b44d1d213
          Author: Jacques Nadeau <jacques.drill@gmail.com>
          Date: 2015-11-19T02:45:56Z

          DRILL-4241: initial commit

          commit dc9c0d097ab7cfd1b722a9065509a5167155a3b1
          Author: Todd Lipcon <todd@cloudera.com>
          Date: 2015-11-19T22:15:58Z

          DRILL-4241: Add pushdown of column projections

          commit 6fe5b352de522670b77985334d7b1af0f15c7467
          Author: Jacques Nadeau <jacques@apache.org>
          Date: 2015-11-19T22:21:14Z

          DRILL-4241: Add table metadata and DROP table support

          commit 756b5dfb5e2de3b5c45b5943451d7e73ab0b65e6
          Author: Todd Lipcon <todd@cloudera.com>
          Date: 2015-11-19T23:00:00Z

          DRILL-4241: Improve record reader and type mappings

          commit 4765e2d499510a844e24b33782b0402d6ea640d7
          Author: Steven Phillips <smp@apache.org>
          Date: 2015-11-20T00:00:40Z

          DRILL-4241: Add parallelization and assignment

          commit 4e8517898ed966f08ac8c0b1b20f7a9b08d2aae3
          Author: Todd Lipcon <todd@cloudera.com>
          Date: 2015-11-20T02:56:52Z

          DRILL-4241: Various Fixes

          • ReaderFix scanning of tables with >4096 rows
          • Fixing the build, make RAT and checkstyle happy.
          • Fix last batch of query not to NPE

          commit a7ff45079379dda2fb49bff71e1b12e4e37c7dda
          Author: Todd Lipcon <todd@cloudera.com>
          Date: 2015-11-20T00:15:36Z

          DRILL-4241: Rewrite RecordReader to support NULLs and be less Java-like

          commit 5f31e7a0a6e57db4d71214b52413b2e5937f545a
          Author: Amit Hadke <amit.hadke@gmail.com>
          Date: 2015-11-20T00:16:17Z

          DRILL-4241: Create table with multiple tablets in kudu test.

          commit 48a988a4ef7cd85dfd8dbf8f348b8fc9c9185ce7
          Author: Todd Lipcon <todd@cloudera.com>
          Date: 2015-11-20T00:19:45Z

          DRILL-4241: Timestamps should divide by 1000 in reader

          commit 77921147bcc2d9ebd6dac3241109c90337440868
          Author: Steven Phillips <smp@apache.org>
          Date: 2015-11-20T00:54:41Z

          DRILL-4241: Add wait stats and estimated row count in RecordReader

          commit 64c8e99a05f69af6d6118531d2d9081e843d1b0f
          Author: Jacques Nadeau <jacques@apache.org>
          Date: 2015-11-20T02:20:03Z

          DRILL-4241: Add Single Tablet Writer

          • Also move to a test bootstrap
          • Update to the latest kudu and Drill
          • Add plugin to Drill distribution
          • Checkstyle and directory cleanup

          githubbot ASF GitHub Bot added a comment - GitHub user jacques-n opened a pull request: https://github.com/apache/drill/pull/314 DRILL-4241 : Add Kudu reader Implements an experimental Kudu reader & writer. You can merge this pull request into a Git repository by running: $ git pull https://github.com/jacques-n/drill DRILL-4241 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/drill/pull/314.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #314 commit bc9da49f93b90507c94a6eab3f83187b44d1d213 Author: Jacques Nadeau <jacques.drill@gmail.com> Date: 2015-11-19T02:45:56Z DRILL-4241 : initial commit commit dc9c0d097ab7cfd1b722a9065509a5167155a3b1 Author: Todd Lipcon <todd@cloudera.com> Date: 2015-11-19T22:15:58Z DRILL-4241 : Add pushdown of column projections commit 6fe5b352de522670b77985334d7b1af0f15c7467 Author: Jacques Nadeau <jacques@apache.org> Date: 2015-11-19T22:21:14Z DRILL-4241 : Add table metadata and DROP table support commit 756b5dfb5e2de3b5c45b5943451d7e73ab0b65e6 Author: Todd Lipcon <todd@cloudera.com> Date: 2015-11-19T23:00:00Z DRILL-4241 : Improve record reader and type mappings commit 4765e2d499510a844e24b33782b0402d6ea640d7 Author: Steven Phillips <smp@apache.org> Date: 2015-11-20T00:00:40Z DRILL-4241 : Add parallelization and assignment commit 4e8517898ed966f08ac8c0b1b20f7a9b08d2aae3 Author: Todd Lipcon <todd@cloudera.com> Date: 2015-11-20T02:56:52Z DRILL-4241 : Various Fixes ReaderFix scanning of tables with >4096 rows Fixing the build, make RAT and checkstyle happy. Fix last batch of query not to NPE commit a7ff45079379dda2fb49bff71e1b12e4e37c7dda Author: Todd Lipcon <todd@cloudera.com> Date: 2015-11-20T00:15:36Z DRILL-4241 : Rewrite RecordReader to support NULLs and be less Java-like commit 5f31e7a0a6e57db4d71214b52413b2e5937f545a Author: Amit Hadke <amit.hadke@gmail.com> Date: 2015-11-20T00:16:17Z DRILL-4241 : Create table with multiple tablets in kudu test. commit 48a988a4ef7cd85dfd8dbf8f348b8fc9c9185ce7 Author: Todd Lipcon <todd@cloudera.com> Date: 2015-11-20T00:19:45Z DRILL-4241 : Timestamps should divide by 1000 in reader commit 77921147bcc2d9ebd6dac3241109c90337440868 Author: Steven Phillips <smp@apache.org> Date: 2015-11-20T00:54:41Z DRILL-4241 : Add wait stats and estimated row count in RecordReader commit 64c8e99a05f69af6d6118531d2d9081e843d1b0f Author: Jacques Nadeau <jacques@apache.org> Date: 2015-11-20T02:20:03Z DRILL-4241 : Add Single Tablet Writer Also move to a test bootstrap Update to the latest kudu and Drill Add plugin to Drill distribution Checkstyle and directory cleanup
          jnadeau Jacques Nadeau added a comment -

          jaltekruse, can you please review?

          jnadeau Jacques Nadeau added a comment - jaltekruse , can you please review?
          githubbot ASF GitHub Bot added a comment -

          Github user StevenMPhillips commented on the pull request:

          https://github.com/apache/drill/pull/314#issuecomment-168588400

          +1

          As long as this doesn't break the build, I say we go ahead and merge it. We can add tests and fixes later.

          githubbot ASF GitHub Bot added a comment - Github user StevenMPhillips commented on the pull request: https://github.com/apache/drill/pull/314#issuecomment-168588400 +1 As long as this doesn't break the build, I say we go ahead and merge it. We can add tests and fixes later.
          githubbot ASF GitHub Bot added a comment -

          Github user jaltekruse commented on the pull request:

          https://github.com/apache/drill/pull/314#issuecomment-168730376

          I agree with Steven, at the hackathon we already did some basic performance and correctness verification at reasonable scale. +1

          githubbot ASF GitHub Bot added a comment - Github user jaltekruse commented on the pull request: https://github.com/apache/drill/pull/314#issuecomment-168730376 I agree with Steven, at the hackathon we already did some basic performance and correctness verification at reasonable scale. +1
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/drill/pull/314

          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/drill/pull/314
          jnadeau Jacques Nadeau added a comment -

          Merged in 392d1f7e9398fb1bee8f67b0d49a82436c3145fb

          jnadeau Jacques Nadeau added a comment - Merged in 392d1f7e9398fb1bee8f67b0d49a82436c3145fb
          githubbot ASF GitHub Bot added a comment -

          Github user tdunning commented on the pull request:

          https://github.com/apache/drill/pull/314#issuecomment-185636190

          How could this have been merged? There is a huge double standard going on here.

          This code has NO comments. No tests. No documentation. No design. It isn't nearly good enough to pass the reviews that are required for others to contribute code.

          How can it be merged without any kind of significant review?

          githubbot ASF GitHub Bot added a comment - Github user tdunning commented on the pull request: https://github.com/apache/drill/pull/314#issuecomment-185636190 How could this have been merged? There is a huge double standard going on here. This code has NO comments. No tests. No documentation. No design. It isn't nearly good enough to pass the reviews that are required for others to contribute code. How can it be merged without any kind of significant review?
          githubbot ASF GitHub Bot added a comment -

          Github user jacques-n commented on the pull request:

          https://github.com/apache/drill/pull/314#issuecomment-185738887

          The Kudu plugin was contributed by six different developers, three of which are Drill PMC members and two more which are PMC members of other Apache projects. The code remained available for five days for review and received two plus ones and no negative feedback. It is modeled after the HBase plugin and works the same. It is unfortunate that there aren't integrated tests (due to the fact that there wasn't an easy way to provide integrated tests such as mini hbase cluster) but it was and is regularly manually tested. Due to the light testing, we are communicating it as experimental to users.

          Suggesting it didn't go through review when you have that large a group of developers involved is weird. Assuming that the user api isn't in dispute (which no one here disputed most likely because Kudu looks exactly like an Oracle table), providing experimental plugins increases the breadth of Drill's appeal and thus broadens and strengthens the community.

          githubbot ASF GitHub Bot added a comment - Github user jacques-n commented on the pull request: https://github.com/apache/drill/pull/314#issuecomment-185738887 The Kudu plugin was contributed by six different developers, three of which are Drill PMC members and two more which are PMC members of other Apache projects. The code remained available for five days for review and received two plus ones and no negative feedback. It is modeled after the HBase plugin and works the same. It is unfortunate that there aren't integrated tests (due to the fact that there wasn't an easy way to provide integrated tests such as mini hbase cluster) but it was and is regularly manually tested. Due to the light testing, we are communicating it as experimental to users. Suggesting it didn't go through review when you have that large a group of developers involved is weird. Assuming that the user api isn't in dispute (which no one here disputed most likely because Kudu looks exactly like an Oracle table), providing experimental plugins increases the breadth of Drill's appeal and thus broadens and strengthens the community.

          People

            jnadeau Jacques Nadeau
            jnadeau Jacques Nadeau
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: