Details

    • Type: New Feature
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 6.0
    • Fix Version/s: 6.0
    • Component/s: clients - java
    • Labels:
      None

      Description

      This ticket is to create a JDBC Driver (thin client) for the new SQL interface (SOLR-7560). As part of this ticket a driver will be added to the Solrj libary under the package: org.apache.solr.client.solrj.io.sql

      Initial implementation will include basic Driver, Connection, Statement and ResultSet implementations.

      Future releases can build on this implementation to support a wide range of JDBC clients and tools.

      Syntax using parallel Map/Reduce for aggregations:

      Properties props = new Properties();
      props.put("aggregationMode", "map_reduce");
      props.put("numWorkers", "10");
      Connection con = DriverManager.getConnection("jdbc:solr://<zkhost:port>?collection=<collection>", props);
      Statement stmt = con.createStatement();
      ResultSet rs = stmt.executeQuery("select a, sum(b) from tablex group by a having sum(b) > 100");
      while(rs.next()) {
          String a = rs.getString("a");
          double sumB = rs.getDouble("sum(b)");
      }
      

      Syntax using JSON facet API for aggregations:

      Properties props = new Properties();
      props.put("aggregationMode", "facet");
      Connection con = DriverManager.getConnection("jdbc:solr://<zkhost:port>?collection=<collection>", props);
      Statement stmt = con.createStatement();
      ResultSet rs = stmt.executeQuery("select a, sum(b) from tablex group by a having sum(b) > 100");
      while(rs.next()) {
          String a = rs.getString("a");
          double sumB = rs.getDouble("sum(b)");
      }
      
      1. SOLR-7986.patch
        37 kB
        Joel Bernstein
      2. SOLR-7986.patch
        41 kB
        Joel Bernstein
      3. SOLR-7986.patch
        44 kB
        Joel Bernstein
      4. SOLR-7986.patch
        44 kB
        Joel Bernstein
      5. SOLR-7986.patch
        41 kB
        Joel Bernstein
      6. SOLR-7986.patch
        84 kB
        Joel Bernstein
      7. SOLR-7986.patch
        94 kB
        Joel Bernstein
      8. SOLR-7986-SPI.patch
        2 kB
        Uwe Schindler
      9. SOLR-7986.patch
        95 kB
        Joel Bernstein
      10. SOLR-7986.patch
        100 kB
        Joel Bernstein

        Issue Links

          Activity

          Hide
          susheel2777@gmail.com Susheel Kumar added a comment -

          Hi Joel,

          Created SOLR-8184 and attached the patch with tests and linked to SOLR-8125.

          Thanks,
          Susheel

          On Tue, Oct 20, 2015 at 9:15 PM, Joel Bernstein (JIRA) <jira@apache.org>

          Show
          susheel2777@gmail.com Susheel Kumar added a comment - Hi Joel, Created SOLR-8184 and attached the patch with tests and linked to SOLR-8125 . Thanks, Susheel On Tue, Oct 20, 2015 at 9:15 PM, Joel Bernstein (JIRA) <jira@apache.org>
          Hide
          susheel2777@gmail.com Susheel Kumar added a comment -

          Hi Kevin,

          I have created SOLR-8184 for negative tests and i see some commonality
          between the tests from 8179 & 8184. Interestingly the missing zookeeper
          test doesn't get pass in either patch.

          Thanks,
          Susheel

          On Wed, Oct 21, 2015 at 12:57 PM, Kevin Risden (JIRA) <jira@apache.org>

          Show
          susheel2777@gmail.com Susheel Kumar added a comment - Hi Kevin, I have created SOLR-8184 for negative tests and i see some commonality between the tests from 8179 & 8184. Interestingly the missing zookeeper test doesn't get pass in either patch. Thanks, Susheel On Wed, Oct 21, 2015 at 12:57 PM, Kevin Risden (JIRA) <jira@apache.org>
          Hide
          risdenk Kevin Risden added a comment -

          Susheel Kumar I worked on some of the DriverImpl logic in SOLR-8179. There is a test now for the empty ZK host. I didn't add too many other tests.

          Show
          risdenk Kevin Risden added a comment - Susheel Kumar I worked on some of the DriverImpl logic in SOLR-8179 . There is a test now for the empty ZK host. I didn't add too many other tests.
          Hide
          joel.bernstein Joel Bernstein added a comment -

          Hi Susheel Kumar, I just noticed your comment on this ticket. Thanks for adding the tests.

          Let's create a new ticket for these tests and link it to SOLR-8125.

          Show
          joel.bernstein Joel Bernstein added a comment - Hi Susheel Kumar , I just noticed your comment on this ticket. Thanks for adding the tests. Let's create a new ticket for these tests and link it to SOLR-8125 .
          Hide
          susheel2777@gmail.com Susheel Kumar added a comment -

          Hi Joel,

          I added locally 3 bad connection string tests. Please let me know your opinion before I provide a patch
          a) The first below test "testConnectionStringWithMissingZKHost" throws SolrException than SQLException. Is that right behaviour. Just wanted to confirm.
          b) Is it okay to add more public multiple tests method OR have one test method and calling various private test methods inside it. Any preference since I see the later is more common .
          c) The test "testConnectionStringWithWrongCollection" goes thru various retries before it fails. Below the console messages. Is that right behavior.

          Thanks,
          Susheel

          @Test
          public void testConnectionStringWithMissingZKHost() throws Exception

          { //should throw Solr exception as per current design exception.expect(SolrException.class); String zkHost = zkServer.getZkAddress(); Properties props = new Properties(); //bad connection string: missing zkHost Connection con = DriverManager.getConnection("jdbc:solr://" + "?collection=collection1", props); }

          @Test
          public void testConnectionStringJumbled() throws Exception

          { //should throw SQL exception exception.expect(SQLException.class); Properties props = new Properties(); String zkHost = zkServer.getZkAddress(); //Bad connection string: string jumbled Connection con = DriverManager.getConnection("solr:jdbc://" + zkHost + "?collection=collection1", props); }

          @Test
          public void testConnectionStringWithWrongCollection() throws Exception

          { //should throw SQL exception exception.expect(SQLException.class); Properties props = new Properties(); String zkHost = zkServer.getZkAddress(); //Bad connection string: wrong collection name Connection con = DriverManager.getConnection("jdbc:solr://" + zkHost + "?collection=mycollection", props); Statement stmt = con.createStatement(); ResultSet rs = stmt.executeQuery("select id, a_i, a_s, a_f from mycollection order by a_i desc limit 2"); }

          testConnectionStringWithWrongCollection console messages.


          Sep 22, 2015 5:26:28 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
          WARNING: Will linger awaiting termination of 6 leaked thread(s).
          95116 WARN (TEST-JdbcTest.testConnectionStringWithWrongCollection-seed#[8ED3B3162E3AB02D]-SendThread(127.0.0.1:55137)) [n:127.0.0.1:55092_cb_i%2Fof c:collection1 s:shard1 r:core_node4 x:collection1] o.a.z.ClientCnxn Session 0x14ff6f24fcf0012 for server null, unexpected error, closing socket connection and attempting reconnect
          java.net.ConnectException: Connection refused
          at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
          at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
          at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
          at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
          96786 WARN (TEST-JdbcTest.testConnectionStringWithWrongCollection-seed#[8ED3B3162E3AB02D]-SendThread(127.0.0.1:55137)) [n:127.0.0.1:55092_cb_i%2Fof c:collection1 s:shard1 r:core_node4 x:collection1] o.a.z.ClientCnxn Session 0x14ff6f24fcf0012 for server null, unexpected error, closing socket connection and attempting reconnect
          java.net.ConnectException: Connection refused
          at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
          at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
          at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
          at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
          98838 WARN (TEST-JdbcTest.testConnectionStringWithWrongCollection-seed#[8ED3B3162E3AB02D]-SendThread(127.0.0.1:55137)) [n:127.0.0.1:55092_cb_i%2Fof c:collection1 s:shard1 r:core_node4 x:collection1] o.a.z.ClientCnxn Session 0x14ff6f24fcf0012 for server null, unexpected error, closing socket connection and attempting reconnect
          java.net.ConnectException: Connection refused
          at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
          at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
          at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
          at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
          100696 WARN (TEST-JdbcTest.testConnectionStringWithWrongCollection-seed#[8ED3B3162E3AB02D]-SendThread(127.0.0.1:55137)) [n:127.0.0.1:55092_cb_i%2Fof c:collection1 s:shard1 r:core_node4 x:collection1] o.a.z.ClientCnxn Session 0x14ff6f24fcf0012 for server null, unexpected error, closing socket connection and attempting reconnect
          java.net.ConnectException: Connection refused
          at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
          at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
          at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
          at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
          102136 WARN (TEST-JdbcTest.testConnectionStringWithWrongCollection-seed#[8ED3B3162E3AB02D]-SendThread(127.0.0.1:55137)) [n:127.0.0.1:55092_cb_i%2Fof c:collection1 s:shard1 r:core_node4 x:collection1] o.a.z.ClientCnxn Session 0x14ff6f24fcf0012 for server null, unexpected error, closing socket connection and attempting reconnect
          java.net.ConnectException: Connection refused
          .......
          ......
          .......

          Show
          susheel2777@gmail.com Susheel Kumar added a comment - Hi Joel, I added locally 3 bad connection string tests. Please let me know your opinion before I provide a patch a) The first below test "testConnectionStringWithMissingZKHost" throws SolrException than SQLException. Is that right behaviour. Just wanted to confirm. b) Is it okay to add more public multiple tests method OR have one test method and calling various private test methods inside it. Any preference since I see the later is more common . c) The test "testConnectionStringWithWrongCollection" goes thru various retries before it fails. Below the console messages. Is that right behavior. Thanks, Susheel @Test public void testConnectionStringWithMissingZKHost() throws Exception { //should throw Solr exception as per current design exception.expect(SolrException.class); String zkHost = zkServer.getZkAddress(); Properties props = new Properties(); //bad connection string: missing zkHost Connection con = DriverManager.getConnection("jdbc:solr://" + "?collection=collection1", props); } @Test public void testConnectionStringJumbled() throws Exception { //should throw SQL exception exception.expect(SQLException.class); Properties props = new Properties(); String zkHost = zkServer.getZkAddress(); //Bad connection string: string jumbled Connection con = DriverManager.getConnection("solr:jdbc://" + zkHost + "?collection=collection1", props); } @Test public void testConnectionStringWithWrongCollection() throws Exception { //should throw SQL exception exception.expect(SQLException.class); Properties props = new Properties(); String zkHost = zkServer.getZkAddress(); //Bad connection string: wrong collection name Connection con = DriverManager.getConnection("jdbc:solr://" + zkHost + "?collection=mycollection", props); Statement stmt = con.createStatement(); ResultSet rs = stmt.executeQuery("select id, a_i, a_s, a_f from mycollection order by a_i desc limit 2"); } testConnectionStringWithWrongCollection console messages. Sep 22, 2015 5:26:28 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks WARNING: Will linger awaiting termination of 6 leaked thread(s). 95116 WARN (TEST-JdbcTest.testConnectionStringWithWrongCollection-seed# [8ED3B3162E3AB02D] -SendThread(127.0.0.1:55137)) [n:127.0.0.1:55092_cb_i%2Fof c:collection1 s:shard1 r:core_node4 x:collection1] o.a.z.ClientCnxn Session 0x14ff6f24fcf0012 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081) 96786 WARN (TEST-JdbcTest.testConnectionStringWithWrongCollection-seed# [8ED3B3162E3AB02D] -SendThread(127.0.0.1:55137)) [n:127.0.0.1:55092_cb_i%2Fof c:collection1 s:shard1 r:core_node4 x:collection1] o.a.z.ClientCnxn Session 0x14ff6f24fcf0012 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081) 98838 WARN (TEST-JdbcTest.testConnectionStringWithWrongCollection-seed# [8ED3B3162E3AB02D] -SendThread(127.0.0.1:55137)) [n:127.0.0.1:55092_cb_i%2Fof c:collection1 s:shard1 r:core_node4 x:collection1] o.a.z.ClientCnxn Session 0x14ff6f24fcf0012 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081) 100696 WARN (TEST-JdbcTest.testConnectionStringWithWrongCollection-seed# [8ED3B3162E3AB02D] -SendThread(127.0.0.1:55137)) [n:127.0.0.1:55092_cb_i%2Fof c:collection1 s:shard1 r:core_node4 x:collection1] o.a.z.ClientCnxn Session 0x14ff6f24fcf0012 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081) 102136 WARN (TEST-JdbcTest.testConnectionStringWithWrongCollection-seed# [8ED3B3162E3AB02D] -SendThread(127.0.0.1:55137)) [n:127.0.0.1:55092_cb_i%2Fof c:collection1 s:shard1 r:core_node4 x:collection1] o.a.z.ClientCnxn Session 0x14ff6f24fcf0012 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused ....... ...... .......
          Hide
          joel.bernstein Joel Bernstein added a comment -

          Thanks Uwe Schindler for your help on this ticket!

          Show
          joel.bernstein Joel Bernstein added a comment - Thanks Uwe Schindler for your help on this ticket!
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 1703867 from Joel Bernstein in branch 'dev/trunk'
          [ https://svn.apache.org/r1703867 ]

          SOLR-7986: JDBC Driver for SQL Interface

          Show
          jira-bot ASF subversion and git services added a comment - Commit 1703867 from Joel Bernstein in branch 'dev/trunk' [ https://svn.apache.org/r1703867 ] SOLR-7986 : JDBC Driver for SQL Interface
          Hide
          joel.bernstein Joel Bernstein added a comment -

          Just wrote a small test client with the JDBC Driver and it worked well. I think this is far enough along to commit to trunk. Only a very small part of the JDBC spec is implemented but it's already useful and I think it provides a good foundation for future work.

          Show
          joel.bernstein Joel Bernstein added a comment - Just wrote a small test client with the JDBC Driver and it worked well. I think this is far enough along to commit to trunk. Only a very small part of the JDBC spec is implemented but it's already useful and I think it provides a good foundation for future work.
          Hide
          joel.bernstein Joel Bernstein added a comment - - edited

          Thanks Uwe, I believe I've got the right SuppressForbidden.

          Show
          joel.bernstein Joel Bernstein added a comment - - edited Thanks Uwe, I believe I've got the right SuppressForbidden.
          Hide
          thetaphi Uwe Schindler added a comment -

          Hi Joel,
          yes, @SuppressForbidden is the way to go. Choose the right one (there is one in Lucene and one in Solrj/Solr). I'd use the latter one, otherwise you add a dependency to Lucene to Solrj.
          I think the forbidden API was added to make sure, we only use SLF4J, see SOLR-7825.

          Show
          thetaphi Uwe Schindler added a comment - Hi Joel, yes, @SuppressForbidden is the way to go. Choose the right one (there is one in Lucene and one in Solrj/Solr). I'd use the latter one, otherwise you add a dependency to Lucene to Solrj. I think the forbidden API was added to make sure, we only use SLF4J, see SOLR-7825 .
          Hide
          joel.bernstein Joel Bernstein added a comment - - edited

          Patch that passes precommit

          Show
          joel.bernstein Joel Bernstein added a comment - - edited Patch that passes precommit
          Hide
          joel.bernstein Joel Bernstein added a comment -

          I found the @SuppressForbidden annotation which allowed precommitt to pass. Currently this method is not implemented so java.util.logging.Logger is never instantiated.

          Show
          joel.bernstein Joel Bernstein added a comment - I found the @SuppressForbidden annotation which allowed precommitt to pass. Currently this method is not implemented so java.util.logging.Logger is never instantiated.
          Hide
          joel.bernstein Joel Bernstein added a comment -

          Uwe Schindler, I ran into a forbidden API's issue in Driver.getParentLogger() which returns java.util.logging.Logger. Any thought's on how to handle this?

          Show
          joel.bernstein Joel Bernstein added a comment - Uwe Schindler , I ran into a forbidden API's issue in Driver.getParentLogger() which returns java.util.logging.Logger. Any thought's on how to handle this?
          Hide
          joel.bernstein Joel Bernstein added a comment -

          Just tested without the META-INF resource and it does work with Class.forName.

          Show
          joel.bernstein Joel Bernstein added a comment - Just tested without the META-INF resource and it does work with Class.forName.
          Hide
          thetaphi Uwe Schindler added a comment -

          Looks good. So I assume it works without the Class#forName() now.

          The reason why you must have the static initializer is seen in source code: http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/7u40-b43/java/sql/DriverManager.java#510

          As you see it just iterates the driver instances, but does not register them. This was not 100% clear from the docs.

          Show
          thetaphi Uwe Schindler added a comment - Looks good. So I assume it works without the Class#forName() now. The reason why you must have the static initializer is seen in source code: http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/7u40-b43/java/sql/DriverManager.java#510 As you see it just iterates the driver instances, but does not register them. This was not 100% clear from the docs.
          Hide
          joel.bernstein Joel Bernstein added a comment -

          New patch that includes Uwe's patch (which worked perfectly). Also the connect() method now returns null if the url is not accepted.

          The call to DriverManager.registerDriver has been moved to a static initializer. I tried removing this call altogether but the driver won't load without it.

          Show
          joel.bernstein Joel Bernstein added a comment - New patch that includes Uwe's patch (which worked perfectly). Also the connect() method now returns null if the url is not accepted. The call to DriverManager.registerDriver has been moved to a static initializer. I tried removing this call altogether but the driver won't load without it.
          Hide
          joel.bernstein Joel Bernstein added a comment -

          Ok, thanks!

          Show
          joel.bernstein Joel Bernstein added a comment - Ok, thanks!
          Hide
          thetaphi Uwe Schindler added a comment - - edited

          One thing about your Driver:

          According to spec, the driver should return "null" when calling the method connect(), if the URL does not fit the driver. So basically do: if (!acceptURL(url)) return null;
          This is made for very early consumers that rely on that behaviour, see docs:

          Attempts to make a database connection to the given URL. The driver should return "null" if it realizes it is the wrong kind of driver to connect to the given URL. This will be common, as when the JDBC driver manager is asked to connect to a given URL it passes the URL to each loaded driver in turn.
          The driver should throw an SQLException if it is the right driver to connect to the given URL but has trouble connecting to the database.
          The java.util.Properties argument can be used to pass arbitrary string tag/value pairs as connection arguments. Normally at least "user" and "password" properties should be included in the Properties object.

          Your driver will completely fail with string parsing exceptions, I assume. This will prevent users with for example Oracle drivers on classpath, too, to connect to their database - fatal failure!

          Show
          thetaphi Uwe Schindler added a comment - - edited One thing about your Driver: According to spec, the driver should return "null" when calling the method connect(), if the URL does not fit the driver. So basically do: if (!acceptURL(url)) return null; This is made for very early consumers that rely on that behaviour, see docs: Attempts to make a database connection to the given URL. The driver should return "null" if it realizes it is the wrong kind of driver to connect to the given URL. This will be common, as when the JDBC driver manager is asked to connect to a given URL it passes the URL to each loaded driver in turn. The driver should throw an SQLException if it is the right driver to connect to the given URL but has trouble connecting to the database. The java.util.Properties argument can be used to pass arbitrary string tag/value pairs as connection arguments. Normally at least "user" and "password" properties should be included in the Properties object. Your driver will completely fail with string parsing exceptions, I assume. This will prevent users with for example Oracle drivers on classpath, too, to connect to their database - fatal failure!
          Hide
          thetaphi Uwe Schindler added a comment - - edited

          Most drivers I have seen (e.g. Sybase, Oracle) do both. Inside static initialization, so Java 5 code that does not do SPI can still load it (does not affect us, because we require minimum Java 7). Java 7+ for sure always loads via SPI. It may only fail when you have fckd up your classloaders, because it checks context class loader.

          So I would do both: Add the SPI control file and also add the register call to {{static { }}} block. The people can fall back to simple class.forName()

          Example mysql: This is static init for legacy users with Java 5: http://grepcode.com/file/repo1.maven.org/maven2/mysql/mysql-connector-java/5.1.35/com/mysql/jdbc/Driver.java#47

          The MySQL driver currently has no META-INF because it is still on Java 5.

          Don't forget to have a public ctor, too! Otherwise the SPI parser cannot create the driver.

          Show
          thetaphi Uwe Schindler added a comment - - edited Most drivers I have seen (e.g. Sybase, Oracle) do both. Inside static initialization, so Java 5 code that does not do SPI can still load it (does not affect us, because we require minimum Java 7). Java 7+ for sure always loads via SPI. It may only fail when you have fckd up your classloaders, because it checks context class loader. So I would do both: Add the SPI control file and also add the register call to {{static { }}} block. The people can fall back to simple class.forName() Example mysql: This is static init for legacy users with Java 5: http://grepcode.com/file/repo1.maven.org/maven2/mysql/mysql-connector-java/5.1.35/com/mysql/jdbc/Driver.java#47 The MySQL driver currently has no META-INF because it is still on Java 5. Don't forget to have a public ctor, too! Otherwise the SPI parser cannot create the driver.
          Hide
          joel.bernstein Joel Bernstein added a comment -

          Thanks for the patch Uwe! I'll play around with DriverManager.registerDriver and see if it can just be removed. And if not I'll move it to a static initialization block.

          Show
          joel.bernstein Joel Bernstein added a comment - Thanks for the patch Uwe! I'll play around with DriverManager.registerDriver and see if it can just be removed. And if not I'll move it to a static initialization block.
          Hide
          thetaphi Uwe Schindler added a comment -

          Patch to add the JDBC Service Provider.

          One thing: I think you can now remove the DriverManager.registerDriver from the constructor, but I am not sure. Just try it out.

          Most "legacy" drivers do this in the static initialization block - the newInstance() your had confused me...

          Show
          thetaphi Uwe Schindler added a comment - Patch to add the JDBC Service Provider. One thing: I think you can now remove the DriverManager.registerDriver from the constructor, but I am not sure. Just try it out. Most "legacy" drivers do this in the static initialization block - the newInstance() your had confused me...
          Hide
          thetaphi Uwe Schindler added a comment -

          Same as adding codecs to Lucene. Just create META-INF/services folder inside resources and add a file named with the abstract Driver class (java.sql.Driver - in Lucene its similar: org.apache.lucene.codecs.Codec as filename). Inside the file add the License header using # line prefix plus a single line with the implementation class: org.apache.solr.client.solrj.io.sql.DriverImpl). This file must of course be added to the solrj.jar file.

          I can do that for you. I will post a separate patch. Once you have done this, you can remove the following line from the docs and possible tests:

          Class.forName("org.apache.solr.client.solrj.io.sql.DriverImpl").newInstance();
          

          This is no longer needed then. If you call DriverManager.getConnection(), the SPI framework behind the SQL infratsructure scans classpath for all META-INF/services/java.sql.Driver files and loads the classes inside as instances and register them as Driver. Afterwards it can resolve the connection URI.

          Show
          thetaphi Uwe Schindler added a comment - Same as adding codecs to Lucene. Just create META-INF/services folder inside resources and add a file named with the abstract Driver class (java.sql.Driver - in Lucene its similar: org.apache.lucene.codecs.Codec as filename). Inside the file add the License header using # line prefix plus a single line with the implementation class: org.apache.solr.client.solrj.io.sql.DriverImpl). This file must of course be added to the solrj.jar file. I can do that for you. I will post a separate patch. Once you have done this, you can remove the following line from the docs and possible tests: Class .forName( "org.apache.solr.client.solrj.io.sql.DriverImpl" ).newInstance(); This is no longer needed then. If you call DriverManager.getConnection(), the SPI framework behind the SQL infratsructure scans classpath for all META-INF/services/java.sql.Driver files and loads the classes inside as instances and register them as Driver. Afterwards it can resolve the connection URI.
          Hide
          joel.bernstein Joel Bernstein added a comment - - edited

          Uwe, I was reading the spec on the META-INF resource and I'm not quite sure what's the right way to add a META-INF service to the Solrj jar. I'll do so more research but was wondering if you already knew the right approach?

          Show
          joel.bernstein Joel Bernstein added a comment - - edited Uwe, I was reading the spec on the META-INF resource and I'm not quite sure what's the right way to add a META-INF service to the Solrj jar. I'll do so more research but was wondering if you already knew the right approach?
          Hide
          joel.bernstein Joel Bernstein added a comment -
          Show
          joel.bernstein Joel Bernstein added a comment - Uwe Schindler
          Hide
          joel.bernstein Joel Bernstein added a comment -

          New patch that throws UnsupportedOperationException from appropriate methods.

          Show
          joel.bernstein Joel Bernstein added a comment - New patch that throws UnsupportedOperationException from appropriate methods.
          Hide
          joel.bernstein Joel Bernstein added a comment -

          Patch which includes the first set of tests.

          Show
          joel.bernstein Joel Bernstein added a comment - Patch which includes the first set of tests.
          Hide
          joel.bernstein Joel Bernstein added a comment -

          New patch with cleaned up caching.

          Show
          joel.bernstein Joel Bernstein added a comment - New patch with cleaned up caching.
          Hide
          joel.bernstein Joel Bernstein added a comment -

          Thanks, I'll take a look!

          Show
          joel.bernstein Joel Bernstein added a comment - Thanks, I'll take a look!
          Show
          thetaphi Uwe Schindler added a comment - See http://docs.oracle.com/javase/7/docs/api/java/sql/DriverManager.html introduction.
          Hide
          thetaphi Uwe Schindler added a comment - - edited

          For a full compliant JDBC driver you should also add a META-INF resource, so it can be loaded by DriverManager without doing the forName() on the base class. By that you can call the driver without any initialization on the Solr-specific driver, just by providing the URL to DriverManager.

          Show
          thetaphi Uwe Schindler added a comment - - edited For a full compliant JDBC driver you should also add a META-INF resource, so it can be loaded by DriverManager without doing the forName() on the base class. By that you can call the driver without any initialization on the Solr-specific driver, just by providing the URL to DriverManager.
          Hide
          joel.bernstein Joel Bernstein added a comment -

          New patch that registers the driver properly.

          Show
          joel.bernstein Joel Bernstein added a comment - New patch that registers the driver properly.
          Hide
          joel.bernstein Joel Bernstein added a comment -

          New patch handles client caching properly.

          Show
          joel.bernstein Joel Bernstein added a comment - New patch handles client caching properly.
          Hide
          joel.bernstein Joel Bernstein added a comment -

          First basic implementation. No tests, so I have no idea if this works.

          The switch between Map/Reduce and JSON facet API for aggregation would get set when the Connection is created and can't be changed for a connection currently.

          This is not as flexible as I would like, but for tools like Tableau that's how it will have to work.

          For more programmable clients we can add a switch to the ConnectionImpl, to switch back and forth.

          Show
          joel.bernstein Joel Bernstein added a comment - First basic implementation. No tests, so I have no idea if this works. The switch between Map/Reduce and JSON facet API for aggregation would get set when the Connection is created and can't be changed for a connection currently. This is not as flexible as I would like, but for tools like Tableau that's how it will have to work. For more programmable clients we can add a switch to the ConnectionImpl, to switch back and forth.
          Hide
          joel.bernstein Joel Bernstein added a comment -

          Patch with initial classes and skeleton methods.

          Very few of these methods will actually be implemented is the ticket.

          Show
          joel.bernstein Joel Bernstein added a comment - Patch with initial classes and skeleton methods. Very few of these methods will actually be implemented is the ticket.
          Hide
          joel.bernstein Joel Bernstein added a comment - - edited

          Some thoughts on the design:

          1) The Connection class should be SolrCloud aware. This means that the Database it connects to is a SolrCloud Collection with each node running the /sql handler. This Collection will be the facade for the distributed database. SQL queries can refer to Tables (Collections) that reside in the same ZooKeeper or a remote Zookeeper.

          2) The Statement and ResultSet can be a wrapper around a SolrStream class. The SolrStream class can be used to send SQL request to the /sql handler.

          3) The Statement will open and close the SolrStream.

          4) The ResultSet will iterate the Tuples from the SolrStream.

          Show
          joel.bernstein Joel Bernstein added a comment - - edited Some thoughts on the design: 1) The Connection class should be SolrCloud aware. This means that the Database it connects to is a SolrCloud Collection with each node running the /sql handler. This Collection will be the facade for the distributed database. SQL queries can refer to Tables (Collections) that reside in the same ZooKeeper or a remote Zookeeper. 2) The Statement and ResultSet can be a wrapper around a SolrStream class. The SolrStream class can be used to send SQL request to the /sql handler. 3) The Statement will open and close the SolrStream. 4) The ResultSet will iterate the Tuples from the SolrStream.

            People

            • Assignee:
              Unassigned
              Reporter:
              joel.bernstein Joel Bernstein
            • Votes:
              1 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development