Uploaded image for project: 'Calcite'
  1. Calcite
  2. CALCITE-913

Avatica: transport of array fields fails

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.5.0
    • Component/s: avatica
    • Labels:
      None

      Description

      When using the avatica remote driver, reading a resultset which contains a non-null ARRAY-type field fails (both in json & protobuf).

      This can be reproduced by adding the following test to RemoteMetaTest.java:

        @Test public void testArrays() throws SQLException {
          try (AvaticaConnection conn = (AvaticaConnection) DriverManager.getConnection(url);
               Statement stmt = conn.createStatement()) {
      
            ResultSet resultSet =
                stmt.executeQuery("select * from (values ('a', array['b', 'c']));");
      
            assertTrue(resultSet.next());
          }
        }
      

      For example with json this gives the following error:

      2015-10-07 16:09:08.406:WARN:oejs.HttpChannel:qtp1931444790-20: /
      java.lang.RuntimeException: com.fasterxml.jackson.databind.JsonMappingException: Conflicting getter definitions for property "updateCount": org.hsqldb.result.Result#isUpdateCount(0 params) vs org.hsqldb.result.Result#getUpdateCount(0 params) (through reference chain: org.apache.calcite.avatica.remote.ExecuteResponse["results"]->java.util.ArrayList[0]->org.apache.calcite.avatica.remote.ResultSetResponse["firstFrame"]->org.apache.calcite.avatica.Frame["rows"]->java.util.ArrayList[0]->org.hsqldb.jdbc.JDBCArray["resultSet"]->org.hsqldb.jdbc.JDBCResultSet["result"])
      	at org.apache.calcite.avatica.remote.JsonHandler.handle(JsonHandler.java:61)
      	at org.apache.calcite.avatica.remote.JsonHandler.apply(JsonHandler.java:46)
      	at org.apache.calcite.avatica.server.AvaticaHandler.handle(AvaticaHandler.java:66)
      	at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)
      	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
      	at org.eclipse.jetty.server.Server.handle(Server.java:497)
      	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
      	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:245)
      	at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
      	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
      	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
      	at java.lang.Thread.run(Thread.java:745)
      Caused by: 
      com.fasterxml.jackson.databind.JsonMappingException: Conflicting getter definitions for property "updateCount": org.hsqldb.result.Result#isUpdateCount(0 params) vs org.hsqldb.result.Result#getUpdateCount(0 params) (through reference chain: org.apache.calcite.avatica.remote.ExecuteResponse["results"]->java.util.ArrayList[0]->org.apache.calcite.avatica.remote.ResultSetResponse["firstFrame"]->org.apache.calcite.avatica.Frame["rows"]->java.util.ArrayList[0]->org.hsqldb.jdbc.JDBCArray["resultSet"]->org.hsqldb.jdbc.JDBCResultSet["result"])
      	at com.fasterxml.jackson.databind.SerializerProvider._createAndCacheUntypedSerializer(SerializerProvider.java:897)
      	at com.fasterxml.jackson.databind.SerializerProvider.findValueSerializer(SerializerProvider.java:429)
      ...
      

      The above is with hsqldb's array implementation. With another one it went into infinite recursion.

        Issue Links

          Activity

          Hide
          julianhyde Julian Hyde added a comment -

          Can you try this on the latest master? I made some improvements to the handling of arrays in CALCITE-910.

          Show
          julianhyde Julian Hyde added a comment - Can you try this on the latest master? I made some improvements to the handling of arrays in CALCITE-910 .
          Hide
          bruno Bruno Dumon added a comment -

          Tested, but still fails.

          Looking at where Meta.Frame is constructed, in JdbcResultSet.frame() and JdbcResultSet.getValue(), I don't see any array handling in there, thus a java.sql.Array object will end up as-is in the Meta.Frame, which is then serialized by jackson, where it fails. It should probably be translated to a List.

          Show
          bruno Bruno Dumon added a comment - Tested, but still fails. Looking at where Meta.Frame is constructed, in JdbcResultSet.frame() and JdbcResultSet.getValue(), I don't see any array handling in there, thus a java.sql.Array object will end up as-is in the Meta.Frame, which is then serialized by jackson, where it fails. It should probably be translated to a List.
          Hide
          julianhyde Julian Hyde added a comment -

          Agreed.

          Show
          julianhyde Julian Hyde added a comment - Agreed.
          Hide
          elserj Josh Elser added a comment -

          Sniping this one if I may..

          After a quick hack to JdbcResultSet to serialize the Array to a List instead, it looks like the ColumnMetaData in the Signature might also be off. Specifically, it has a ScalarType instead of an ArrayType. Digging into that some more.

          Show
          elserj Josh Elser added a comment - Sniping this one if I may.. After a quick hack to JdbcResultSet to serialize the Array to a List instead, it looks like the ColumnMetaData in the Signature might also be off. Specifically, it has a ScalarType instead of an ArrayType. Digging into that some more.
          Hide
          julianhyde Julian Hyde added a comment -

          You're welcome to snipe!

          I suspect that the metadata will always be less than perfect (e.g. the provider may say that the "employees" column is an ARRAY but may not say of what). And java.sql.Struct doesn't provide much metadata - no column names or types. So the serialization should go by the types of objects it sees come back from the provider.

          I think it's worth dealing with array, multiset and struct as one. In JSON or protobuf, I think they would each be a list of values and a type flag (array, multiset, struct) which would probably be a Rep value.

          Show
          julianhyde Julian Hyde added a comment - You're welcome to snipe! I suspect that the metadata will always be less than perfect (e.g. the provider may say that the "employees" column is an ARRAY but may not say of what). And java.sql.Struct doesn't provide much metadata - no column names or types. So the serialization should go by the types of objects it sees come back from the provider. I think it's worth dealing with array, multiset and struct as one. In JSON or protobuf, I think they would each be a list of values and a type flag (array, multiset, struct) which would probably be a Rep value.
          Hide
          elserj Josh Elser added a comment -

          I suspect that the metadata will always be less than perfect (e.g. the provider may say that the "employees" column is an ARRAY but may not say of what)

          Get out of my head! I just ran into this conundrum

          Show
          elserj Josh Elser added a comment - I suspect that the metadata will always be less than perfect (e.g. the provider may say that the "employees" column is an ARRAY but may not say of what) Get out of my head! I just ran into this conundrum
          Hide
          julianhyde Julian Hyde added a comment -

          I was thinking of fixing this late last night and instead I ran through what needed to be done. Cooled off my ardour somewhat.

          Show
          julianhyde Julian Hyde added a comment - I was thinking of fixing this late last night and instead I ran through what needed to be done. Cooled off my ardour somewhat.
          Hide
          elserj Josh Elser added a comment -

          Link to a stab that at least makes Bruno's original unit test pass. Had to introduce another level of indirection into protobuf (to deal with repeated fields), but it was relatively straightforward all in all.

          Might have missed something then

          Show
          elserj Josh Elser added a comment - Link to a stab that at least makes Bruno's original unit test pass. Had to introduce another level of indirection into protobuf (to deal with repeated fields), but it was relatively straightforward all in all. Might have missed something then
          Hide
          julianhyde Julian Hyde added a comment -
          Show
          julianhyde Julian Hyde added a comment - Fixed in http://git-wip-us.apache.org/repos/asf/incubator-calcite/commit/bf178d55 . Thanks for the patch!
          Hide
          jcamachorodriguez Jesus Camacho Rodriguez added a comment -

          Resolved in release 1.5.0 (2015-11-10)

          Show
          jcamachorodriguez Jesus Camacho Rodriguez added a comment - Resolved in release 1.5.0 (2015-11-10)

            People

            • Assignee:
              elserj Josh Elser
              Reporter:
              bruno Bruno Dumon
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development