Apache Drill
  1. Apache Drill
  2. DRILL-310

Queries with multi level map type causes NPE

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Labels:
      None

      Description

      This issue is related to JIRA#: 307, which is the task to get rid of _MAP from drill sql.

      Easy way to reproduce this issue is to use the "donuts" schema and try and select a multi level map type as follows

      SELECT BATTERS.BATTER.ID from "/tmp/donuts.json";

      Following is the stack trace

      java.lang.NullPointerException
      at org.eigenbase.sql.validate.SqlValidatorUtil.lookupFieldType(SqlValidatorUtil.java:99)
      at org.eigenbase.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:4328)
      at org.eigenbase.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:4239)
      at org.eigenbase.sql.SqlIdentifier.accept(SqlIdentifier.java:266)
      at org.eigenbase.sql.validate.SqlValidatorImpl.deriveTypeImpl(SqlValidatorImpl.java:1521)
      at org.eigenbase.sql.validate.SqlValidatorImpl.deriveType(SqlValidatorImpl.java:1503)
      at org.eigenbase.sql.validate.SqlValidatorImpl.expandSelectItem(SqlValidatorImpl.java:435)
      at org.eigenbase.sql.validate.SqlValidatorImpl.validateSelectList(SqlValidatorImpl.java:3309)
      at org.eigenbase.sql.validate.SqlValidatorImpl.validateSelect(SqlValidatorImpl.java:3045)
      at org.eigenbase.sql.validate.SelectNamespace.validateImpl(SelectNamespace.java:69)
      at org.eigenbase.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:90)
      at org.eigenbase.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:802)
      at org.eigenbase.sql.validate.SqlValidatorImpl.validateQuery(SqlValidatorImpl.java:790)
      at org.eigenbase.sql.SqlSelect.validate(SqlSelect.java:152)
      at org.eigenbase.sql.validate.SqlValidatorImpl.validateScopedExpression(SqlValidatorImpl.java:753)
      at org.eigenbase.sql.validate.SqlValidatorImpl.validate(SqlValidatorImpl.java:444)
      at org.eigenbase.sql2rel.SqlToRelConverter.convertQuery(SqlToRelConverter.java:445)
      at net.hydromatic.optiq.prepare.Prepare.prepareSql(Prepare.java:162)
      at net.hydromatic.optiq.prepare.Prepare.prepareSql(Prepare.java:131)
      at net.hydromatic.optiq.prepare.OptiqPrepareImpl.prepare2_(OptiqPrepareImpl.java:256)
      at net.hydromatic.optiq.prepare.OptiqPrepareImpl.prepare_(OptiqPrepareImpl.java:196)
      at net.hydromatic.optiq.prepare.OptiqPrepareImpl.prepareSql(OptiqPrepareImpl.java:169)
      at net.hydromatic.optiq.jdbc.OptiqStatement.parseQuery(OptiqStatement.java:402)
      at net.hydromatic.optiq.jdbc.OptiqStatement.execute(OptiqStatement.java:192)
      at sqlline.SqlLine$Commands.execute(SqlLine.java:3825)
      at sqlline.SqlLine$Commands.sql(SqlLine.java:3738)
      at sqlline.SqlLine.dispatch(SqlLine.java:882)
      at sqlline.SqlLine.begin(SqlLine.java:717)
      at sqlline.SqlLine.mainWithInputRedirection(SqlLine.java:460)
      at sqlline.SqlLine.main(SqlLine.java:443)

        Activity

        Hide
        Timothy Chen added a comment -

        Drill actually doesn't support repeated map type yet, so the json reader only reads one row of the map.

        Show
        Timothy Chen added a comment - Drill actually doesn't support repeated map type yet, so the json reader only reads one row of the map.
        Hide
        Mehant Baid added a comment -

        A workaround is to use the following query

        SELECT BATTERS['BATTER']['ID'] from "/tmp/donuts.json";

        This query works in the same way as SELECT _MAP['BATTERS.BATTER.ID'] from "/tmp/donuts.json" works.

        I see a strange behavior while executing this query and I am not sure if this is a known issue or not.

        Consider the following dataset
        {
        "ID": 35,
        "BATTERS":
        {
        "BATTER":
        [

        { "ID": "1001", "TYPE": "Regular" }

        ,

        { "ID": "1002", "TYPE": "Chocolate" }

        ,

        { "ID": "1003", "TYPE": "Blueberry" }

        ,

        { "ID": "1004", "TYPE": "Devil's Food" }

        ]
        }
        }

        The query SELECT BATTERS['BATTER']['ID'] from "/tmp/donuts.json" would yield only the last record in the repeated type, 1004 in this case. This behavior would reproduce without my changes for JIRA 307 (Remove _MAP from Drill sql) with the equivalent query.

        I briefly looked at BatchLoaderMap (which generates results from value vectors and passes to JDBC driver) and it seems that the record batch it gets contains only the last value in the repeated type, so Project must be the problem here.

        Show
        Mehant Baid added a comment - A workaround is to use the following query SELECT BATTERS ['BATTER'] ['ID'] from "/tmp/donuts.json"; This query works in the same way as SELECT _MAP ['BATTERS.BATTER.ID'] from "/tmp/donuts.json" works. I see a strange behavior while executing this query and I am not sure if this is a known issue or not. Consider the following dataset { "ID": 35, "BATTERS": { "BATTER": [ { "ID": "1001", "TYPE": "Regular" } , { "ID": "1002", "TYPE": "Chocolate" } , { "ID": "1003", "TYPE": "Blueberry" } , { "ID": "1004", "TYPE": "Devil's Food" } ] } } The query SELECT BATTERS ['BATTER'] ['ID'] from "/tmp/donuts.json" would yield only the last record in the repeated type, 1004 in this case. This behavior would reproduce without my changes for JIRA 307 (Remove _MAP from Drill sql) with the equivalent query. I briefly looked at BatchLoaderMap (which generates results from value vectors and passes to JDBC driver) and it seems that the record batch it gets contains only the last value in the repeated type, so Project must be the problem here.

          People

          • Assignee:
            Mehant Baid
            Reporter:
            Mehant Baid
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:

              Development