Apache Drill
  1. Apache Drill
  2. DRILL-487

Joining across different data sources throw 'schema change' exception

    Details

      Description

      Cross data source queries are throwing exception. Examples include between a file and Hive table, a hive table and HBase table, 2 files of different formats.

      Here is a simple example. I will add more detailed log.

      select c.c_custkey from
      dfs.`/home/mapr/tpch/sf-0.1_tpc-h_DataParquet/customer.parquet` c,
      dfs.`/home/mapr/sample-data/userprofilejson/user1.json` u
      where c.c_custkey = u.id;

      user1.json is below.
      {
      "id":8362,
      "account":

      { "name":"Alice", "email":"Alice@gmail.com" }

      ,
      "address":

      { "street":"123 Ballmer Av", "zipcode":"12345", "state":"CA" }

      }

      Exception:
      message: "Screen received stop request sent. < SchemaChangeException:[ org.apache.drill.exec.exception.ClassTransformationException: Failure generating transformation classes for value: \n \npackage org.apache.drill.exec.test.generated;\n\nimport org.apache.drill.exec.exception.SchemaChangeException;\nimport org.apache.drill.exec.ops.FragmentContext;\nimport org.apache.drill.exec.physical.impl.join.JoinStatus;\nimport org.apache.drill.exec.record.RecordBatch;\nimport org.apache.drill.exec.record.VectorContainer;\nimport org.apache.drill.exec.vector.BigIntVector;\nimport org.apache.drill.exec.vector.Float8Vector;\nimport org.apache.drill.exec.vector.NullableBigIntVector;\nimport org.apache.drill.exec.vector.NullableIntVector;\nimport org.apache.drill.exec.vector.NullableVarCharVector;\nimport org.apache.drill.exec.vector.VarBinaryVector;\n\npublic class JoinWorkerGen141 {\n\n JoinStatus status;\n VectorContainer outgoing;\n

        Activity

        Hide
        Jacques Nadeau added a comment -

        Fixed in 838fda8

        Show
        Jacques Nadeau added a comment - Fixed in 838fda8
        Hide
        Aman Sinha added a comment -

        Submitted patch with the following fixes:

        • Setup mappings appropriately for MergeJoin's compare functions.
        • Throw exception instead of assert when no join conditions are present otherwise if assertions are not enabled (which is the case with SqlLine), the execution continues until we encounter the code compilation error.
        • (Re) Enable PushFilterPastJoinRule (not sure why this was disabled). Without this rule, only JOIN..ON clause would get the proper join conditions. If the join conditions were present in the WHERE clause, it was not getting pushed to the join.
        Show
        Aman Sinha added a comment - Submitted patch with the following fixes: Setup mappings appropriately for MergeJoin's compare functions. Throw exception instead of assert when no join conditions are present otherwise if assertions are not enabled (which is the case with SqlLine), the execution continues until we encounter the code compilation error. (Re) Enable PushFilterPastJoinRule (not sure why this was disabled). Without this rule, only JOIN..ON clause would get the proper join conditions. If the join conditions were present in the WHERE clause, it was not getting pushed to the join.
        Hide
        Aman Sinha added a comment -

        The underlying problem is that the generated code for MergeJoin fails to compile. Following is the error from the logs:

        Caused by: org.codehaus.commons.compiler.CompileException: Line 43, Column 13: Method must not return a value
        at org.codehaus.janino.UnitCompiler.compileError(UnitCompiler.java:9014) ~[janino-2.6.1.jar:2.6.1]

        The generated code for JoinWorker has an invalid doSetup() method: (note line 43 which is returning value 0 whereas this method should not return anything):
        35: public void doSetup(FragmentContext context, JoinStatus status, VectorContainer outgoing)
        36: throws SchemaChangeException
        37: {
        38: {
        39: this.status = (status);
        40: this.outgoing = (outgoing);
        41: this.incomingLeft = status.left;
        42: this.incomingRight = status.right;
        43: return 0;
        44: return 0;

        Show
        Aman Sinha added a comment - The underlying problem is that the generated code for MergeJoin fails to compile. Following is the error from the logs: Caused by: org.codehaus.commons.compiler.CompileException: Line 43, Column 13: Method must not return a value at org.codehaus.janino.UnitCompiler.compileError(UnitCompiler.java:9014) ~ [janino-2.6.1.jar:2.6.1] The generated code for JoinWorker has an invalid doSetup() method: (note line 43 which is returning value 0 whereas this method should not return anything): 35: public void doSetup(FragmentContext context, JoinStatus status, VectorContainer outgoing) 36: throws SchemaChangeException 37: { 38: { 39: this.status = (status); 40: this.outgoing = (outgoing); 41: this.incomingLeft = status.left; 42: this.incomingRight = status.right; 43: return 0; 44: return 0;

          People

          • Assignee:
            Aman Sinha
            Reporter:
            Neeraja
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development