Uploaded image for project: 'Calcite'
  1. Calcite
  2. CALCITE-470

CSV Example README Tutorial Doesn't Work

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.0.0-incubating
    • Component/s: None
    • Labels:
      None

      Description

      While kicking the tires I encountered the following failure:

      0: jdbc:calcite:model=target/test-classes/mod> !connect jdbc:calcite:model=target/test-classes/smart.json admin admin
      1: jdbc:calcite:model=target/test-classes/sma> EXPLAIN PLAN FOR SELECT * FROM emps;
      +------+
      | PLAN |
      +------+
      | CsvTableScan(table=[[SALES, EMPS]], fields=[[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]])
       |
      +------+
      1 row selected (0.342 seconds)
      1: jdbc:calcite:model=target/test-classes/sma> SELECT depts.name, count(*)
      . . . . . . . . . . . . . . . . . . . . . . .> FROM emps JOIN depts USING (deptno)
      . . . . . . . . . . . . . . . . . . . . . . .> GROUP BY depts.name;
      Nov 19, 2014 10:45:14 PM org.apache.calcite.sql.validate.SqlValidatorException <init>
      SEVERE: org.apache.calcite.sql.validate.SqlValidatorException: Column 'DEPTNO' matched using NATURAL keyword or USING clause has incompatible types: cannot compare 'JavaType(class java.lang.String)' to 'JavaType(class java.lang.Integer)'
      Nov 19, 2014 10:45:14 PM org.apache.calcite.runtime.CalciteException <init>
      SEVERE: org.apache.calcite.runtime.CalciteContextException: From line 2, column 29 to line 2, column 34: Column 'DEPTNO' matched using NATURAL keyword or USING clause has incompatible types: cannot compare 'JavaType(class java.lang.String)' to 'JavaType(class java.lang.Integer)'
      Error: while executing SQL: SELECT depts.name, count(*)
      FROM emps JOIN depts USING (deptno)
      GROUP BY depts.name (state=,code=0)
      

      Given a pointer to what needs to be addressed, I can try and provide a patch for this.

      1. CALCITE-470-2.patch
        0.6 kB
        Larry McCay
      2. CALCITE-470-3.patch
        0.9 kB
        Larry McCay
      3. EMPS.csv.gz
        0.3 kB
        Larry McCay
      4. OPTIQ-470.patch
        0.4 kB
        Larry McCay

        Activity

        Hide
        lmccay Larry McCay added a comment -

        I've found the issue and will post a patch.

        Show
        lmccay Larry McCay added a comment - I've found the issue and will post a patch.
        Hide
        vladimirsitnikov Vladimir Sitnikov added a comment -

        Can you please
        1) Adjust EMPS table instead so DEPTNO column is int
        2) Make CsvEnumerator complain if it encounters unknown column type hint. Currently it just ignores ":Integer" and assumes the unknown type is String

        Show
        vladimirsitnikov Vladimir Sitnikov added a comment - Can you please 1) Adjust EMPS table instead so DEPTNO column is int 2) Make CsvEnumerator complain if it encounters unknown column type hint. Currently it just ignores ":Integer" and assumes the unknown type is String
        Hide
        lmccay Larry McCay added a comment -

        Would you like it to just log a complaint and continue to assume it is string or throw a RuntimeException?
        Also, it seems the EMPS table is gzipped - I assume we want to keep it zipped?

        Show
        lmccay Larry McCay added a comment - Would you like it to just log a complaint and continue to assume it is string or throw a RuntimeException? Also, it seems the EMPS table is gzipped - I assume we want to keep it zipped?
        Hide
        vladimirsitnikov Vladimir Sitnikov added a comment -

        I think log and assume string is fine.
        I guess gz is intentional to show that the adapter supports such files.

        Show
        vladimirsitnikov Vladimir Sitnikov added a comment - I think log and assume string is fine. I guess gz is intentional to show that the adapter supports such files.
        Hide
        lmccay Larry McCay added a comment -

        Okay - second patch:

        1. sysout a warning message that an unknown type was found and that it is being assumed to be string.
        2. changed the EMPS.csv.gz file to be int rather than the DEPTS.csv to be string.

        Attaching the binary gz file separately.

        Show
        lmccay Larry McCay added a comment - Okay - second patch: 1. sysout a warning message that an unknown type was found and that it is being assumed to be string. 2. changed the EMPS.csv.gz file to be int rather than the DEPTS.csv to be string. Attaching the binary gz file separately.
        Hide
        lmccay Larry McCay added a comment -

        This file needs to replace the one in example/csv/src/test/resources/sales/EMPS.csv.gz

        Show
        lmccay Larry McCay added a comment - This file needs to replace the one in example/csv/src/test/resources/sales/EMPS.csv.gz
        Hide
        lmccay Larry McCay added a comment -

        Hi Vladimir Strisovsky - I've changed the patch and attached the updated binary file that includes the EMPS.csv.

        Show
        lmccay Larry McCay added a comment - Hi Vladimir Strisovsky - I've changed the patch and attached the updated binary file that includes the EMPS.csv.
        Hide
        vladimirsitnikov Vladimir Sitnikov added a comment -

        Your fix hit not exactly the right place in CsvEnumerator.
        RowConverter is used at the query execution time, while the original message happens at the query parse time.

        Can you please fix CsvEnumerator.deduceRowType?
        I mean adding a check after fieldType = CsvFieldType.of(typeString);.

        In warning message please include: file name, column name, and the unresolved type in question.

        Show
        vladimirsitnikov Vladimir Sitnikov added a comment - Your fix hit not exactly the right place in CsvEnumerator . RowConverter is used at the query execution time, while the original message happens at the query parse time. Can you please fix CsvEnumerator.deduceRowType ? I mean adding a check after fieldType = CsvFieldType.of(typeString); . In warning message please include: file name, column name, and the unresolved type in question.
        Hide
        lmccay Larry McCay added a comment -

        Okay.
        Is there any value in leaving the warning in the RowConverter as well - or should I remove that?

        Show
        lmccay Larry McCay added a comment - Okay. Is there any value in leaving the warning in the RowConverter as well - or should I remove that?
        Hide
        vlsi Vladimir Sitnikov added a comment -

        I do not think there is

        Show
        vlsi Vladimir Sitnikov added a comment - I do not think there is
        Hide
        lmccay Larry McCay added a comment - - edited

        Hi Vladimir Sitnikov - This patch version addresses the unknown type from the correct location. Resulting in a message like the following:

        "WARNING: found unknown type: Integer in file: /Users/larry/Projects/calcite-forked/example/csv/target/test-classes/sales/DEPTS.csv for column: DEPTNO"

        Show
        lmccay Larry McCay added a comment - - edited Hi Vladimir Sitnikov - This patch version addresses the unknown type from the correct location. Resulting in a message like the following: "WARNING: found unknown type: Integer in file: /Users/larry/Projects/calcite-forked/example/csv/target/test-classes/sales/DEPTS.csv for column: DEPTNO"
        Show
        vladimirsitnikov Vladimir Sitnikov added a comment - Commited as https://git-wip-us.apache.org/repos/asf?p=incubator-calcite.git;a=commitdiff;h=ca980a469702224c458c9e3473e679ff491c3578
        Hide
        julianhyde Julian Hyde added a comment -

        Closing now that 1.0.0-incubating has been released.

        Show
        julianhyde Julian Hyde added a comment - Closing now that 1.0.0-incubating has been released.

          People

          • Assignee:
            vladimirsitnikov Vladimir Sitnikov
            Reporter:
            lmccay Larry McCay
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development