[DRILL-7741] Columns are missing when using convert_from function

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.13.0
• Fix Version/s: 1.18.0
• Labels: ready-to-commit
• Assignee: Vova Vysotskyi
• Reporter: Vova Vysotskyi
• Reviewer: Bohdan Kazydub
• Created: 18/May/20 10:17
• Updated: 27/May/20 03:48
• Resolved: 25/May/20 22:36

Description

The query returns wrong results when convert_from function and additional columns may appear in new batches. Here is a simple test case for this issue:

  @Test
  public void testConvertFromJson() throws Exception {
    String fileName = "table.tsv";
    try (BufferedWriter writer = new BufferedWriter(new FileWriter(new File(dirTestWatcher.getRootDir(), fileName)))) {
      for (int i = 0; i < JSONRecordReader.DEFAULT_ROWS_PER_BATCH; i++) {
        writer.write("{\"id\":\"1\"}\n");
      }
      writer.write("{\"id\":\"2\",\"v\":[\"abc\"]}");
    }

    String sql = "SELECT t.m.id AS id, t.m.v[0] v FROM \n" +
        "(SELECT convert_from(columns[0], 'json') AS m FROM dfs.`%s`) t\n" +
        "where t.m.id='2'";

    testBuilder()
        .sqlQuery(sql, fileName)
        .unOrdered()
        .baselineColumns("id", "v")
        .baselineValues("2", "abc")
        .go();
  }

Currently, the "v" column is null since OK_NEW_SCHEMA wasn't returned from the project when the complex writer was used.