Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-6374

test tpcds-q98.test has some incorrect data

    XMLWordPrintableJSON

    Details

    • Type: Test
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: Impala 2.9.0
    • Fix Version/s: Impala 3.1.0
    • Component/s: Infrastructure
    • Labels:
      None

      Description

      I happened to look through the unit tests and it looks like tpcds-q98.test has some bad data in it, but it is verifying correctly.

      One example (among maybe 12 or so) is on line 469:
      line 468: 'AAAAAAAAEKGDAAAA','Houses should ','Books','mystery',1.77,3341.80,1.96
      line 469: 'AAAAAAAAFFDDAAAA',','Books','mystery',2.79,4237.23,2.49

      Note that the 2nd field for line 468 looks normal, but line 469 has just a single quote.

      I believe this is happening on all strings that end with a comma for this test. The correct result for this line (I believe) should be (note the comma after Poor):
      'AAAAAAAAFFDDAAAA','French, civil hours must report essential values. Reasonable, complete judges vary clearly homes; often pleasant women would watch. Poor,','Books','mystery',2.79,4237.23,2.48

      My guess as to why this is happening is some code in test_result_verifier.py, specifically in the part that says:
      for col_val in row_string.split(','):

      1. This is a bit tricky because we need to handle the case where a comma may be in
      2. the middle of a string. We detect this by finding a split that starts with an
      3. opening string character but that doesn't end in a string character. It is
      4. possible for the first character to be a single-quote, so handle that case
        if (col_val.startswith("'") and not col_val.endswith("'")) or (col_val == "'"):

        Attachments

          Activity

            People

            • Assignee:
              tarmstrong Tim Armstrong
              Reporter:
              sjc362000 Stephen Carlin
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: