Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-4615

test_avro_schema_resolution.py fails with wrong results

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: Impala 2.8.0, Impala 2.9.0
    • Fix Version/s: Impala 2.9.0
    • Component/s: Backend

      Description

      Jenkins console output:

      02:07:54 ---------------------------- Captured stderr setup -----------------------------
      02:07:54 -- connecting to: localhost:21000
      02:07:54 SET sync_ddl=False;
      02:07:54 -- executing against localhost:21000
      02:07:54 DROP DATABASE IF EXISTS `test_avro_schema_resolution_cad70577` CASCADE;
      02:07:54 
      02:07:54 SET sync_ddl=False;
      02:07:54 -- executing against localhost:21000
      02:07:54 CREATE DATABASE `test_avro_schema_resolution_cad70577`;
      02:07:54 
      02:07:54 MainThread: Created database "test_avro_schema_resolution_cad70577" for test ID "query_test/test_avro_schema_resolution.py::TestAvroSchemaResolution::()::test_avro_schema_resolution[exec_option: {'disable_codegen': True, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0, 'batch_size': 0, 'num_nodes': 0} | table_format: avro/snap/block]"
      02:07:54 ----------------------------- Captured stderr call -----------------------------
      02:07:54 -- executing against localhost:21000
      02:07:54 use test_avro_schema_resolution_cad70577;
      02:07:54 
      02:07:54 SET disable_codegen=True;
      02:07:54 SET abort_on_error=1;
      02:07:54 SET exec_single_node_rows_threshold=0;
      02:07:54 SET batch_size=0;
      02:07:54 SET num_nodes=0;
      02:07:54 -- executing against localhost:21000
      02:07:54 select * from functional_avro_snap.schema_resolution_test;
      02:07:54 
      02:07:54 MainThread: Skipping type verification of Avro-format table.
      02:07:54 -- executing against localhost:21000
      02:07:54 select * from functional_avro_snap.no_avro_schema where year = 2009 order by id limit 1
      02:07:54 union all
      02:07:54 select * from functional_avro_snap.no_avro_schema where year = 2010 order by id limit 1;
      02:07:54 
      02:07:54 MainThread: Skipping type verification of Avro-format table.
      02:07:54 -- executing against localhost:21000
      02:07:54 select count(*) from functional_avro_snap.avro_coldef;
      02:07:54 
      02:07:54 MainThread: Skipping type verification of Avro-format table.
      02:07:54 MainThread: Comparing QueryTestResults (expected vs actual):
      02:07:54 10 != 0
      02:07:54  TestAvroSchemaResolution.test_avro_schema_resolution[exec_option: {'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0, 'batch_size': 0, 'num_nodes': 0} | table_format: avro/snap/block] 
      02:07:54 [gw2] linux2 -- Python 2.7.5 /data/jenkins/workspace/impala-cdh5-2.7.0_5.10.0-exhaustive-rhel7/repos/Impala/bin/../infra/python/env/bin/python
      02:07:54 query_test/test_avro_schema_resolution.py:35: in test_avro_schema_resolution
      02:07:54     self.run_test_case('QueryTest/avro-schema-resolution', vector, unique_database)
      02:07:54 common/impala_test_suite.py:353: in run_test_case
      02:07:54     self.__verify_results_and_errors(vector, test_section, result, use_db)
      02:07:54 common/impala_test_suite.py:238: in __verify_results_and_errors
      02:07:54     replace_filenames_with_placeholder)
      02:07:54 common/test_result_verifier.py:404: in verify_raw_results
      02:07:54     VERIFIER_MAP[verifier](expected, actual)
      02:07:54 common/test_result_verifier.py:231: in verify_query_result_is_equal
      02:07:54     assert expected_results == actual_results
      02:07:54 E   assert Comparing QueryTestResults (expected vs actual):
      02:07:54 E     10 != 0
      02:07:54 ---------------------------- Captured stderr setup -----------------------------
      02:07:54 SET sync_ddl=False;
      02:07:54 -- executing against localhost:21000
      02:07:54 DROP DATABASE IF EXISTS `test_avro_schema_resolution_9eabe99a` CASCADE;
      02:07:54 
      02:07:54 SET sync_ddl=False;
      02:07:54 -- executing against localhost:21000
      02:07:54 CREATE DATABASE `test_avro_schema_resolution_9eabe99a`;
      02:07:54 
      02:07:54 MainThread: Created database "test_avro_schema_resolution_9eabe99a" for test ID "query_test/test_avro_schema_resolution.py::TestAvroSchemaResolution::()::test_avro_schema_resolution[exec_option: {'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0, 'batch_size': 0, 'num_nodes': 0} | table_format: avro/snap/block]"
      02:07:54 ----------------------------- Captured stderr call -----------------------------
      02:07:54 -- executing against localhost:21000
      02:07:54 use test_avro_schema_resolution_9eabe99a;
      02:07:54 
      02:07:54 SET disable_codegen=False;
      02:07:54 SET abort_on_error=1;
      02:07:54 SET exec_single_node_rows_threshold=0;
      02:07:54 SET batch_size=0;
      02:07:54 SET num_nodes=0;
      02:07:54 -- executing against localhost:21000
      02:07:54 select * from functional_avro_snap.schema_resolution_test;
      02:07:54 
      02:07:54 MainThread: Skipping type verification of Avro-format table.
      02:07:54 -- executing against localhost:21000
      02:07:54 select * from functional_avro_snap.no_avro_schema where year = 2009 order by id limit 1
      02:07:54 union all
      02:07:54 select * from functional_avro_snap.no_avro_schema where year = 2010 order by id limit 1;
      02:07:54 
      02:07:54 MainThread: Skipping type verification of Avro-format table.
      02:07:54 -- executing against localhost:21000
      02:07:54 select count(*) from functional_avro_snap.avro_coldef;
      02:07:54 
      02:07:54 MainThread: Skipping type verification of Avro-format table.
      02:07:54 MainThread: Comparing QueryTestResults (expected vs actual):
      02:07:54 10 != 0
      02:07:54  generated xml file: /data/jenkins/workspace/impala-cdh5-2.7.0_5.10.0-exhaustive-rhel7/repos/Impala/logs/ee_tests/results/TEST-impala-parallel.xml 
      02:07:54 =========================== short test summary info ============================
      

        Issue Links

          Activity

          Hide
          alex.behm Alexander Behm added a comment -

          Michael Ho I'm not really sure what might have introduced this problem, can you help me take a look? Thanks! It might be related to IMPALA-4614

          Show
          alex.behm Alexander Behm added a comment - Michael Ho I'm not really sure what might have introduced this problem, can you help me take a look? Thanks! It might be related to IMPALA-4614
          Hide
          alex.behm Alexander Behm added a comment -

          Nvm, only in an internal branch.

          Show
          alex.behm Alexander Behm added a comment - Nvm, only in an internal branch.
          Hide
          tarmstrong Tim Armstrong added a comment -

          Alexander Behm did you figure out what happened here? I'm running into this reproducible on my own branch, which has changed nothing related to avro.

          Show
          tarmstrong Tim Armstrong added a comment - Alexander Behm did you figure out what happened here? I'm running into this reproducible on my own branch, which has changed nothing related to avro.
          Hide
          alex.behm Alexander Behm added a comment -

          Unfortunately, I have not been able to figure out what happened, but maybe it's time to take a closer look.

          Show
          alex.behm Alexander Behm added a comment - Unfortunately, I have not been able to figure out what happened, but maybe it's time to take a closer look.
          Hide
          mikesbrown Michael Brown added a comment -

          I saw this on a private build based off commit Qualify min() in header.

          Show
          mikesbrown Michael Brown added a comment - I saw this on a private build based off commit Qualify min() in header .
          Hide
          kwho Michael Ho added a comment -

          Won't get to it this week. Unassigning for now in case someone can take a look into whether there is something wrong with data loading.

          Show
          kwho Michael Ho added a comment - Won't get to it this week. Unassigning for now in case someone can take a look into whether there is something wrong with data loading.
          Hide
          alex.behm Alexander Behm added a comment -

          Lars, please take a look. Let me know if you are swamped and I'll find someone else.

          Show
          alex.behm Alexander Behm added a comment - Lars, please take a look. Let me know if you are swamped and I'll find someone else.
          Hide
          lv Lars Volker added a comment -

          I found out what caused this and pushed a change here: https://gerrit.cloudera.org/#/c/6317/

          The reason we didn't see this more often is that we usually run with SKIP_METADATA_LOAD=1. However, we can fall back to SKIP_METADATA_LOAD=0 if check-schema-diff.sh fails, which can happen on private branches and private builds.

          The final clue came from this blog post by Eric Lin. Kudos, Eric!

          Show
          lv Lars Volker added a comment - I found out what caused this and pushed a change here: https://gerrit.cloudera.org/#/c/6317/ The reason we didn't see this more often is that we usually run with SKIP_METADATA_LOAD=1 . However, we can fall back to SKIP_METADATA_LOAD=0 if check-schema-diff.sh fails, which can happen on private branches and private builds. The final clue came from this blog post by Eric Lin . Kudos, Eric!
          Hide
          lv Lars Volker added a comment -

          IMPALA-4615: Fix create_table.sql command order

          INSERT OVERWRITE commands in Hive will only affect partitions that Hive
          knows about. If an external table gets dropped and recreated, then
          'MSCK REPAIR TABLE' needs to be executed to recover any preexisting
          partitions. Otherwise, an INSERT OVERWRITE will not remove the data
          files in those partitions and will fail to move the new data in place.

          More information can be found here:
          http://www.ericlin.me/hive-insert-overwrite-does-not-remove-existing-data

          I tested the fix by running the following commands, making sure that the
          second run of the .sql script completed without errors and validating
          the number of lines was correct (10) after both runs.

          export JDBC_URL="jdbc:hive2://$

          {HS2_HOST_PORT}

          /default;"
          export HS2_HOST_PORT=localhost:11050

          beeline -n $USER -u "$

          {JDBC_URL}" -f ${IMPALA_HOME}/testdata/avro_schema_resolution/create_table.sql
          beeline -n $USER -u "${JDBC_URL}

          " -f $

          {IMPALA_HOME}

          /testdata/avro_schema_resolution/create_table.sql

          Change-Id: I0f68eeb75ba2f43b96b8f3d82f902e291d3bd396
          Reviewed-on: http://gerrit.cloudera.org:8080/6317
          Reviewed-by: Lars Volker <lv@cloudera.com>
          Tested-by: Impala Public Jenkins

          Show
          lv Lars Volker added a comment - IMPALA-4615 : Fix create_table.sql command order INSERT OVERWRITE commands in Hive will only affect partitions that Hive knows about. If an external table gets dropped and recreated, then 'MSCK REPAIR TABLE' needs to be executed to recover any preexisting partitions. Otherwise, an INSERT OVERWRITE will not remove the data files in those partitions and will fail to move the new data in place. More information can be found here: http://www.ericlin.me/hive-insert-overwrite-does-not-remove-existing-data I tested the fix by running the following commands, making sure that the second run of the .sql script completed without errors and validating the number of lines was correct (10) after both runs. export JDBC_URL="jdbc:hive2://$ {HS2_HOST_PORT} /default;" export HS2_HOST_PORT=localhost:11050 beeline -n $USER -u "$ {JDBC_URL}" -f ${IMPALA_HOME}/testdata/avro_schema_resolution/create_table.sql beeline -n $USER -u "${JDBC_URL} " -f $ {IMPALA_HOME} /testdata/avro_schema_resolution/create_table.sql Change-Id: I0f68eeb75ba2f43b96b8f3d82f902e291d3bd396 Reviewed-on: http://gerrit.cloudera.org:8080/6317 Reviewed-by: Lars Volker <lv@cloudera.com> Tested-by: Impala Public Jenkins

            People

            • Assignee:
              lv Lars Volker
              Reporter:
              alex.behm Alexander Behm
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development