Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-11192

test_scanner_fuzz.py runs super slow on ORC format

    XMLWordPrintableJSON

Details

    • Test
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • Impala 4.1.0
    • Infrastructure
    • None
    • ghx-label-1

    Description

      I recently need to iterate test_scanner_fuzz.py multiple times and find it takes more than 0.5h to run it once (only for ORC).

      $ time -p impala-py.test --skip_hbase --table_formats=orc/def/block tests/query_test/test_scanners_fuzz.py

real 2155.47
user 2779.64
sys 193.76

      Looking into a Jenkins job, it shows that ORC tests are much slower than other formats:

      Test name Duration Status
      test_fuzz_alltypes[... 'debug_action': '-1:OPEN:SET_DENY_RESERVATION_PROBABILITY@0.5' \\| table_format: avro/snap/block] 31 sec Passed
      test_fuzz_alltypes[... 'debug_action': '-1:OPEN:SET_DENY_RESERVATION_PROBABILITY@0.5' \\| table_format: orc/def/block] 2 min 1 sec Passed
      test_fuzz_alltypes[... 'debug_action': '-1:OPEN:SET_DENY_RESERVATION_PROBABILITY@0.5' \\| table_format: parquet/none] 35 sec Passed
      test_fuzz_alltypes[... 'debug_action': '-1:OPEN:SET_DENY_RESERVATION_PROBABILITY@0.5' \\| table_format: text/none] 48 sec Passed
      test_fuzz_alltypes[... 'debug_action': '-1:OPEN:SET_DENY_RESERVATION_PROBABILITY@1.0' \\| table_format: avro/snap/block] 40 sec Passed
      test_fuzz_alltypes[... 'debug_action': '-1:OPEN:SET_DENY_RESERVATION_PROBABILITY@1.0' \\| table_format: orc/def/block] 2 min 55 sec Passed
      test_fuzz_alltypes[... 'debug_action': '-1:OPEN:SET_DENY_RESERVATION_PROBABILITY@1.0' \\| table_format: parquet/none] 22 sec Passed
      test_fuzz_alltypes[... 'debug_action': '-1:OPEN:SET_DENY_RESERVATION_PROBABILITY@1.0' \\| table_format: text/none] 29 sec Passed
      test_fuzz_alltypes[... 'debug_action': None, \\| table_format: avro/snap/block] 32 sec Passed
      test_fuzz_alltypes[... 'debug_action': None, \\| table_format: orc/def/block] 3 min 25 sec Passed
      test_fuzz_alltypes[... 'debug_action': None, \\| table_format: parquet/none] 29 sec Passed
      test_fuzz_alltypes[... 'debug_action': None, \\| table_format: text/none] 20 sec Passed
      test_fuzz_decimal_tbl[... 'debug_action': '-1:OPEN:SET_DENY_RESERVATION_PROBABILITY@0.5' \\| table_format: avro/snap/block] 20 sec Passed
      test_fuzz_decimal_tbl[... 'debug_action': '-1:OPEN:SET_DENY_RESERVATION_PROBABILITY@0.5' \\| table_format: orc/def/block] 1 min 35 sec Passed
      test_fuzz_decimal_tbl[... 'debug_action': '-1:OPEN:SET_DENY_RESERVATION_PROBABILITY@0.5' \\| table_format: parquet/none] 22 sec Passed
      test_fuzz_decimal_tbl[... 'debug_action': '-1:OPEN:SET_DENY_RESERVATION_PROBABILITY@0.5' \\| table_format: text/none] 18 sec Passed
      test_fuzz_decimal_tbl[... 'debug_action': '-1:OPEN:SET_DENY_RESERVATION_PROBABILITY@1.0' \\| table_format: avro/snap/block] 20 sec Passed
      test_fuzz_decimal_tbl[... 'debug_action': '-1:OPEN:SET_DENY_RESERVATION_PROBABILITY@1.0' \\| table_format: orc/def/block] 1 min 16 sec Passed
      test_fuzz_decimal_tbl[... 'debug_action': '-1:OPEN:SET_DENY_RESERVATION_PROBABILITY@1.0' \\| table_format: parquet/none] 17 sec Passed
      test_fuzz_decimal_tbl[... 'debug_action': '-1:OPEN:SET_DENY_RESERVATION_PROBABILITY@1.0' \\| table_format: text/none] 16 sec Passed
      test_fuzz_decimal_tbl[... 'debug_action': None \\| table_format: avro/snap/block] 19 sec Passed
      test_fuzz_decimal_tbl[... 'debug_action': None \\| table_format: orc/def/block] 1 min 4 sec Passed
      test_fuzz_decimal_tbl[... 'debug_action': None \\| table_format: parquet/none] 22 sec Passed
      test_fuzz_decimal_tbl[... 'debug_action': None \\| table_format: text/none] 29 sec Passed
      test_fuzz_nested_types[... 'debug_action': '-1:OPEN:SET_DENY_RESERVATION_PROBABILITY@0.5' \\| table_format: avro/snap/block] 4 sec Skipped
      test_fuzz_nested_types[... 'debug_action': '-1:OPEN:SET_DENY_RESERVATION_PROBABILITY@0.5' \\| table_format: orc/def/block] 1 min 35 sec Passed
      test_fuzz_nested_types[... 'debug_action': '-1:OPEN:SET_DENY_RESERVATION_PROBABILITY@0.5' \\| table_format: parquet/none] 18 sec Passed
      test_fuzz_nested_types[... 'debug_action': '-1:OPEN:SET_DENY_RESERVATION_PROBABILITY@0.5' \\| table_format: text/none] 4.7 sec Skipped
      test_fuzz_nested_types[... 'debug_action': '-1:OPEN:SET_DENY_RESERVATION_PROBABILITY@1.0' \\| table_format: avro/snap/block] 5.2 sec Skipped
      test_fuzz_nested_types[... 'debug_action': '-1:OPEN:SET_DENY_RESERVATION_PROBABILITY@1.0' \\| table_format: orc/def/block] 1 min 59 sec Passed
      test_fuzz_nested_types[... 'debug_action': '-1:OPEN:SET_DENY_RESERVATION_PROBABILITY@1.0' \\| table_format: parquet/none] 17 sec Passed
      test_fuzz_nested_types[... 'debug_action': '-1:OPEN:SET_DENY_RESERVATION_PROBABILITY@1.0' \\| table_format: text/none] 4.3 sec Skipped
      test_fuzz_nested_types[... 'debug_action': None \\| table_format: avro/snap/block] 4.1 sec Skipped
      test_fuzz_nested_types[... 'debug_action': None \\| table_format: orc/def/block] 2 min 30 sec Passed
      test_fuzz_nested_types[... 'debug_action': None \\| table_format: parquet/none] 18 sec Passed
      test_fuzz_nested_types[... 'debug_action': None \\| table_format: text/none] 3.5 sec Skipped
      test_fuzz_uncompressed_parquet_orc[... 'debug_action': '-1:OPEN:SET_DENY_RESERVATION_PROBABILITY@0.5' \\| table_format: avro/snap/block] 4 sec Skipped
      test_fuzz_uncompressed_parquet_orc[... 'debug_action': '-1:OPEN:SET_DENY_RESERVATION_PROBABILITY@0.5' \\| table_format: orc/def/block] 17 min Passed
      test_fuzz_uncompressed_parquet_orc[... 'debug_action': '-1:OPEN:SET_DENY_RESERVATION_PROBABILITY@0.5' \\| table_format: parquet/none] 20 sec Passed
      test_fuzz_uncompressed_parquet_orc[... 'debug_action': '-1:OPEN:SET_DENY_RESERVATION_PROBABILITY@0.5' \\| table_format: text/none] 3.1 sec Skipped
      test_fuzz_uncompressed_parquet_orc[... 'debug_action': '-1:OPEN:SET_DENY_RESERVATION_PROBABILITY@1.0' \\| table_format: avro/snap/block] 3.8 sec Skipped
      test_fuzz_uncompressed_parquet_orc[... 'debug_action': '-1:OPEN:SET_DENY_RESERVATION_PROBABILITY@1.0' \\| table_format: orc/def/block] 9 min 51 sec Passed
      test_fuzz_uncompressed_parquet_orc[... 'debug_action': '-1:OPEN:SET_DENY_RESERVATION_PROBABILITY@1.0' \\| table_format: parquet/none] 19 sec Passed
      test_fuzz_uncompressed_parquet_orc[... 'debug_action': '-1:OPEN:SET_DENY_RESERVATION_PROBABILITY@1.0' \\| table_format: text/none] 4.4 sec Skipped
      test_fuzz_uncompressed_parquet_orc[... 'debug_action': None \\| table_format: avro/snap/block] 3.9 sec Skipped
      test_fuzz_uncompressed_parquet_orc[... 'debug_action': None \\| table_format: orc/def/block] 9 min 27 sec Passed
      test_fuzz_uncompressed_parquet_orc[... 'debug_action': None \\| table_format: parquet/none] 19 sec Passed
      test_fuzz_uncompressed_parquet_orc[... 'debug_action': None \\| table_format: text/none] 4.3 sec Skipped

      Tests on other formats take less than 1min for each. ORC tests usually takes several minutes.

      CC boroknagyz

      Attachments

        Activity

          People

            stigahuang Quanlong Huang
            stigahuang Quanlong Huang
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: