Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-6331

slow s3 operations in unit tests

    XMLWordPrintableJSON

    Details

    • Epic Color:
      ghx-label-1

      Description

      Seeing some slowness for certain fs tests when run against S3. Since they in the hour range, I think its useful to track it down and distinguish between a test infrastructure issue vs. an impala/s3 issue.
      Here are logs from a recent run (note the time gaps in the log entries):

      20:07:37 query_test/test_hdfs_file_mods.py::TestHdfsFileMods::test_file_modifications[modification_type: move_file | exec_option:

      {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0}

      | table_format: text/none] PASSED
      20:08:19 query_test/test_hdfs_file_mods.py::TestHdfsFileMods::test_file_modifications[modification_type: append | exec_option:

      {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0}

      | table_format: text/none] PASSED
      22:59:55 query_test/test_insert.py::TestInsertQueries::test_insert[exec_option:

      {'sync_ddl': 0, 'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': True, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0}

      | table_format: text/none] PASSED
      03:40:11 query_test/test_insert.py::TestInsertQueries::test_insert[exec_option:

      {'sync_ddl': 0, 'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0}

      | table_format: text/none] PASSED
      03:41:12 query_test/test_insert.py::TestInsertQueries::test_insert_overwrite[exec_option:

      {'sync_ddl': 0, 'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': True, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0}

      | table_format: text/none] PASSED

      There are several other tests that take in the 10 minute to minute range. Highlighting above the longest ones (> hour).

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              vukercegovac Vuk Ercegovac
            • Votes:
              1 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: