Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-4775

discrepancy_searcher.py logging overwriting itself

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: Impala 2.9.0
    • Fix Version/s: Impala 2.9.0
    • Component/s: Infrastructure
    • Labels:
      None

      Description

      If I run discrepancy_searcher.py the log is overwritten near the end of execution and will only contain a single line:

      13:02:47 INFO:hiveserver2[265]:Closing active operation
      

      tail -F proves this is happening. From 1 execution of discrepancy_searcher.py:

      13:02:47 INFO:discrepancy_searcher[799]:Results matched (1 rows)
      tail: /tmp/discrepancy_searcher.py.log: file truncated
      13:02:47 INFO:hiveserver2[265]:Closing active operation
      

      This is a "regression" due to this commit:

      IMPALA-4467: Add support for DML statements in stress test
      

      ... in which the Impyla version changed from 0.11.2 to 0.14.0.

        Activity

        Hide
        lv Lars Volker added a comment -

        I'm changing the "Fix Version/s" to 2.9 as part of a bulk edit. If you think this is wrong, please set the version to the correct one.

        Show
        lv Lars Volker added a comment - I'm changing the "Fix Version/s" to 2.9 as part of a bulk edit. If you think this is wrong, please set the version to the correct one.
        Hide
        mikesbrown Michael Brown added a comment -
        commit fe2be25245280492c9a6b5b4e20e066b244fb808
        Author: Michael Brown <mikeb@cloudera.com>
        Date:   Tue Jan 17 12:25:03 2017 -0800
        
            IMPALA-4775: minor adjustments to python test infra logging
        
            - Set up log handler to append, not truncate. This was the cause of
              IMPALA-4775.
        
            Other improvements:
            - Log a thread name, not thread ID. Thread names are more useful.
            - Use ISO 8601-like timestamps
        
            I tested that running disrepancy_searcher.py doesn't overwrite its logs
            anymore. One such command that could reproduce it is:
        
            tests/comparison/discrepancy_searcher.py \
              --use-postgresql \
              --query-count 1 \
              --db-name tpch_kudu
        
            I also ensured the stress test (concurrent_select.py) still logged to
            its file.
        
            Change-Id: I2b7af5b2be20f3c6f38d25612f6888433c62d693
            Reviewed-on: http://gerrit.cloudera.org:8080/5746
            Reviewed-by: Alex Behm <alex.behm@cloudera.com>
            Tested-by: Impala Public Jenkins
        
        Show
        mikesbrown Michael Brown added a comment - commit fe2be25245280492c9a6b5b4e20e066b244fb808 Author: Michael Brown <mikeb@cloudera.com> Date: Tue Jan 17 12:25:03 2017 -0800 IMPALA-4775: minor adjustments to python test infra logging - Set up log handler to append, not truncate. This was the cause of IMPALA-4775. Other improvements: - Log a thread name, not thread ID. Thread names are more useful. - Use ISO 8601-like timestamps I tested that running disrepancy_searcher.py doesn't overwrite its logs anymore. One such command that could reproduce it is: tests/comparison/discrepancy_searcher.py \ --use-postgresql \ --query-count 1 \ --db-name tpch_kudu I also ensured the stress test (concurrent_select.py) still logged to its file. Change-Id: I2b7af5b2be20f3c6f38d25612f6888433c62d693 Reviewed-on: http://gerrit.cloudera.org:8080/5746 Reviewed-by: Alex Behm <alex.behm@cloudera.com> Tested-by: Impala Public Jenkins
        Show
        mikesbrown Michael Brown added a comment - https://gerrit.cloudera.org/#/c/5746/
        Hide
        mikesbrown Michael Brown added a comment -

        I didn't catch this when vetting the Impyla upgrade because I didn't test discrepancy_searcher.py as an entry point.

        Show
        mikesbrown Michael Brown added a comment - I didn't catch this when vetting the Impyla upgrade because I didn't test discrepancy_searcher.py as an entry point.

          People

          • Assignee:
            mikesbrown Michael Brown
            Reporter:
            mikesbrown Michael Brown
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development