Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-8063

Excessive logging from BeeswaxConnection::get_state() bloats JUnitXML output

    XMLWordPrintableJSON

    Details

      Description

      BeeswaxConnection has logging for each call of get_state:

       

      def get_state(self, operation_handle):
        LOG.info("-- getting state for operation: %s" % operation_handle)
        return self.__beeswax_client.get_state(operation_handle.get_handle())
      

      With IMPALA-7625, ImpalaTestSuite::wait_for_state() calls this more frequently:

       

       

      def wait_for_state(self, handle, expected_state, timeout):
        """Waits for the given 'query_handle' to reach the 'expected_state'. If it does not
        reach the given state within 'timeout' seconds, the method throws an AssertionError.
        """
        start_time = time.time()
        actual_state = self.client.get_state(handle)
        while actual_state != expected_state and time.time() - start_time < timeout:
          actual_state = self.client.get_state(handle)
        if actual_state != expected_state:
          raise Timeout("query '%s' did not reach expected state '%s', last known state '%s'"
                        % (handle.get_handle().id, expected_state, actual_state))
      

      When running our tests in exhaustive mode, that increases the size of the logging significantly. For example:

       

      Before this change:
      $ ls -l TEST-impala-parallel.xml
      -rw-rw-r-- 1 joe joe 34254745 Jan 6 23:23 TEST-impala-parallel.xml
      $ grep "getting state for operation" TEST-impala-parallel.xml | wc -
      l
      1044
      
      After this change:
      $ ls -l TEST-impala-parallel.xml
      -rw-rw-r-- 1 joe joe 159187682 Jan 9 14:51 TEST-impala-parallel.xml
      $ grep "getting state for operation" TEST-impala-parallel.xml | wc -
      l
      1167084
      

       

       

      We should reduce this logging. Bloated JUnitXML files add burden to developer workstations and any Jenkins infrastructure parsing them.

        Attachments

          Activity

            People

            • Assignee:
              joemcdonnell Joe McDonnell
              Reporter:
              joemcdonnell Joe McDonnell
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: