Details
-
New Feature
-
Status: Open
-
P3
-
Resolution: Unresolved
-
None
-
None
-
None
Description
I'm looking at a flake in a postcommit suite that happened last week in Python Dataflow integration test (https://issues.apache.org/jira/browse/BEAM-5415), and the existing logs are not sufficient to debug the problem. The test suite failed, however:
1. Test Result section in Jenkins postcommit only includes unit tests, and does not include any integration tests.
2. The only available information about the failure is buried in the Full Console Log and is not very informative (copy-pasted below).
3. There is a line in the console log pointing to: "XML: /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/nosetests.xml", however as a non-committer I don't have access to Jenkins machines.
This makes rootcausing integration tests failures quite difficult, since we cannot even reliably identify an ID of a failed Dataflow job from the test output.
It would be helpful to capture and persist in Jenkins the invocation of integration test, and all the console output that SDK produces during test execution.
Sample available logs from console output.
13:41:54 test_leader_board_it (apache_beam.examples.complete.game.leader_board_it_test.LeaderBoardIT) ... ERROR
13:41:54
13:41:54 ======================================================================
13:41:54 ERROR: test_leader_board_it (apache_beam.examples.complete.game.leader_board_it_test.LeaderBoardIT)
13:41:54 ----------------------------------------------------------------------
13:41:54 Traceback (most recent call last):
13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/nose/plugins/multiprocess.py", line 812, in run
13:41:54 test(orig)
13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/nose/case.py", line 45, in _call_
13:41:54 return self.run(*arg, **kwarg)
13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/nose/case.py", line 133, in run
13:41:54 self.runTest(result)
13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/nose/case.py", line 151, in runTest
13:41:54 test(result)
13:41:54 File "/usr/lib/python2.7/unittest/case.py", line 393, in _call_
13:41:54 return self.run(*args, **kwds)
13:41:54 File "/usr/lib/python2.7/unittest/case.py", line 329, in run
13:41:54 testMethod()
13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/examples/complete/game/leader_board_it_test.py", line 161, in test_leader_board_it
13:41:54 self.test_pipeline.get_full_options_as_args(**extra_opts))
13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/examples/complete/game/leader_board.py", line 345, in run
13:41:54 'total_score': 'INTEGER',
13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/pipeline.py", line 414, in _exit_
13:41:54 self.run().wait_until_finish()
13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/pipeline.py", line 394, in run
13:41:54 self.to_runner_api(), self.runner, self._options).run(False)
13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/pipeline.py", line 407, in run
13:41:54 return self.runner.run_pipeline(self)
13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/runners/dataflow/test_dataflow_runner.py", line 68, in run_pipeline
13:41:54 self.result.cancel()
13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py", line 1167, in cancel
13:41:54 self._update_job()
13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py", line 1078, in _update_job
13:41:54 self._job = self._runner.dataflow_client.get_job(self.job_id())
13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/utils/retry.py", line 184, in wrapper
13:41:54 return fun(*args, **kwargs)
13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/runners/dataflow/internal/apiclient.py", line 629, in get_job
13:41:54 response = self._client.projects_locations_jobs.Get(request)
13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py", line 659, in Get
13:41:54 config, request, global_params=global_params)
13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/apitools/base/py/base_api.py", line 720, in _RunMethod
13:41:54 http, http_request, **opts)
13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/apitools/base/py/http_wrapper.py", line 346, in MakeRequest
13:41:54 check_response_func=check_response_func)
13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/apitools/base/py/http_wrapper.py", line 396, in _MakeRequestNoRetry
13:41:54 redirections=redirections, connection_type=connection_type)
13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/oauth2client/transport.py", line 193, in new_request
13:41:54 redirections, connection_type)
13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/oauth2client/transport.py", line 282, in request
13:41:54 connection_type=connection_type)
13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/httplib2/_init_.py", line 1694, in request
13:41:54 (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/httplib2/_init_.py", line 1434, in _request
13:41:54 (response, content) = self._conn_request(conn, request_uri, method, body, headers)
13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/httplib2/_init_.py", line 1390, in _conn_request
13:41:54 response = conn.getresponse()
13:41:54 File "/usr/lib/python2.7/httplib.py", line 1136, in getresponse
13:41:54 response.begin()
13:41:54 File "/usr/lib/python2.7/httplib.py", line 453, in begin
13:41:54 version, status, reason = self._read_status()
13:41:54 File "/usr/lib/python2.7/httplib.py", line 409, in _read_status
13:41:54 line = self.fp.readline(_MAXLINE + 1)
13:41:54 File "/usr/lib/python2.7/socket.py", line 480, in readline
13:41:54 data = self._sock.recv(self._rbufsize)
13:41:54 File "/usr/lib/python2.7/ssl.py", line 756, in recv
13:41:54 return self.read(buflen)
13:41:54 File "/usr/lib/python2.7/ssl.py", line 643, in read
13:41:54 v = self._sslobj.read(len)
13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/nose/plugins/multiprocess.py", line 276, in signalhandler
13:41:54 raise TimedOutException()
13:41:54 TimedOutException: 'test_leader_board_it (apache_beam.examples.complete.game.leader_board_it_test.LeaderBoardIT)'
13:41:54
13:41:54 ----------------------------------------------------------------------
13:41:54 XML: /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/nosetests.xml
13:41:54 ----------------------------------------------------------------------
13:41:54 Ran 14 tests in 3001.679s
13:41:54
13:41:54 FAILED (errors=1)
13:41:54
