Details
-
Bug
-
Status: Resolved
-
P2
-
Resolution: Fixed
-
None
-
None
Description
Saw this in a PostCommit test today. Likely a flake, but it's a strange failure mode and we may need to investigate this.
13:32:02 ====================================================================== 13:32:02 FAIL: test_streaming_wordcount_it (apache_beam.examples.streaming_wordcount_it_test.StreamingWordCountIT) 13:32:02 ---------------------------------------------------------------------- 13:32:02 Traceback (most recent call last): 13:32:02 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify_PR/src/sdks/python/apache_beam/examples/streaming_wordcount_it_test.py", line 110, in test_streaming_wordcount_it 13:32:02 self.test_pipeline.get_full_options_as_args(**extra_opts)) 13:32:02 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify_PR/src/sdks/python/apache_beam/examples/streaming_wordcount.py", line 101, in run 13:32:02 result = p.run() 13:32:02 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify_PR/src/sdks/python/apache_beam/pipeline.py", line 419, in run 13:32:02 return self.runner.run_pipeline(self, self._options) 13:32:02 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify_PR/src/sdks/python/apache_beam/runners/dataflow/test_dataflow_runner.py", line 68, in run_pipeline 13:32:02 hc_assert_that(self.result, pickler.loads(on_success_matcher)) 13:32:02 AssertionError: 13:32:02 Expected: (Test pipeline expected terminated in state: RUNNING and Expected 500 messages.) 13:32:02 but: Expected 500 messages. Got 508 messages. Diffs (item, count): 13:32:02 Expected but not in actual: [] 13:32:02 Unexpected: [(u'476: 1', 1), (u'416: 1', 1), (u'245: 1', 1), (u'478: 1', 1), (u'58: 1', 1), (u'364: 1', 1), (u'77: 1', 1), (u'283: 1', 1)] 13:32:02 13:32:02 -------------------- >> begin captured logging << -------------------- 13:32:02 google.auth.transport._http_client: DEBUG: Making request: GET http://169.254.169.254 13:32:02 google.auth.transport._http_client: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/project/project-id 13:32:02 google.auth.transport.requests: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true 13:32:02 urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): metadata.google.internal:80 13:32:02 urllib3.connectionpool: DEBUG: http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/default/?recursive=true HTTP/1.1" 200 144 13:32:02 google.auth.transport.requests: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/844138762903-compute@developer.gserviceaccount.com/token 13:32:02 urllib3.connectionpool: DEBUG: http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/844138762903-compute@developer.gserviceaccount.com/token HTTP/1.1" 200 176 13:32:02 google.auth.transport._http_client: DEBUG: Making request: GET http://169.254.169.254 13:32:02 google.auth.transport._http_client: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/project/project-id 13:32:02 google.auth.transport.requests: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true 13:32:02 urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): metadata.google.internal:80 13:32:02 urllib3.connectionpool: DEBUG: http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/default/?recursive=true HTTP/1.1" 200 144 13:32:02 google.auth.transport.requests: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/844138762903-compute@developer.gserviceaccount.com/token 13:32:02 urllib3.connectionpool: DEBUG: http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/844138762903-compute@developer.gserviceaccount.com/token HTTP/1.1" 200 176 13:32:02 root: DEBUG: Injecting 500 numbers to topic projects/apache-beam-testing/topics/wc_topic_inputb7a2edca-b896-496f-9ab7-4fdfee001097 13:32:02 google.cloud.pubsub_v1.publisher._batch.thread: DEBUG: Monitor is waking up 13:32:02 root: DEBUG: Connecting using Google Application Default Credentials. 13:32:02 root: INFO: Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/pipeline.pb... 13:32:02 oauth2client.transport: INFO: Attempting refresh to obtain initial access_token 13:32:02 oauth2client.transport: INFO: Attempting refresh to obtain initial access_token 13:32:02 google.cloud.pubsub_v1.publisher._batch.thread: DEBUG: gRPC Publish took 0.253291845322 seconds. 13:32:02 root: INFO: Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/pipeline.pb in 0 seconds. 13:32:02 root: INFO: Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/requirements.txt... 13:32:02 root: INFO: Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/requirements.txt in 0 seconds. 13:32:02 root: INFO: Executing command: ['/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify_PR/src/build/gradleenv/1922375555/bin/python', '-m', 'pip', 'download', '--dest', '/tmp/dataflow-requirements-cache', '-r', 'postcommit_requirements.txt', '--exists-action', 'i', '--no-binary', ':all:'] 13:32:02 root: INFO: Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/PyHamcrest-1.9.0.tar.gz... 13:32:02 root: INFO: Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/PyHamcrest-1.9.0.tar.gz in 0 seconds. 13:32:02 root: INFO: Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/mock-3.0.5.tar.gz... 13:32:02 root: INFO: Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/mock-3.0.5.tar.gz in 0 seconds. 13:32:02 root: INFO: Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/mock-3.0.4.tar.gz... 13:32:02 root: INFO: Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/mock-3.0.4.tar.gz in 0 seconds. 13:32:02 root: INFO: Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/setuptools-41.0.1.zip... 13:32:02 root: INFO: Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/setuptools-41.0.1.zip in 0 seconds. 13:32:02 root: INFO: Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/pbr-5.2.0.tar.gz... 13:32:02 root: INFO: Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/pbr-5.2.0.tar.gz in 0 seconds. 13:32:02 root: INFO: Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/mock-2.0.0.tar.gz... 13:32:02 root: INFO: Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/mock-2.0.0.tar.gz in 0 seconds. 13:32:02 root: INFO: Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/mock-3.0.3.tar.gz... 13:32:02 root: INFO: Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/mock-3.0.3.tar.gz in 0 seconds. 13:32:02 root: INFO: Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/six-1.12.0.tar.gz... 13:32:02 root: INFO: Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/six-1.12.0.tar.gz in 0 seconds. 13:32:02 root: INFO: Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/funcsigs-1.0.2.tar.gz... 13:32:02 root: INFO: Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/funcsigs-1.0.2.tar.gz in 0 seconds. 13:32:02 root: INFO: Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/pickled_main_session... 13:32:02 root: INFO: Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/pickled_main_session in 0 seconds. 13:32:02 root: INFO: Copying Beam SDK "build/apache-beam.tar.gz" to staging location. 13:32:02 root: INFO: Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/dataflow_python_sdk.tar... 13:32:02 root: INFO: Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/dataflow_python_sdk.tar in 0 seconds. 13:32:02 root: INFO: Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/dataflow-worker.jar... 13:32:02 root: INFO: Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/dataflow-worker.jar in 3 seconds. 13:32:02 root: WARNING: Discarding unparseable args: ['--sleep_secs=20', '--kms_key_name=projects/apache-beam-testing/locations/global/keyRings/beam-it/cryptoKeys/test'] 13:32:02 root: WARNING: Discarding unparseable args: ['--sleep_secs=20', '--kms_key_name=projects/apache-beam-testing/locations/global/keyRings/beam-it/cryptoKeys/test'] 13:32:02 root: DEBUG: JOB: { 13:32:02 "environment": { 13:32:02 "clusterManagerApiService": "compute.googleapis.com", 13:32:02 "dataset": "bigquery.googleapis.com/cloud_dataflow", 13:32:02 "experiments": [ 13:32:02 "use_staged_dataflow_worker_jar", 13:32:02 "use_multiple_sdk_containers" 13:32:02 ], 13:32:02 "sdkPipelineOptions": { 13:32:02 "display_data": [ 13:32:02 { 13:32:02 "key": "requirements_file", 13:32:02 "namespace": "apache_beam.options.pipeline_options.PipelineOptions", 13:32:02 "type": "STRING", 13:32:02 "value": "postcommit_requirements.txt" 13:32:02 }, 13:32:02 { 13:32:02 "key": "sdk_location", 13:32:02 "namespace": "apache_beam.options.pipeline_options.PipelineOptions", 13:32:02 "type": "STRING", 13:32:02 "value": "build/apache-beam.tar.gz" 13:32:02 }, 13:32:02 { 13:32:02 "key": "save_main_session", 13:32:02 "namespace": "apache_beam.options.pipeline_options.PipelineOptions", 13:32:02 "type": "BOOLEAN", 13:32:02 "value": true 13:32:02 }, 13:32:02 { 13:32:02 "key": "num_workers", 13:32:02 "namespace": "apache_beam.options.pipeline_options.PipelineOptions", 13:32:02 "type": "INTEGER", 13:32:02 "value": 1 13:32:02 }, 13:32:02 { 13:32:02 "key": "runner", 13:32:02 "namespace": "apache_beam.options.pipeline_options.PipelineOptions", 13:32:02 "type": "STRING", 13:32:02 "value": "TestDataflowRunner" 13:32:02 }, 13:32:02 { 13:32:02 "key": "staging_location", 13:32:02 "namespace": "apache_beam.options.pipeline_options.PipelineOptions", 13:32:02 "type": "STRING", 13:32:02 "value": "gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255" 13:32:02 }, 13:32:02 { 13:32:02 "key": "dataflow_worker_jar", 13:32:02 "namespace": "apache_beam.options.pipeline_options.PipelineOptions", 13:32:02 "type": "STRING", 13:32:02 "value": "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify_PR/src/runners/google-cloud-dataflow-java/worker/build/libs/beam-runners-google-cloud-dataflow-java-fn-api-worker-2.14.0-SNAPSHOT.jar" 13:32:02 }, 13:32:02 { 13:32:02 "key": "wait_until_finish_duration", 13:32:02 "namespace": "apache_beam.options.pipeline_options.PipelineOptions", 13:32:02 "type": "INTEGER", 13:32:02 "value": 180000 13:32:02 }, 13:32:02 { 13:32:02 "key": "project", 13:32:02 "namespace": "apache_beam.options.pipeline_options.PipelineOptions", 13:32:02 "type": "STRING", 13:32:02 "value": "apache-beam-testing" 13:32:02 }, 13:32:02 { 13:32:02 "key": "streaming", 13:32:02 "namespace": "apache_beam.options.pipeline_options.PipelineOptions", 13:32:02 "type": "BOOLEAN", 13:32:02 "value": true 13:32:02 }, 13:32:02 { 13:32:02 "key": "dataflow_kms_key", 13:32:02 "namespace": "apache_beam.options.pipeline_options.PipelineOptions", 13:32:02 "type": "STRING", 13:32:02 "value": "projects/apache-beam-testing/locations/global/keyRings/beam-it/cryptoKeys/test" 13:32:02 }, 13:32:02 { 13:32:02 "key": "experiments", 13:32:02 "namespace": "apache_beam.options.pipeline_options.PipelineOptions", 13:32:02 "type": "STRING", 13:32:02 "value": "['use_staged_dataflow_worker_jar', 'use_multiple_sdk_containers']" 13:32:02 }, 13:32:02 { 13:32:02 "key": "temp_location", 13:32:02 "namespace": "apache_beam.options.pipeline_options.PipelineOptions", 13:32:02 "type": "STRING", 13:32:02 "value": "gs://temp-storage-for-end-to-end-tests/temp-it/beamapp-jenkins-0517194018-707115.1558122018.707255" 13:32:02 }, 13:32:02 { 13:32:02 "key": "beam_plugins", 13:32:02 "namespace": "apache_beam.options.pipeline_options.PipelineOptions", 13:32:02 "type": "STRING", 13:32:02 "value": "['apache_beam.io.filesystem.FileSystem', 'apache_beam.io.hadoopfilesystem.HadoopFileSystem', 'apache_beam.io.localfilesystem.LocalFileSystem', 'apache_beam.io.gcp.gcsfilesystem.GCSFileSystem', 'apache_beam.io.filesystem_test.TestingFileSystem', 'apache_beam.runners.interactive.display.pipeline_graph_renderer.PipelineGraphRenderer', 'apache_beam.runners.interactive.display.pipeline_graph_renderer.MuteRenderer', 'apache_beam.runners.interactive.display.pipeline_graph_renderer.TextRenderer', 'apache_beam.runners.interactive.display.pipeline_graph_renderer.PydotRenderer']" 13:32:02 }, 13:32:02 { 13:32:02 "key": "job_name", 13:32:02 "namespace": "apache_beam.options.pipeline_options.PipelineOptions", 13:32:02 "type": "STRING", 13:32:02 "value": "beamapp-jenkins-0517194018-707115" 13:32:02 } 13:32:02 ], 13:32:02 "options": { 13:32:02 "beam_plugins": [ 13:32:02 "apache_beam.io.filesystem.FileSystem", 13:32:02 "apache_beam.io.hadoopfilesystem.HadoopFileSystem", 13:32:02 "apache_beam.io.localfilesystem.LocalFileSystem", 13:32:02 "apache_beam.io.gcp.gcsfilesystem.GCSFileSystem", 13:32:02 "apache_beam.io.filesystem_test.TestingFileSystem", 13:32:02 "apache_beam.runners.interactive.display.pipeline_graph_renderer.PipelineGraphRenderer", 13:32:02 "apache_beam.runners.interactive.display.pipeline_graph_renderer.MuteRenderer", 13:32:02 "apache_beam.runners.interactive.display.pipeline_graph_renderer.TextRenderer", 13:32:02 "apache_beam.runners.interactive.display.pipeline_graph_renderer.PydotRenderer" 13:32:02 ], 13:32:02 "dataflow_endpoint": "https://dataflow.googleapis.com", 13:32:02 "dataflow_kms_key": "projects/apache-beam-testing/locations/global/keyRings/beam-it/cryptoKeys/test", 13:32:02 "dataflow_worker_jar": "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify_PR/src/runners/google-cloud-dataflow-java/worker/build/libs/beam-runners-google-cloud-dataflow-java-fn-api-worker-2.14.0-SNAPSHOT.jar", 13:32:02 "direct_runner_bundle_repeat": 0, 13:32:02 "direct_runner_use_stacked_bundle": true, 13:32:02 "dry_run": false, 13:32:02 "enable_streaming_engine": false, 13:32:02 "environment_cache_millis": 0, 13:32:02 "experiments": [ 13:32:02 "use_staged_dataflow_worker_jar", 13:32:02 "use_multiple_sdk_containers" 13:32:02 ], 13:32:02 "fake_flag": false, 13:32:02 "job_name": "beamapp-jenkins-0517194018-707115", 13:32:02 "m_flag": false, 13:32:02 "male": false, 13:32:02 "mock_flag": false, 13:32:02 "no_auth": false, 13:32:02 "num_workers": 1, 13:32:02 "pipelineUrl": "gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/pipeline.pb", 13:32:02 "pipeline_type_check": true, 13:32:02 "profile_cpu": false, 13:32:02 "profile_memory": false, 13:32:02 "profile_sample_rate": 1.0, 13:32:02 "project": "apache-beam-testing", 13:32:02 "region": "us-central1", 13:32:02 "requirements_file": "postcommit_requirements.txt", 13:32:02 "runner": "TestDataflowRunner", 13:32:02 "runtime_type_check": false, 13:32:02 "save_main_session": true, 13:32:02 "sdk_location": "build/apache-beam.tar.gz", 13:32:02 "sdk_worker_parallelism": 0, 13:32:02 "staging_location": "gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255", 13:32:02 "streaming": true, 13:32:02 "style": "scrambled", 13:32:02 "temp_location": "gs://temp-storage-for-end-to-end-tests/temp-it/beamapp-jenkins-0517194018-707115.1558122018.707255", 13:32:02 "type_check_strictness": "DEFAULT_TO_ANY", 13:32:02 "update": false, 13:32:02 "wait_until_finish_duration": 180000 13:32:02 } 13:32:02 }, 13:32:02 "serviceKmsKeyName": "projects/apache-beam-testing/locations/global/keyRings/beam-it/cryptoKeys/test", 13:32:02 "tempStoragePrefix": "storage.googleapis.com/temp-storage-for-end-to-end-tests/temp-it/beamapp-jenkins-0517194018-707115.1558122018.707255", 13:32:02 "userAgent": { 13:32:02 "name": "Apache Beam Python 2.7 SDK", 13:32:02 "version": "2.14.0.dev" 13:32:02 }, 13:32:02 "version": { 13:32:02 "job_type": "FNAPI_STREAMING", 13:32:02 "major": "7" 13:32:02 }, 13:32:02 "workerPools": [ 13:32:02 { 13:32:02 "autoscalingSettings": {}, 13:32:02 "dataDisks": [ 13:32:02 {} 13:32:02 ], 13:32:02 "kind": "harness", 13:32:02 "numWorkers": 1, 13:32:02 "packages": [ 13:32:02 { 13:32:02 "location": "storage.googleapis.com/temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/requirements.txt", 13:32:02 "name": "requirements.txt" 13:32:02 }, 13:32:02 { 13:32:02 "location": "storage.googleapis.com/temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/PyHamcrest-1.9.0.tar.gz", 13:32:02 "name": "PyHamcrest-1.9.0.tar.gz" 13:32:02 }, 13:32:02 { 13:32:02 "location": "storage.googleapis.com/temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/mock-3.0.5.tar.gz", 13:32:02 "name": "mock-3.0.5.tar.gz" 13:32:02 }, 13:32:02 { 13:32:02 "location": "storage.googleapis.com/temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/mock-3.0.4.tar.gz", 13:32:02 "name": "mock-3.0.4.tar.gz" 13:32:02 }, 13:32:02 { 13:32:02 "location": "storage.googleapis.com/temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/setuptools-41.0.1.zip", 13:32:02 "name": "setuptools-41.0.1.zip" 13:32:02 }, 13:32:02 { 13:32:02 "location": "storage.googleapis.com/temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/pbr-5.2.0.tar.gz", 13:32:02 "name": "pbr-5.2.0.tar.gz" 13:32:02 }, 13:32:02 { 13:32:02 "location": "storage.googleapis.com/temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/mock-2.0.0.tar.gz", 13:32:02 "name": "mock-2.0.0.tar.gz" 13:32:02 }, 13:32:02 { 13:32:02 "location": "storage.googleapis.com/temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/mock-3.0.3.tar.gz", 13:32:02 "name": "mock-3.0.3.tar.gz" 13:32:02 }, 13:32:02 { 13:32:02 "location": "storage.googleapis.com/temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/six-1.12.0.tar.gz", 13:32:02 "name": "six-1.12.0.tar.gz" 13:32:02 }, 13:32:02 { 13:32:02 "location": "storage.googleapis.com/temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/funcsigs-1.0.2.tar.gz", 13:32:02 "name": "funcsigs-1.0.2.tar.gz" 13:32:02 }, 13:32:02 { 13:32:02 "location": "storage.googleapis.com/temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/pickled_main_session", 13:32:02 "name": "pickled_main_session" 13:32:02 }, 13:32:02 { 13:32:02 "location": "storage.googleapis.com/temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/dataflow_python_sdk.tar", 13:32:02 "name": "dataflow_python_sdk.tar" 13:32:02 }, 13:32:02 { 13:32:02 "location": "storage.googleapis.com/temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/dataflow-worker.jar", 13:32:02 "name": "dataflow-worker.jar" 13:32:02 } 13:32:02 ], 13:32:02 "taskrunnerSettings": { 13:32:02 "parallelWorkerSettings": { 13:32:02 "baseUrl": "https://dataflow.googleapis.com", 13:32:02 "servicePath": "https://dataflow.googleapis.com" 13:32:02 } 13:32:02 }, 13:32:02 "workerHarnessContainerImage": "gcr.io/cloud-dataflow/v1beta3/python-fnapi:beam-master-20190509" 13:32:02 } 13:32:02 ] 13:32:02 }, 13:32:02 "name": "beamapp-jenkins-0517194018-707115", 13:32:02 "steps": [ 13:32:02 { 13:32:02 "kind": "ParallelRead", 13:32:02 "name": "s1", 13:32:02 "properties": { 13:32:02 "display_data": [ 13:32:02 { 13:32:02 "key": "source", 13:32:02 "label": "Read Source", 13:32:02 "namespace": "apache_beam.io.iobase.Read", 13:32:02 "shortValue": "_PubSubSource", 13:32:02 "type": "STRING", 13:32:02 "value": "apache_beam.io.gcp.pubsub._PubSubSource" 13:32:02 }, 13:32:02 { 13:32:02 "key": "with_attributes", 13:32:02 "label": "With Attributes", 13:32:02 "namespace": "apache_beam.io.gcp.pubsub._PubSubSource", 13:32:02 "type": "BOOLEAN", 13:32:02 "value": false 13:32:02 }, 13:32:02 { 13:32:02 "key": "subscription", 13:32:02 "label": "Pubsub Subscription", 13:32:02 "namespace": "apache_beam.io.gcp.pubsub._PubSubSource", 13:32:02 "type": "STRING", 13:32:02 "value": "projects/apache-beam-testing/subscriptions/wc_subscription_inputb7a2edca-b896-496f-9ab7-4fdfee001097" 13:32:02 } 13:32:02 ], 13:32:02 "format": "pubsub", 13:32:02 "output_info": [ 13:32:02 { 13:32:02 "encoding": { 13:32:02 "@type": "kind:windowed_value", 13:32:02 "component_encodings": [ 13:32:02 { 13:32:02 "@type": "kind:bytes" 13:32:02 }, 13:32:02 { 13:32:02 "@type": "kind:global_window" 13:32:02 } 13:32:02 ], 13:32:02 "is_wrapper": true 13:32:02 }, 13:32:02 "output_name": "out", 13:32:02 "user_name": "ReadFromPubSub/Read.out" 13:32:02 } 13:32:02 ], 13:32:02 "pubsub_subscription": "projects/apache-beam-testing/subscriptions/wc_subscription_inputb7a2edca-b896-496f-9ab7-4fdfee001097", 13:32:02 "user_name": "ReadFromPubSub/Read" 13:32:02 } 13:32:02 }, 13:32:02 { 13:32:02 "kind": "ParallelDo", 13:32:02 "name": "s2", 13:32:02 "properties": { 13:32:02 "display_data": [ 13:32:02 { 13:32:02 "key": "fn", 13:32:02 "label": "Transform Function", 13:32:02 "namespace": "apache_beam.transforms.core.CallableWrapperDoFn", 13:32:02 "type": "STRING", 13:32:02 "value": "<lambda>" 13:32:02 }, 13:32:02 { 13:32:02 "key": "fn", 13:32:02 "label": "Transform Function", 13:32:02 "namespace": "apache_beam.transforms.core.ParDo", 13:32:02 "shortValue": "CallableWrapperDoFn", 13:32:02 "type": "STRING", 13:32:02 "value": "apache_beam.transforms.core.CallableWrapperDoFn" 13:32:02 } 13:32:02 ], 13:32:02 "non_parallel_inputs": {}, 13:32:02 "output_info": [ 13:32:02 { 13:32:02 "encoding": { 13:32:02 "@type": "kind:windowed_value", 13:32:02 "component_encodings": [ 13:32:02 { 13:32:02 "@type": "FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/", 13:32:02 "component_encodings": [ 13:32:02 { 13:32:02 "@type": "FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/", 13:32:02 "component_encodings": [], 13:32:02 "pipeline_proto_coder_id": "ref_Coder_FastPrimitivesCoder_3" 13:32:02 }, 13:32:02 { 13:32:02 "@type": "FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/", 13:32:02 "component_encodings": [], 13:32:02 "pipeline_proto_coder_id": "ref_Coder_FastPrimitivesCoder_3" 13:32:02 } 13:32:02 ], 13:32:02 "is_pair_like": true, 13:32:02 "pipeline_proto_coder_id": "ref_Coder_FastPrimitivesCoder_3" 13:32:02 }, 13:32:02 { 13:32:02 "@type": "kind:global_window" 13:32:02 } 13:32:02 ], 13:32:02 "is_wrapper": true 13:32:02 }, 13:32:02 "output_name": "out", 13:32:02 "user_name": "decode.out" 13:32:02 } 13:32:02 ], 13:32:02 "parallel_input": { 13:32:02 "@type": "OutputReference", 13:32:02 "output_name": "out", 13:32:02 "step_name": "s1" 13:32:02 }, 13:32:02 "serialized_fn": "ref_AppliedPTransform_decode_4", 13:32:02 "user_name": "decode" 13:32:02 } 13:32:02 }, 13:32:02 { 13:32:02 "kind": "ParallelDo", 13:32:02 "name": "s3", 13:32:02 "properties": { 13:32:02 "display_data": [ 13:32:02 { 13:32:02 "key": "fn", 13:32:02 "label": "Transform Function", 13:32:02 "namespace": "apache_beam.transforms.core.ParDo", 13:32:02 "shortValue": "WordExtractingDoFn", 13:32:02 "type": "STRING", 13:32:02 "value": "apache_beam.examples.wordcount.WordExtractingDoFn" 13:32:02 } 13:32:02 ], 13:32:02 "non_parallel_inputs": {}, 13:32:02 "output_info": [ 13:32:02 { 13:32:02 "encoding": { 13:32:02 "@type": "kind:windowed_value", 13:32:02 "component_encodings": [ 13:32:02 { 13:32:02 "@type": "StrUtf8Coder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlzBJUWhJWkWziAeVyGDZmMhY20hU5IeAAajEkY=", 13:32:02 "component_encodings": [], 13:32:02 "pipeline_proto_coder_id": "ref_Coder_StrUtf8Coder_4" 13:32:02 }, 13:32:02 { 13:32:02 "@type": "kind:global_window" 13:32:02 } 13:32:02 ], 13:32:02 "is_wrapper": true 13:32:02 }, 13:32:02 "output_name": "out", 13:32:02 "user_name": "split.out" 13:32:02 } 13:32:02 ], 13:32:02 "parallel_input": { 13:32:02 "@type": "OutputReference", 13:32:02 "output_name": "out", 13:32:02 "step_name": "s2" 13:32:02 }, 13:32:02 "serialized_fn": "ref_AppliedPTransform_split_5", 13:32:02 "user_name": "split" 13:32:02 } 13:32:02 }, 13:32:02 { 13:32:02 "kind": "ParallelDo", 13:32:02 "name": "s4", 13:32:02 "properties": { 13:32:02 "display_data": [ 13:32:02 { 13:32:02 "key": "fn", 13:32:02 "label": "Transform Function", 13:32:02 "namespace": "apache_beam.transforms.core.CallableWrapperDoFn", 13:32:02 "type": "STRING", 13:32:02 "value": "<lambda>" 13:32:02 }, 13:32:02 { 13:32:02 "key": "fn", 13:32:02 "label": "Transform Function", 13:32:02 "namespace": "apache_beam.transforms.core.ParDo", 13:32:02 "shortValue": "CallableWrapperDoFn", 13:32:02 "type": "STRING", 13:32:02 "value": "apache_beam.transforms.core.CallableWrapperDoFn" 13:32:02 } 13:32:02 ], 13:32:02 "non_parallel_inputs": {}, 13:32:02 "output_info": [ 13:32:02 { 13:32:02 "encoding": { 13:32:02 "@type": "kind:windowed_value", 13:32:02 "component_encodings": [ 13:32:02 { 13:32:02 "@type": "kind:pair", 13:32:02 "component_encodings": [ 13:32:02 { 13:32:02 "@type": "StrUtf8Coder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlzBJUWhJWkWziAeVyGDZmMhY20hU5IeAAajEkY=", 13:32:02 "component_encodings": [], 13:32:02 "pipeline_proto_coder_id": "ref_Coder_StrUtf8Coder_4" 13:32:02 }, 13:32:02 { 13:32:02 "@type": "kind:varint" 13:32:02 } 13:32:02 ], 13:32:02 "is_pair_like": true 13:32:02 }, 13:32:02 { 13:32:02 "@type": "kind:global_window" 13:32:02 } 13:32:02 ], 13:32:02 "is_wrapper": true 13:32:02 }, 13:32:02 "output_name": "out", 13:32:02 "user_name": "pair_with_one.out" 13:32:02 } 13:32:02 ], 13:32:02 "parallel_input": { 13:32:02 "@type": "OutputReference", 13:32:02 "output_name": "out", 13:32:02 "step_name": "s3" 13:32:02 }, 13:32:02 "serialized_fn": "ref_AppliedPTransform_pair_with_one_6", 13:32:02 "user_name": "pair_with_one" 13:32:02 } 13:32:02 }, 13:32:02 { 13:32:02 "kind": "ParallelDo", 13:32:02 "name": "s5", 13:32:02 "properties": { 13:32:02 "display_data": [ 13:32:02 { 13:32:02 "key": "fn", 13:32:02 "label": "Transform Function", 13:32:02 "namespace": "apache_beam.transforms.core.WindowInto", 13:32:02 "shortValue": "WindowIntoFn", 13:32:02 "type": "STRING", 13:32:02 "value": "apache_beam.transforms.core.WindowIntoFn" 13:32:02 } 13:32:02 ], 13:32:02 "non_parallel_inputs": {}, 13:32:02 "output_info": [ 13:32:02 { 13:32:02 "encoding": { 13:32:02 "@type": "kind:windowed_value", 13:32:02 "component_encodings": [ 13:32:02 { 13:32:02 "@type": "kind:pair", 13:32:02 "component_encodings": [ 13:32:02 { 13:32:02 "@type": "StrUtf8Coder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlzBJUWhJWkWziAeVyGDZmMhY20hU5IeAAajEkY=", 13:32:02 "component_encodings": [], 13:32:02 "pipeline_proto_coder_id": "ref_Coder_StrUtf8Coder_4" 13:32:02 }, 13:32:02 { 13:32:02 "@type": "kind:varint" 13:32:02 } 13:32:02 ], 13:32:02 "is_pair_like": true 13:32:02 }, 13:32:02 { 13:32:02 "@type": "kind:interval_window" 13:32:02 } 13:32:02 ], 13:32:02 "is_wrapper": true 13:32:02 }, 13:32:02 "output_name": "out", 13:32:02 "user_name": "WindowInto(WindowIntoFn).out" 13:32:02 } 13:32:02 ], 13:32:02 "parallel_input": { 13:32:02 "@type": "OutputReference", 13:32:02 "output_name": "out", 13:32:02 "step_name": "s4" 13:32:02 }, 13:32:02 "serialized_fn": "<string of 436 bytes>", 13:32:02 "user_name": "WindowInto(WindowIntoFn)" 13:32:02 } 13:32:02 }, 13:32:02 { 13:32:02 "kind": "GroupByKey", 13:32:02 "name": "s6", 13:32:02 "properties": { 13:32:02 "display_data": [], 13:32:02 "output_info": [ 13:32:02 { 13:32:02 "encoding": { 13:32:02 "@type": "kind:windowed_value", 13:32:02 "component_encodings": [ 13:32:02 { 13:32:02 "@type": "kind:pair", 13:32:02 "component_encodings": [ 13:32:02 { 13:32:02 "@type": "StrUtf8Coder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlzBJUWhJWkWziAeVyGDZmMhY20hU5IeAAajEkY=", 13:32:02 "component_encodings": [], 13:32:02 "pipeline_proto_coder_id": "ref_Coder_StrUtf8Coder_4" 13:32:02 }, 13:32:02 { 13:32:02 "@type": "kind:stream", 13:32:02 "component_encodings": [ 13:32:02 { 13:32:02 "@type": "kind:varint" 13:32:02 } 13:32:02 ], 13:32:02 "is_stream_like": true 13:32:02 } 13:32:02 ], 13:32:02 "is_pair_like": true 13:32:02 }, 13:32:02 { 13:32:02 "@type": "kind:interval_window" 13:32:02 } 13:32:02 ], 13:32:02 "is_wrapper": true 13:32:02 }, 13:32:02 "output_name": "out", 13:32:02 "user_name": "group.out" 13:32:02 } 13:32:02 ], 13:32:02 "parallel_input": { 13:32:02 "@type": "OutputReference", 13:32:02 "output_name": "out", 13:32:02 "step_name": "s5" 13:32:02 }, 13:32:02 "serialized_fn": "%0AH%22F%0A%1Fref_Coder_IntervalWindowCoder_1%12%23%0A%21%0A%1F%0A%1Dbeam%3Acoder%3Ainterval_window%3Av1j%5D%0A%2C%0A%2A%0A%20beam%3Awindowfn%3Afixed_windows%3Av0.1%1A%06%0A%02%08%0F%12%00%10%01%1A%1Fref_Coder_IntervalWindowCoder_1%22%02%3A%00%28%010%018%01H%01", 13:32:02 "user_name": "group" 13:32:02 } 13:32:02 }, 13:32:02 { 13:32:02 "kind": "ParallelDo", 13:32:02 "name": "s7", 13:32:02 "properties": { 13:32:02 "display_data": [ 13:32:02 { 13:32:02 "key": "fn", 13:32:02 "label": "Transform Function", 13:32:02 "namespace": "apache_beam.transforms.core.CallableWrapperDoFn", 13:32:02 "type": "STRING", 13:32:02 "value": "count_ones" 13:32:02 }, 13:32:02 { 13:32:02 "key": "fn", 13:32:02 "label": "Transform Function", 13:32:02 "namespace": "apache_beam.transforms.core.ParDo", 13:32:02 "shortValue": "CallableWrapperDoFn", 13:32:02 "type": "STRING", 13:32:02 "value": "apache_beam.transforms.core.CallableWrapperDoFn" 13:32:02 } 13:32:02 ], 13:32:02 "non_parallel_inputs": {}, 13:32:02 "output_info": [ 13:32:02 { 13:32:02 "encoding": { 13:32:02 "@type": "kind:windowed_value", 13:32:02 "component_encodings": [ 13:32:02 { 13:32:02 "@type": "kind:pair", 13:32:02 "component_encodings": [ 13:32:02 { 13:32:02 "@type": "StrUtf8Coder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlzBJUWhJWkWziAeVyGDZmMhY20hU5IeAAajEkY=", 13:32:02 "component_encodings": [], 13:32:02 "pipeline_proto_coder_id": "ref_Coder_StrUtf8Coder_4" 13:32:02 }, 13:32:02 { 13:32:02 "@type": "FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/", 13:32:02 "component_encodings": [ 13:32:02 { 13:32:02 "@type": "FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/", 13:32:02 "component_encodings": [], 13:32:02 "pipeline_proto_coder_id": "ref_Coder_FastPrimitivesCoder_3" 13:32:02 }, 13:32:02 { 13:32:02 "@type": "FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/", 13:32:02 "component_encodings": [], 13:32:02 "pipeline_proto_coder_id": "ref_Coder_FastPrimitivesCoder_3" 13:32:02 } 13:32:02 ], 13:32:02 "is_pair_like": true, 13:32:02 "pipeline_proto_coder_id": "ref_Coder_FastPrimitivesCoder_3" 13:32:02 } 13:32:02 ], 13:32:02 "is_pair_like": true 13:32:02 }, 13:32:02 { 13:32:02 "@type": "kind:interval_window" 13:32:02 } 13:32:02 ], 13:32:02 "is_wrapper": true 13:32:02 }, 13:32:02 "output_name": "out", 13:32:02 "user_name": "count.out" 13:32:02 } 13:32:02 ], 13:32:02 "parallel_input": { 13:32:02 "@type": "OutputReference", 13:32:02 "output_name": "out", 13:32:02 "step_name": "s6" 13:32:02 }, 13:32:02 "serialized_fn": "ref_AppliedPTransform_count_9", 13:32:02 "user_name": "count" 13:32:02 } 13:32:02 }, 13:32:02 { 13:32:02 "kind": "ParallelDo", 13:32:02 "name": "s8", 13:32:02 "properties": { 13:32:02 "display_data": [ 13:32:02 { 13:32:02 "key": "fn", 13:32:02 "label": "Transform Function", 13:32:02 "namespace": "apache_beam.transforms.core.CallableWrapperDoFn", 13:32:02 "type": "STRING", 13:32:02 "value": "format_result" 13:32:02 }, 13:32:02 { 13:32:02 "key": "fn", 13:32:02 "label": "Transform Function", 13:32:02 "namespace": "apache_beam.transforms.core.ParDo", 13:32:02 "shortValue": "CallableWrapperDoFn", 13:32:02 "type": "STRING", 13:32:02 "value": "apache_beam.transforms.core.CallableWrapperDoFn" 13:32:02 } 13:32:02 ], 13:32:02 "non_parallel_inputs": {}, 13:32:02 "output_info": [ 13:32:02 { 13:32:02 "encoding": { 13:32:02 "@type": "kind:windowed_value", 13:32:02 "component_encodings": [ 13:32:02 { 13:32:02 "@type": "FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/", 13:32:02 "component_encodings": [ 13:32:02 { 13:32:02 "@type": "FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/", 13:32:02 "component_encodings": [], 13:32:02 "pipeline_proto_coder_id": "ref_Coder_FastPrimitivesCoder_3" 13:32:02 }, 13:32:02 { 13:32:02 "@type": "FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/", 13:32:02 "component_encodings": [], 13:32:02 "pipeline_proto_coder_id": "ref_Coder_FastPrimitivesCoder_3" 13:32:02 } 13:32:02 ], 13:32:02 "is_pair_like": true, 13:32:02 "pipeline_proto_coder_id": "ref_Coder_FastPrimitivesCoder_3" 13:32:02 }, 13:32:02 { 13:32:02 "@type": "kind:interval_window" 13:32:02 } 13:32:02 ], 13:32:02 "is_wrapper": true 13:32:02 }, 13:32:02 "output_name": "out", 13:32:02 "user_name": "format.out" 13:32:02 } 13:32:02 ], 13:32:02 "parallel_input": { 13:32:02 "@type": "OutputReference", 13:32:02 "output_name": "out", 13:32:02 "step_name": "s7" 13:32:02 }, 13:32:02 "serialized_fn": "ref_AppliedPTransform_format_10", 13:32:02 "user_name": "format" 13:32:02 } 13:32:02 }, 13:32:02 { 13:32:02 "kind": "ParallelDo", 13:32:02 "name": "s9", 13:32:02 "properties": { 13:32:02 "display_data": [ 13:32:02 { 13:32:02 "key": "fn", 13:32:02 "label": "Transform Function", 13:32:02 "namespace": "apache_beam.transforms.core.CallableWrapperDoFn", 13:32:02 "type": "STRING", 13:32:02 "value": "<lambda>" 13:32:02 }, 13:32:02 { 13:32:02 "key": "fn", 13:32:02 "label": "Transform Function", 13:32:02 "namespace": "apache_beam.transforms.core.ParDo", 13:32:02 "shortValue": "CallableWrapperDoFn", 13:32:02 "type": "STRING", 13:32:02 "value": "apache_beam.transforms.core.CallableWrapperDoFn" 13:32:02 } 13:32:02 ], 13:32:02 "non_parallel_inputs": {}, 13:32:02 "output_info": [ 13:32:02 { 13:32:02 "encoding": { 13:32:02 "@type": "kind:windowed_value", 13:32:02 "component_encodings": [ 13:32:02 { 13:32:02 "@type": "kind:bytes" 13:32:02 }, 13:32:02 { 13:32:02 "@type": "kind:interval_window" 13:32:02 } 13:32:02 ], 13:32:02 "is_wrapper": true 13:32:02 }, 13:32:02 "output_name": "out", 13:32:02 "user_name": "encode.out" 13:32:02 } 13:32:02 ], 13:32:02 "parallel_input": { 13:32:02 "@type": "OutputReference", 13:32:02 "output_name": "out", 13:32:02 "step_name": "s8" 13:32:02 }, 13:32:02 "serialized_fn": "ref_AppliedPTransform_encode_11", 13:32:02 "user_name": "encode" 13:32:02 } 13:32:02 }, 13:32:02 { 13:32:02 "kind": "ParallelWrite", 13:32:02 "name": "s10", 13:32:02 "properties": { 13:32:02 "display_data": [], 13:32:02 "encoding": { 13:32:02 "@type": "kind:windowed_value", 13:32:02 "component_encodings": [ 13:32:02 { 13:32:02 "@type": "kind:bytes" 13:32:02 }, 13:32:02 { 13:32:02 "@type": "kind:global_window" 13:32:02 } 13:32:02 ], 13:32:02 "is_wrapper": true 13:32:02 }, 13:32:02 "format": "pubsub", 13:32:02 "parallel_input": { 13:32:02 "@type": "OutputReference", 13:32:02 "output_name": "out", 13:32:02 "step_name": "s9" 13:32:02 }, 13:32:02 "pubsub_topic": "projects/apache-beam-testing/topics/wc_topic_outputb7a2edca-b896-496f-9ab7-4fdfee001097", 13:32:02 "user_name": "WriteToPubSub/Write/NativeWrite" 13:32:02 } 13:32:02 } 13:32:02 ], 13:32:02 "type": "JOB_TYPE_STREAMING" 13:32:02 } 13:32:02 root: INFO: Create job: <Job 13:32:02 createTime: u'2019-05-17T19:40:28.428825Z' 13:32:02 currentStateTime: u'1970-01-01T00:00:00Z' 13:32:02 id: u'2019-05-17_12_40_27-16144646158774694719' 13:32:02 location: u'us-central1' 13:32:02 name: u'beamapp-jenkins-0517194018-707115' 13:32:02 projectId: u'apache-beam-testing' 13:32:02 stageStates: [] 13:32:02 startTime: u'2019-05-17T19:40:28.428825Z' 13:32:02 steps: [] 13:32:02 tempFiles: [] 13:32:02 type: TypeValueValuesEnum(JOB_TYPE_STREAMING, 2)> 13:32:02 root: INFO: Created job with id: [2019-05-17_12_40_27-16144646158774694719] 13:32:02 root: INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-05-17_12_40_27-16144646158774694719?project=apache-beam-testing 13:32:02 root: INFO: Job 2019-05-17_12_40_27-16144646158774694719 is in state JOB_STATE_RUNNING 13:32:02 root: INFO: 2019-05-17T19:40:31.054Z: JOB_MESSAGE_DETAILED: Checking permissions granted to controller Service Account. 13:32:02 root: INFO: 2019-05-17T19:40:31.699Z: JOB_MESSAGE_BASIC: Worker configuration: n1-standard-4 in us-central1-a. 13:32:02 root: INFO: 2019-05-17T19:40:32.189Z: JOB_MESSAGE_DETAILED: Expanding SplittableParDo operations into optimizable parts. 13:32:02 root: INFO: 2019-05-17T19:40:32.192Z: JOB_MESSAGE_DETAILED: Expanding CollectionToSingleton operations into optimizable parts. 13:32:02 root: INFO: 2019-05-17T19:40:32.202Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts. 13:32:02 root: INFO: 2019-05-17T19:40:32.208Z: JOB_MESSAGE_DETAILED: Expanding SplittableProcessKeyed operations into optimizable parts. 13:32:02 root: INFO: 2019-05-17T19:40:32.211Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into streaming Read/Write steps 13:32:02 root: INFO: 2019-05-17T19:40:32.219Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information. 13:32:02 root: INFO: 2019-05-17T19:40:32.232Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations 13:32:02 root: INFO: 2019-05-17T19:40:32.235Z: JOB_MESSAGE_DETAILED: Fusing consumer decode into ReadFromPubSub/Read 13:32:02 root: INFO: 2019-05-17T19:40:32.238Z: JOB_MESSAGE_DETAILED: Fusing consumer pair_with_one into split 13:32:02 root: INFO: 2019-05-17T19:40:32.240Z: JOB_MESSAGE_DETAILED: Fusing consumer count into group/MergeBuckets 13:32:02 root: INFO: 2019-05-17T19:40:32.243Z: JOB_MESSAGE_DETAILED: Fusing consumer WriteToPubSub/Write/NativeWrite into encode 13:32:02 root: INFO: 2019-05-17T19:40:32.245Z: JOB_MESSAGE_DETAILED: Fusing consumer encode into format 13:32:02 root: INFO: 2019-05-17T19:40:32.247Z: JOB_MESSAGE_DETAILED: Fusing consumer group/MergeBuckets into group/ReadStream 13:32:02 root: INFO: 2019-05-17T19:40:32.250Z: JOB_MESSAGE_DETAILED: Fusing consumer format into count 13:32:02 root: INFO: 2019-05-17T19:40:32.252Z: JOB_MESSAGE_DETAILED: Fusing consumer group/WriteStream into WindowInto(WindowIntoFn) 13:32:02 root: INFO: 2019-05-17T19:40:32.254Z: JOB_MESSAGE_DETAILED: Fusing consumer split into decode 13:32:02 root: INFO: 2019-05-17T19:40:32.256Z: JOB_MESSAGE_DETAILED: Fusing consumer WindowInto(WindowIntoFn) into pair_with_one 13:32:02 root: INFO: 2019-05-17T19:40:32.268Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph. 13:32:02 root: INFO: 2019-05-17T19:40:32.283Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps. 13:32:02 root: INFO: 2019-05-17T19:40:32.296Z: JOB_MESSAGE_DEBUG: Assigning stage ids. 13:32:02 root: INFO: 2019-05-17T19:40:32.457Z: JOB_MESSAGE_DEBUG: Executing wait step start2 13:32:02 root: INFO: 2019-05-17T19:40:32.490Z: JOB_MESSAGE_DEBUG: Starting worker pool setup. 13:32:02 root: INFO: 2019-05-17T19:40:32.496Z: JOB_MESSAGE_BASIC: Starting 1 workers... 13:32:02 root: INFO: 2019-05-17T19:40:34.639Z: JOB_MESSAGE_BASIC: Executing operation group/ReadStream+group/MergeBuckets+count+format+encode+WriteToPubSub/Write/NativeWrite 13:32:02 root: INFO: 2019-05-17T19:40:34.639Z: JOB_MESSAGE_BASIC: Executing operation ReadFromPubSub/Read+decode+split+pair_with_one+WindowInto(WindowIntoFn)+group/WriteStream 13:32:02 root: INFO: 2019-05-17T19:41:03.592Z: JOB_MESSAGE_DETAILED: Workers have started successfully. 13:32:02 root: INFO: 2019-05-17T19:41:47.135Z: JOB_MESSAGE_DEBUG: Executing input step topology_init_attach_disk_input_step 13:32:02 root: WARNING: Timing out on waiting for job 2019-05-17_12_40_27-16144646158774694719 after 184 seconds 13:32:02 google.auth.transport._http_client: DEBUG: Making request: GET http://169.254.169.254 13:32:02 google.auth.transport._http_client: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/project/project-id 13:32:02 google.auth.transport.requests: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true 13:32:02 urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): metadata.google.internal:80 13:32:02 urllib3.connectionpool: DEBUG: http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/default/?recursive=true HTTP/1.1" 200 144 13:32:02 google.auth.transport.requests: DEBUG: Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/844138762903-compute@developer.gserviceaccount.com/token 13:32:02 urllib3.connectionpool: DEBUG: http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/844138762903-compute@developer.gserviceaccount.com/token HTTP/1.1" 200 176
Attachments
Issue Links
- links to