Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-12207

Python PostCommits failing portableWordCountSparkRunnerBatch

    XMLWordPrintableJSON

Details

    Description

      The first run in the Python 3.8 suite with this particular failure appears to be https://ci-beam.apache.org/job/beam_PostCommit_Python38/1104

      06:26:36 OK (SKIP=6)
06:26:36 
06:26:38 FAILURE: Build failed with an exception.
06:26:38 
06:26:38 * What went wrong:
06:26:38 Execution failed for task ':sdks:python:test-suites:portable:py38:portableWordCountSparkRunnerBatch'.
06:26:38 > Process 'command 'sh'' finished with non-zero exit value 1
06:26:38 
06:26:38 * Try:
06:26:38 Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
06:26:38 
06:26:38 * Get more help at https://help.gradle.org
06:26:38 
06:26:38 Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
06:26:38 Use '--warning-mode all' to show the individual deprecation warnings.
06:26:38 See https://docs.gradle.org/6.8.3/userguide/command_line_interface.html#sec:command_line_warnings
06:26:38 
06:26:38 BUILD FAILED in 1h 26m 11s

      05:25:34 > Task :sdks:python:test-suites:portable:py38:portableWordCountSparkRunnerBatch FAILED
05:25:34 INFO:apache_beam.runners.worker.worker_pool_main:Listening for workers at localhost:42005
05:25:34 WARNING:root:Make sure that locally built Python SDK docker image has Python 3.8 interpreter.
05:25:34 INFO:root:Default Python SDK image for environment is apache/beam_python3.8_sdk:2.30.0.dev
05:25:34 INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function lift_combiners at 0x7f71dcdf5a60> ====================
05:25:34 INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function sort_stages at 0x7f71dcdfa1f0> ====================
05:25:34 INFO:apache_beam.utils.subprocess_server:Starting service with ['java' '-jar' '/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python38/src/runners/spark/2/job-server/build/libs/beam-runners-spark-job-server-2.30.0-SNAPSHOT.jar' '--spark-master-url' 'local[4]' '--artifacts-dir' '/tmp/beam-tempgetxx5hw/artifactsw4djif3r' '--job-port' '42069' '--artifact-port' '0' '--expansion-port' '0']
05:25:34 INFO:apache_beam.utils.subprocess_server:b'21/04/16 12:25:29 INFO org.apache.beam.runners.jobsubmission.JobServerDriver: ArtifactStagingService started on localhost:46287'
05:25:34 WARNING:root:Waiting for grpc channel to be ready at localhost:42069.
05:25:34 INFO:apache_beam.utils.subprocess_server:b'21/04/16 12:25:30 INFO org.apache.beam.runners.jobsubmission.JobServerDriver: Java ExpansionService started on localhost:41861'
05:25:34 INFO:apache_beam.utils.subprocess_server:b'21/04/16 12:25:30 INFO org.apache.beam.runners.jobsubmission.JobServerDriver: JobService started on localhost:42069'
05:25:34 INFO:apache_beam.utils.subprocess_server:b'21/04/16 12:25:30 INFO org.apache.beam.runners.jobsubmission.JobServerDriver: Job server now running, terminate with Ctrl+C'
05:25:34 WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['--parallelism=2']
05:25:34 INFO:apache_beam.utils.subprocess_server:b'21/04/16 12:25:31 INFO org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService: Staging artifacts for job_7c9a3a39-f855-455c-8f36-bcd6bfd8152e.'
05:25:34 INFO:apache_beam.utils.subprocess_server:b'21/04/16 12:25:31 INFO org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService: Resolving artifacts for job_7c9a3a39-f855-455c-8f36-bcd6bfd8152e.ref_Environment_default_environment_1.'
05:25:34 INFO:apache_beam.utils.subprocess_server:b'21/04/16 12:25:31 INFO org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService: Getting 1 artifacts for job_7c9a3a39-f855-455c-8f36-bcd6bfd8152e.null.'
05:25:34 INFO:apache_beam.utils.subprocess_server:b'21/04/16 12:25:31 INFO org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService: Artifacts fully staged for job_7c9a3a39-f855-455c-8f36-bcd6bfd8152e.'
05:25:34 INFO:apache_beam.utils.subprocess_server:b'21/04/16 12:25:32 INFO org.apache.beam.runners.spark.SparkJobInvoker: Invoking job BeamApp-jenkins-0416122532-da1b901c_d29eb552-6c06-4531-b0bd-fd234973ff23'
05:25:34 INFO:apache_beam.utils.subprocess_server:b'21/04/16 12:25:32 INFO org.apache.beam.runners.jobsubmission.JobInvocation: Starting job invocation BeamApp-jenkins-0416122532-da1b901c_d29eb552-6c06-4531-b0bd-fd234973ff23'
05:25:34 INFO:apache_beam.runners.portability.portable_runner:Environment "LOOPBACK" has started a component necessary for the execution. Be sure to run the pipeline using
05:25:34   with Pipeline() as p:
05:25:34     p.apply(..)
05:25:34 This ensures that the pipeline finishes before this program exits.
05:25:34 INFO:apache_beam.runners.portability.portable_runner:Job state changed to STOPPED
05:25:34 INFO:apache_beam.runners.portability.portable_runner:Job state changed to STARTING
05:25:34 INFO:apache_beam.runners.portability.portable_runner:Job state changed to RUNNING
05:25:34 INFO:apache_beam.utils.subprocess_server:b'21/04/16 12:25:33 ERROR org.apache.beam.runners.jobsubmission.JobInvocation: Error during job invocation BeamApp-jenkins-0416122532-da1b901c_d29eb552-6c06-4531-b0bd-fd234973ff23.'
05:25:34 INFO:apache_beam.utils.subprocess_server:b'java.lang.NullPointerException'
05:25:34 INFO:apache_beam.utils.subprocess_server:b'\tat org.apache.beam.runners.spark.SparkPipelineRunner.run(SparkPipelineRunner.java:120)'
05:25:34 INFO:apache_beam.utils.subprocess_server:b'\tat org.apache.beam.runners.jobsubmission.JobInvocation.runPipeline(JobInvocation.java:86)'
05:25:34 INFO:apache_beam.utils.subprocess_server:b'\tat org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)'
05:25:34 INFO:apache_beam.utils.subprocess_server:b'\tat org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:57)'
05:25:34 INFO:apache_beam.utils.subprocess_server:b'\tat org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)'
05:25:34 INFO:apache_beam.utils.subprocess_server:b'\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)'
05:25:34 INFO:apache_beam.utils.subprocess_server:b'\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)'
05:25:34 INFO:apache_beam.utils.subprocess_server:b'\tat java.lang.Thread.run(Thread.java:748)'
05:25:34 ERROR:root:java.lang.NullPointerException
05:25:34 INFO:apache_beam.runners.portability.portable_runner:Job state changed to FAILED
05:25:34 Traceback (most recent call last):
05:25:34   File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
05:25:34     return _run_code(code, main_globals, None,
05:25:34   File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
05:25:34     exec(code, run_globals)
05:25:34   File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python38/src/sdks/python/apache_beam/examples/wordcount.py", line 94, in <module>
05:25:34     run()
05:25:34   File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python38/src/sdks/python/apache_beam/examples/wordcount.py", line 89, in run
05:25:34     output | 'Write' >> WriteToText(known_args.output)
05:25:34   File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python38/src/sdks/python/apache_beam/pipeline.py", line 580, in __exit__
05:25:34     self.result.wait_until_finish()
05:25:34   File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python38/src/sdks/python/apache_beam/runners/portability/portable_runner.py", line 602, in wait_until_finish
05:25:34     raise self._runtime_exception
05:25:34 RuntimeError: Pipeline BeamApp-jenkins-0416122532-da1b901c_d29eb552-6c06-4531-b0bd-fd234973ff23 failed in state FAILED: java.lang.NullPointerException

      Attachments

        Issue Links

          links to

          Web Link GitHub Pull Request #14616

          Activity

            People

              ibzib Kyle Weaver
              bhulette Brian Hulette
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1h 10m
                  1h 10m