Details
-
Bug
-
Status: Resolved
-
P0
-
Resolution: Fixed
-
2.11.0
-
None
-
Ubuntu 18.04, Beam 2.11.0, Flink 1.7.2, Docker 18.09.4, on a 2-node cluster with identical VM images
Description
Setup
2-node cluster: `master` and `worker`. `master` runs Flink JobManager, `worker` runs Flink TaskManager.
Problem description
When running Flink TaskManager on `master` and running the word count example provided by Beam as
python sdks/python/apache_beam/examples/wordcount.py --input=gs://<input file location>-output=gs://<output file location>--runner=PortableRunner --job_endpoint=localhost:8099 --environment_config='<docker image location>'
on `master`, the job executes correctly.
However, when running the Flink TaskManger on `worker` (the second node), the `worker` outputs:
java.util.concurrent.ExecutionException: java.io.FileNotFoundException: /tmp/beam-artifact-staging/job_8ff51268-48fb-42e3-b14c-f2e387e26e5a/MANIFEST (No such file or directory)
and crashes (full log in [1]).
The docker image has been created via:
./gradlew :beam-sdks-python-container:docker
The job server has been set up on `master` via:
./gradlew :beam-runners-flink-1.7-job-server:runShadow -PflinkMasterUrl=localhost:8081
which logs
> Task :beam-runners-flink-1.7-job-server:runShadow
Listening for transport dt_socket at address: 5005
[main] INFO org.apache.beam.runners.flink.FlinkJobServerDriver - ArtifactStagingService started on localhost:8098
[main] INFO org.apache.beam.runners.flink.FlinkJobServerDriver - JobService started on localhost:8099
<============-> 98% EXECUTING [3m 0s]
> :beam-runners-flink-1.7-job-server:runShadow
> IDLE
> IDLE
> IDLE
The default Flink example
./bin/flink run ./examples/batch/WordCount.jarworks correctly on the 2-node cluster.
works correctly on the 2-node cluster.
Expected behavior
Either the job executes correctly and returns as in the single-node setup or additional configuration is needed, which is not reflected in the docs. Please let me know if I missed something.
[1]
tail -111111f /root/flink-1.7.2/log/flink-root-taskexecutor-0-VM-0-16-ubuntu.log 2019-04-11 13:11:59,497 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - -------------------------------------------------------------------------------- 2019-04-11 13:11:59,498 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - Starting TaskManager (Version: 1.7.2, Rev:ceba8af, Date:11.02.2019 @ 14:17:09 UTC) 2019-04-11 13:11:59,498 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - OS current user: root 2019-04-11 13:11:59,499 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - Current Hadoop/Kerberos user: <no hadoop dependency found> 2019-04-11 13:11:59,499 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - JVM: OpenJDK 64-Bit Server VM - Oracle Corporation - 1.8/25.191-b12 2019-04-11 13:11:59,499 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - Maximum heap size: 922 MiBytes 2019-04-11 13:11:59,499 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - JAVA_HOME: (not set) 2019-04-11 13:11:59,499 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - No Hadoop Dependency available 2019-04-11 13:11:59,499 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - JVM Options: 2019-04-11 13:11:59,499 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - -XX:+UseG1GC 2019-04-11 13:11:59,499 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - -Xms922M 2019-04-11 13:11:59,499 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - -Xmx922M 2019-04-11 13:11:59,500 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - -XX:MaxDirectMemorySize=8388607T 2019-04-11 13:11:59,500 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - -Dlog.file=/root/flink-1.7.2/log/flink-root-taskexecutor-0-VM-0-16-ubuntu.log 2019-04-11 13:11:59,500 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - -Dlog4j.configuration=file:/root/flink-1.7.2/conf/log4j.properties 2019-04-11 13:11:59,500 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - -Dlogback.configurationFile=file:/root/flink-1.7.2/conf/logback.xml 2019-04-11 13:11:59,500 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - Program Arguments: 2019-04-11 13:11:59,500 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - --configDir 2019-04-11 13:11:59,500 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - /root/flink-1.7.2/conf 2019-04-11 13:11:59,500 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - Classpath: /root/flink-1.7.2/lib/flink-python_2.11-1.7.2.jar:/root/flink-1.7.2/lib/log4j-1.2.17.jar:/root/flink-1.7.2/lib/slf4j-log4j12-1.7.15.jar:/root/flink-1.7.2/lib/flink-dist_2.11-1.7.2.jar::: 2019-04-11 13:11:59,500 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - -------------------------------------------------------------------------------- 2019-04-11 13:11:59,502 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - Registered UNIX signal handlers for [TERM, HUP, INT] 2019-04-11 13:11:59,505 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - Maximum number of open file descriptors is 1048576. 2019-04-11 13:11:59,517 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.rpc.address, 172.26.0.4 2019-04-11 13:11:59,517 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.rpc.port, 6123 2019-04-11 13:11:59,517 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.heap.size, 1024m 2019-04-11 13:11:59,517 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.heap.size, 1024m 2019-04-11 13:11:59,517 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.numberOfTaskSlots, 4 2019-04-11 13:11:59,517 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: parallelism.default, 2 2019-04-11 13:11:59,518 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: rest.port, 8081 2019-04-11 13:11:59,523 INFO org.apache.flink.core.fs.FileSystem - Hadoop is not in the classpath/dependencies. The extended set of supported File Systems via Hadoop is not available. 2019-04-11 13:11:59,547 INFO org.apache.flink.runtime.security.modules.HadoopModuleFactory - Cannot create Hadoop Security Module because Hadoop cannot be found in the Classpath. 2019-04-11 13:11:59,568 INFO org.apache.flink.runtime.security.SecurityUtils - Cannot install HadoopSecurityContext because Hadoop cannot be found in the Classpath. 2019-04-11 13:11:59,754 WARN org.apache.flink.configuration.Configuration - Config uses deprecated configuration key 'jobmanager.rpc.address' instead of proper key 'rest.address' 2019-04-11 13:11:59,758 INFO org.apache.flink.runtime.util.LeaderRetrievalUtils - Trying to select the network interface and address to use by connecting to the leading JobManager. 2019-04-11 13:11:59,759 INFO org.apache.flink.runtime.util.LeaderRetrievalUtils - TaskManager will try to connect for 10000 milliseconds before falling back to heuristics 2019-04-11 13:11:59,762 INFO org.apache.flink.runtime.net.ConnectionUtils - Retrieved new target address /172.26.0.4:6123. 2019-04-11 13:11:59,769 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - TaskManager will use hostname/address '172.26.0.16' (172.26.0.16) for communication. 2019-04-11 13:11:59,772 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils - Trying to start actor system at 172.26.0.16:0 2019-04-11 13:12:00,135 INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started 2019-04-11 13:12:00,201 INFO akka.remote.Remoting - Starting remoting 2019-04-11 13:12:00,303 INFO akka.remote.Remoting - Remoting started; listening on addresses :[akka.tcp://flink@172.26.0.16:39493] 2019-04-11 13:12:00,310 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils - Actor system started at akka.tcp://flink@172.26.0.16:39493 2019-04-11 13:12:00,316 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - Trying to start actor system at 172.26.0.16:0 2019-04-11 13:12:00,332 INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started 2019-04-11 13:12:00,338 INFO akka.remote.Remoting - Starting remoting 2019-04-11 13:12:00,354 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - Actor system started at akka.tcp://flink-metrics@172.26.0.16:41573 2019-04-11 13:12:00,356 INFO akka.remote.Remoting - Remoting started; listening on addresses :[akka.tcp://flink-metrics@172.26.0.16:41573] 2019-04-11 13:12:00,367 INFO org.apache.flink.runtime.metrics.MetricRegistryImpl - No metrics reporter configured, no metrics will be exposed/reported. 2019-04-11 13:12:00,373 INFO org.apache.flink.runtime.blob.PermanentBlobCache - Created BLOB cache storage directory /tmp/blobStore-b1adf8f9-213f-4fcf-86ee-be140bc76181 2019-04-11 13:12:00,376 INFO org.apache.flink.runtime.blob.TransientBlobCache - Created BLOB cache storage directory /tmp/blobStore-3bdadd52-faad-4807-b06a-6e6924f1b1d0 2019-04-11 13:12:00,380 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner - Starting TaskManager with ResourceID: 76eff45c35decbc12d43e53d88883a7f 2019-04-11 13:12:00,384 INFO org.apache.flink.runtime.io.network.netty.NettyConfig - NettyConfig [server address: /172.26.0.16, server port: 0, ssl enabled: false, memory segment size (bytes): 32768, transport type: NIO, number of server threads: 4 (manual), number of client threads: 4 (manual), server connect backlog: 0 (use Netty's default), client connect timeout (sec): 120, send/receive buffer size (bytes): 0 (use Netty's default)] 2019-04-11 13:12:00,423 INFO org.apache.flink.runtime.taskexecutor.TaskManagerServices - Temporary file directory '/tmp': total 49 GB, usable 38 GB (77.55% usable) 2019-04-11 13:12:00,492 INFO org.apache.flink.runtime.io.network.buffer.NetworkBufferPool - Allocated 102 MB for network buffer pool (number of memory segments: 3278, bytes per segment: 32768). 2019-04-11 13:12:00,542 INFO org.apache.flink.runtime.query.QueryableStateUtils - Could not load Queryable State Client Proxy. Probable reason: flink-queryable-state-runtime is not in the classpath. To enable Queryable State, please move the flink-queryable-state-runtime jar from the opt to the lib folder. 2019-04-11 13:12:00,542 INFO org.apache.flink.runtime.query.QueryableStateUtils - Could not load Queryable State Server. Probable reason: flink-queryable-state-runtime is not in the classpath. To enable Queryable State, please move the flink-queryable-state-runtime jar from the opt to the lib folder. 2019-04-11 13:12:00,543 INFO org.apache.flink.runtime.io.network.NetworkEnvironment - Starting the network environment and its components. 2019-04-11 13:12:00,578 INFO org.apache.flink.runtime.io.network.netty.NettyClient - Successful initialization (took 33 ms). 2019-04-11 13:12:00,624 INFO org.apache.flink.runtime.io.network.netty.NettyServer - Successful initialization (took 45 ms). Listening on SocketAddress /172.26.0.16:33565. 2019-04-11 13:12:00,625 WARN org.apache.flink.runtime.taskmanager.TaskManagerLocation - No hostname could be resolved for the IP address 172.26.0.16, using IP address as host name. Local input split assignment (such as for HDFS files) may be impacted. 2019-04-11 13:12:00,625 INFO org.apache.flink.runtime.taskexecutor.TaskManagerServices - Limiting managed memory to 0.7 of the currently free heap space (640 MB), memory will be allocated lazily. 2019-04-11 13:12:00,629 INFO org.apache.flink.runtime.io.disk.iomanager.IOManager - I/O manager uses directory /tmp/flink-io-eb728574-61d9-4170-a020-5eeed20b8551 for spill files. 2019-04-11 13:12:00,693 INFO org.apache.flink.runtime.taskexecutor.TaskManagerConfiguration - Messages have a max timeout of 10000 ms 2019-04-11 13:12:00,701 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.runtime.taskexecutor.TaskExecutor at akka://flink/user/taskmanager_0 . 2019-04-11 13:12:00,717 INFO org.apache.flink.runtime.taskexecutor.JobLeaderService - Start job leader service. 2019-04-11 13:12:00,718 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Connecting to ResourceManager akka.tcp://flink@172.26.0.4:6123/user/resourcemanager(00000000000000000000000000000000). 2019-04-11 13:12:00,718 INFO org.apache.flink.runtime.filecache.FileCache - User file cache uses directory /tmp/flink-dist-cache-d80719fe-a81a-47b6-9560-59766fb86afe 2019-04-11 13:12:00,907 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Resolved ResourceManager address, beginning registration 2019-04-11 13:12:00,907 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Registration at ResourceManager attempt 1 (timeout=100ms) 2019-04-11 13:12:00,965 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Successful registration at resource manager akka.tcp://flink@172.26.0.4:6123/user/resourcemanager under registration id 28630de4ae4181854d3b7143d9a37af1. 2019-04-11 13:22:13,394 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Receive slot request AllocationID\{8810509925a2184bdabc6d58bd808b17} for job a0f3ed507058c39f6261f1e8953d9363 from resource manager with leader id 00000000000000000000000000000000. 2019-04-11 13:22:13,395 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Allocated slot for AllocationID\{8810509925a2184bdabc6d58bd808b17}. 2019-04-11 13:22:13,396 INFO org.apache.flink.runtime.taskexecutor.JobLeaderService - Add job a0f3ed507058c39f6261f1e8953d9363 for job leader monitoring. 2019-04-11 13:22:13,397 INFO org.apache.flink.runtime.taskexecutor.JobLeaderService - Try to register at job manager akka.tcp://flink@172.26.0.4:6123/user/jobmanager_0 with leader id 00000000-0000-0000-0000-000000000000. 2019-04-11 13:22:13,414 INFO org.apache.flink.runtime.taskexecutor.JobLeaderService - Resolved JobManager address, beginning registration 2019-04-11 13:22:13,414 INFO org.apache.flink.runtime.taskexecutor.JobLeaderService - Registration at JobManager attempt 1 (timeout=100ms) 2019-04-11 13:22:13,433 INFO org.apache.flink.runtime.taskexecutor.JobLeaderService - Successful registration at job manager akka.tcp://flink@172.26.0.4:6123/user/jobmanager_0 for job a0f3ed507058c39f6261f1e8953d9363. 2019-04-11 13:22:13,434 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Establish JobManager connection for job a0f3ed507058c39f6261f1e8953d9363. 2019-04-11 13:22:13,437 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Offer reserved slots to the leader of job a0f3ed507058c39f6261f1e8953d9363. 2019-04-11 13:22:13,454 INFO org.apache.flink.runtime.taskexecutor.slot.TaskSlotTable - Activate slot AllocationID\{8810509925a2184bdabc6d58bd808b17}. 2019-04-11 13:22:13,481 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Received task DataSource (Impulse) (1/1). 2019-04-11 13:22:13,482 INFO org.apache.flink.runtime.taskmanager.Task - DataSource (Impulse) (1/1) (ad6774374f4abba501b8f319af3d9ac1) switched from CREATED to DEPLOYING. 2019-04-11 13:22:13,482 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Received task DataSource (Impulse) (1/1). 2019-04-11 13:22:13,483 INFO org.apache.flink.runtime.taskmanager.Task - Creating FileSystem stream leak safety net for task DataSource (Impulse) (1/1) (ad6774374f4abba501b8f319af3d9ac1) [DEPLOYING] 2019-04-11 13:22:13,483 INFO org.apache.flink.runtime.taskmanager.Task - DataSource (Impulse) (1/1) (94014197941e34830c0d79aff847f000) switched from CREATED to DEPLOYING. 2019-04-11 13:22:13,484 INFO org.apache.flink.runtime.taskmanager.Task - Creating FileSystem stream leak safety net for task DataSource (Impulse) (1/1) (94014197941e34830c0d79aff847f000) [DEPLOYING] 2019-04-11 13:22:13,486 INFO org.apache.flink.runtime.taskmanager.Task - Loading JAR files for task DataSource (Impulse) (1/1) (94014197941e34830c0d79aff847f000) [DEPLOYING]. 2019-04-11 13:22:13,487 INFO org.apache.flink.runtime.taskmanager.Task - Loading JAR files for task DataSource (Impulse) (1/1) (ad6774374f4abba501b8f319af3d9ac1) [DEPLOYING]. 2019-04-11 13:22:13,489 INFO org.apache.flink.runtime.blob.BlobClient - Downloading a0f3ed507058c39f6261f1e8953d9363/p-f784164b3f975e48e5bdfa7f321e70fbe4b0e186-ca919340fcac9cb4b2e93b7b6225711b from /172.26.0.4:45541 2019-04-11 13:22:14,085 INFO org.apache.flink.runtime.taskmanager.Task - Registering task at network: DataSource (Impulse) (1/1) (94014197941e34830c0d79aff847f000) [DEPLOYING]. 2019-04-11 13:22:14,085 INFO org.apache.flink.runtime.taskmanager.Task - Registering task at network: DataSource (Impulse) (1/1) (ad6774374f4abba501b8f319af3d9ac1) [DEPLOYING]. 2019-04-11 13:22:14,103 INFO org.apache.flink.runtime.taskmanager.Task - DataSource (Impulse) (1/1) (94014197941e34830c0d79aff847f000) switched from DEPLOYING to RUNNING. 2019-04-11 13:22:14,103 INFO org.apache.flink.runtime.taskmanager.Task - DataSource (Impulse) (1/1) (ad6774374f4abba501b8f319af3d9ac1) switched from DEPLOYING to RUNNING. 2019-04-11 13:22:14,325 INFO org.apache.flink.runtime.taskmanager.Task - DataSource (Impulse) (1/1) (94014197941e34830c0d79aff847f000) switched from RUNNING to FINISHED. 2019-04-11 13:22:14,325 INFO org.apache.flink.runtime.taskmanager.Task - Freeing task resources for DataSource (Impulse) (1/1) (94014197941e34830c0d79aff847f000). 2019-04-11 13:22:14,326 INFO org.apache.flink.runtime.taskmanager.Task - Ensuring all FileSystem streams are closed for task DataSource (Impulse) (1/1) (94014197941e34830c0d79aff847f000) [FINISHED] 2019-04-11 13:22:14,326 INFO org.apache.flink.runtime.taskmanager.Task - DataSource (Impulse) (1/1) (ad6774374f4abba501b8f319af3d9ac1) switched from RUNNING to FINISHED. 2019-04-11 13:22:14,326 INFO org.apache.flink.runtime.taskmanager.Task - Freeing task resources for DataSource (Impulse) (1/1) (ad6774374f4abba501b8f319af3d9ac1). 2019-04-11 13:22:14,327 INFO org.apache.flink.runtime.taskmanager.Task - Ensuring all FileSystem streams are closed for task DataSource (Impulse) (1/1) (ad6774374f4abba501b8f319af3d9ac1) [FINISHED] 2019-04-11 13:22:14,327 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Un-registering task and sending final execution state FINISHED to JobManager for task DataSource (Impulse) 94014197941e34830c0d79aff847f000. 2019-04-11 13:22:14,333 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Un-registering task and sending final execution state FINISHED to JobManager for task DataSource (Impulse) ad6774374f4abba501b8f319af3d9ac1. 2019-04-11 13:22:14,341 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Received task CHAIN MapPartition (MapPartition at [2]write/Write/WriteImpl/DoOnce/\{FlatMap(<lambda at core.py:2123>), Map(decode)}) -> FlatMap (FlatMap at ExtractOutput[0]) (1/1). 2019-04-11 13:22:14,343 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Received task CHAIN MapPartition (MapPartition at [1]read/Read/Split) -> FlatMap (FlatMap at ExtractOutput[0]) (1/1). 2019-04-11 13:22:14,343 INFO org.apache.flink.runtime.taskmanager.Task - CHAIN MapPartition (MapPartition at [2]write/Write/WriteImpl/DoOnce/\{FlatMap(<lambda at core.py:2123>), Map(decode)}) -> FlatMap (FlatMap at ExtractOutput[0]) (1/1) (c3350586427cb04999d1c23038e2b1fc) switched from CREATED to DEPLOYING. 2019-04-11 13:22:14,344 INFO org.apache.flink.runtime.taskmanager.Task - Creating FileSystem stream leak safety net for task CHAIN MapPartition (MapPartition at [2]write/Write/WriteImpl/DoOnce/\{FlatMap(<lambda at core.py:2123>), Map(decode)}) -> FlatMap (FlatMap at ExtractOutput[0]) (1/1) (c3350586427cb04999d1c23038e2b1fc) [DEPLOYING] 2019-04-11 13:22:14,348 INFO org.apache.flink.runtime.taskmanager.Task - Loading JAR files for task CHAIN MapPartition (MapPartition at [2]write/Write/WriteImpl/DoOnce/\{FlatMap(<lambda at core.py:2123>), Map(decode)}) -> FlatMap (FlatMap at ExtractOutput[0]) (1/1) (c3350586427cb04999d1c23038e2b1fc) [DEPLOYING]. 2019-04-11 13:22:14,350 INFO org.apache.flink.runtime.taskmanager.Task - Registering task at network: CHAIN MapPartition (MapPartition at [2]write/Write/WriteImpl/DoOnce/\{FlatMap(<lambda at core.py:2123>), Map(decode)}) -> FlatMap (FlatMap at ExtractOutput[0]) (1/1) (c3350586427cb04999d1c23038e2b1fc) [DEPLOYING]. 2019-04-11 13:22:14,350 INFO org.apache.flink.runtime.taskmanager.Task - CHAIN MapPartition (MapPartition at [2]write/Write/WriteImpl/DoOnce/\{FlatMap(<lambda at core.py:2123>), Map(decode)}) -> FlatMap (FlatMap at ExtractOutput[0]) (1/1) (c3350586427cb04999d1c23038e2b1fc) switched from DEPLOYING to RUNNING. 2019-04-11 13:22:14,352 WARN org.apache.flink.metrics.MetricGroup - The operator name MapPartition (MapPartition at [2]write/Write/WriteImpl/DoOnce/\{FlatMap(<lambda at core.py:2123>), Map(decode)}) exceeded the 80 characters length limit and was truncated. 2019-04-11 13:22:14,352 INFO org.apache.flink.runtime.taskmanager.Task - CHAIN MapPartition (MapPartition at [1]read/Read/Split) -> FlatMap (FlatMap at ExtractOutput[0]) (1/1) (c9bb2f2368faf10656e3ef9ce40e578f) switched from CREATED to DEPLOYING. 2019-04-11 13:22:14,353 INFO org.apache.flink.runtime.taskmanager.Task - Creating FileSystem stream leak safety net for task CHAIN MapPartition (MapPartition at [1]read/Read/Split) -> FlatMap (FlatMap at ExtractOutput[0]) (1/1) (c9bb2f2368faf10656e3ef9ce40e578f) [DEPLOYING] 2019-04-11 13:22:14,356 INFO org.apache.flink.runtime.taskmanager.Task - Loading JAR files for task CHAIN MapPartition (MapPartition at [1]read/Read/Split) -> FlatMap (FlatMap at ExtractOutput[0]) (1/1) (c9bb2f2368faf10656e3ef9ce40e578f) [DEPLOYING]. 2019-04-11 13:22:14,356 INFO org.apache.flink.runtime.taskmanager.Task - Registering task at network: CHAIN MapPartition (MapPartition at [1]read/Read/Split) -> FlatMap (FlatMap at ExtractOutput[0]) (1/1) (c9bb2f2368faf10656e3ef9ce40e578f) [DEPLOYING]. 2019-04-11 13:22:14,365 INFO org.apache.flink.runtime.taskmanager.Task - CHAIN MapPartition (MapPartition at [1]read/Read/Split) -> FlatMap (FlatMap at ExtractOutput[0]) (1/1) (c9bb2f2368faf10656e3ef9ce40e578f) switched from DEPLOYING to RUNNING. 2019-04-11 13:22:15,156 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.rpc.address, 172.26.0.4 2019-04-11 13:22:15,156 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.rpc.port, 6123 2019-04-11 13:22:15,156 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.heap.size, 1024m 2019-04-11 13:22:15,156 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.heap.size, 1024m 2019-04-11 13:22:15,156 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.numberOfTaskSlots, 4 2019-04-11 13:22:15,156 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: parallelism.default, 2 2019-04-11 13:22:15,157 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: rest.port, 8081 2019-04-11 13:22:16,544 INFO org.apache.beam.runners.fnexecution.artifact.BeamFileSystemArtifactRetrievalService - GetManifest for /tmp/beam-artifact-staging/job_8ff51268-48fb-42e3-b14c-f2e387e26e5a/MANIFEST 2019-04-11 13:22:16,545 INFO org.apache.beam.runners.fnexecution.artifact.BeamFileSystemArtifactRetrievalService - Loading manifest for retrieval token /tmp/beam-artifact-staging/job_8ff51268-48fb-42e3-b14c-f2e387e26e5a/MANIFEST 2019-04-11 13:22:16,552 INFO org.apache.beam.runners.fnexecution.artifact.BeamFileSystemArtifactRetrievalService - GetManifest for /tmp/beam-artifact-staging/job_8ff51268-48fb-42e3-b14c-f2e387e26e5a/MANIFEST failed java.util.concurrent.ExecutionException: java.io.FileNotFoundException: /tmp/beam-artifact-staging/job_8ff51268-48fb-42e3-b14c-f2e387e26e5a/MANIFEST (No such file or directory) at org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:500) at org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:459) at org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:76) at org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:142) at org.apache.beam.vendor.guava.v20_0.com.google.common.cache.LocalCache$Segment.getAndRecordStats(LocalCache.java:2373) at org.apache.beam.vendor.guava.v20_0.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2337) at org.apache.beam.vendor.guava.v20_0.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2295) at org.apache.beam.vendor.guava.v20_0.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2208) at org.apache.beam.vendor.guava.v20_0.com.google.common.cache.LocalCache.get(LocalCache.java:4053) at org.apache.beam.vendor.guava.v20_0.com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4057) at org.apache.beam.vendor.guava.v20_0.com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4986) at org.apache.beam.runners.fnexecution.artifact.BeamFileSystemArtifactRetrievalService.getManifest(BeamFileSystemArtifactRetrievalService.java:80) at org.apache.beam.model.jobmanagement.v1.ArtifactRetrievalServiceGrpc$MethodHandlers.invoke(ArtifactRetrievalServiceGrpc.java:298) at org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) at org.apache.beam.vendor.grpc.v1p13p1.io.grpc.PartialForwardingServerCallListener.onHalfClose(PartialForwardingServerCallListener.java:35) at org.apache.beam.vendor.grpc.v1p13p1.io.grpc.ForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java:23) at org.apache.beam.vendor.grpc.v1p13p1.io.grpc.ForwardingServerCallListener$SimpleForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java:40) at org.apache.beam.vendor.grpc.v1p13p1.io.grpc.Contexts$ContextualizedServerCallListener.onHalfClose(Contexts.java:86) at org.apache.beam.vendor.grpc.v1p13p1.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) at org.apache.beam.vendor.grpc.v1p13p1.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:707) at org.apache.beam.vendor.grpc.v1p13p1.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) at org.apache.beam.vendor.grpc.v1p13p1.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.FileNotFoundException: /tmp/beam-artifact-staging/job_8ff51268-48fb-42e3-b14c-f2e387e26e5a/MANIFEST (No such file or directory) at java.io.FileInputStream.open0(Native Method) at java.io.FileInputStream.open(FileInputStream.java:195) at java.io.FileInputStream.<init>(FileInputStream.java:138) at org.apache.beam.sdk.io.LocalFileSystem.open(LocalFileSystem.java:114) at org.apache.beam.sdk.io.LocalFileSystem.open(LocalFileSystem.java:81) at org.apache.beam.sdk.io.FileSystems.open(FileSystems.java:251) at org.apache.beam.runners.fnexecution.artifact.BeamFileSystemArtifactRetrievalService.loadManifest(BeamFileSystemArtifactRetrievalService.java:185) at org.apache.beam.runners.fnexecution.artifact.BeamFileSystemArtifactRetrievalService.loadManifest(BeamFileSystemArtifactRetrievalService.java:180) at org.apache.beam.runners.fnexecution.artifact.BeamFileSystemArtifactRetrievalService$1.load(BeamFileSystemArtifactRetrievalService.java:171) at org.apache.beam.runners.fnexecution.artifact.BeamFileSystemArtifactRetrievalService$1.load(BeamFileSystemArtifactRetrievalService.java:168) at org.apache.beam.vendor.guava.v20_0.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3628) at org.apache.beam.vendor.guava.v20_0.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2336) ... 19 more