[FLINK-9842] Job submission fails via CLI with SSL enabled

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.5.1, 1.6.0
• Fix Version/s: 1.5.2, 1.6.0
• Component/s: Command Line Client, Runtime / Coordination
• Labels: pull-request-available regression
• Assignee: Chesnay Schepler
• Reporter: Nico Kruber
• Created: 13/Jul/18 08:14
• Updated: 28/Feb/19 13:30
• Resolved: 16/Jul/18 18:03

Description

There's a regression in Flink 1.5.1 which leads to the job submission via CLI failing with SSL enabled (1.5.0 works). Tried with the WordCount example:

Client log:

2018-07-16 11:11:12,688 INFO  org.apache.flink.client.cli.CliFrontend                       - --------------------------------------------------------------------------------
2018-07-16 11:11:12,690 INFO  org.apache.flink.client.cli.CliFrontend                       -  Starting Command Line Client (Version: 1.5.1, Rev:3488f8b, Date:10.07.2018 @ 11:51:27 GMT)
2018-07-16 11:11:12,690 INFO  org.apache.flink.client.cli.CliFrontend                       -  OS current user: nico
2018-07-16 11:11:12,690 INFO  org.apache.flink.client.cli.CliFrontend                       -  Current Hadoop/Kerberos user: <no hadoop dependency found>
2018-07-16 11:11:12,690 INFO  org.apache.flink.client.cli.CliFrontend                       -  JVM: OpenJDK 64-Bit Server VM - Oracle Corporation - 1.8/25.171-b11
2018-07-16 11:11:12,690 INFO  org.apache.flink.client.cli.CliFrontend                       -  Maximum heap size: 3534 MiBytes
2018-07-16 11:11:12,691 INFO  org.apache.flink.client.cli.CliFrontend                       -  JAVA_HOME: /usr/lib64/jvm/java
2018-07-16 11:11:12,691 INFO  org.apache.flink.client.cli.CliFrontend                       -  No Hadoop Dependency available
2018-07-16 11:11:12,691 INFO  org.apache.flink.client.cli.CliFrontend                       -  JVM Options:
2018-07-16 11:11:12,691 INFO  org.apache.flink.client.cli.CliFrontend                       -     -Dlog.file=/home/nico/Downloads/flink-1.5.1/log/flink-nico-client-nico-work.fritz.box.log
2018-07-16 11:11:12,691 INFO  org.apache.flink.client.cli.CliFrontend                       -     -Dlog4j.configuration=file:/home/nico/Downloads/flink-1.5.1/conf/log4j-cli.properties
2018-07-16 11:11:12,691 INFO  org.apache.flink.client.cli.CliFrontend                       -     -Dlogback.configurationFile=file:/home/nico/Downloads/flink-1.5.1/conf/logback.xml
2018-07-16 11:11:12,691 INFO  org.apache.flink.client.cli.CliFrontend                       -  Program Arguments:
2018-07-16 11:11:12,691 INFO  org.apache.flink.client.cli.CliFrontend                       -     run
2018-07-16 11:11:12,692 INFO  org.apache.flink.client.cli.CliFrontend                       -     ./examples/streaming/WordCount.jar
2018-07-16 11:11:12,692 INFO  org.apache.flink.client.cli.CliFrontend                       -     --input
2018-07-16 11:11:12,692 INFO  org.apache.flink.client.cli.CliFrontend                       -     LICENSE
2018-07-16 11:11:12,692 INFO  org.apache.flink.client.cli.CliFrontend                       -  Classpath: /home/nico/Downloads/flink-1.5.1/lib/flink-python_2.11-1.5.1.jar:/home/nico/Downloads/flink-1.5.1/lib/log4j-1.2.17.jar:/home/nico/Downloads/flink-1.5.1/lib/slf4j-log4j12-1.7.7.jar:/home/nico/Downloads/flink-1.5.1/lib/flink-dist_2.11-1.5.1.jar:::
2018-07-16 11:11:12,692 INFO  org.apache.flink.client.cli.CliFrontend                       - --------------------------------------------------------------------------------
2018-07-16 11:11:12,698 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: jobmanager.rpc.address, localhost
2018-07-16 11:11:12,698 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: jobmanager.rpc.port, 6123
2018-07-16 11:11:12,698 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: jobmanager.heap.mb, 1024
2018-07-16 11:11:12,699 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: taskmanager.heap.mb, 1024
2018-07-16 11:11:12,699 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: taskmanager.numberOfTaskSlots, 1
2018-07-16 11:11:12,699 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: parallelism.default, 1
2018-07-16 11:11:12,700 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: rest.port, 8081
2018-07-16 11:11:12,701 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: security.ssl.enabled, true
2018-07-16 11:11:12,702 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: security.ssl.keystore, /home/nico/Downloads/ssl/node.keystore
2018-07-16 11:11:12,702 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: security.ssl.keystore-password, ******
2018-07-16 11:11:12,702 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: security.ssl.key-password, ******
2018-07-16 11:11:12,702 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: security.ssl.truststore, /home/nico/Downloads/ssl/ca.truststore
2018-07-16 11:11:12,702 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: security.ssl.truststore-password, ******
2018-07-16 11:11:12,706 WARN  org.apache.flink.client.cli.CliFrontend                       - Could not load CLI class org.apache.flink.yarn.cli.FlinkYarnSessionCli.
java.lang.NoClassDefFoundError: org/apache/hadoop/conf/Configuration
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:264)
	at org.apache.flink.client.cli.CliFrontend.loadCustomCommandLine(CliFrontend.java:1208)
	at org.apache.flink.client.cli.CliFrontend.loadCustomCommandLines(CliFrontend.java:1164)
	at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1090)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.conf.Configuration
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 5 more
2018-07-16 11:11:12,720 INFO  org.apache.flink.core.fs.FileSystem                           - Hadoop is not in the classpath/dependencies. The extended set of supported File Systems via Hadoop is not available.
2018-07-16 11:11:12,901 INFO  org.apache.flink.runtime.security.modules.HadoopModuleFactory  - Cannot create Hadoop Security Module because Hadoop cannot be found in the Classpath.
2018-07-16 11:11:12,929 INFO  org.apache.flink.runtime.security.SecurityUtils               - Cannot install HadoopSecurityContext because Hadoop cannot be found in the Classpath.
2018-07-16 11:11:12,987 INFO  org.apache.flink.client.cli.CliFrontend                       - Running 'run' command.
2018-07-16 11:11:12,992 INFO  org.apache.flink.client.cli.CliFrontend                       - Building program from JAR file
2018-07-16 11:11:13,027 WARN  org.apache.flink.configuration.Configuration                  - Config uses deprecated configuration key 'jobmanager.rpc.address' instead of proper key 'rest.address'
2018-07-16 11:11:13,374 INFO  org.apache.flink.runtime.rest.RestClient                      - Rest client endpoint started.
2018-07-16 11:11:13,376 INFO  org.apache.flink.client.cli.CliFrontend                       - Starting execution of program
2018-07-16 11:11:13,377 INFO  org.apache.flink.client.program.rest.RestClusterClient        - Starting program in interactive mode (detached: false)
2018-07-16 11:11:13,397 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: jobmanager.rpc.address, localhost
2018-07-16 11:11:13,398 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: jobmanager.rpc.port, 6123
2018-07-16 11:11:13,398 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: jobmanager.heap.mb, 1024
2018-07-16 11:11:13,398 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: taskmanager.heap.mb, 1024
2018-07-16 11:11:13,398 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: taskmanager.numberOfTaskSlots, 1
2018-07-16 11:11:13,398 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: parallelism.default, 1
2018-07-16 11:11:13,399 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: rest.port, 8081
2018-07-16 11:11:13,400 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: security.ssl.enabled, true
2018-07-16 11:11:13,400 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: security.ssl.keystore, /home/nico/Downloads/ssl/node.keystore
2018-07-16 11:11:13,400 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: security.ssl.keystore-password, ******
2018-07-16 11:11:13,400 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: security.ssl.key-password, ******
2018-07-16 11:11:13,400 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: security.ssl.truststore, /home/nico/Downloads/ssl/ca.truststore
2018-07-16 11:11:13,401 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: security.ssl.truststore-password, ******
2018-07-16 11:11:13,409 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: jobmanager.rpc.address, localhost
2018-07-16 11:11:13,409 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: jobmanager.rpc.port, 6123
2018-07-16 11:11:13,409 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: jobmanager.heap.mb, 1024
2018-07-16 11:11:13,409 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: taskmanager.heap.mb, 1024
2018-07-16 11:11:13,409 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: taskmanager.numberOfTaskSlots, 1
2018-07-16 11:11:13,410 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: parallelism.default, 1
2018-07-16 11:11:13,410 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: rest.port, 8081
2018-07-16 11:11:13,410 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: security.ssl.enabled, true
2018-07-16 11:11:13,410 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: security.ssl.keystore, /home/nico/Downloads/ssl/node.keystore
2018-07-16 11:11:13,410 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: security.ssl.keystore-password, ******
2018-07-16 11:11:13,411 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: security.ssl.key-password, ******
2018-07-16 11:11:13,411 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: security.ssl.truststore, /home/nico/Downloads/ssl/ca.truststore
2018-07-16 11:11:13,411 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: security.ssl.truststore-password, ******
2018-07-16 11:11:13,443 INFO  org.apache.flink.api.java.typeutils.TypeExtractor             - class org.apache.flink.streaming.api.functions.source.TimestampedFileInputSplit does not contain a setter for field modificationTime
2018-07-16 11:11:13,443 INFO  org.apache.flink.api.java.typeutils.TypeExtractor             - Class class org.apache.flink.streaming.api.functions.source.TimestampedFileInputSplit cannot be used as a POJO type because not all fields are valid POJO fields, and must be processed as GenericType. Please read the Flink documentation on "Data Types & Serialization" for details of the effect on performance.
2018-07-16 11:11:13,647 INFO  org.apache.flink.client.program.rest.RestClusterClient        - Submitting job 2de97496915cc5d32a8035d561c25a5c (detached: false).
2018-07-16 11:11:14,697 INFO  org.apache.flink.runtime.rest.RestClient                      - Shutting down rest endpoint.
2018-07-16 11:11:14,702 INFO  org.apache.flink.runtime.rest.RestClient                      - Rest endpoint shutdown complete.
2018-07-16 11:11:14,702 ERROR org.apache.flink.client.cli.CliFrontend                       - Error while running the command.
org.apache.flink.client.program.ProgramInvocationException: Could not retrieve the execution result.
	at org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:257)
	at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:464)
	at org.apache.flink.streaming.api.environment.StreamContextEnvironment.execute(StreamContextEnvironment.java:66)
	at org.apache.flink.streaming.examples.wordcount.WordCount.main(WordCount.java:89)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:528)
	at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:420)
	at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:404)
	at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:785)
	at org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:279)
	at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:214)
	at org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1025)
	at org.apache.flink.client.cli.CliFrontend.lambda$main$9(CliFrontend.java:1101)
	at org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)
	at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1101)
Caused by: org.apache.flink.runtime.client.JobSubmissionException: Failed to submit JobGraph.
	at org.apache.flink.client.program.rest.RestClusterClient.lambda$submitJob$8(RestClusterClient.java:370)
	at java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:870)
	at java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:852)
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
	at org.apache.flink.runtime.concurrent.FutureUtils.lambda$retryOperationWithDelay$5(FutureUtils.java:214)
	at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
	at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
	at java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:561)
	at java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:929)
	at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:442)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.CompletionException: org.apache.flink.runtime.concurrent.FutureUtils$RetryException: Could not complete the operation. Exception is not retryable.
	at java.util.concurrent.CompletableFuture.encodeRelay(CompletableFuture.java:326)
	at java.util.concurrent.CompletableFuture.completeRelay(CompletableFuture.java:338)
	at java.util.concurrent.CompletableFuture.uniRelay(CompletableFuture.java:911)
	at java.util.concurrent.CompletableFuture$UniRelay.tryFire(CompletableFuture.java:899)
	... 12 more
Caused by: org.apache.flink.runtime.concurrent.FutureUtils$RetryException: Could not complete the operation. Exception is not retryable.
	... 10 more
Caused by: java.util.concurrent.CompletionException: org.apache.flink.runtime.rest.util.RestClientException: [Job submission failed.]
	at java.util.concurrent.CompletableFuture.encodeRelay(CompletableFuture.java:326)
	at java.util.concurrent.CompletableFuture.completeRelay(CompletableFuture.java:338)
	at java.util.concurrent.CompletableFuture.uniRelay(CompletableFuture.java:911)
	at java.util.concurrent.CompletableFuture.uniCompose(CompletableFuture.java:953)
	at java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:926)
	... 4 more
Caused by: org.apache.flink.runtime.rest.util.RestClientException: [Job submission failed.]
	at org.apache.flink.runtime.rest.RestClient.parseResponse(RestClient.java:309)
	at org.apache.flink.runtime.rest.RestClient.lambda$submitRequest$3(RestClient.java:293)
	at java.util.concurrent.CompletableFuture.uniCompose(CompletableFuture.java:952)
	... 5 more

JM log:

2018-07-16 11:10:52,992 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         - --------------------------------------------------------------------------------
2018-07-16 11:10:52,997 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -  Starting StandaloneSessionClusterEntrypoint (Version: 1.5.1, Rev:3488f8b, Date:10.07.2018 @ 11:51:27 GMT)
2018-07-16 11:10:52,999 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -  OS current user: nico
2018-07-16 11:10:53,000 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -  Current Hadoop/Kerberos user: <no hadoop dependency found>
2018-07-16 11:10:53,000 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -  JVM: OpenJDK 64-Bit Server VM - Oracle Corporation - 1.8/25.171-b11
2018-07-16 11:10:53,001 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -  Maximum heap size: 981 MiBytes
2018-07-16 11:10:53,001 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -  JAVA_HOME: /usr/lib64/jvm/java
2018-07-16 11:10:53,001 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -  No Hadoop Dependency available
2018-07-16 11:10:53,002 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -  JVM Options:
2018-07-16 11:10:53,002 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -     -Xms1024m
2018-07-16 11:10:53,004 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -     -Xmx1024m
2018-07-16 11:10:53,005 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -     -Dlog.file=/home/nico/Downloads/flink-1.5.1/log/flink-nico-standalonesession-58-nico-work.fritz.box.log
2018-07-16 11:10:53,005 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -     -Dlog4j.configuration=file:/home/nico/Downloads/flink-1.5.1/conf/log4j.properties
2018-07-16 11:10:53,005 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -     -Dlogback.configurationFile=file:/home/nico/Downloads/flink-1.5.1/conf/logback.xml
2018-07-16 11:10:53,005 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -  Program Arguments:
2018-07-16 11:10:53,006 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -     --configDir
2018-07-16 11:10:53,009 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -     /home/nico/Downloads/flink-1.5.1/conf
2018-07-16 11:10:53,010 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -     --executionMode
2018-07-16 11:10:53,010 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -     cluster
2018-07-16 11:10:53,010 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -  Classpath: /home/nico/Downloads/flink-1.5.1/lib/flink-python_2.11-1.5.1.jar:/home/nico/Downloads/flink-1.5.1/lib/log4j-1.2.17.jar:/home/nico/Downloads/flink-1.5.1/lib/slf4j-log4j12-1.7.7.jar:/home/nico/Downloads/flink-1.5.1/lib/flink-dist_2.11-1.5.1.jar:::
2018-07-16 11:10:53,010 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         - --------------------------------------------------------------------------------
2018-07-16 11:10:53,012 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         - Registered UNIX signal handlers for [TERM, HUP, INT]
2018-07-16 11:10:53,044 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: jobmanager.rpc.address, localhost
2018-07-16 11:10:53,050 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: jobmanager.rpc.port, 6123
2018-07-16 11:10:53,050 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: jobmanager.heap.mb, 1024
2018-07-16 11:10:53,055 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: taskmanager.heap.mb, 1024
2018-07-16 11:10:53,055 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: taskmanager.numberOfTaskSlots, 1
2018-07-16 11:10:53,056 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: parallelism.default, 1
2018-07-16 11:10:53,057 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: rest.port, 8081
2018-07-16 11:10:53,065 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: security.ssl.enabled, true
2018-07-16 11:10:53,066 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: security.ssl.keystore, /home/nico/Downloads/ssl/node.keystore
2018-07-16 11:10:53,069 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: security.ssl.keystore-password, ******
2018-07-16 11:10:53,070 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: security.ssl.key-password, ******
2018-07-16 11:10:53,073 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: security.ssl.truststore, /home/nico/Downloads/ssl/ca.truststore
2018-07-16 11:10:53,074 INFO  org.apache.flink.configuration.GlobalConfiguration            - Loading configuration property: security.ssl.truststore-password, ******
2018-07-16 11:10:53,442 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         - Starting StandaloneSessionClusterEntrypoint.
2018-07-16 11:10:53,442 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         - Install default filesystem.
2018-07-16 11:10:53,465 INFO  org.apache.flink.core.fs.FileSystem                           - Hadoop is not in the classpath/dependencies. The extended set of supported File Systems via Hadoop is not available.
2018-07-16 11:10:53,488 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         - Install security context.
2018-07-16 11:10:53,514 INFO  org.apache.flink.runtime.security.modules.HadoopModuleFactory  - Cannot create Hadoop Security Module because Hadoop cannot be found in the Classpath.
2018-07-16 11:10:53,550 INFO  org.apache.flink.runtime.security.SecurityUtils               - Cannot install HadoopSecurityContext because Hadoop cannot be found in the Classpath.
2018-07-16 11:10:53,553 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         - Initializing cluster services.
2018-07-16 11:10:53,587 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         - Trying to start actor system at localhost:6123
2018-07-16 11:10:55,809 INFO  akka.event.slf4j.Slf4jLogger                                  - Slf4jLogger started
2018-07-16 11:10:55,993 INFO  akka.remote.Remoting                                          - Starting remoting
2018-07-16 11:10:56,499 INFO  akka.remote.Remoting                                          - Remoting started; listening on addresses :[akka.ssl.tcp://flink@localhost:6123]
2018-07-16 11:10:56,508 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         - Actor system started at akka.ssl.tcp://flink@localhost:6123
2018-07-16 11:10:56,568 WARN  org.apache.flink.configuration.Configuration                  - Config uses deprecated configuration key 'jobmanager.rpc.address' instead of proper key 'rest.address'
2018-07-16 11:10:56,593 INFO  org.apache.flink.runtime.blob.BlobServer                      - Created BLOB server storage directory /tmp/blobStore-769607e6-0636-4c5c-a148-fb32a8eb8ce2
2018-07-16 11:10:57,301 INFO  org.apache.flink.runtime.blob.BlobServer                      - Enabling ssl for the blob server
2018-07-16 11:10:57,312 INFO  org.apache.flink.runtime.blob.BlobServer                      - Started BLOB server at 0.0.0.0:40163 - max concurrent requests: 50 - max backlog: 1000
2018-07-16 11:10:57,409 INFO  org.apache.flink.runtime.metrics.MetricRegistryImpl           - No metrics reporter configured, no metrics will be exposed/reported.
2018-07-16 11:10:57,426 INFO  org.apache.flink.runtime.dispatcher.FileArchivedExecutionGraphStore  - Initializing FileArchivedExecutionGraphStore: Storage directory /tmp/executionGraphStore-fd2a296e-9e71-430b-a4c8-2d05255d7c78, expiration time 3600000, maximum cache size 52428800 bytes.
2018-07-16 11:10:57,609 INFO  org.apache.flink.runtime.blob.TransientBlobCache              - Created BLOB cache storage directory /tmp/blobStore-ec8844ea-b919-4366-b18f-497c58f6b0f7
2018-07-16 11:10:57,642 WARN  org.apache.flink.configuration.Configuration                  - Config uses deprecated configuration key 'jobmanager.rpc.address' instead of proper key 'rest.address'
2018-07-16 11:10:57,692 WARN  org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint    - Upload directory /tmp/flink-web-dd08b7a4-8a2f-4053-972b-f9b40ff4e79c/flink-web-upload does not exist, or has been deleted externally. Previously uploaded files are no longer available.
2018-07-16 11:10:57,701 INFO  org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint    - Created directory /tmp/flink-web-dd08b7a4-8a2f-4053-972b-f9b40ff4e79c/flink-web-upload for file uploads.
2018-07-16 11:10:57,710 INFO  org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint    - Starting rest endpoint.
2018-07-16 11:10:59,406 INFO  org.apache.flink.runtime.webmonitor.WebMonitorUtils           - Determined location of main cluster component log file: /home/nico/Downloads/flink-1.5.1/log/flink-nico-standalonesession-58-nico-work.fritz.box.log
2018-07-16 11:10:59,406 INFO  org.apache.flink.runtime.webmonitor.WebMonitorUtils           - Determined location of main cluster component stdout file: /home/nico/Downloads/flink-1.5.1/log/flink-nico-standalonesession-58-nico-work.fritz.box.out
2018-07-16 11:10:59,809 INFO  org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint    - Rest endpoint listening at localhost:8081
2018-07-16 11:10:59,809 INFO  org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint    - https://localhost:8081 was granted leadership with leaderSessionID=00000000-0000-0000-0000-000000000000
2018-07-16 11:10:59,809 INFO  org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint    - Web frontend listening at https://localhost:8081.
2018-07-16 11:10:59,862 INFO  org.apache.flink.runtime.rpc.akka.AkkaRpcService              - Starting RPC endpoint for org.apache.flink.runtime.resourcemanager.StandaloneResourceManager at akka://flink/user/resourcemanager .
2018-07-16 11:11:00,257 INFO  org.apache.flink.runtime.rpc.akka.AkkaRpcService              - Starting RPC endpoint for org.apache.flink.runtime.dispatcher.StandaloneDispatcher at akka://flink/user/dispatcher .
2018-07-16 11:11:00,345 INFO  org.apache.flink.runtime.resourcemanager.StandaloneResourceManager  - ResourceManager akka.ssl.tcp://flink@localhost:6123/user/resourcemanager was granted leadership with fencing token 00000000000000000000000000000000
2018-07-16 11:11:00,346 INFO  org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager  - Starting the SlotManager.
2018-07-16 11:11:00,431 INFO  org.apache.flink.runtime.dispatcher.StandaloneDispatcher      - Dispatcher akka.ssl.tcp://flink@localhost:6123/user/dispatcher was granted leadership with fencing token 00000000-0000-0000-0000-000000000000
2018-07-16 11:11:00,431 INFO  org.apache.flink.runtime.dispatcher.StandaloneDispatcher      - Recovering all persisted jobs.
2018-07-16 11:11:04,083 INFO  org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager  - Registering TaskManager f2a9473f4feee80fb38e3dd276e0f83a under fb8c534b0f3152daddcc44a9f90c99e0 at the SlotManager.
2018-07-16 11:11:05,094 INFO  org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager  - Registering TaskManager 4f232803cb360a229e8fa984602883e6 under a4b98f1d88a137245010a8e7f7b35338 at the SlotManager.
2018-07-16 11:11:14,620 ERROR org.apache.flink.runtime.blob.BlobServerConnection            - Error while executing BLOB connection.
javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
	at sun.security.ssl.InputRecord.handleUnknownRecord(InputRecord.java:710)
	at sun.security.ssl.InputRecord.read(InputRecord.java:527)
	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983)
	at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385)
	at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:938)
	at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
	at sun.security.ssl.AppInputStream.read(AppInputStream.java:71)
	at org.apache.flink.runtime.blob.BlobServerConnection.run(BlobServerConnection.java:106)
2018-07-16 11:11:14,621 ERROR org.apache.flink.runtime.rest.handler.job.JobSubmitHandler    - Exception occurred in REST handler.
org.apache.flink.runtime.rest.handler.RestHandlerException: Job submission failed.
	at org.apache.flink.runtime.rest.handler.job.JobSubmitHandler.lambda$handleRequest$2(JobSubmitHandler.java:106)
	at java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:870)
	at java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:852)
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
	at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1595)
	at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:39)
	at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:415)
	at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
	at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: java.util.concurrent.CompletionException: org.apache.flink.runtime.rest.handler.RestHandlerException: Could not upload job files.
	at org.apache.flink.runtime.rest.handler.job.JobSubmitHandler.lambda$uploadJobGraphFiles$4(JobSubmitHandler.java:151)
	at java.util.concurrent.CompletableFuture.biApply(CompletableFuture.java:1105)
	at java.util.concurrent.CompletableFuture$BiApply.tryFire(CompletableFuture.java:1070)
	... 8 more
Caused by: org.apache.flink.runtime.rest.handler.RestHandlerException: Could not upload job files.
	... 11 more
Caused by: java.io.IOException: PUT operation failed: Unrecognized response: 21.
	at org.apache.flink.runtime.blob.BlobClient.putInputStream(BlobClient.java:445)
	at org.apache.flink.runtime.blob.BlobClient.uploadJarFiles(BlobClient.java:554)
	at org.apache.flink.runtime.rest.handler.job.JobSubmitHandler.lambda$uploadJobGraphFiles$4(JobSubmitHandler.java:146)
	... 10 more
Caused by: java.io.IOException: Unrecognized response: 21.
	at org.apache.flink.runtime.blob.BlobClient.receiveAndCheckPutResponse(BlobClient.java:516)
	at org.apache.flink.runtime.blob.BlobClient.putInputStream(BlobClient.java:441)
	... 12 more

Issue Links

• blocks: FLINK-9839 End-to-end test: Streaming job with SSL
• is related to: FLINK-9261 Regression - Flink CLI and Web UI not working when SSL is enabled
• links to: GitHub Pull Request #6340