Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-10910

Harden Kubernetes e2e test

    XMLWordPrintableJSON

Details

    Description

      The Kubernetes test (e2e test) sometimes fails with the following output:

      ==============================================================================
Running 'Run Kubernetes test'
==============================================================================
TEST_DATA_DIR: /home/admin/flink/flink-end-to-end-tests/test-scripts/temp-test-directory-40594844780
Flink dist directory: /home/admin/flink-1.7.0
/home/admin/flink/flink-end-to-end-tests/test-scripts/test_kubernetes_embedded_job.sh: line 44: none: command not f
ound
Using flink dist: ../../flink-dist/target/flink-*-bin
./
./flink-1.7-SNAPSHOT/
./flink-1.7-SNAPSHOT/LICENSE
./flink-1.7-SNAPSHOT/examples/
./flink-1.7-SNAPSHOT/examples/gelly/
./flink-1.7-SNAPSHOT/examples/gelly/flink-gelly-examples_2.11-1.7-SNAPSHOT.jar
./flink-1.7-SNAPSHOT/examples/streaming/
./flink-1.7-SNAPSHOT/examples/streaming/WindowJoin.jar
./flink-1.7-SNAPSHOT/examples/streaming/SocketWindowWordCount.jar
./flink-1.7-SNAPSHOT/examples/streaming/StateMachineExample.jar
./flink-1.7-SNAPSHOT/examples/streaming/Kafka010Example.jar
./flink-1.7-SNAPSHOT/examples/streaming/Kafka011Example.jar
./flink-1.7-SNAPSHOT/examples/streaming/SessionWindowing.jar
./flink-1.7-SNAPSHOT/examples/streaming/IncrementalLearning.jar
./flink-1.7-SNAPSHOT/examples/streaming/KafkaExample.jar
./flink-1.7-SNAPSHOT/examples/streaming/WordCount.jar
./flink-1.7-SNAPSHOT/examples/streaming/Twitter.jar
./flink-1.7-SNAPSHOT/examples/streaming/Iteration.jar
./flink-1.7-SNAPSHOT/examples/streaming/TopSpeedWindowing.jar
./flink-1.7-SNAPSHOT/examples/batch/
./flink-1.7-SNAPSHOT/examples/batch/KMeans.jar
./flink-1.7-SNAPSHOT/examples/batch/PageRank.jar
./flink-1.7-SNAPSHOT/examples/batch/WebLogAnalysis.jar
./flink-1.7-SNAPSHOT/examples/batch/WordCount.jar
./flink-1.7-SNAPSHOT/examples/batch/EnumTriangles.jar
./flink-1.7-SNAPSHOT/examples/batch/DistCp.jar
./flink-1.7-SNAPSHOT/examples/batch/TransitiveClosure.jar
./flink-1.7-SNAPSHOT/examples/batch/ConnectedComponents.jar
./flink-1.7-SNAPSHOT/examples/python/
./flink-1.7-SNAPSHOT/examples/python/streaming/
./flink-1.7-SNAPSHOT/examples/python/streaming/fibonacci.py
./flink-1.7-SNAPSHOT/examples/python/streaming/word_count.py
./flink-1.7-SNAPSHOT/examples/python/batch/
./flink-1.7-SNAPSHOT/examples/python/batch/TriangleEnumeration.py
./flink-1.7-SNAPSHOT/examples/python/batch/TPCHQuery3.py
./flink-1.7-SNAPSHOT/examples/python/batch/WebLogAnalysis.py
./flink-1.7-SNAPSHOT/examples/python/batch/TPCHQuery10.py
./flink-1.7-SNAPSHOT/examples/python/batch/WordCount.py
./flink-1.7-SNAPSHOT/examples/python/batch/__init__.py
./flink-1.7-SNAPSHOT/log/
./flink-1.7-SNAPSHOT/opt/
./flink-1.7-SNAPSHOT/opt/flink-metrics-ganglia-1.7-SNAPSHOT.jar
./flink-1.7-SNAPSHOT/opt/flink-table_2.11-1.7-SNAPSHOT.jar
./flink-1.7-SNAPSHOT/opt/flink-metrics-dropwizard-1.7-SNAPSHOT.jar
./flink-1.7-SNAPSHOT/opt/flink-metrics-prometheus-1.7-SNAPSHOT.jar
./flink-1.7-SNAPSHOT/opt/flink-cep_2.11-1.7-SNAPSHOT.jar
./flink-1.7-SNAPSHOT/opt/flink-metrics-graphite-1.7-SNAPSHOT.jar
./flink-1.7-SNAPSHOT/opt/flink-gelly-scala_2.11-1.7-SNAPSHOT.jar
./flink-1.7-SNAPSHOT/opt/flink-s3-fs-hadoop-1.7-SNAPSHOT.jar
./flink-1.7-SNAPSHOT/opt/flink-metrics-statsd-1.7-SNAPSHOT.jar
./flink-1.7-SNAPSHOT/opt/flink-queryable-state-runtime_2.11-1.7-SNAPSHOT.jar
./flink-1.7-SNAPSHOT/opt/flink-streaming-python_2.11-1.7-SNAPSHOT.jar
./flink-1.7-SNAPSHOT/opt/flink-ml_2.11-1.7-SNAPSHOT.jar
./flink-1.7-SNAPSHOT/opt/flink-gelly_2.11-1.7-SNAPSHOT.jar
./flink-1.7-SNAPSHOT/opt/flink-swift-fs-hadoop-1.7-SNAPSHOT.jar
./flink-1.7-SNAPSHOT/opt/flink-cep-scala_2.11-1.7-SNAPSHOT.jar
./flink-1.7-SNAPSHOT/opt/flink-metrics-slf4j-1.7-SNAPSHOT.jar
./flink-1.7-SNAPSHOT/opt/flink-s3-fs-presto-1.7-SNAPSHOT.jar
./flink-1.7-SNAPSHOT/opt/flink-metrics-datadog-1.7-SNAPSHOT.jar
./flink-1.7-SNAPSHOT/opt/flink-sql-client_2.11-1.7-SNAPSHOT.jar
./flink-1.7-SNAPSHOT/NOTICE
./flink-1.7-SNAPSHOT/README.txt
./flink-1.7-SNAPSHOT/lib/
./flink-1.7-SNAPSHOT/lib/flink-python_2.11-1.7-SNAPSHOT.jar
./flink-1.7-SNAPSHOT/lib/log4j-1.2.17.jar
./flink-1.7-SNAPSHOT/lib/flink-shaded-hadoop2-uber-1.7-SNAPSHOT.jar
./flink-1.7-SNAPSHOT/lib/flink-dist_2.11-1.7-SNAPSHOT.jar
./flink-1.7-SNAPSHOT/lib/slf4j-log4j12-1.7.15.jar
./flink-1.7-SNAPSHOT/conf/
./flink-1.7-SNAPSHOT/conf/log4j-cli.properties
./flink-1.7-SNAPSHOT/conf/logback-console.xml
./flink-1.7-SNAPSHOT/conf/logback.xml
./flink-1.7-SNAPSHOT/conf/logback-yarn.xml
./flink-1.7-SNAPSHOT/conf/flink-conf.yaml
./flink-1.7-SNAPSHOT/conf/sql-client-defaults.yaml
./flink-1.7-SNAPSHOT/conf/log4j-yarn-session.properties
./flink-1.7-SNAPSHOT/conf/masters
./flink-1.7-SNAPSHOT/conf/log4j.properties
./flink-1.7-SNAPSHOT/conf/zoo.cfg
./flink-1.7-SNAPSHOT/conf/log4j-console.properties
./flink-1.7-SNAPSHOT/conf/slaves
./flink-1.7-SNAPSHOT/bin/
./flink-1.7-SNAPSHOT/bin/mesos-appmaster-job.sh
./flink-1.7-SNAPSHOT/bin/flink
./flink-1.7-SNAPSHOT/bin/flink.bat
./flink-1.7-SNAPSHOT/bin/taskmanager.sh
./flink-1.7-SNAPSHOT/bin/mesos-taskmanager.sh
./flink-1.7-SNAPSHOT/bin/stop-zookeeper-quorum.sh
./flink-1.7-SNAPSHOT/bin/flink-console.sh
./flink-1.7-SNAPSHOT/bin/zookeeper.sh
./flink-1.7-SNAPSHOT/bin/start-cluster.sh
./flink-1.7-SNAPSHOT/bin/standalone-job.sh
./flink-1.7-SNAPSHOT/bin/pyflink-stream.sh
./flink-1.7-SNAPSHOT/bin/jobmanager.sh
./flink-1.7-SNAPSHOT/bin/historyserver.sh
./flink-1.7-SNAPSHOT/bin/config.sh
./flink-1.7-SNAPSHOT/bin/start-zookeeper-quorum.sh
./flink-1.7-SNAPSHOT/bin/start-scala-shell.sh
./flink-1.7-SNAPSHOT/bin/yarn-session.sh
./flink-1.7-SNAPSHOT/bin/pyflink.bat
./flink-1.7-SNAPSHOT/bin/flink-daemon.sh
./flink-1.7-SNAPSHOT/bin/pyflink.sh
./flink-1.7-SNAPSHOT/bin/sql-client.sh
./flink-1.7-SNAPSHOT/bin/mesos-appmaster.sh
./flink-1.7-SNAPSHOT/bin/stop-cluster.sh
./flink-1.7-SNAPSHOT/bin/start-cluster.bat
Sending build context to Docker daemon   472 MB
Step 1/16 : FROM java:8-jre-alpine
 ---> fdc893b19a14
Step 2/16 : RUN apk add --no-cache bash snappy
 ---> Running in 9b6a826cdbd6
fetch http://dl-cdn.alpinelinux.org/alpine/v3.4/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.4/community/x86_64/APKINDEX.tar.gz
(1/7) Installing ncurses-terminfo-base (6.0_p20171125-r0)
(2/7) Installing ncurses-terminfo (6.0_p20171125-r0)
(3/7) Installing ncurses-libs (6.0_p20171125-r0)
(4/7) Installing readline (6.3.008-r4)
(5/7) Installing bash (4.3.42-r5)
Executing bash-4.3.42-r5.post-install
(6/7) Installing libstdc++ (5.3.0-r0)
(7/7) Installing snappy (1.1.3-r0)
Executing busybox-1.24.2-r13.trigger
OK: 115 MiB in 41 packages
 ---> 194f5306312e
Removing intermediate container 9b6a826cdbd6
Step 3/16 : ENV FLINK_INSTALL_PATH /opt
 ---> Running in f44ed28c3ecd
 ---> 61495cda8d7e
Removing intermediate container f44ed28c3ecd
Step 4/16 : ENV FLINK_HOME $FLINK_INSTALL_PATH/flink
 ---> Running in 106db4c7c3e3
 ---> 04fcb3422dfa
Removing intermediate container 106db4c7c3e3
Step 5/16 : ENV FLINK_LIB_DIR $FLINK_HOME/lib
 ---> Running in 51030ee2a57e
 ---> 7c290e53f0f1
Removing intermediate container 51030ee2a57e
Step 6/16 : ENV PATH $PATH:$FLINK_HOME/bin
 ---> Running in 1f89c045de97
 ---> 81cec8fe782c
Removing intermediate container 1f89c045de97
Step 7/16 : ARG flink_dist=NOT_SET
 ---> Running in bfd5fac37dea
 ---> 1dfd4009b044
Removing intermediate container bfd5fac37dea
Step 8/16 : ARG job_jar=NOT_SET
 ---> Running in 712432ce7471
 ---> 17cba67f1748
Removing intermediate container 712432ce7471
Step 9/16 : ADD $flink_dist $FLINK_INSTALL_PATH
 ---> be5be560416b
Removing intermediate container b6b1c4ae0855
Step 10/16 : ADD $job_jar $FLINK_INSTALL_PATH/job.jar
 ---> 1299878dc57c
Removing intermediate container 0fea418feed1
Step 11/16 : RUN set -x &&   ln -s $FLINK_INSTALL_PATH/flink-* $FLINK_HOME &&   ln -s $FLINK_INSTALL_PATH/job.jar $
FLINK_LIB_DIR &&   addgroup -S flink && adduser -D -S -H -G flink -h $FLINK_HOME flink &&   chown -R flink:flink $F
LINK_INSTALL_PATH/flink-* &&   chown -h flink:flink $FLINK_HOME
 ---> Running in 53d9785301eb
+ ln -s /opt/flink-1.7-SNAPSHOT /opt/flink
+ ln -s /opt/job.jar /opt/flink/lib
+ addgroup -S flink
+ adduser -D -S -H -G flink -h /opt/flink flink
+ chown -R flink:flink /opt/flink-1.7-SNAPSHOT
+ chown -h flink:flink /opt/flink
 ---> 98c9da70c69c
Removing intermediate container 53d9785301eb
Step 12/16 : COPY docker-entrypoint.sh /
 ---> 8491578301f5
Removing intermediate container 6798845d05c7
Step 13/16 : USER flink
 ---> Running in c0af68cc8c68
 ---> f6073e2f0aee
Removing intermediate container c0af68cc8c68
Step 14/16 : EXPOSE 8081 6123
 ---> Running in 9048cc411ae5
 ---> 765a20289069
Removing intermediate container 9048cc411ae5
Step 15/16 : ENTRYPOINT /docker-entrypoint.sh
 ---> Running in bfa44c4924c3
 ---> e542e35e7612
Removing intermediate container bfa44c4924c3
Step 16/16 : CMD --help
 ---> Running in d2ac82b536b2
 ---> effc5b265663
Removing intermediate container d2ac82b536b2
Successfully built effc5b265663
The connection to the server 172.31.47.225:8443 was refused - did you specify the right host or port?
The connection to the server 172.31.47.225:8443 was refused - did you specify the right host or port?
The connection to the server 172.31.47.225:8443 was refused - did you specify the right host or port?
The connection to the server 172.31.47.225:8443 was refused - did you specify the right host or port?
The connection to the server 172.31.47.225:8443 was refused - did you specify the right host or port?
error: Filespec must match the canonical format: [[namespace/]pod:]file/path
sort: cannot read: '/home/admin/flink/flink-end-to-end-tests/test-scripts/temp-test-directory-40594844780/out/kubernetes_wc_out*': No such file or directory
FAIL WordCount: Output hash mismatch.  Got d41d8cd98f00b204e9800998ecf8427e, expected e682ec6622b5e83f2eb614617d5ab2cf.
head hexdump of actual:
head: cannot open '/home/admin/flink/flink-end-to-end-tests/test-scripts/temp-test-directory-40594844780/out/kubernetes_wc_out*' for reading: No such file or directory
The connection to the server 172.31.47.225:8443 was refused - did you specify the right host or port?
The connection to the server 172.31.47.225:8443 was refused - did you specify the right host or port?
The connection to the server 172.31.47.225:8443 was refused - did you specify the right host or port?
[FAIL] Test script contains errors.
Checking for errors...
No errors in log files.
Checking for exceptions...
No exceptions in log files.
Checking for non-empty .out files...
grep: /home/admin/flink-1.7.0/log/*.out: No such file or directory
No non-empty .out files.

[FAIL] 'Run Kubernetes test' failed after 1 minutes and 6 seconds! Test exited with exit code 1

      We should try to harden this test.

      Attachments

        Issue Links

          links to

          Web Link GitHub Pull Request #7507

          Activity

            People

              dwysakowicz Dawid Wysakowicz
              trohrmann Till Rohrmann
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 20m
                  20m