Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-10910

Harden Kubernetes e2e test

    XMLWordPrintableJSON

    Details

      Description

      The Kubernetes test (e2e test) sometimes fails with the following output:

      ==============================================================================
      Running 'Run Kubernetes test'
      ==============================================================================
      TEST_DATA_DIR: /home/admin/flink/flink-end-to-end-tests/test-scripts/temp-test-directory-40594844780
      Flink dist directory: /home/admin/flink-1.7.0
      /home/admin/flink/flink-end-to-end-tests/test-scripts/test_kubernetes_embedded_job.sh: line 44: none: command not f
      ound
      Using flink dist: ../../flink-dist/target/flink-*-bin
      ./
      ./flink-1.7-SNAPSHOT/
      ./flink-1.7-SNAPSHOT/LICENSE
      ./flink-1.7-SNAPSHOT/examples/
      ./flink-1.7-SNAPSHOT/examples/gelly/
      ./flink-1.7-SNAPSHOT/examples/gelly/flink-gelly-examples_2.11-1.7-SNAPSHOT.jar
      ./flink-1.7-SNAPSHOT/examples/streaming/
      ./flink-1.7-SNAPSHOT/examples/streaming/WindowJoin.jar
      ./flink-1.7-SNAPSHOT/examples/streaming/SocketWindowWordCount.jar
      ./flink-1.7-SNAPSHOT/examples/streaming/StateMachineExample.jar
      ./flink-1.7-SNAPSHOT/examples/streaming/Kafka010Example.jar
      ./flink-1.7-SNAPSHOT/examples/streaming/Kafka011Example.jar
      ./flink-1.7-SNAPSHOT/examples/streaming/SessionWindowing.jar
      ./flink-1.7-SNAPSHOT/examples/streaming/IncrementalLearning.jar
      ./flink-1.7-SNAPSHOT/examples/streaming/KafkaExample.jar
      ./flink-1.7-SNAPSHOT/examples/streaming/WordCount.jar
      ./flink-1.7-SNAPSHOT/examples/streaming/Twitter.jar
      ./flink-1.7-SNAPSHOT/examples/streaming/Iteration.jar
      ./flink-1.7-SNAPSHOT/examples/streaming/TopSpeedWindowing.jar
      ./flink-1.7-SNAPSHOT/examples/batch/
      ./flink-1.7-SNAPSHOT/examples/batch/KMeans.jar
      ./flink-1.7-SNAPSHOT/examples/batch/PageRank.jar
      ./flink-1.7-SNAPSHOT/examples/batch/WebLogAnalysis.jar
      ./flink-1.7-SNAPSHOT/examples/batch/WordCount.jar
      ./flink-1.7-SNAPSHOT/examples/batch/EnumTriangles.jar
      ./flink-1.7-SNAPSHOT/examples/batch/DistCp.jar
      ./flink-1.7-SNAPSHOT/examples/batch/TransitiveClosure.jar
      ./flink-1.7-SNAPSHOT/examples/batch/ConnectedComponents.jar
      ./flink-1.7-SNAPSHOT/examples/python/
      ./flink-1.7-SNAPSHOT/examples/python/streaming/
      ./flink-1.7-SNAPSHOT/examples/python/streaming/fibonacci.py
      ./flink-1.7-SNAPSHOT/examples/python/streaming/word_count.py
      ./flink-1.7-SNAPSHOT/examples/python/batch/
      ./flink-1.7-SNAPSHOT/examples/python/batch/TriangleEnumeration.py
      ./flink-1.7-SNAPSHOT/examples/python/batch/TPCHQuery3.py
      ./flink-1.7-SNAPSHOT/examples/python/batch/WebLogAnalysis.py
      ./flink-1.7-SNAPSHOT/examples/python/batch/TPCHQuery10.py
      ./flink-1.7-SNAPSHOT/examples/python/batch/WordCount.py
      ./flink-1.7-SNAPSHOT/examples/python/batch/__init__.py
      ./flink-1.7-SNAPSHOT/log/
      ./flink-1.7-SNAPSHOT/opt/
      ./flink-1.7-SNAPSHOT/opt/flink-metrics-ganglia-1.7-SNAPSHOT.jar
      ./flink-1.7-SNAPSHOT/opt/flink-table_2.11-1.7-SNAPSHOT.jar
      ./flink-1.7-SNAPSHOT/opt/flink-metrics-dropwizard-1.7-SNAPSHOT.jar
      ./flink-1.7-SNAPSHOT/opt/flink-metrics-prometheus-1.7-SNAPSHOT.jar
      ./flink-1.7-SNAPSHOT/opt/flink-cep_2.11-1.7-SNAPSHOT.jar
      ./flink-1.7-SNAPSHOT/opt/flink-metrics-graphite-1.7-SNAPSHOT.jar
      ./flink-1.7-SNAPSHOT/opt/flink-gelly-scala_2.11-1.7-SNAPSHOT.jar
      ./flink-1.7-SNAPSHOT/opt/flink-s3-fs-hadoop-1.7-SNAPSHOT.jar
      ./flink-1.7-SNAPSHOT/opt/flink-metrics-statsd-1.7-SNAPSHOT.jar
      ./flink-1.7-SNAPSHOT/opt/flink-queryable-state-runtime_2.11-1.7-SNAPSHOT.jar
      ./flink-1.7-SNAPSHOT/opt/flink-streaming-python_2.11-1.7-SNAPSHOT.jar
      ./flink-1.7-SNAPSHOT/opt/flink-ml_2.11-1.7-SNAPSHOT.jar
      ./flink-1.7-SNAPSHOT/opt/flink-gelly_2.11-1.7-SNAPSHOT.jar
      ./flink-1.7-SNAPSHOT/opt/flink-swift-fs-hadoop-1.7-SNAPSHOT.jar
      ./flink-1.7-SNAPSHOT/opt/flink-cep-scala_2.11-1.7-SNAPSHOT.jar
      ./flink-1.7-SNAPSHOT/opt/flink-metrics-slf4j-1.7-SNAPSHOT.jar
      ./flink-1.7-SNAPSHOT/opt/flink-s3-fs-presto-1.7-SNAPSHOT.jar
      ./flink-1.7-SNAPSHOT/opt/flink-metrics-datadog-1.7-SNAPSHOT.jar
      ./flink-1.7-SNAPSHOT/opt/flink-sql-client_2.11-1.7-SNAPSHOT.jar
      ./flink-1.7-SNAPSHOT/NOTICE
      ./flink-1.7-SNAPSHOT/README.txt
      ./flink-1.7-SNAPSHOT/lib/
      ./flink-1.7-SNAPSHOT/lib/flink-python_2.11-1.7-SNAPSHOT.jar
      ./flink-1.7-SNAPSHOT/lib/log4j-1.2.17.jar
      ./flink-1.7-SNAPSHOT/lib/flink-shaded-hadoop2-uber-1.7-SNAPSHOT.jar
      ./flink-1.7-SNAPSHOT/lib/flink-dist_2.11-1.7-SNAPSHOT.jar
      ./flink-1.7-SNAPSHOT/lib/slf4j-log4j12-1.7.15.jar
      ./flink-1.7-SNAPSHOT/conf/
      ./flink-1.7-SNAPSHOT/conf/log4j-cli.properties
      ./flink-1.7-SNAPSHOT/conf/logback-console.xml
      ./flink-1.7-SNAPSHOT/conf/logback.xml
      ./flink-1.7-SNAPSHOT/conf/logback-yarn.xml
      ./flink-1.7-SNAPSHOT/conf/flink-conf.yaml
      ./flink-1.7-SNAPSHOT/conf/sql-client-defaults.yaml
      ./flink-1.7-SNAPSHOT/conf/log4j-yarn-session.properties
      ./flink-1.7-SNAPSHOT/conf/masters
      ./flink-1.7-SNAPSHOT/conf/log4j.properties
      ./flink-1.7-SNAPSHOT/conf/zoo.cfg
      ./flink-1.7-SNAPSHOT/conf/log4j-console.properties
      ./flink-1.7-SNAPSHOT/conf/slaves
      ./flink-1.7-SNAPSHOT/bin/
      ./flink-1.7-SNAPSHOT/bin/mesos-appmaster-job.sh
      ./flink-1.7-SNAPSHOT/bin/flink
      ./flink-1.7-SNAPSHOT/bin/flink.bat
      ./flink-1.7-SNAPSHOT/bin/taskmanager.sh
      ./flink-1.7-SNAPSHOT/bin/mesos-taskmanager.sh
      ./flink-1.7-SNAPSHOT/bin/stop-zookeeper-quorum.sh
      ./flink-1.7-SNAPSHOT/bin/flink-console.sh
      ./flink-1.7-SNAPSHOT/bin/zookeeper.sh
      ./flink-1.7-SNAPSHOT/bin/start-cluster.sh
      ./flink-1.7-SNAPSHOT/bin/standalone-job.sh
      ./flink-1.7-SNAPSHOT/bin/pyflink-stream.sh
      ./flink-1.7-SNAPSHOT/bin/jobmanager.sh
      ./flink-1.7-SNAPSHOT/bin/historyserver.sh
      ./flink-1.7-SNAPSHOT/bin/config.sh
      ./flink-1.7-SNAPSHOT/bin/start-zookeeper-quorum.sh
      ./flink-1.7-SNAPSHOT/bin/start-scala-shell.sh
      ./flink-1.7-SNAPSHOT/bin/yarn-session.sh
      ./flink-1.7-SNAPSHOT/bin/pyflink.bat
      ./flink-1.7-SNAPSHOT/bin/flink-daemon.sh
      ./flink-1.7-SNAPSHOT/bin/pyflink.sh
      ./flink-1.7-SNAPSHOT/bin/sql-client.sh
      ./flink-1.7-SNAPSHOT/bin/mesos-appmaster.sh
      ./flink-1.7-SNAPSHOT/bin/stop-cluster.sh
      ./flink-1.7-SNAPSHOT/bin/start-cluster.bat
      Sending build context to Docker daemon   472 MB
      Step 1/16 : FROM java:8-jre-alpine
       ---> fdc893b19a14
      Step 2/16 : RUN apk add --no-cache bash snappy
       ---> Running in 9b6a826cdbd6
      fetch http://dl-cdn.alpinelinux.org/alpine/v3.4/main/x86_64/APKINDEX.tar.gz
      fetch http://dl-cdn.alpinelinux.org/alpine/v3.4/community/x86_64/APKINDEX.tar.gz
      (1/7) Installing ncurses-terminfo-base (6.0_p20171125-r0)
      (2/7) Installing ncurses-terminfo (6.0_p20171125-r0)
      (3/7) Installing ncurses-libs (6.0_p20171125-r0)
      (4/7) Installing readline (6.3.008-r4)
      (5/7) Installing bash (4.3.42-r5)
      Executing bash-4.3.42-r5.post-install
      (6/7) Installing libstdc++ (5.3.0-r0)
      (7/7) Installing snappy (1.1.3-r0)
      Executing busybox-1.24.2-r13.trigger
      OK: 115 MiB in 41 packages
       ---> 194f5306312e
      Removing intermediate container 9b6a826cdbd6
      Step 3/16 : ENV FLINK_INSTALL_PATH /opt
       ---> Running in f44ed28c3ecd
       ---> 61495cda8d7e
      Removing intermediate container f44ed28c3ecd
      Step 4/16 : ENV FLINK_HOME $FLINK_INSTALL_PATH/flink
       ---> Running in 106db4c7c3e3
       ---> 04fcb3422dfa
      Removing intermediate container 106db4c7c3e3
      Step 5/16 : ENV FLINK_LIB_DIR $FLINK_HOME/lib
       ---> Running in 51030ee2a57e
       ---> 7c290e53f0f1
      Removing intermediate container 51030ee2a57e
      Step 6/16 : ENV PATH $PATH:$FLINK_HOME/bin
       ---> Running in 1f89c045de97
       ---> 81cec8fe782c
      Removing intermediate container 1f89c045de97
      Step 7/16 : ARG flink_dist=NOT_SET
       ---> Running in bfd5fac37dea
       ---> 1dfd4009b044
      Removing intermediate container bfd5fac37dea
      Step 8/16 : ARG job_jar=NOT_SET
       ---> Running in 712432ce7471
       ---> 17cba67f1748
      Removing intermediate container 712432ce7471
      Step 9/16 : ADD $flink_dist $FLINK_INSTALL_PATH
       ---> be5be560416b
      Removing intermediate container b6b1c4ae0855
      Step 10/16 : ADD $job_jar $FLINK_INSTALL_PATH/job.jar
       ---> 1299878dc57c
      Removing intermediate container 0fea418feed1
      Step 11/16 : RUN set -x &&   ln -s $FLINK_INSTALL_PATH/flink-* $FLINK_HOME &&   ln -s $FLINK_INSTALL_PATH/job.jar $
      FLINK_LIB_DIR &&   addgroup -S flink && adduser -D -S -H -G flink -h $FLINK_HOME flink &&   chown -R flink:flink $F
      LINK_INSTALL_PATH/flink-* &&   chown -h flink:flink $FLINK_HOME
       ---> Running in 53d9785301eb
      + ln -s /opt/flink-1.7-SNAPSHOT /opt/flink
      + ln -s /opt/job.jar /opt/flink/lib
      + addgroup -S flink
      + adduser -D -S -H -G flink -h /opt/flink flink
      + chown -R flink:flink /opt/flink-1.7-SNAPSHOT
      + chown -h flink:flink /opt/flink
       ---> 98c9da70c69c
      Removing intermediate container 53d9785301eb
      Step 12/16 : COPY docker-entrypoint.sh /
       ---> 8491578301f5
      Removing intermediate container 6798845d05c7
      Step 13/16 : USER flink
       ---> Running in c0af68cc8c68
       ---> f6073e2f0aee
      Removing intermediate container c0af68cc8c68
      Step 14/16 : EXPOSE 8081 6123
       ---> Running in 9048cc411ae5
       ---> 765a20289069
      Removing intermediate container 9048cc411ae5
      Step 15/16 : ENTRYPOINT /docker-entrypoint.sh
       ---> Running in bfa44c4924c3
       ---> e542e35e7612
      Removing intermediate container bfa44c4924c3
      Step 16/16 : CMD --help
       ---> Running in d2ac82b536b2
       ---> effc5b265663
      Removing intermediate container d2ac82b536b2
      Successfully built effc5b265663
      The connection to the server 172.31.47.225:8443 was refused - did you specify the right host or port?
      The connection to the server 172.31.47.225:8443 was refused - did you specify the right host or port?
      The connection to the server 172.31.47.225:8443 was refused - did you specify the right host or port?
      The connection to the server 172.31.47.225:8443 was refused - did you specify the right host or port?
      The connection to the server 172.31.47.225:8443 was refused - did you specify the right host or port?
      error: Filespec must match the canonical format: [[namespace/]pod:]file/path
      sort: cannot read: '/home/admin/flink/flink-end-to-end-tests/test-scripts/temp-test-directory-40594844780/out/kubernetes_wc_out*': No such file or directory
      FAIL WordCount: Output hash mismatch.  Got d41d8cd98f00b204e9800998ecf8427e, expected e682ec6622b5e83f2eb614617d5ab2cf.
      head hexdump of actual:
      head: cannot open '/home/admin/flink/flink-end-to-end-tests/test-scripts/temp-test-directory-40594844780/out/kubernetes_wc_out*' for reading: No such file or directory
      The connection to the server 172.31.47.225:8443 was refused - did you specify the right host or port?
      The connection to the server 172.31.47.225:8443 was refused - did you specify the right host or port?
      The connection to the server 172.31.47.225:8443 was refused - did you specify the right host or port?
      [FAIL] Test script contains errors.
      Checking for errors...
      No errors in log files.
      Checking for exceptions...
      No exceptions in log files.
      Checking for non-empty .out files...
      grep: /home/admin/flink-1.7.0/log/*.out: No such file or directory
      No non-empty .out files.
      
      [FAIL] 'Run Kubernetes test' failed after 1 minutes and 6 seconds! Test exited with exit code 1
      

      We should try to harden this test.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                dwysakowicz Dawid Wysakowicz
                Reporter:
                trohrmann Till Rohrmann
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 20m
                  20m