Details
-
Improvement
-
Status: Closed
-
Major
-
Resolution: Abandoned
-
1.8.0
-
None
-
None
Description
This issue is intended to be the umbrella issue for speeding up build and test times of our modules. At the moment building Flink takes more than 1 hour on my machine. Here are the modules sorted in descending order with respect to build+test times (second column in seconds):
flink-connector-kafka-0.11 766 flink-tests 425 flink-runtime 391 flink-yarn-tests 331 flink-table 207 flink-connector-kafka-0.9 189 flink-connector-cassandra 135 flink-connector-kafka-0.8 132 flink-connector-kafka-0.10 130 flink-gelly-examples 76 flink-hbase 60 flink-core 58.251 flink-storm-examples 56.796 flink-streaming-java 56.080 flink-ml 53.349 flink-connector-filesystem 47.698 flink-connector-elasticsearch5 47.143 flink-connector-elasticsearch2 45.406 flink-connector-elasticsearch 40.804 flink-streaming-scala 40.493 flink-storm 40.107 flink-scala-shell 38.566 flink-scala 34.964 flink-gelly 33.714 flink-fs-tests 30.466 flink-python 26.919d flink-java 24.899 flink-gelly-scala 23.406 flink-dist 21.778 flink-mesos 21.524 flink-hadoop-compatibility 20.106 flink-examples-streaming 19.951 flink-shaded-hadoop2 19.782 flink-s3-fs-presto 18.731 flink-s3-fs-hadoop 17.070 flink-optimizer 17.034 flink-hadoop-fs 15.857 flink-avro 15.409 flink-shaded-yarn-tests 15.147 flink-yarn 14.962 flink-swift-fs-hadoop 13.709 flink-streaming-python 13.163 flink-connector-wikiedits 12.515 flink-queryable-state-runtime 12.474 flink-statebackend-rocksdb 11.987 flink-shaded-hadoop2-uber 11.527 flink-connector-elasticsearch6 10.880 flink-runtime-web 10.384 flink-orc 9.916 flink-cep 8.963 flink-sql-client 8.944 flink-examples-batch 8.555 flink-cep-scala 8.270 flink-clients 5.966 flink-examples-table 5.769 flink-connector-kafka-base 5.754 flink-high-parallelism-iterations-test 4.707 flink-parquet 3.640 flink-hcatalog 3.436 flink-jdbc 3.296 flink 3.257 flink-connector-rabbitmq 2.779 flink-test-utils 2.686 flink-metrics-statsd 2.667 flink-elasticsearch6-test 2.251 flink-metrics-jmx 2.167 flink-elasticsearch5-test 2.144 flink-elasticsearch2-test 2.114 flink-metrics-dropwizard 1.898 flink-connector-elasticsearch-base 1.726 flink-metrics-datadog 1.676 flink-elasticsearch1-test 1.652 flink-mapr-fs 1.551 flink-json 1.526 flink-docs 1.441 flink-metrics-core 1.373 flink-avro-confluent-registry 1.345 flink-metrics-prometheus 1.340 flink-connector-twitter 1.276 flink-annotations 1.183 flink-queryable-state-test 1.121 force-shading 1.114 flink-queryable-state-client-java 1.085 flink-test-utils-junit 1.058 flink-sql-client-test 1.049 flink-confluent-schema-registry 1.001 flink-quickstart 0.939 flink-datastream-allround-test 0.924 flink-container 0.740 flink-stream-state-ttl-test 0.679 flink-metrics-slf4j 0.663 flink-stream-stateful-job-upgrade-test 0.623 flink-shaded-curator 0.544 flink-connector-nifi 0.487 flink-quickstart-java 0.451 flink-bucketing-sink-test 0.369 flink-local-recovery-and-allocation-test 0.220 flink-stream-sql-test 0.191 flink-quickstart-test 0.191 flink-streaming-file-sink-test 0.182 flink-dataset-allround-test 0.175 flink-distributed-cache-via-blob 0.163 flink-metrics-graphite 0.160 flink-examples 0.155 flink-parent-child-classloading-test 0.154 flink-quickstart-scala 0.135 flink-shaded-hadoop 0.120 flink-filesystems 0.081 flink-metrics 0.076 flink-test-utils-parent 0.074 flink-queryable-state 0.067 flink-formats 0.061 flink-state-backends 0.058 flink-end-to-end-tests 0.058 flink-libraries 0.056 flink-connectors 0.054 flink-contrib 0.052
As one can see flink-connector-kafka-0.11 alone takes more than 12 minutes to build and run all tests. I think it would be good to revisit some of these tests to see whether they can be sped up.