Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-6867

Elasticsearch 1.x ITCase still instable due to embedded node instability

    Details

      Description

      The integration tests for Elasticsearch 1.x seems to still be instable, being that the test is failing more frequently recently. One example is [1].

      The last attempt to fix this was FLINK-5772, in which the test attempts was increased to 3. This doesn't seem to fix the issue. In the worst scenario, since the root cause is an instability with Elasticsearch 1.x's embedded node, and ES 1.x is a very old version that is usually recommended to be upgraded from, we could also consider removing the IT test for ES 1.x.

      [1] https://travis-ci.org/apache/flink/jobs/240444523

        Issue Links

          Activity

          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user tzulitai opened a pull request:

          https://github.com/apache/flink/pull/4191

          FLINK-6867 [elasticsearch] Fix instable Elasticsearch 1.x ITCases

          The root cause for the instability in ES 1.x is that we're testing both embedded client mode and transport client mode against the same Elasticsearch environment. This PR ignores the embedded mode testing, as that method is generally discouraged anyways.

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/tzulitai/flink FLINK-6867

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/flink/pull/4191.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #4191


          commit ee09a7b99adbbadec584c5df1297139303f0c604
          Author: Tzu-Li (Gordon) Tai <tzulitai@apache.org>
          Date: 2017-06-27T08:40:51Z

          FLINK-6867 [elasticsearch] Fix instable Elasticsearch 1.x ITCases

          The root cause for the instability in ES 1.x is that we're testing both
          embedded client mode and transport client mode against the same
          Elasticsearch environment. This commit ignores the embedded mode
          testing, as that method is generally discouraged anyways.


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user tzulitai opened a pull request: https://github.com/apache/flink/pull/4191 FLINK-6867 [elasticsearch] Fix instable Elasticsearch 1.x ITCases The root cause for the instability in ES 1.x is that we're testing both embedded client mode and transport client mode against the same Elasticsearch environment. This PR ignores the embedded mode testing, as that method is generally discouraged anyways. You can merge this pull request into a Git repository by running: $ git pull https://github.com/tzulitai/flink FLINK-6867 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/4191.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4191 commit ee09a7b99adbbadec584c5df1297139303f0c604 Author: Tzu-Li (Gordon) Tai <tzulitai@apache.org> Date: 2017-06-27T08:40:51Z FLINK-6867 [elasticsearch] Fix instable Elasticsearch 1.x ITCases The root cause for the instability in ES 1.x is that we're testing both embedded client mode and transport client mode against the same Elasticsearch environment. This commit ignores the embedded mode testing, as that method is generally discouraged anyways.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user zentol commented on a diff in the pull request:

          https://github.com/apache/flink/pull/4191#discussion_r124260636

          — Diff: flink-connectors/flink-connector-elasticsearch/src/test/java/org/apache/flink/streaming/connectors/elasticsearch/ElasticsearchSinkITCase.java —
          @@ -67,7 +68,11 @@ public void testTransportClientFails() throws Exception{

          /**

          • Tests that the Elasticsearch sink works properly using an embedded node to connect to Elasticsearch.
            + *
            + * NOTE: This is ignored, since executing both embedded client mode and transport client mode introduces
            + * instability for node discovery in ES 1.x. See FLINK-6867.
            */
            + @Ignore
              • End diff –

          Do we expect to ever enable this test again? if not, let's just remove it.

          Show
          githubbot ASF GitHub Bot added a comment - Github user zentol commented on a diff in the pull request: https://github.com/apache/flink/pull/4191#discussion_r124260636 — Diff: flink-connectors/flink-connector-elasticsearch/src/test/java/org/apache/flink/streaming/connectors/elasticsearch/ElasticsearchSinkITCase.java — @@ -67,7 +68,11 @@ public void testTransportClientFails() throws Exception{ /** Tests that the Elasticsearch sink works properly using an embedded node to connect to Elasticsearch. + * + * NOTE: This is ignored, since executing both embedded client mode and transport client mode introduces + * instability for node discovery in ES 1.x. See FLINK-6867 . */ + @Ignore End diff – Do we expect to ever enable this test again? if not, let's just remove it.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user zentol commented on a diff in the pull request:

          https://github.com/apache/flink/pull/4191#discussion_r124260908

          — Diff: flink-connectors/flink-connector-elasticsearch/src/test/java/org/apache/flink/streaming/connectors/elasticsearch/ElasticsearchSinkITCase.java —
          @@ -67,7 +68,11 @@ public void testTransportClientFails() throws Exception{

          /**

          • Tests that the Elasticsearch sink works properly using an embedded node to connect to Elasticsearch.
            + *
            + * NOTE: This is ignored, since executing both embedded client mode and transport client mode introduces
            + * instability for node discovery in ES 1.x. See FLINK-6867.
            */
            + @Ignore
              • End diff –

          that would also resolve the checkstyle violation

          Show
          githubbot ASF GitHub Bot added a comment - Github user zentol commented on a diff in the pull request: https://github.com/apache/flink/pull/4191#discussion_r124260908 — Diff: flink-connectors/flink-connector-elasticsearch/src/test/java/org/apache/flink/streaming/connectors/elasticsearch/ElasticsearchSinkITCase.java — @@ -67,7 +68,11 @@ public void testTransportClientFails() throws Exception{ /** Tests that the Elasticsearch sink works properly using an embedded node to connect to Elasticsearch. + * + * NOTE: This is ignored, since executing both embedded client mode and transport client mode introduces + * instability for node discovery in ES 1.x. See FLINK-6867 . */ + @Ignore End diff – that would also resolve the checkstyle violation
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user tzulitai commented on a diff in the pull request:

          https://github.com/apache/flink/pull/4191#discussion_r124284249

          — Diff: flink-connectors/flink-connector-elasticsearch/src/test/java/org/apache/flink/streaming/connectors/elasticsearch/ElasticsearchSinkITCase.java —
          @@ -67,7 +68,11 @@ public void testTransportClientFails() throws Exception{

          /**

          • Tests that the Elasticsearch sink works properly using an embedded node to connect to Elasticsearch.
            + *
            + * NOTE: This is ignored, since executing both embedded client mode and transport client mode introduces
            + * instability for node discovery in ES 1.x. See FLINK-6867.
            */
            + @Ignore
              • End diff –

          I would not expect it to be enabled again, at least for the Travis runs.
          At the rate of the new Elasticsearch versions (there's already a 6.x now), we might also consider dropping 1.x support soon.

          Makes sense, will remove it completely

          Show
          githubbot ASF GitHub Bot added a comment - Github user tzulitai commented on a diff in the pull request: https://github.com/apache/flink/pull/4191#discussion_r124284249 — Diff: flink-connectors/flink-connector-elasticsearch/src/test/java/org/apache/flink/streaming/connectors/elasticsearch/ElasticsearchSinkITCase.java — @@ -67,7 +68,11 @@ public void testTransportClientFails() throws Exception{ /** Tests that the Elasticsearch sink works properly using an embedded node to connect to Elasticsearch. + * + * NOTE: This is ignored, since executing both embedded client mode and transport client mode introduces + * instability for node discovery in ES 1.x. See FLINK-6867 . */ + @Ignore End diff – I would not expect it to be enabled again, at least for the Travis runs. At the rate of the new Elasticsearch versions (there's already a 6.x now), we might also consider dropping 1.x support soon. Makes sense, will remove it completely
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user tzulitai commented on the issue:

          https://github.com/apache/flink/pull/4191

          Merging once Travis gives green.

          Show
          githubbot ASF GitHub Bot added a comment - Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/4191 Merging once Travis gives green.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/flink/pull/4191

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/4191
          Hide
          tzulitai Tzu-Li (Gordon) Tai added a comment -

          Fixed for master via 2d275e0c26b6e9b13df89450e7ff6291cae5bede.
          Fixed for 1.3 via 0d02f26e76823731415936ccc211844a6057408a.

          Show
          tzulitai Tzu-Li (Gordon) Tai added a comment - Fixed for master via 2d275e0c26b6e9b13df89450e7ff6291cae5bede. Fixed for 1.3 via 0d02f26e76823731415936ccc211844a6057408a.

            People

            • Assignee:
              tzulitai Tzu-Li (Gordon) Tai
              Reporter:
              tzulitai Tzu-Li (Gordon) Tai
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development