Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-6207

Duplicate type serializers for async snapshots of CopyOnWriteStateTable

    Details

      Description

      TypeSerializer are used for copy-on-write and the parallel snapshots in the CopyOnWriteStateTable. For stateful serializers, this can lead to race conditions. Snapshots need to duplicate the serializers before using them.

        Issue Links

          Activity

          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user StefanRRichter opened a pull request:

          https://github.com/apache/flink/pull/3634

          FLINK-6207 Duplicate TypeSerializers for async snapshots of CopyOnW…

          This PR fixes a problem for stateful serializers for async snapshots in the `CopyOnWriteStateTable`. Serializers must be duplicated before they are used in the async snapshot thread.

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/StefanRRichter/flink fix-serialier-duplicate

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/flink/pull/3634.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #3634


          commit 63f661067aa90a4b53161fb1bf4e84fcf54b9c10
          Author: Stefan Richter <s.richter@data-artisans.com>
          Date: 2017-03-28T14:56:27Z

          FLINK-6207 Duplicate TypeSerializers for async snapshots of CopyOnWriteStateTable


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user StefanRRichter opened a pull request: https://github.com/apache/flink/pull/3634 FLINK-6207 Duplicate TypeSerializers for async snapshots of CopyOnW… This PR fixes a problem for stateful serializers for async snapshots in the `CopyOnWriteStateTable`. Serializers must be duplicated before they are used in the async snapshot thread. You can merge this pull request into a Git repository by running: $ git pull https://github.com/StefanRRichter/flink fix-serialier-duplicate Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/3634.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3634 commit 63f661067aa90a4b53161fb1bf4e84fcf54b9c10 Author: Stefan Richter <s.richter@data-artisans.com> Date: 2017-03-28T14:56:27Z FLINK-6207 Duplicate TypeSerializers for async snapshots of CopyOnWriteStateTable
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user StefanRRichter commented on the issue:

          https://github.com/apache/flink/pull/3634

          CC @StephanEwen

          Show
          githubbot ASF GitHub Bot added a comment - Github user StefanRRichter commented on the issue: https://github.com/apache/flink/pull/3634 CC @StephanEwen
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user tillrohrmann commented on a diff in the pull request:

          https://github.com/apache/flink/pull/3634#discussion_r108473910

          — Diff: flink-runtime/src/test/java/org/apache/flink/runtime/state/heap/CopyOnWriteStateTableTest.java —
          @@ -380,6 +385,77 @@ public void testCopyOnWriteContracts()

          { Assert.assertTrue(originalState5 == stateTable.get(5, 1)); }

          + /**
          — End diff –

          The `CopyOnWriteStateTableTest` could extend the `TestLogger`. That way we get better logging output on Travis.

          Show
          githubbot ASF GitHub Bot added a comment - Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/3634#discussion_r108473910 — Diff: flink-runtime/src/test/java/org/apache/flink/runtime/state/heap/CopyOnWriteStateTableTest.java — @@ -380,6 +385,77 @@ public void testCopyOnWriteContracts() { Assert.assertTrue(originalState5 == stateTable.get(5, 1)); } + /** — End diff – The `CopyOnWriteStateTableTest` could extend the `TestLogger`. That way we get better logging output on Travis.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user StefanRRichter commented on the issue:

          https://github.com/apache/flink/pull/3634

          Thanks for the fast review @tillrohrmann ! I will let the test extend the `TestLogger` as suggested and then merge this.

          Show
          githubbot ASF GitHub Bot added a comment - Github user StefanRRichter commented on the issue: https://github.com/apache/flink/pull/3634 Thanks for the fast review @tillrohrmann ! I will let the test extend the `TestLogger` as suggested and then merge this.
          Hide
          srichter Stefan Richter added a comment -

          fixed in 89866a5
          backported to 1.2 in bb3e26f

          Show
          srichter Stefan Richter added a comment - fixed in 89866a5 backported to 1.2 in bb3e26f
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user StefanRRichter closed the pull request at:

          https://github.com/apache/flink/pull/3634

          Show
          githubbot ASF GitHub Bot added a comment - Github user StefanRRichter closed the pull request at: https://github.com/apache/flink/pull/3634

            People

            • Assignee:
              srichter Stefan Richter
              Reporter:
              srichter Stefan Richter
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development