Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-5556

BarrierBuffer resets bytes written on spiller roll over

    Details

      Description

      When rolling over a spilled sequence of buffers, the tracker bytes written are reset to 0. They are reported to the checkpoint listener right after this operation, which results in the reported buffered bytes always being 0.

        Issue Links

          Activity

          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user uce opened a pull request:

          https://github.com/apache/flink/pull/3164

          FLINK-5556 [checkpointing] Report correct buffered bytes during alignment

          During stream alignment via the `BarrierTracker`, we were reporting the wrong number of spilled bytes, because the `bytesWritten` in `BufferSpiller` were reset before the checkpoint notification happens.

          Instead of `bufferSpiller.getBytesWritten()` I use `SpilledBufferOrEventSequence.size()` if `currentBuffered != null`.

          I adjusted one `BarrierBufferTest` to check the expected number of buffered bytes. Before the check was only for `>= 0`.

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/uce/flink 5556-reset_bytes

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/flink/pull/3164.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #3164


          commit 591fa4c600ef70432e8162584ab89dd5c0508da6
          Author: Ufuk Celebi <uce@apache.org>
          Date: 2017-01-18T17:14:16Z

          FLINK-5556 [checkpointing] Report correct buffered bytes during alignment


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user uce opened a pull request: https://github.com/apache/flink/pull/3164 FLINK-5556 [checkpointing] Report correct buffered bytes during alignment During stream alignment via the `BarrierTracker`, we were reporting the wrong number of spilled bytes, because the `bytesWritten` in `BufferSpiller` were reset before the checkpoint notification happens. Instead of `bufferSpiller.getBytesWritten()` I use `SpilledBufferOrEventSequence.size()` if `currentBuffered != null`. I adjusted one `BarrierBufferTest` to check the expected number of buffered bytes. Before the check was only for `>= 0`. You can merge this pull request into a Git repository by running: $ git pull https://github.com/uce/flink 5556-reset_bytes Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/3164.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3164 commit 591fa4c600ef70432e8162584ab89dd5c0508da6 Author: Ufuk Celebi <uce@apache.org> Date: 2017-01-18T17:14:16Z FLINK-5556 [checkpointing] Report correct buffered bytes during alignment
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user StephanEwen commented on the issue:

          https://github.com/apache/flink/pull/3164

          Very good, +1 to merge this!

          Show
          githubbot ASF GitHub Bot added a comment - Github user StephanEwen commented on the issue: https://github.com/apache/flink/pull/3164 Very good, +1 to merge this!
          Hide
          uce Ufuk Celebi added a comment -

          Fixed in a4853ba (release-1.2), e1b2cd0 (master).

          Show
          uce Ufuk Celebi added a comment - Fixed in a4853ba (release-1.2), e1b2cd0 (master).
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/flink/pull/3164

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/3164

            People

            • Assignee:
              uce Ufuk Celebi
              Reporter:
              uce Ufuk Celebi
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development