Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-6290

SharedBuffer is improperly released when multiple edges between entries

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 1.3.0
    • Fix Version/s: 1.3.0
    • Component/s: CEP
    • Labels:
      None

      Description

      Below test right now fails:

      	@Test
      	public void testClearingSharedBufferWithMultipleEdgesBetweenEntries() {
      		SharedBuffer<String, Event> sharedBuffer = new SharedBuffer<>(Event.createTypeSerializer());
      		int numberEvents = 8;
      		Event[] events = new Event[numberEvents];
      		final long timestamp = 1L;
      
      		for (int i = 0; i < numberEvents; i++) {
      			events[i] = new Event(i + 1, "e" + (i + 1), i);
      		}
      
      		sharedBuffer.put("start", events[1], timestamp, DeweyNumber.fromString("1"));
      		sharedBuffer.put("branching", events[2], timestamp, "start", events[1], timestamp, DeweyNumber.fromString("1.0"));
      		sharedBuffer.put("branching", events[3], timestamp, "start", events[1], timestamp, DeweyNumber.fromString("1.1"));
      		sharedBuffer.put("branching", events[3], timestamp, "branching", events[2], timestamp, DeweyNumber.fromString("1.0.0"));
      		sharedBuffer.put("branching", events[4], timestamp, "branching", events[3], timestamp, DeweyNumber.fromString("1.0.0.0"));
      		sharedBuffer.put("branching", events[4], timestamp, "branching", events[3], timestamp, DeweyNumber.fromString("1.1.0"));
      
      		//simulate IGNORE (next event can point to events[2])
      		sharedBuffer.lock("branching", events[2], timestamp);
      
      		sharedBuffer.release("branching", events[4], timestamp);
      
      		//There should be still events[1] and events[2] in the buffer
      		assertFalse(sharedBuffer.isEmpty());
      	}
      

      The problem is with the SharedBuffer#internalRemove method:

      private void internalRemove(final SharedBufferEntry<K, V> entry) {
      		Stack<SharedBufferEntry<K, V>> entriesToRemove = new Stack<>();
      		entriesToRemove.add(entry);
      
      		while (!entriesToRemove.isEmpty()) {
      			SharedBufferEntry<K, V> currentEntry = entriesToRemove.pop();
      
      			if (currentEntry.getReferenceCounter() == 0) {
      				currentEntry.remove();
      
      				for (SharedBufferEdge<K, V> edge: currentEntry.getEdges()) {
      					if (edge.getTarget() != null) {
      						edge.getTarget().decreaseReferenceCounter();
      						entriesToRemove.push(edge.getTarget());
      					}
      				}
      			}
      		}
      	}
      

      When currentEntry has multiple edges to the same entry. The entry will be added twice to the entriesToRemove and it's edges will be removed twice and the second edge can potentially change the referenceCounter for both of those entries to 0. Resulting in removing this entry twice.

        Issue Links

          Activity

          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user dawidwys opened a pull request:

          https://github.com/apache/flink/pull/3706

          FLINK-6290 SharedBuffer is improperly released when multiple edges …

          …between entries

          Thanks for contributing to Apache Flink. Before you open your pull request, please take the following check list into consideration.
          If your changes take all of the items into account, feel free to open your pull request. For more information and/or questions please refer to the [How To Contribute guide](http://flink.apache.org/how-to-contribute.html).
          In addition to going through the list, please provide a meaningful description of your changes.

          • [ ] General
          • The pull request references the related JIRA issue ("[FLINK-XXX] Jira title text")
          • The pull request addresses only one issue
          • Each commit in the PR has a meaningful commit message (including the JIRA id)
          • [ ] Documentation
          • Documentation has been added for new functionality
          • Old documentation affected by the pull request has been updated
          • JavaDoc for public methods has been added
          • [ ] Tests & Build
          • Functionality added by the pull request is covered by tests
          • `mvn clean verify` has been executed successfully locally or a Travis build has passed

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/dawidwys/flink cep-sharedbuffer-bug

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/flink/pull/3706.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #3706



          Show
          githubbot ASF GitHub Bot added a comment - GitHub user dawidwys opened a pull request: https://github.com/apache/flink/pull/3706 FLINK-6290 SharedBuffer is improperly released when multiple edges … …between entries Thanks for contributing to Apache Flink. Before you open your pull request, please take the following check list into consideration. If your changes take all of the items into account, feel free to open your pull request. For more information and/or questions please refer to the [How To Contribute guide] ( http://flink.apache.org/how-to-contribute.html ). In addition to going through the list, please provide a meaningful description of your changes. [ ] General The pull request references the related JIRA issue (" [FLINK-XXX] Jira title text") The pull request addresses only one issue Each commit in the PR has a meaningful commit message (including the JIRA id) [ ] Documentation Documentation has been added for new functionality Old documentation affected by the pull request has been updated JavaDoc for public methods has been added [ ] Tests & Build Functionality added by the pull request is covered by tests `mvn clean verify` has been executed successfully locally or a Travis build has passed You can merge this pull request into a Git repository by running: $ git pull https://github.com/dawidwys/flink cep-sharedbuffer-bug Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/3706.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3706
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user StephanEwen commented on the issue:

          https://github.com/apache/flink/pull/3706

          Thanks for the patch @dawidwys - good fix, nice test.

          Merging this...

          Show
          githubbot ASF GitHub Bot added a comment - Github user StephanEwen commented on the issue: https://github.com/apache/flink/pull/3706 Thanks for the patch @dawidwys - good fix, nice test. Merging this...
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/flink/pull/3706

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/3706
          Hide
          StephanEwen Stephan Ewen added a comment -

          Fixed via c0ea74a0a4eedf4fe73e8f383e837ea373216476

          Show
          StephanEwen Stephan Ewen added a comment - Fixed via c0ea74a0a4eedf4fe73e8f383e837ea373216476

            People

            • Assignee:
              dawidwys Dawid Wysakowicz
              Reporter:
              dawidwys Dawid Wysakowicz
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development