Uploaded image for project: 'Metron'
  1. Metron
  2. METRON-569

Enrichment topology duplicates messages

    Details

    • Type: Bug
    • Status: Done
    • Priority: Major
    • Resolution: Done
    • Affects Version/s: None
    • Fix Version/s: 0.4.0
    • Labels:
      None
    • Flags:
      Important

      Description

      When running the 'enrichment' topology, I get duplicate message being indexed. For example, I put 100 messages into the 'enrichment' Kafka queue and I get 175 messages onto the 'indexing' Kafka queue. This happens when I am running the 'enrichment' topology with one or more enrichment bolt.

      This is an acking issue within the JoinBolt class. When a message does not "complete" the join (like when it is the first message in a pair of message to get joined) it does not get acked. This means that this message will get replayed through Storm, causing message duplication further down the road and tons of additional overhead. Adding the correct acking resolves this problem.

      I will add the PR for this shortly.

        Issue Links

          Activity

          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user cestella commented on the issue:

          https://github.com/apache/incubator-metron/pull/359

          Also, @DomenicPuzio would you mind renaming this PR to METRON-569: Change acking to prevent duplicate tuples in enrichment topology so that the PR comments get replicated properly?

          Show
          githubbot ASF GitHub Bot added a comment - Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/359 Also, @DomenicPuzio would you mind renaming this PR to METRON-569 : Change acking to prevent duplicate tuples in enrichment topology so that the PR comments get replicated properly?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user DomenicPuzio commented on the issue:

          https://github.com/apache/incubator-metron/pull/359

          @cestella, I made the change to the PR name; thanks for the tip there.

          I completely agree that we don't want to miss out on catching failures due to an enrichment source (like MySQL) or in the enrichment infrastructure. However, after we have put a tuple into the JoinBolt's cache, isn't it already past the point where these pitfalls could occur? If there is a failure in an enrichment, then Storm will time out while waiting for an ack that never takes place, so this message will be re-sent; but if the message is already in the cache, isn't its journey complete?

          My thought with placing the ack there was (1) at this stage of the topology, that tuple has been correctly processed, and (2) for simplicity's sake so that we wouldn't have to modify `streamMessageMap`. I do agree that acking everything after the join has taken place also makes a lot of sense, and I can work on that if you would like.

          We saw the duplicate data while running this in our development environment in EC2. Perhaps this is due to different ack timeout settings in the Storm topologies? We've repeated the duplication of data many times on our end.

          Show
          githubbot ASF GitHub Bot added a comment - Github user DomenicPuzio commented on the issue: https://github.com/apache/incubator-metron/pull/359 @cestella, I made the change to the PR name; thanks for the tip there. I completely agree that we don't want to miss out on catching failures due to an enrichment source (like MySQL) or in the enrichment infrastructure. However, after we have put a tuple into the JoinBolt's cache, isn't it already past the point where these pitfalls could occur? If there is a failure in an enrichment, then Storm will time out while waiting for an ack that never takes place, so this message will be re-sent; but if the message is already in the cache, isn't its journey complete? My thought with placing the ack there was (1) at this stage of the topology, that tuple has been correctly processed, and (2) for simplicity's sake so that we wouldn't have to modify `streamMessageMap`. I do agree that acking everything after the join has taken place also makes a lot of sense, and I can work on that if you would like. We saw the duplicate data while running this in our development environment in EC2. Perhaps this is due to different ack timeout settings in the Storm topologies? We've repeated the duplication of data many times on our end.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user cestella commented on the issue:

          https://github.com/apache/incubator-metron/pull/359

          Hey @DomenicPuzio thanks man

          The problem with placing the ack there is that in the situation where the enrichment adapter worker gets killed, we would like the data to replay from the anchor point (the SplitBolt) because Storm will have respawned the worker elsewhere and we'd like that enrichment to be captured. If you keep the ack where you have it, the message will never join if the enrichment worker gets killed, so the message will ultimately get dropped, which isn't ideal.

          Ultimately, I think we can all agree that we need to ack at LEAST the tuple on the `message` stream, but I think that we want to ack it AFTER the join has happened.

          Regarding seeing dup data, I believe you. I am just trying to wrap my head around how it happens, but, as they say, the proof of the pudding is in the eating.

          Show
          githubbot ASF GitHub Bot added a comment - Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/359 Hey @DomenicPuzio thanks man The problem with placing the ack there is that in the situation where the enrichment adapter worker gets killed, we would like the data to replay from the anchor point (the SplitBolt) because Storm will have respawned the worker elsewhere and we'd like that enrichment to be captured. If you keep the ack where you have it, the message will never join if the enrichment worker gets killed, so the message will ultimately get dropped, which isn't ideal. Ultimately, I think we can all agree that we need to ack at LEAST the tuple on the `message` stream, but I think that we want to ack it AFTER the join has happened. Regarding seeing dup data, I believe you. I am just trying to wrap my head around how it happens, but, as they say, the proof of the pudding is in the eating.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user cestella commented on the issue:

          https://github.com/apache/incubator-metron/pull/359

          It occurs to me that what may be happening is that an enrichment may be taking longer than the timeout that storm is using to wait on that ack. If that is the case, I could see duplicated data.

          Imagine the following situation, with the storm timeout `x` and an enrichment taking `x + 1`. The enrichment would finish and send the enriched data from the enrichment adapter to the join bolt but storm would've already triggered a replay. The enrichment completing would have triggered the join to happen and the joined message to be emitted and the replay would trigger another copy of the message.

          In this case, I'd suggest ensuring that either your enrichment is capped at maximum http://storm.apache.org/releases/current/javadocs/org/apache/storm/Config.html#TOPOLOGY_MESSAGE_TIMEOUT_SECS or adjusting the message timeout in storm to be higher than this setting in storm.

          Show
          githubbot ASF GitHub Bot added a comment - Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/359 It occurs to me that what may be happening is that an enrichment may be taking longer than the timeout that storm is using to wait on that ack. If that is the case, I could see duplicated data. Imagine the following situation, with the storm timeout `x` and an enrichment taking `x + 1`. The enrichment would finish and send the enriched data from the enrichment adapter to the join bolt but storm would've already triggered a replay. The enrichment completing would have triggered the join to happen and the joined message to be emitted and the replay would trigger another copy of the message. In this case, I'd suggest ensuring that either your enrichment is capped at maximum http://storm.apache.org/releases/current/javadocs/org/apache/storm/Config.html#TOPOLOGY_MESSAGE_TIMEOUT_SECS or adjusting the message timeout in storm to be higher than this setting in storm.
          Hide
          nickwallen Nick Allen added a comment -

          It makes sense to me that this problem occurs when an enrichment takes longer than the storm timeout. Bravo for teasing that out.

          The join bolt has a cache containing the original messages that it is expecting enrichments for. This cache is invalidated after some period of time. If the cache invalidation time is less than the storm timeout, then in the scenario described, the 'late' enrichment message would reach the join bolt, but the join bolt would have already forgotten about the original message. The join bolt would then correctly ignore the 'late' enrichment message.

          To address the potential for misconfiguration, we could set the cache invalidation time to be some fraction of the storm timeout.

          Show
          nickwallen Nick Allen added a comment - It makes sense to me that this problem occurs when an enrichment takes longer than the storm timeout. Bravo for teasing that out. The join bolt has a cache containing the original messages that it is expecting enrichments for. This cache is invalidated after some period of time. If the cache invalidation time is less than the storm timeout, then in the scenario described, the 'late' enrichment message would reach the join bolt, but the join bolt would have already forgotten about the original message. The join bolt would then correctly ignore the 'late' enrichment message. To address the potential for misconfiguration, we could set the cache invalidation time to be some fraction of the storm timeout.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user DomenicPuzio commented on the issue:

          https://github.com/apache/incubator-metron/pull/359

          Sorry for the delayed response, @cestella - I was taking some time off for the holiday.

          While I don't believe custom enrichment is pushing us over the timeout (we've clocked it at 15ms at the worst), that's certainly a possibility and something that we should also guard against. With this in mind, we now have two tasks to look into:
          1. Ensuring all messages get acked after a successful join by building out the `streamMessageMap`
          2. Making sure that our cache refresh is set to some fraction of the Storm timeout interval

          Are these both items that we would like to complete? Should we split these into separate JIRAs and PRs? Should I work on the acking changes within this PR?

          Show
          githubbot ASF GitHub Bot added a comment - Github user DomenicPuzio commented on the issue: https://github.com/apache/incubator-metron/pull/359 Sorry for the delayed response, @cestella - I was taking some time off for the holiday. While I don't believe custom enrichment is pushing us over the timeout (we've clocked it at 15ms at the worst), that's certainly a possibility and something that we should also guard against. With this in mind, we now have two tasks to look into: 1. Ensuring all messages get acked after a successful join by building out the `streamMessageMap` 2. Making sure that our cache refresh is set to some fraction of the Storm timeout interval Are these both items that we would like to complete? Should we split these into separate JIRAs and PRs? Should I work on the acking changes within this PR?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user cestella commented on the issue:

          https://github.com/apache/incubator-metron/pull/359

          Yep, so my personal suggestion is to make sure that all messages get acked on a successful join by augmenting `streamMessageMap` and ensure that the cache refresh is longer than the storm timeout (if it is shorter, then it'll never join because the partial messages will timeout before the final message comes through).

          Anyone else have thoughts?

          Show
          githubbot ASF GitHub Bot added a comment - Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/359 Yep, so my personal suggestion is to make sure that all messages get acked on a successful join by augmenting `streamMessageMap` and ensure that the cache refresh is longer than the storm timeout (if it is shorter, then it'll never join because the partial messages will timeout before the final message comes through). Anyone else have thoughts?
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user merrimanr opened a pull request:

          https://github.com/apache/metron/pull/603

          METRON-569: Enrichment topology duplicates messages

            1. Contributor Comments
              This issue was discovered a while ago but a PR with a fix was never accepted. In summary, the JoinBolt is acking the last tuple of a split message instead of the tuple containing the original message sent directly from the SplitBolt. This is a problem because the tuple with the original message is anchored in the SplitBolt and the other tuples that go to enrichment bolts are not.

          The major change in this PR involves storing the tuples in the cache instead of just the message. This way the tuple from the SplitBolt can be retrieved and acked once all the parts have been received in the JoinBolt.

          A couple other minor changes are also included. I refactored the JoinBoltTest to make it easier to follow and maintain. If anyone thinks this is worse than what we currently have I can revert that. I also bumped up the default cache sizes for the JoinBolts from 10,000 to 100,000. The low initial cache size was causing failures in full dev if I let it run for a while.

            1. Pull Request Checklist

          Thank you for submitting a contribution to Apache Metron.
          Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions.
          Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides.

          In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following:

              1. For all changes:
              1. For code changes:
          • [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed?
          • [x] Have you included steps or a guide to how the change may be verified and tested manually?
          • [x] Have you ensured that the full suite of tests and checks have been executed in the root incubating-metron folder via:
            ```
            mvn -q clean integration-test install && build_utils/verify_licenses.sh
            ```
          • [x] Have you written or updated unit tests and or integration tests to verify your changes?
          • [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
          • [x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?
              1. For documentation related changes:
          • [x] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`:

          ```
          cd site-book
          mvn site
          ```

                1. Note:
                  Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
                  It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request.

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/merrimanr/incubator-metron tuple-fix

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/metron/pull/603.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #603


          commit c9ef4f3d9bf023801754267716f108af96827077
          Author: merrimanr <merrimanr@gmail.com>
          Date: 2017-05-18T15:17:11Z

          added logging for unexpected cache issues

          commit 35f9d6878e07ed604212b0af1877b57cad5871e2
          Author: merrimanr <merrimanr@gmail.com>
          Date: 2017-05-22T20:04:52Z

          Merge remote-tracking branch 'mirror/master' into tuple-fix

          commit 8c99f70c871a150d7d763eb5c16716f01f3bec06
          Author: merrimanr <merrimanr@gmail.com>
          Date: 2017-05-22T20:05:23Z

          saving tuples instead of messages

          commit 99ee88697d853aae0f57904660f3ebaf9ea39dee
          Author: merrimanr <merrimanr@gmail.com>
          Date: 2017-05-22T20:15:30Z

          resolve compile errors

          commit b87889f8b3814b8b98569fd2441fc0aa6dbb58e5
          Author: merrimanr <merrimanr@gmail.com>
          Date: 2017-05-31T19:59:09Z

          updated unit tests

          commit 6cd343ede3f9a95d8ad4ed04e7cf323ba2706d55
          Author: merrimanr <merrimanr@gmail.com>
          Date: 2017-05-31T20:00:44Z

          Merge remote-tracking branch 'mirror/master' into tuple-fix

          1. Conflicts:
          2. metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/bolt/JoinBolt.java

          commit ed5a19728998afb97eb5fc7f11e1a8a1b806be2b
          Author: merrimanr <merrimanr@gmail.com>
          Date: 2017-06-01T14:26:59Z

          Resolved merge conflicts and changed default join cache size to 100000


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user merrimanr opened a pull request: https://github.com/apache/metron/pull/603 METRON-569 : Enrichment topology duplicates messages Contributor Comments This issue was discovered a while ago but a PR with a fix was never accepted. In summary, the JoinBolt is acking the last tuple of a split message instead of the tuple containing the original message sent directly from the SplitBolt. This is a problem because the tuple with the original message is anchored in the SplitBolt and the other tuples that go to enrichment bolts are not. The major change in this PR involves storing the tuples in the cache instead of just the message. This way the tuple from the SplitBolt can be retrieved and acked once all the parts have been received in the JoinBolt. A couple other minor changes are also included. I refactored the JoinBoltTest to make it easier to follow and maintain. If anyone thinks this is worse than what we currently have I can revert that. I also bumped up the default cache sizes for the JoinBolts from 10,000 to 100,000. The low initial cache size was causing failures in full dev if I let it run for a while. Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines] ( https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235 ) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines] ( https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview ) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: For all changes: [ ] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira] ( https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel ). [ ] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? For code changes: [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed? [x] Have you included steps or a guide to how the change may be verified and tested manually? [x] Have you ensured that the full suite of tests and checks have been executed in the root incubating-metron folder via: ``` mvn -q clean integration-test install && build_utils/verify_licenses.sh ``` [x] Have you written or updated unit tests and or integration tests to verify your changes? [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0] ( http://www.apache.org/legal/resolved.html#category-a)? [x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? For documentation related changes: [x] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci] ( https://travis-ci.org ) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/merrimanr/incubator-metron tuple-fix Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/603.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #603 commit c9ef4f3d9bf023801754267716f108af96827077 Author: merrimanr <merrimanr@gmail.com> Date: 2017-05-18T15:17:11Z added logging for unexpected cache issues commit 35f9d6878e07ed604212b0af1877b57cad5871e2 Author: merrimanr <merrimanr@gmail.com> Date: 2017-05-22T20:04:52Z Merge remote-tracking branch 'mirror/master' into tuple-fix commit 8c99f70c871a150d7d763eb5c16716f01f3bec06 Author: merrimanr <merrimanr@gmail.com> Date: 2017-05-22T20:05:23Z saving tuples instead of messages commit 99ee88697d853aae0f57904660f3ebaf9ea39dee Author: merrimanr <merrimanr@gmail.com> Date: 2017-05-22T20:15:30Z resolve compile errors commit b87889f8b3814b8b98569fd2441fc0aa6dbb58e5 Author: merrimanr <merrimanr@gmail.com> Date: 2017-05-31T19:59:09Z updated unit tests commit 6cd343ede3f9a95d8ad4ed04e7cf323ba2706d55 Author: merrimanr <merrimanr@gmail.com> Date: 2017-05-31T20:00:44Z Merge remote-tracking branch 'mirror/master' into tuple-fix Conflicts: metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/bolt/JoinBolt.java commit ed5a19728998afb97eb5fc7f11e1a8a1b806be2b Author: merrimanr <merrimanr@gmail.com> Date: 2017-06-01T14:26:59Z Resolved merge conflicts and changed default join cache size to 100000
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user cestella commented on a diff in the pull request:

          https://github.com/apache/metron/pull/603#discussion_r120106060

          — Diff: metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/bolt/JoinBolt.java —
          @@ -141,11 +141,12 @@ public void execute(Tuple tuple) {
          collector.emit( "message"
          , tuple
          , new Values( key

          • , joinMessages(streamMessageMap)
            + , joinMessages(streamMessageMap, this.messageGetStrategy)
            )
            );
            cache.invalidate(key);
          • collector.ack(tuple);
            + Tuple messageTuple = streamMessageMap.get("message:");
            + collector.ack(messageTuple);
              • End diff –

          Do we need to ack every tuple in `streamMessageMap`?

          Show
          githubbot ASF GitHub Bot added a comment - Github user cestella commented on a diff in the pull request: https://github.com/apache/metron/pull/603#discussion_r120106060 — Diff: metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/bolt/JoinBolt.java — @@ -141,11 +141,12 @@ public void execute(Tuple tuple) { collector.emit( "message" , tuple , new Values( key , joinMessages(streamMessageMap) + , joinMessages(streamMessageMap, this.messageGetStrategy) ) ); cache.invalidate(key); collector.ack(tuple); + Tuple messageTuple = streamMessageMap.get("message:"); + collector.ack(messageTuple); End diff – Do we need to ack every tuple in `streamMessageMap`?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user merrimanr commented on a diff in the pull request:

          https://github.com/apache/metron/pull/603#discussion_r120115152

          — Diff: metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/bolt/JoinBolt.java —
          @@ -141,11 +141,12 @@ public void execute(Tuple tuple) {
          collector.emit( "message"
          , tuple
          , new Values( key

          • , joinMessages(streamMessageMap)
            + , joinMessages(streamMessageMap, this.messageGetStrategy)
            )
            );
            cache.invalidate(key);
          • collector.ack(tuple);
            + Tuple messageTuple = streamMessageMap.get("message:");
            + collector.ack(messageTuple);
              • End diff –

          I don't believe so. The tuple that comes straight from the SplitBolt (the "message" stream) is the only tuple that is anchored. Tuples that go to EnrichmentBolts are not anchored.

          Show
          githubbot ASF GitHub Bot added a comment - Github user merrimanr commented on a diff in the pull request: https://github.com/apache/metron/pull/603#discussion_r120115152 — Diff: metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/bolt/JoinBolt.java — @@ -141,11 +141,12 @@ public void execute(Tuple tuple) { collector.emit( "message" , tuple , new Values( key , joinMessages(streamMessageMap) + , joinMessages(streamMessageMap, this.messageGetStrategy) ) ); cache.invalidate(key); collector.ack(tuple); + Tuple messageTuple = streamMessageMap.get("message:"); + collector.ack(messageTuple); End diff – I don't believe so. The tuple that comes straight from the SplitBolt (the "message" stream) is the only tuple that is anchored. Tuples that go to EnrichmentBolts are not anchored.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user cestella commented on a diff in the pull request:

          https://github.com/apache/metron/pull/603#discussion_r120116377

          — Diff: metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/bolt/JoinBolt.java —
          @@ -141,11 +141,12 @@ public void execute(Tuple tuple) {
          collector.emit( "message"
          , tuple
          , new Values( key

          • , joinMessages(streamMessageMap)
            + , joinMessages(streamMessageMap, this.messageGetStrategy)
            )
            );
            cache.invalidate(key);
          • collector.ack(tuple);
            + Tuple messageTuple = streamMessageMap.get("message:");
            + collector.ack(messageTuple);
              • End diff –

          Sounds good!

          Show
          githubbot ASF GitHub Bot added a comment - Github user cestella commented on a diff in the pull request: https://github.com/apache/metron/pull/603#discussion_r120116377 — Diff: metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/bolt/JoinBolt.java — @@ -141,11 +141,12 @@ public void execute(Tuple tuple) { collector.emit( "message" , tuple , new Values( key , joinMessages(streamMessageMap) + , joinMessages(streamMessageMap, this.messageGetStrategy) ) ); cache.invalidate(key); collector.ack(tuple); + Tuple messageTuple = streamMessageMap.get("message:"); + collector.ack(messageTuple); End diff – Sounds good!
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user cestella commented on the issue:

          https://github.com/apache/metron/pull/603

          +1 by inspection, great catch!

          Show
          githubbot ASF GitHub Bot added a comment - Github user cestella commented on the issue: https://github.com/apache/metron/pull/603 +1 by inspection, great catch!
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user cestella commented on the issue:

          https://github.com/apache/metron/pull/359

          I think #603 should address this now.

          Show
          githubbot ASF GitHub Bot added a comment - Github user cestella commented on the issue: https://github.com/apache/metron/pull/359 I think #603 should address this now.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/metron/pull/603

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/metron/pull/603
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user merrimanr commented on the issue:

          https://github.com/apache/metron/pull/359

          @DomenicPuzio, this has been resolved by https://github.com/apache/metron/pull/603. Can we close this PR?

          Show
          githubbot ASF GitHub Bot added a comment - Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/359 @DomenicPuzio, this has been resolved by https://github.com/apache/metron/pull/603 . Can we close this PR?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user DomenicPuzio commented on the issue:

          https://github.com/apache/metron/pull/359

          Yep, we can wrap it up! Thanks, all!

          Show
          githubbot ASF GitHub Bot added a comment - Github user DomenicPuzio commented on the issue: https://github.com/apache/metron/pull/359 Yep, we can wrap it up! Thanks, all!
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user DomenicPuzio closed the pull request at:

          https://github.com/apache/metron/pull/359

          Show
          githubbot ASF GitHub Bot added a comment - Github user DomenicPuzio closed the pull request at: https://github.com/apache/metron/pull/359

            People

            • Assignee:
              Unassigned
              Reporter:
              DomenicPuzio Domenic Puzio
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - 1h
                1h
                Remaining:
                Remaining Estimate - 1h
                1h
                Logged:
                Time Spent - Not Specified
                Not Specified

                  Development