Uploaded image for project: 'CouchDB'
  1. CouchDB
  2. COUCHDB-3269

view response can 'hang' with filter and limit specified

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.0
    • Fix Version/s: 2.1.0
    • Component/s: None
    • Labels:
      None

      Description

      If you query a large view with both 'filter' and 'limit' specified, couchdb will not promptly complete the view response after the requested number of rows has been returned.

      This is because view completion happens on receipt of a 'change' or 'completed' message from a worker. In the case of a filtered view request, the worker might instead return a 'no_pass' message, indicating that no documents passed the filter at a given sequence.

      Add a clause in fabric_view_changes to handle this special case and end the view response promptly where possible.

        Issue Links

          Activity

          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user rnewson opened a pull request:

          https://github.com/apache/couchdb-fabric/pull/82

          Handle no_pass message when limit is 0

          A view query that uses both a filter and a limit value can appear to
          'hang' because the workers continue to run even after the number of
          requested items have been returned, as workers that find no more
          matches return 'no_pass' messages not 'change' messages, and thus the
          short-circuiting code is never activated.

          COUCHDB-3269

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/cloudant/couchdb-fabric 3269-filter-and-limit

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/couchdb-fabric/pull/82.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #82


          commit 58c810ede8e929c372ac575044311bf1bdb90d8a
          Author: Robert Newson <rnewson@apache.org>
          Date: 2017-01-09T12:30:02Z

          Handle no_pass message when limit is 0

          A view query that uses both a filter and a limit value can appear to
          'hang' because the workers continue to run even after the number of
          requested items have been returned, as workers that find no more
          matches return 'no_pass' messages not 'change' messages, and thus the
          short-circuiting code is never activated.

          COUCHDB-3269


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user rnewson opened a pull request: https://github.com/apache/couchdb-fabric/pull/82 Handle no_pass message when limit is 0 A view query that uses both a filter and a limit value can appear to 'hang' because the workers continue to run even after the number of requested items have been returned, as workers that find no more matches return 'no_pass' messages not 'change' messages, and thus the short-circuiting code is never activated. COUCHDB-3269 You can merge this pull request into a Git repository by running: $ git pull https://github.com/cloudant/couchdb-fabric 3269-filter-and-limit Alternatively you can review and apply these changes as the patch at: https://github.com/apache/couchdb-fabric/pull/82.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #82 commit 58c810ede8e929c372ac575044311bf1bdb90d8a Author: Robert Newson <rnewson@apache.org> Date: 2017-01-09T12:30:02Z Handle no_pass message when limit is 0 A view query that uses both a filter and a limit value can appear to 'hang' because the workers continue to run even after the number of requested items have been returned, as workers that find no more matches return 'no_pass' messages not 'change' messages, and thus the short-circuiting code is never activated. COUCHDB-3269
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user rnewson closed the pull request at:

          https://github.com/apache/couchdb-fabric/pull/82

          Show
          githubbot ASF GitHub Bot added a comment - Github user rnewson closed the pull request at: https://github.com/apache/couchdb-fabric/pull/82
          Hide
          rnewson Robert Newson added a comment -

          This isn't quite right. What I've seen is a _changes request that 'hangs' despite sending as many rows as requested by the limit parameter.

          In the hang cases, filter and limit and since are specified, and I think it's the combination of all three that gets us here.

          The current fabric_view_changes code around no_pass looks right to me on reflection. We can't update offset until we get a hit from a shard since offset is the distance to the first row from the beginning of the view, and without updating offset for each worker, maybe_stop will not stop.

          So, what must be happening is that the combination of the filter and the since value means at least one worker has not seen a hit yet. That is takes so long to hit one implies that there are no hits in that shard range at all and we're just processing the feed from it, sending no_pass messages, until we hit the end.

          Not immediately clear how we short-circuit correctly here.

          Show
          rnewson Robert Newson added a comment - This isn't quite right. What I've seen is a _changes request that 'hangs' despite sending as many rows as requested by the limit parameter. In the hang cases, filter and limit and since are specified, and I think it's the combination of all three that gets us here. The current fabric_view_changes code around no_pass looks right to me on reflection. We can't update offset until we get a hit from a shard since offset is the distance to the first row from the beginning of the view, and without updating offset for each worker, maybe_stop will not stop. So, what must be happening is that the combination of the filter and the since value means at least one worker has not seen a hit yet. That is takes so long to hit one implies that there are no hits in that shard range at all and we're just processing the feed from it, sending no_pass messages, until we hit the end. Not immediately clear how we short-circuit correctly here.
          Hide
          rnewson Robert Newson added a comment -

          maybe not so wrong. the 'offset' field in collector in fabric_view_changes doesn't mean what I initially thought. Patch is not too far off, but it's still wrong. We reused the same record as we do in regular views where it does mean what I thought (the offset of the first row from the beginning of the view).

          Show
          rnewson Robert Newson added a comment - maybe not so wrong. the 'offset' field in collector in fabric_view_changes doesn't mean what I initially thought. Patch is not too far off, but it's still wrong. We reused the same record as we do in regular views where it does mean what I thought (the offset of the first row from the beginning of the view).
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user rnewson opened a pull request:

          https://github.com/apache/couchdb-fabric/pull/83

          Handle no_pass message when limit is 0

          A view query that uses both a filter and a limit value can appear to
          'hang' because the workers continue to run even after the number of
          requested items have been returned, as workers that find no more
          matches return 'no_pass' messages not 'change' messages, and thus the
          short-circuiting code is never activated.

          COUCHDB-3269

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/cloudant/couchdb-fabric 3269-filter-and-limit

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/couchdb-fabric/pull/83.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #83


          commit f45be70172beb6b46b1e3a758527ea87787571c7
          Author: Robert Newson <rnewson@apache.org>
          Date: 2017-01-09T12:30:02Z

          Handle no_pass message when limit is 0

          A view query that uses both a filter and a limit value can appear to
          'hang' because the workers continue to run even after the number of
          requested items have been returned, as workers that find no more
          matches return 'no_pass' messages not 'change' messages, and thus the
          short-circuiting code is never activated.

          COUCHDB-3269


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user rnewson opened a pull request: https://github.com/apache/couchdb-fabric/pull/83 Handle no_pass message when limit is 0 A view query that uses both a filter and a limit value can appear to 'hang' because the workers continue to run even after the number of requested items have been returned, as workers that find no more matches return 'no_pass' messages not 'change' messages, and thus the short-circuiting code is never activated. COUCHDB-3269 You can merge this pull request into a Git repository by running: $ git pull https://github.com/cloudant/couchdb-fabric 3269-filter-and-limit Alternatively you can review and apply these changes as the patch at: https://github.com/apache/couchdb-fabric/pull/83.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #83 commit f45be70172beb6b46b1e3a758527ea87787571c7 Author: Robert Newson <rnewson@apache.org> Date: 2017-01-09T12:30:02Z Handle no_pass message when limit is 0 A view query that uses both a filter and a limit value can appear to 'hang' because the workers continue to run even after the number of requested items have been returned, as workers that find no more matches return 'no_pass' messages not 'change' messages, and thus the short-circuiting code is never activated. COUCHDB-3269
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 998cf2d66aeb0a436f2d836f3f50ee0c647137fc in couchdb-fabric's branch refs/heads/master from Robert Newson
          [ https://git-wip-us.apache.org/repos/asf?p=couchdb-fabric.git;h=998cf2d ]

          Handle no_pass message when limit is 0

          A view query that uses both a filter and a limit value can appear to
          'hang' because the workers continue to run even after the number of
          requested items have been returned, as workers that find no more
          matches return 'no_pass' messages not 'change' messages, and thus the
          short-circuiting code is never activated.

          COUCHDB-3269

          Show
          jira-bot ASF subversion and git services added a comment - Commit 998cf2d66aeb0a436f2d836f3f50ee0c647137fc in couchdb-fabric's branch refs/heads/master from Robert Newson [ https://git-wip-us.apache.org/repos/asf?p=couchdb-fabric.git;h=998cf2d ] Handle no_pass message when limit is 0 A view query that uses both a filter and a limit value can appear to 'hang' because the workers continue to run even after the number of requested items have been returned, as workers that find no more matches return 'no_pass' messages not 'change' messages, and thus the short-circuiting code is never activated. COUCHDB-3269
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/couchdb-fabric/pull/83

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/couchdb-fabric/pull/83
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user eiri opened a pull request:

          https://github.com/apache/couchdb/pull/450

          Bump dependencies for couch, chttpd and fabric

          List of changes:

          chttpd: cb0f20..90648a

          • allow w parameter for attachments

          fabric: 205064..998cf2

          couch: b83f1a..318873

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/cloudant/couchdb bump-dependencies

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/couchdb/pull/450.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #450


          commit 64bd1f16437671c56462ec1b9dcf7115e8dea37b
          Author: Eric Avdey <eiri@eiri.ca>
          Date: 2017-01-18T17:07:35Z

          Bump dependencies for couch, chttpd and fabric

          List of changes:

          chttpd: cb0f20..90648a

          • allow w parameter for attachments

          fabric: 205064..998cf2

          • Handle no_pass message when limit is 0
          • Merge remote-tracking branch 'willholley/3264-alldocs-keys-conflicts'
          • Include conflicts parameter in doc_options

          couch: b83f1a..318873

          • Merge remote branch 'cloudant:validate-count-in-uuids'
          • Return "Bad Request" when count in /_uuids exceeds max
          • Validate that count in /_uuids is positive
          • Merge remote branch 'cloudant:fix-views_tests'
          • Fix couchdb_views tests
          • Don't trap exits in couch_file
          • Merge branch 'COUCHDB-3255-fix-new-revid'
          • Make revision generation deterministic
          • Merge remote branch 'cloudant:add-filter-changes-test'
          • Add tests for filter by filter function
          • Reduce duplicated code, rename vars consistently
          • Prevent consumer from hiding error messages

          Show
          githubbot ASF GitHub Bot added a comment - GitHub user eiri opened a pull request: https://github.com/apache/couchdb/pull/450 Bump dependencies for couch, chttpd and fabric List of changes: chttpd: cb0f20..90648a allow w parameter for attachments fabric: 205064..998cf2 Handle no_pass message when limit is 0 ( COUCHDB-3269 ( https://issues.apache.org/jira/browse/COUCHDB-3269 )) Merge remote-tracking branch 'willholley/3264-alldocs-keys-conflicts' ( COUCHDB-3264 ( https://issues.apache.org/jira/browse/COUCHDB-3264 )) Include conflicts parameter in doc_options ( COUCHDB-3264 ( https://issues.apache.org/jira/browse/COUCHDB-3264 )) couch: b83f1a..318873 Merge remote branch 'cloudant:validate-count-in-uuids' ( COUCHDB-3273 ( https://issues.apache.org/jira/browse/COUCHDB-3273 )) Return "Bad Request" when count in /_uuids exceeds max Validate that count in /_uuids is positive Merge remote branch 'cloudant:fix-views_tests' Fix couchdb_views tests ( COUCHDB-3259 ( https://issues.apache.org/jira/browse/COUCHDB-3259 )) Don't trap exits in couch_file ( COUCHDB-3259 ( https://issues.apache.org/jira/browse/COUCHDB-3259 )) Merge branch ' COUCHDB-3255 -fix-new-revid' ( COUCHDB-3255 ( https://issues.apache.org/jira/browse/COUCHDB-3255 )) Make revision generation deterministic ( COUCHDB-3255 ( https://issues.apache.org/jira/browse/COUCHDB-3255 )) Merge remote branch 'cloudant:add-filter-changes-test' Add tests for filter by filter function Reduce duplicated code, rename vars consistently Prevent consumer from hiding error messages You can merge this pull request into a Git repository by running: $ git pull https://github.com/cloudant/couchdb bump-dependencies Alternatively you can review and apply these changes as the patch at: https://github.com/apache/couchdb/pull/450.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #450 commit 64bd1f16437671c56462ec1b9dcf7115e8dea37b Author: Eric Avdey <eiri@eiri.ca> Date: 2017-01-18T17:07:35Z Bump dependencies for couch, chttpd and fabric List of changes: chttpd: cb0f20..90648a allow w parameter for attachments fabric: 205064..998cf2 Handle no_pass message when limit is 0 Merge remote-tracking branch 'willholley/3264-alldocs-keys-conflicts' Include conflicts parameter in doc_options couch: b83f1a..318873 Merge remote branch 'cloudant:validate-count-in-uuids' Return "Bad Request" when count in /_uuids exceeds max Validate that count in /_uuids is positive Merge remote branch 'cloudant:fix-views_tests' Fix couchdb_views tests Don't trap exits in couch_file Merge branch ' COUCHDB-3255 -fix-new-revid' Make revision generation deterministic Merge remote branch 'cloudant:add-filter-changes-test' Add tests for filter by filter function Reduce duplicated code, rename vars consistently Prevent consumer from hiding error messages
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user eiri closed the pull request at:

          https://github.com/apache/couchdb/pull/450

          Show
          githubbot ASF GitHub Bot added a comment - Github user eiri closed the pull request at: https://github.com/apache/couchdb/pull/450
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 998cf2d66aeb0a436f2d836f3f50ee0c647137fc in couchdb-fabric's branch refs/heads/2971-count-distinct from Robert Newson
          [ https://git-wip-us.apache.org/repos/asf?p=couchdb-fabric.git;h=998cf2d ]

          Handle no_pass message when limit is 0

          A view query that uses both a filter and a limit value can appear to
          'hang' because the workers continue to run even after the number of
          requested items have been returned, as workers that find no more
          matches return 'no_pass' messages not 'change' messages, and thus the
          short-circuiting code is never activated.

          COUCHDB-3269

          Show
          jira-bot ASF subversion and git services added a comment - Commit 998cf2d66aeb0a436f2d836f3f50ee0c647137fc in couchdb-fabric's branch refs/heads/2971-count-distinct from Robert Newson [ https://git-wip-us.apache.org/repos/asf?p=couchdb-fabric.git;h=998cf2d ] Handle no_pass message when limit is 0 A view query that uses both a filter and a limit value can appear to 'hang' because the workers continue to run even after the number of requested items have been returned, as workers that find no more matches return 'no_pass' messages not 'change' messages, and thus the short-circuiting code is never activated. COUCHDB-3269

            People

            • Assignee:
              Unassigned
              Reporter:
              rnewson Robert Newson
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development