Uploaded image for project: 'CouchDB'
  1. CouchDB
  2. COUCHDB-3046

Improve reduce function overflow protection

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Database Core
    • Labels:
      None

      Description

      The protection algorithm:

      https://github.com/apache/couchdb/blob/master/share/server/views.js#L36-L41

      When enabled, looks at couchjs' reduce command input and output line lengths (as stringy-fied json). If 2*len(output) > len(input) and len(output) > 200 then an error is triggered.

      There a few issues in that scheme:

      • No mode to first log failures only. This way user can handle bad reduce functions as a warning rather than as a failed query result.
      • Input line contains the length of the reduce function code itself. A large reduce function body (say 100KB) might skew the result and allow allow through reduce function with larger than needed output (without tripping the error).
      • On the other hand, output size checking threshold is too small = 200. It prevents functions using single large accumulator object (say with fields like .sum, .count, .stddev, and so on) from working. The size of output will be > 200 but, even though it won't be growing it will still be prevented from running.

        Issue Links

          Activity

          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user nickva commented on the issue:

          https://github.com/apache/couchdb/pull/425

          Thanks for taking a look @wohali and @kxepal

          There is a JIRA issue already: https://issues.apache.org/jira/browse/COUCHDB-3046.

          Also rebased on master.

          Show
          githubbot ASF GitHub Bot added a comment - Github user nickva commented on the issue: https://github.com/apache/couchdb/pull/425 Thanks for taking a look @wohali and @kxepal There is a JIRA issue already: https://issues.apache.org/jira/browse/COUCHDB-3046 . Also rebased on master.
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 3937fb1ef2cb4377571f488e7a20e003b5e8c69c in couchdb's branch refs/heads/master from Nick Vatamaniuc
          [ https://git-wip-us.apache.org/repos/asf?p=couchdb.git;h=3937fb1 ]

          Improve reduce function overlow handling

          Add `log` as another option for `query_config.reduce_limit` instead of `true`
          or `false` only. This warns user without crashing the view.

          Account for reduce functions code size when checking for overlflow. Code size
          was previously implicitly included in input size, skewing the result of
          overflow check.

          Increase threshold for overflow check. Before it was only 200. This
          should prevent false positives in case users have a large accumulator object to
          collect various stats.

          Jira: COUCHDB-3046

          Show
          jira-bot ASF subversion and git services added a comment - Commit 3937fb1ef2cb4377571f488e7a20e003b5e8c69c in couchdb's branch refs/heads/master from Nick Vatamaniuc [ https://git-wip-us.apache.org/repos/asf?p=couchdb.git;h=3937fb1 ] Improve reduce function overlow handling Add `log` as another option for `query_config.reduce_limit` instead of `true` or `false` only. This warns user without crashing the view. Account for reduce functions code size when checking for overlflow. Code size was previously implicitly included in input size, skewing the result of overflow check. Increase threshold for overflow check. Before it was only 200. This should prevent false positives in case users have a large accumulator object to collect various stats. Jira: COUCHDB-3046

            People

            • Assignee:
              Unassigned
              Reporter:
              vatamane Nick Vatamaniuc
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:

                Development