Uploaded image for project: 'CouchDB'
  1. CouchDB
  2. COUCHDB-3274

eof in couch_file can be incorrect after error

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      It's possible for a file:write call to write some of the requested data but then return an error (

      {error, enospc}

      , for example).

      In 2010 we started tracking eof in couch_file state, where previously we called file:position(Fd, eof) and used that value, which was always correct. In success cases we advance the eof value by the correct amount but in error cases we don't advance it at all.

      A simple fix is to call file:position(Fd, eof) in the error cases and set the eof field to that value.

      The upshot of this bug is that data written to the file once eof is wrong is essentially corrupt in that we'll look for it at the wrong offsets. Given this would typically when a filesystem runs completely out of space, in most cases it would not cause issues (we wouldn't be able to write a header that pointed to this) but if space were then recovered (and the couch_file had not closed or crashed) then corruption seems assured.

      I have a patch for this and I certainly welcome opinions on severity (and, indeed, if I'm flat wrong).

        Issue Links

          Activity

          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 604edd10e28e144c57826630b1d92ab41a9377b9 in couchdb-couch's branch refs/heads/3274-couch-file-eof from Robert Newson
          [ https://git-wip-us.apache.org/repos/asf?p=couchdb-couch.git;h=604edd1 ]

          Reset EOF if a partial write was possible

          We can't know if one or more bytes were written by a file:write/2 call
          that results in an error and so it is not correct to leave #file.eof
          at its original value. In the event of error, use file:position(Fd,
          eof) to find the new, true length of the file, and update #file{}
          accordingly.

          COUCHDB-3274

          Show
          jira-bot ASF subversion and git services added a comment - Commit 604edd10e28e144c57826630b1d92ab41a9377b9 in couchdb-couch's branch refs/heads/3274-couch-file-eof from Robert Newson [ https://git-wip-us.apache.org/repos/asf?p=couchdb-couch.git;h=604edd1 ] Reset EOF if a partial write was possible We can't know if one or more bytes were written by a file:write/2 call that results in an error and so it is not correct to leave #file.eof at its original value. In the event of error, use file:position(Fd , eof) to find the new, true length of the file, and update #file{} accordingly. COUCHDB-3274
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user rnewson opened a pull request:

          https://github.com/apache/couchdb-couch/pull/222

          Reset EOF if a partial write was possible

          We can't know if one or more bytes were written by a file:write/2 call
          that results in an error and so it is not correct to leave #file.eof
          at its original value. In the event of error, use file:position(Fd,
          eof) to find the new, true length of the file, and update #file{}
          accordingly.

          COUCHDB-3274

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/cloudant/couchdb-couch 3274-couch-file-eof

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/couchdb-couch/pull/222.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #222


          commit 604edd10e28e144c57826630b1d92ab41a9377b9
          Author: Robert Newson <rnewson@apache.org>
          Date: 2017-01-19T23:12:00Z

          Reset EOF if a partial write was possible

          We can't know if one or more bytes were written by a file:write/2 call
          that results in an error and so it is not correct to leave #file.eof
          at its original value. In the event of error, use file:position(Fd,
          eof) to find the new, true length of the file, and update #file{}
          accordingly.

          COUCHDB-3274


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user rnewson opened a pull request: https://github.com/apache/couchdb-couch/pull/222 Reset EOF if a partial write was possible We can't know if one or more bytes were written by a file:write/2 call that results in an error and so it is not correct to leave #file.eof at its original value. In the event of error, use file:position(Fd , eof) to find the new, true length of the file, and update #file{} accordingly. COUCHDB-3274 You can merge this pull request into a Git repository by running: $ git pull https://github.com/cloudant/couchdb-couch 3274-couch-file-eof Alternatively you can review and apply these changes as the patch at: https://github.com/apache/couchdb-couch/pull/222.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #222 commit 604edd10e28e144c57826630b1d92ab41a9377b9 Author: Robert Newson <rnewson@apache.org> Date: 2017-01-19T23:12:00Z Reset EOF if a partial write was possible We can't know if one or more bytes were written by a file:write/2 call that results in an error and so it is not correct to leave #file.eof at its original value. In the event of error, use file:position(Fd , eof) to find the new, true length of the file, and update #file{} accordingly. COUCHDB-3274
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 604edd10e28e144c57826630b1d92ab41a9377b9 in couchdb-couch's branch refs/heads/master from Robert Newson
          [ https://git-wip-us.apache.org/repos/asf?p=couchdb-couch.git;h=604edd1 ]

          Reset EOF if a partial write was possible

          We can't know if one or more bytes were written by a file:write/2 call
          that results in an error and so it is not correct to leave #file.eof
          at its original value. In the event of error, use file:position(Fd,
          eof) to find the new, true length of the file, and update #file{}
          accordingly.

          COUCHDB-3274

          Show
          jira-bot ASF subversion and git services added a comment - Commit 604edd10e28e144c57826630b1d92ab41a9377b9 in couchdb-couch's branch refs/heads/master from Robert Newson [ https://git-wip-us.apache.org/repos/asf?p=couchdb-couch.git;h=604edd1 ] Reset EOF if a partial write was possible We can't know if one or more bytes were written by a file:write/2 call that results in an error and so it is not correct to leave #file.eof at its original value. In the event of error, use file:position(Fd , eof) to find the new, true length of the file, and update #file{} accordingly. COUCHDB-3274
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/couchdb-couch/pull/222

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/couchdb-couch/pull/222
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user eiri opened a pull request:

          https://github.com/apache/couchdb/pull/451

          Updated couch dependencies

          List of changes:

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/cloudant/couchdb bump-dependencies

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/couchdb/pull/451.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #451


          commit 1624f84a4ac2c833736317faddb1ed0c41696668
          Author: Eric Avdey <eiri@eiri.ca>
          Date: 2017-01-20T21:43:18Z

          Updated couch dependencies

          couch: b83f1a..604edd

          • Reset EOF if a partial write was possible
          • Merge remote branch 'cloudant:validate-count-in-uuids'
          • Return "Bad Request" when count in /_uuids exceeds max
          • Validate that count in /_uuids is positive
          • Merge remote branch 'cloudant:fix-views_tests'
          • Fix couchdb_views tests
          • Don't trap exits in couch_file
          • Merge branch 'COUCHDB-3255-fix-new-revid'
          • Make revision generation deterministic
          • Merge remote branch 'cloudant:add-filter-changes-test'
          • Add tests for filter by filter function
          • Reduce duplicated code, rename vars consistently
          • Prevent consumer from hiding error messages

          Show
          githubbot ASF GitHub Bot added a comment - GitHub user eiri opened a pull request: https://github.com/apache/couchdb/pull/451 Updated couch dependencies List of changes: Reset EOF if a partial write was possible ( COUCHDB-3274 ( https://issues.apache.org/jira/browse/COUCHDB-3274 )) Merge remote branch 'cloudant:validate-count-in-uuids' Return "Bad Request" when count in /_uuids exceeds max Validate that count in /_uuids is positive Merge remote branch 'cloudant:fix-views_tests' Fix couchdb_views tests ( COUCHDB-3259 ( https://issues.apache.org/jira/browse/COUCHDB-3259 )) Don't trap exits in couch_file ( COUCHDB-3259 ( https://issues.apache.org/jira/browse/COUCHDB-3259 )) Merge branch ' COUCHDB-3255 -fix-new-revid' ( COUCHDB-3255 ( https://issues.apache.org/jira/browse/COUCHDB-3255 )) Make revision generation deterministic ( COUCHDB-3255 ( https://issues.apache.org/jira/browse/COUCHDB-3255 )) Merge remote branch 'cloudant:add-filter-changes-test' Add tests for filter by filter function Reduce duplicated code, rename vars consistently Prevent consumer from hiding error messages You can merge this pull request into a Git repository by running: $ git pull https://github.com/cloudant/couchdb bump-dependencies Alternatively you can review and apply these changes as the patch at: https://github.com/apache/couchdb/pull/451.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #451 commit 1624f84a4ac2c833736317faddb1ed0c41696668 Author: Eric Avdey <eiri@eiri.ca> Date: 2017-01-20T21:43:18Z Updated couch dependencies couch: b83f1a..604edd Reset EOF if a partial write was possible Merge remote branch 'cloudant:validate-count-in-uuids' Return "Bad Request" when count in /_uuids exceeds max Validate that count in /_uuids is positive Merge remote branch 'cloudant:fix-views_tests' Fix couchdb_views tests Don't trap exits in couch_file Merge branch ' COUCHDB-3255 -fix-new-revid' Make revision generation deterministic Merge remote branch 'cloudant:add-filter-changes-test' Add tests for filter by filter function Reduce duplicated code, rename vars consistently Prevent consumer from hiding error messages

            People

            • Assignee:
              Unassigned
              Reporter:
              rnewson Robert Newson
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:

                Development