CouchDB
  1. CouchDB
  2. COUCHDB-1357

Authentication failure after updating password in user document

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.1.1
    • Fix Version/s: 1.2
    • Component/s: None
    • Labels:
      None

      Description

      From the report at the users mailing list:

      http://s.apache.org/9OG

      Seems like after updating the password in a user doc, the user is not able to login with the new password unless Couch is restarted. Sounds like a caching issue.
      The only case of getting the cache consistent with the _users database content is if the _users database processes crash and after the crash user documents are updated. The cache daemon is ignoring the database crash.

      The following patch updates the daemon to monitor the _users database and crash (letting the supervisor restart it) if the database process crashes.
      Etap test included.

      This might be related to COUCHDB-1212.

        Activity

        Hide
        Pete Vander Giessen added a comment -

        We ran into a similar issue, with 1.1.1. The problem seemed to start with a log entry that looked like so:

        [Thu, 01 Dec 2011 19:22:48 GMT] [error] [<0.31233.1889>] Uncaught
        error in HTTP request:

        {error,system_limit}

        [Thu, 01 Dec 2011 19:22:48 GMT] [info] [<0.31233.1889>] Stacktrace:
        [{erlang,open_port,
        [

        {spawn,"couch_icu_driver"}

        ,[]]},

        {couch_util,drv_port,0}

        ,

        {couch_util,collate,3}

        ,

        {couch_view,less_json_ids,2}

        ,

        {couch_btree,drop_nodes,4}

        ,

        {couch_btree,stream_kp_node,8}

        ,

        {couch_btree,fold,4}

        ,

        {couch_view,fold,4}

        ]

        We had a cluster of 7 or so Windows users replicating 2 databases apiece from a "master" server running on a Red Hat box. Each user also has a separate "user" database that they use to fetch oauth credentials, so there were perhaps 16 databases in use at the time.

        Next time this happens, I'll run netstat on the machine and save the output. For now, I've saved off the logs, and can provide more complete logs upon request.

        Show
        Pete Vander Giessen added a comment - We ran into a similar issue, with 1.1.1. The problem seemed to start with a log entry that looked like so: [Thu, 01 Dec 2011 19:22:48 GMT] [error] [<0.31233.1889>] Uncaught error in HTTP request: {error,system_limit} [Thu, 01 Dec 2011 19:22:48 GMT] [info] [<0.31233.1889>] Stacktrace: [{erlang,open_port, [ {spawn,"couch_icu_driver"} ,[]]}, {couch_util,drv_port,0} , {couch_util,collate,3} , {couch_view,less_json_ids,2} , {couch_btree,drop_nodes,4} , {couch_btree,stream_kp_node,8} , {couch_btree,fold,4} , {couch_view,fold,4} ] We had a cluster of 7 or so Windows users replicating 2 databases apiece from a "master" server running on a Red Hat box. Each user also has a separate "user" database that they use to fetch oauth credentials, so there were perhaps 16 databases in use at the time. Next time this happens, I'll run netstat on the machine and save the output. For now, I've saved off the logs, and can provide more complete logs upon request.
        Hide
        Randall Leeds added a comment -

        Patch looks good.

        Show
        Randall Leeds added a comment - Patch looks good.
        Hide
        Filipe Manana added a comment -

        Pete, I'm not sure this issue relates directly yo your issue.
        From the error {error,system_limit, it seems your system reached the maximum limit of available file descriptors. I have no idea how to increase this limit on Windows, perhaps Dave can help here.

        Show
        Filipe Manana added a comment - Pete, I'm not sure this issue relates directly yo your issue. From the error {error,system_limit, it seems your system reached the maximum limit of available file descriptors. I have no idea how to increase this limit on Windows, perhaps Dave can help here.
        Hide
        Filipe Manana added a comment -

        Pete, there's this Erlang thread ("Increase the handle limit") Dave participated on which covers your issue:
        http://erlang.org/pipermail/erlang-questions/2011-October/thread.html#61859

        Show
        Filipe Manana added a comment - Pete, there's this Erlang thread ("Increase the handle limit") Dave participated on which covers your issue: http://erlang.org/pipermail/erlang-questions/2011-October/thread.html#61859
        Hide
        Dave Cottlehuber added a comment -

        @pete the following might help. There are notes on raising # of FDs on both linux and windows in this thread http://erlang.org/pipermail/erlang-questions/2011-October/thread.html#61859

        Try updating this in couchdb.bat assuming you're not running as a service:
        echo CouchDB %version% - prepare to relax...
        %ERL% -sasl errlog_type error -s couch +A 4 +W w -env ERL_MAX_PORTS 100000 +P 1000000

        If you're running as a service then you'll need to use erlsrv.exe http://wiki.apache.org/couchdb/Quirks_on_Windows and http://www.erlang.org/doc/man/erlsrv.html instead.

        There is also a short test file in http://erlang.org/pipermail/erlang-questions/2011-October/061895.html to help confirm your changes were successful.

        I'm not by my windows box to confirm; I'll check back later.

        Show
        Dave Cottlehuber added a comment - @pete the following might help. There are notes on raising # of FDs on both linux and windows in this thread http://erlang.org/pipermail/erlang-questions/2011-October/thread.html#61859 Try updating this in couchdb.bat assuming you're not running as a service: echo CouchDB %version% - prepare to relax... %ERL% -sasl errlog_type error -s couch +A 4 +W w -env ERL_MAX_PORTS 100000 +P 1000000 If you're running as a service then you'll need to use erlsrv.exe http://wiki.apache.org/couchdb/Quirks_on_Windows and http://www.erlang.org/doc/man/erlsrv.html instead. There is also a short test file in http://erlang.org/pipermail/erlang-questions/2011-October/061895.html to help confirm your changes were successful. I'm not by my windows box to confirm; I'll check back later.
        Hide
        Pete Vander Giessen added a comment -

        @Dave and @Filipe: thank you for the advice. I've been playing around with limits on my Linux server. It looks like I can get things pretty high by juggling the stuff in ulimit, as well as the overall limits in security.conf, in combination w/ ERL_MAX_PORTS and +P ...

        One more question: is there are generic way to tell when a view server has crashed, and we may need to restart couchdb? It sounds like I can make the problem I ran into less likely to occur by increasing file/process limits, but I'd like to add some sort of watchdog process that will let me know that I've hit a limit, and that my _users view server, for example, may be crashed ...

        Thank you,
        ~ Pete

        Show
        Pete Vander Giessen added a comment - @Dave and @Filipe: thank you for the advice. I've been playing around with limits on my Linux server. It looks like I can get things pretty high by juggling the stuff in ulimit, as well as the overall limits in security.conf, in combination w/ ERL_MAX_PORTS and +P ... One more question: is there are generic way to tell when a view server has crashed, and we may need to restart couchdb? It sounds like I can make the problem I ran into less likely to occur by increasing file/process limits, but I'd like to add some sort of watchdog process that will let me know that I've hit a limit, and that my _users view server, for example, may be crashed ... Thank you, ~ Pete
        Hide
        Pete Vander Giessen added a comment -

        I managed to reproduce the bug

        I started couch w/ the following in the invocation:
        -env ERL_MAX_PORTS 256 +P 256

        (I tried setting the ports lower, but couch wouldn't start at all if I set it too low.)

        Then I visited futon on a linux server with about 100 databases. This opened enough files/ports to trigger the system limit.

        Here's what I saw:
        1) Couch continued to run, and my admin user could log in, but I couldn't log in with any users in the _users database.
        2) The log got ... noisy. Here's the first error:
        ]\
        [Thu, 08 Dec 2011 23:36:00 GMT] [error] [<0.84.0>] Unexpected message, restarting couch_server: {'EXIT',
        <0.734.0>,
        {system_limit,
        [{erlang,
        spawn_opt,
        [proc_lib,
        init_p,
        [<0.734.0>,
        [],gen,
        init_it,
        [gen_server,
        <0.734.0>,
        <0.734.0>,
        couch_file,
        {"/opt/Cloudpic/var/lib/couchdb/p-8911029bacfc48c9809a97ef00da45b6.couch",
        [{user_ctx,
        {user_ctx,
        null,
        [],
        undefined}}],
        <0.734.0>,
        #Ref<0.0.0.1816>},
        []]],
        [link]]},

        {proc_lib, start_link, 5}

        ,

        {couch_file, open,2}

        ,

        {couch_db, open_db_file, 2}

        ,

        {couch_db, start_link, 3}

        ,

        {couch_server, '-open_async/5-fun-0-', 4}

        ]}}

        Show
        Pete Vander Giessen added a comment - I managed to reproduce the bug I started couch w/ the following in the invocation: -env ERL_MAX_PORTS 256 +P 256 (I tried setting the ports lower, but couch wouldn't start at all if I set it too low.) Then I visited futon on a linux server with about 100 databases. This opened enough files/ports to trigger the system limit. Here's what I saw: 1) Couch continued to run, and my admin user could log in, but I couldn't log in with any users in the _users database. 2) The log got ... noisy. Here's the first error: ]\ [Thu, 08 Dec 2011 23:36:00 GMT] [error] [<0.84.0>] Unexpected message, restarting couch_server: {'EXIT', <0.734.0>, {system_limit, [{erlang, spawn_opt, [proc_lib, init_p, [<0.734.0>, [],gen, init_it, [gen_server, <0.734.0>, <0.734.0>, couch_file, {"/opt/Cloudpic/var/lib/couchdb/p-8911029bacfc48c9809a97ef00da45b6.couch", [{user_ctx, {user_ctx, null, [], undefined}}], <0.734.0>, #Ref<0.0.0.1816>}, []]], [link] ]}, {proc_lib, start_link, 5} , {couch_file, open,2} , {couch_db, open_db_file, 2} , {couch_db, start_link, 3} , {couch_server, '-open_async/5-fun-0-', 4} ]}}
        Hide
        Filipe Manana added a comment -

        Pete, your scenario makes sense. It will cause all database process to be killed (couch_server dies).

        Show
        Filipe Manana added a comment - Pete, your scenario makes sense. It will cause all database process to be killed (couch_server dies).
        Hide
        Filipe Manana added a comment -

        Updated patch to deal with the case where the auth database is modified during run time. When this happens the new database process most be monitored.
        Also updated the tests to be more comprehensive.

        Show
        Filipe Manana added a comment - Updated patch to deal with the case where the auth database is modified during run time. When this happens the new database process most be monitored. Also updated the tests to be more comprehensive.
        Hide
        Filipe Manana added a comment -

        Fix applied to master and 1.2.x

        Show
        Filipe Manana added a comment - Fix applied to master and 1.2.x

          People

          • Assignee:
            Unassigned
            Reporter:
            Filipe Manana
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development