Ops, forgot to include the adaptations necessary to test/etap/070-couch-db.t

So, testing this with with a server having 1000 DBs, each one with a size of about 1Mb and 100 docs, the response time for _all_dbs is about 0.5s on my system (Ubuntu 9.10, SATA hd 7200rpm)

The DB was populated with the tool at http://github.com/fdmanana/seatoncouch using the following doc template:

{
"_id": "doc#

",
"name": "#

",
"address": "#

",
"age": #

,
"children": #

,
"bio": "#

"
}

Each DB has the security doc:

{
"admins":

,
"readers":

}

$ time ./seatoncouch.rb --doc-tpl fdmanana_doc.tpl --security-doc security_doc.json --dbs 1000 --docs 100
[INFO] Created DB named testdb1
[INFO] Created doc at /testdb1/doc1
[INFO] Created doc at /testdb1/doc2

1. etc... takes more than 1 hour

Measuring the time:

$ time curl http://localhost:5984/_all_dbs
[
"testdb2",
"testdb485",
"testdb497",

1. etc...

real 0m0.498s
user 0m0.000s
sys 0m0.010s

Increasing the number of DBs to 7500:

$ time ./seatoncouch.rb --doc-tpl fdmanana_doc.tpl --security-doc security_doc.json --dbs 6500 --docs 100 --db-start-id 1001
...

$ time curl http://localhost:5984/_all_dbs 2> /dev/null | wc -l
7502

real 0m3.763s
user 0m0.010s
sys 0m0.090s

$ time curl http://localhost:5984/_all_dbs 2> /dev/null | wc -l
7502

real 0m3.804s
user 0m0.020s
sys 0m0.060s

$ time curl http://localhost:5984/_all_dbs 2> /dev/null | wc -l
7502

real 0m3.714s
user 0m0.020s
sys 0m0.100s

Looks like a serious DoS to me, with "only" 7500 databases.

If _all_dbs won't scale, then I think it should be for admins only (ideally with startkey/limit like _all_docs for efficient pagination, but that's a different issue)

Or perhaps it should be possible to replace _all_dbs with a view in a 'real' database for non-admins.

e.g. occasionally you could copy all the _security objects into another database, and generate a view with keys like
emit(['name',name],db)
emit(['role',role],db)
for efficient querying.

(IMHO this is another reason why _security objects should be real docs: so that you can follow a _changes feed on them)

Yes Brian, I do share the same vision.

The idea, which I pretend to discuss soon in the @dev mailing list, is to use a view which will map user names and roles to lists of dbs. This view would exist in the design doc of a special db named "_dbs" (or whatever).

I will send a mail asap to the dev mailing list presenting a partial solution and pointing out some technical issues, as well as to collect some feeback about them.

cheers

Bump to 1.3.x