CouchDB
  1. CouchDB
  2. COUCHDB-1259

Replication ID is not stable if local server has a dynamic port number

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 1.1
    • Fix Version/s: 1.3
    • Component/s: Replication
    • Labels:
      None
    • Skill Level:
      Dont Know

      Description

      I noticed that when Couchbase Mobile running on iOS replicates to/from a remote server (on iriscouch in this case), the replication has to fetch the full _changes feed every time it starts. Filipe helped me track down the problem – the replication ID is coming out different every time. The reason for this is that the local port number, which is one of the inputs to the hash that generates the replication ID, is randomly assigned by the OS. (I.e. it uses a port number of 0 when opening its listener socket.) This is because there could be multiple apps using Couchbase Mobile running on the same device and we can't have their ports colliding.

      The underlying problem is that CouchDB is attempting to generate a unique ID for a particular pair of

      {source, destination}

      databases, but it's basing it on attributes that aren't fundamental to the database and can change, like the hostname or port number.

      One solution, proposed by Filipe and me, is to assign each database (or each server?) a random UUID when it's created, and use that to generate replication IDs.

      Another solution, proposed by Damien, is to have CouchDB let the client work out the replication ID on its own, and set it as a property in the replication document (or the JSON body of a _replicate request.) This is even more flexible and will handle tricky scenarios like full P2P replication where there may be no low-level way to uniquely identify the remote database being synced with.

      1. couchdb-1259.patch
        4 kB
        Filipe Manana
      2. couchdb-1259.patch
        4 kB
        Filipe Manana

        Issue Links

          Activity

          Hide
          Filipe Manana added a comment -

          Thanks Jens for filing this and the detailed description.

          To be more clear, the idea would be to add a UUID to each server and use it as input to the replication id generation instead of the local port number. It would be something similar to what is done for the cookie authentication handler:

          https://github.com/apache/couchdb/blob/trunk/src/couchdb/couch_httpd_auth.erl#L240

          If such uuid doesn't exist in the .ini (replicator section), we generate a new uuid and save it.

          Damien's thought about allowing the client to name them, sounds also very simple - perhaps using the "_replication_id" field in replication documents (which already exists and is currently automatically set by the replication manager).

          Show
          Filipe Manana added a comment - Thanks Jens for filing this and the detailed description. To be more clear, the idea would be to add a UUID to each server and use it as input to the replication id generation instead of the local port number. It would be something similar to what is done for the cookie authentication handler: https://github.com/apache/couchdb/blob/trunk/src/couchdb/couch_httpd_auth.erl#L240 If such uuid doesn't exist in the .ini (replicator section), we generate a new uuid and save it. Damien's thought about allowing the client to name them, sounds also very simple - perhaps using the "_replication_id" field in replication documents (which already exists and is currently automatically set by the replication manager).
          Hide
          Randall Leeds added a comment -

          I'd rather see the replicator respect a naming field. CouchDB core places no specific significance on the replication documents, treating them as any other document in the _local/ namespace. And we've heard a number of times, especially in the last few weeks, about how config files (and specifically ones that change) are ugly.

          I proposed UUIDs at the DB level a long time ago for this reason, and relatedly so that you could trigger push/pull without using HTTP at both sides and have it be the same replication (discover that you are a host via UUID). Configuration files would work to make it server level, but it's hacky. DB-level is a bad idea because sysadmins might copy couch files. Ultimately, the client should identify the replication if it can. I think that's the best solution.

          Show
          Randall Leeds added a comment - I'd rather see the replicator respect a naming field. CouchDB core places no specific significance on the replication documents, treating them as any other document in the _local/ namespace. And we've heard a number of times, especially in the last few weeks, about how config files (and specifically ones that change) are ugly. I proposed UUIDs at the DB level a long time ago for this reason, and relatedly so that you could trigger push/pull without using HTTP at both sides and have it be the same replication (discover that you are a host via UUID). Configuration files would work to make it server level, but it's hacky. DB-level is a bad idea because sysadmins might copy couch files. Ultimately, the client should identify the replication if it can. I think that's the best solution.
          Hide
          Jason Smith added a comment -

          Couch already has a unique identifier: its URL. I'm not sure another per-server UUID buys you much. With a second unique identifier, you can determine that (this couch has moved on the Internet || this couch has a configuration error). Maybe the first condition is more likely. Meh.

          Couch already has a per-server UUID: _config/couch_httpd_auth/secret. Hashing this value can produce a public unique ID.

          Normal users may not read the config, so you have to expose this value to them somehow. Is that a new global handler? Or is it added to the

          {"couchdb":"Welcome"}

          response?

          For these reasons, I'm not sure the solution is to assign a random UUID to the Couch.

          Show
          Jason Smith added a comment - Couch already has a unique identifier: its URL. I'm not sure another per-server UUID buys you much. With a second unique identifier, you can determine that (this couch has moved on the Internet || this couch has a configuration error). Maybe the first condition is more likely. Meh. Couch already has a per-server UUID: _config/couch_httpd_auth/secret. Hashing this value can produce a public unique ID. Normal users may not read the config, so you have to expose this value to them somehow. Is that a new global handler? Or is it added to the {"couchdb":"Welcome"} response? For these reasons, I'm not sure the solution is to assign a random UUID to the Couch.
          Hide
          Jason Smith added a comment -

          The small problem with Damien's suggestion is that, presently, the client (and the replicator) hasn't enough information to produce a good replication ID.

          Randall, I'm not sure I understand what is bad about sysadmins copying files. Shouldn't that be exactly the situation where the replicator notices that it can skip some changes?

          If a sysadmin runs (and then restarts couch!)

          mv old.couch new.couch

          or perhaps

          rsync user@somewhere:/path/to/remote.couch ./new.couch

          and runs a replication, it would be nice for it to continue where it left off. Does that already happen? I think the answer is no: the replication ID depends on the database name. A server ID doesn't help to detect database renames. A DB rename is a URL "rename," no different from changing the port.

          Is it possible to address both bugs at the same time? Can the client (replicator) compute a stable replication ID, without any help from core CouchDB?

          In other discussions, Adam mentioned possibly demoting _security to be merely a cache of a _local/security doc (and promoting _local docs to have revision trees).

          Could the client (replicator) or perhaps Couch create e.g. _local/metadata with a UUID? To calculate the replication ID, check for _local/metadata, and if not available then fall back to the URL, database name, etc. I think that would require no changes to core CouchDB, just the replicator application. And clients (the one creating /_replicator docs) needn't change.

          Show
          Jason Smith added a comment - The small problem with Damien's suggestion is that, presently, the client (and the replicator) hasn't enough information to produce a good replication ID. Randall, I'm not sure I understand what is bad about sysadmins copying files. Shouldn't that be exactly the situation where the replicator notices that it can skip some changes? If a sysadmin runs (and then restarts couch!) mv old.couch new.couch or perhaps rsync user@somewhere:/path/to/remote.couch ./new.couch and runs a replication, it would be nice for it to continue where it left off. Does that already happen? I think the answer is no: the replication ID depends on the database name. A server ID doesn't help to detect database renames. A DB rename is a URL "rename," no different from changing the port. Is it possible to address both bugs at the same time? Can the client (replicator) compute a stable replication ID, without any help from core CouchDB? In other discussions, Adam mentioned possibly demoting _security to be merely a cache of a _local/security doc (and promoting _local docs to have revision trees). Could the client (replicator) or perhaps Couch create e.g. _local/metadata with a UUID? To calculate the replication ID, check for _local/metadata, and if not available then fall back to the URL, database name, etc. I think that would require no changes to core CouchDB, just the replicator application. And clients (the one creating /_replicator docs) needn't change.
          Hide
          Paul Joseph Davis added a comment -

          @Jason

          URL's are nice and all, but they're fairly unstable. What's the URL for a couch on a phone? Or after it changes networks? Or if its behind a proxy? And how would that couch figure out what's URL is?

          Randall's comment about copying files is why its not viable to just generate a UUID for each database. Because its quite likely that a sysadmin would copy that file which would result in to db's having the same UUID making the UUID not so UI.

          Show
          Paul Joseph Davis added a comment - @Jason URL's are nice and all, but they're fairly unstable. What's the URL for a couch on a phone? Or after it changes networks? Or if its behind a proxy? And how would that couch figure out what's URL is? Randall's comment about copying files is why its not viable to just generate a UUID for each database. Because its quite likely that a sysadmin would copy that file which would result in to db's having the same UUID making the UUID not so UI.
          Hide
          Jens Alfke added a comment -

          @Jason: The reason I filed this bug report is that the URL of a database in Couchbase Mobile isn't unique. It's barely meaningful at all; it's of the form "http://127.0.0.1:nnnnn" where "nnnnn" is an upredictable port number assigned by the TCP stack at launch time. That URL isn't even exposed to the outside world because the CouchDB server is only listening on the loopback interface.

          The state that's being represented by a replication ID is the contents of the database (at least as it was when it last synced.) So it seems the ID should be something that sticks to the database itself, not to any ephemeral manifestation like a URL.

          Show
          Jens Alfke added a comment - @Jason: The reason I filed this bug report is that the URL of a database in Couchbase Mobile isn't unique. It's barely meaningful at all; it's of the form "http://127.0.0.1:nnnnn" where "nnnnn" is an upredictable port number assigned by the TCP stack at launch time. That URL isn't even exposed to the outside world because the CouchDB server is only listening on the loopback interface. The state that's being represented by a replication ID is the contents of the database (at least as it was when it last synced.) So it seems the ID should be something that sticks to the database itself, not to any ephemeral manifestation like a URL.
          Hide
          Jason Smith added a comment -

          @Paul, I overlooked or misunderstood Randal's point about duplicating the UUID. Makes sense.

          @Paul and Jens, totally: URLs are unstable. And, in general, on the web, if a URL changes, that is a huge piece of information. That is a huge hint that you need to re-evaluate your assumptions. Exhibit A: the replicator starts from scratch if you give it a new URL. That is the correct general solution.

          Your requirement to change the couch URL and everything is just fine and the application doesn't have to worry--I think that requirement is asking CouchDB to be a bad web citizen. In other words, IMO you have an application-level problem, not a couch problem, except that even if you could determine everything is alright, you can't specify the replication ID.

          I think Damien just time-traveled again.

          BTW, @Jens, you can already give Apache CouchDB a UUID for your own needs. If you have admin access, just put it anywhere, /_config/jens/uuid. If an unprivileged client must know this UUID, then change the "Welcome" mesage in _config/httpd_global_handlers/%2f. Any unprivileged client can find it there. (You could also place the UUID in the authentication realm in _config/httpd/WWW-Authenticate which actually sounds sort of appropriate.) Finally, if you are willing to run a fork (which mobile Couchbase is) then you could add any feature you need which doesn't make sense for Apache CouchDB.

          Show
          Jason Smith added a comment - @Paul, I overlooked or misunderstood Randal's point about duplicating the UUID. Makes sense. @Paul and Jens, totally: URLs are unstable. And, in general, on the web, if a URL changes, that is a huge piece of information. That is a huge hint that you need to re-evaluate your assumptions. Exhibit A: the replicator starts from scratch if you give it a new URL. That is the correct general solution. Your requirement to change the couch URL and everything is just fine and the application doesn't have to worry--I think that requirement is asking CouchDB to be a bad web citizen. In other words, IMO you have an application-level problem, not a couch problem, except that even if you could determine everything is alright, you can't specify the replication ID. I think Damien just time-traveled again. BTW, @Jens, you can already give Apache CouchDB a UUID for your own needs. If you have admin access, just put it anywhere, /_config/jens/uuid. If an unprivileged client must know this UUID, then change the "Welcome" mesage in _config/httpd_global_handlers/%2f. Any unprivileged client can find it there. (You could also place the UUID in the authentication realm in _config/httpd/WWW-Authenticate which actually sounds sort of appropriate.) Finally, if you are willing to run a fork (which mobile Couchbase is) then you could add any feature you need which doesn't make sense for Apache CouchDB.
          Hide
          Jan Lehnardt added a comment -

          link relevant older ticket

          Show
          Jan Lehnardt added a comment - link relevant older ticket
          Hide
          Jens Alfke added a comment -

          @Jason — I see where you're coming from with "bad web citizen" but it refers only to a specific usage of CouchDB as a traditional web server. We have equally valid uses of it that do not act as traditional servers nor have stable URLs, for which we absolutely need this fix. I don't think arguments about whether URLs change or not are productive here in a specific bug report.

          Show
          Jens Alfke added a comment - @Jason — I see where you're coming from with "bad web citizen" but it refers only to a specific usage of CouchDB as a traditional web server. We have equally valid uses of it that do not act as traditional servers nor have stable URLs, for which we absolutely need this fix. I don't think arguments about whether URLs change or not are productive here in a specific bug report.
          Hide
          Jason Smith added a comment -

          Hi, Jens. Yes, I take your point to stick to the specific topic. Thanks.

          To summarize my long rants: A Couch UUID already exists. A database UUID won't work. Allowing the client to provide the replication ID would be great!

          Show
          Jason Smith added a comment - Hi, Jens. Yes, I take your point to stick to the specific topic. Thanks. To summarize my long rants: A Couch UUID already exists. A database UUID won't work. Allowing the client to provide the replication ID would be great!
          Hide
          Filipe Manana added a comment -

          The following patch solves the issue by using the same approach as the auth cookie handler - using a node uuid which is stored in .ini config.

          The new replication ID generation no longer uses the hostname and port pair, but instead this uuid. It fallbacks to the current method when searching for existing checkpoints however (thanks to Randall's replication ID upgrade scheme).

          Besides mobile, desktop couch (Ubuntu One) is another example where the port is dynamic. I've also seen a desktop running normal CouchDB where the ID for a replication was different the first time it was generated from subsequent times - I suspect it was an issue with inet:gethostname/0 but unable to prove it.

          Personally I don't think using the .ini for a single parameter is that bad (unlike things like OAuth tokens for e.g.) like others commented here before - an alternative would be to store it in a local document of the replicator database for e.g.

          I would very much like to see this issue fixed in 1.2.0.

          Show
          Filipe Manana added a comment - The following patch solves the issue by using the same approach as the auth cookie handler - using a node uuid which is stored in .ini config. The new replication ID generation no longer uses the hostname and port pair, but instead this uuid. It fallbacks to the current method when searching for existing checkpoints however (thanks to Randall's replication ID upgrade scheme). Besides mobile, desktop couch (Ubuntu One) is another example where the port is dynamic. I've also seen a desktop running normal CouchDB where the ID for a replication was different the first time it was generated from subsequent times - I suspect it was an issue with inet:gethostname/0 but unable to prove it. Personally I don't think using the .ini for a single parameter is that bad (unlike things like OAuth tokens for e.g.) like others commented here before - an alternative would be to store it in a local document of the replicator database for e.g. I would very much like to see this issue fixed in 1.2.0.
          Hide
          Robert Newson added a comment -

          I've picked up this task and prepared a branch with my work (1259-stable_replication_ids). This patch goes beyond filipe's original and applies to the source and target as well. If couchdb believes either is using a dynamic port (it's configurable, but defaults to any port in the 14192-65535 range), it will ask the server for its uuid (emitted in the / welcome message). If it has one, it uses that instead of the port (specifically it uses

          {Scheme, UserInfo, UUID, Path}

          instead of the full url).

          Show
          Robert Newson added a comment - I've picked up this task and prepared a branch with my work (1259-stable_replication_ids). This patch goes beyond filipe's original and applies to the source and target as well. If couchdb believes either is using a dynamic port (it's configurable, but defaults to any port in the 14192-65535 range), it will ask the server for its uuid (emitted in the / welcome message). If it has one, it uses that instead of the port (specifically it uses {Scheme, UserInfo, UUID, Path} instead of the full url).
          Hide
          Benoit Chesneau added a comment -

          I was reading the comment, and not sure it's a probem. Irt's expected in a p2p world and master-master replication that the node at the end could change. What doesn't changes are the data inside the dbs. Imo tthis is the role of the application to handle port,ips, dns changes. Not couchdb. In short I would close this issue as a wontfix.

          Show
          Benoit Chesneau added a comment - I was reading the comment, and not sure it's a probem. Irt's expected in a p2p world and master-master replication that the node at the end could change. What doesn't changes are the data inside the dbs. Imo tthis is the role of the application to handle port,ips, dns changes. Not couchdb. In short I would close this issue as a wontfix.
          Hide
          Jens Alfke added a comment -

          Benoit: I think you're misunderstanding the issue. This isn't something about P2P. It's just that if the local CouchDB is not listening on a fixed port number, then replications made by that server to/from another server aren't handled efficiently ... even though the local server's port number has nothing at all to do with the replication (since it's the one making the connections.)

          In a real P2P case, this change makes even more sense, because the addresses of the servers are unimportant – as you said, it's the databases and their data that are the important thing. A UUID helps identify those.

          Show
          Jens Alfke added a comment - Benoit: I think you're misunderstanding the issue. This isn't something about P2P. It's just that if the local CouchDB is not listening on a fixed port number, then replications made by that server to/from another server aren't handled efficiently ... even though the local server's port number has nothing at all to do with the replication (since it's the one making the connections.) In a real P2P case, this change makes even more sense, because the addresses of the servers are unimportant – as you said, it's the databases and their data that are the important thing. A UUID helps identify those.
          Hide
          Benoit Chesneau added a comment -

          I don't. The replication is fundamentally a peer to peer protocol. or master to master. whatever.

          It is and should be expected that a node can disappear for many reasons. Relying on a server id is one of the possibilities to restart automatically a replication, other would use other parameters based on the ip and the location, etc... It is the responsibility of the application to know that. As a protocol the replication shouldn't force this way imo. As a client the replicator shouldn't too. It is already doing the best effort imo: ie restarting for the same ip, port and properties of the replication.

          As a side note, this is one of the reason the 0 port could be used here instead of letting your application fix a port. 0 won't change. And you could rely with the application on a unique property in the replication properties.

          Show
          Benoit Chesneau added a comment - I don't. The replication is fundamentally a peer to peer protocol. or master to master. whatever. It is and should be expected that a node can disappear for many reasons. Relying on a server id is one of the possibilities to restart automatically a replication, other would use other parameters based on the ip and the location, etc... It is the responsibility of the application to know that. As a protocol the replication shouldn't force this way imo. As a client the replicator shouldn't too. It is already doing the best effort imo: ie restarting for the same ip, port and properties of the replication. As a side note, this is one of the reason the 0 port could be used here instead of letting your application fix a port. 0 won't change. And you could rely with the application on a unique property in the replication properties.
          Hide
          Jens Alfke added a comment -

          > It is the responsibility of the application to know that. As a protocol the replication shouldn't force this way imo.

          Then how does the application do this? I haven't seen any API for it.

          Also, I don't see how this has anything to do with the case of a leaf node running a server that happens to have a dynamic port assignment. The port this node is running on has absolutely nothing to do with the replication. In the (now obsolete) case of Couchbase Mobile, the server doesn't even accept external requests, so its port number is purely an internal affair.

          I still have a feeling that we're talking about completely different things. But I can't really figure out what your point is...

          Show
          Jens Alfke added a comment - > It is the responsibility of the application to know that. As a protocol the replication shouldn't force this way imo. Then how does the application do this? I haven't seen any API for it. Also, I don't see how this has anything to do with the case of a leaf node running a server that happens to have a dynamic port assignment. The port this node is running on has absolutely nothing to do with the replication. In the (now obsolete) case of Couchbase Mobile, the server doesn't even accept external requests, so its port number is purely an internal affair. I still have a feeling that we're talking about completely different things. But I can't really figure out what your point is...
          Hide
          Benoit Chesneau added a comment - - edited

          We don't speak about different things. We are expecting different behaviours however.

          If you have a dynamic port assignment then how knows the other node the new port ? There are no other way than a ping-pong here. Ping-Pong that can be done in different manner. To know this new port you will have to update the replicator doc. Or the replication client could check on an adress table etc... In any case there will be a dialog that can be different across applications. And this is the point, this part is generally the logic of the application or the application that handle the routing policy. A server id can be used as an adress point but some could decide to use other parameters to associate this replication id to a node.

          Maybe instead of relying on a fixed node id, we could however introduce an arbitrary remote address id fixed on the node that handles the replication. This remote ID will be associated by to an host , port. The layer assigning this address id to the host/port could be switchable, so the application or user could introduce easily its own routing policy. Which could be relying on a server id or not. Eventually if there is an interrest like we do for the auth handlers or uuid generators, we could have different implementations shipped with couchdb. One for example relying on a server id.

          btw your example with couchbase mobile is generally solved by using the replication in pull mode only. So here it is relying on a fixed address to replicate. A central node is then used as a meeting point for all the data coming from other nodes and the replication is filtered per nodes.

          Show
          Benoit Chesneau added a comment - - edited We don't speak about different things. We are expecting different behaviours however. If you have a dynamic port assignment then how knows the other node the new port ? There are no other way than a ping-pong here. Ping-Pong that can be done in different manner. To know this new port you will have to update the replicator doc. Or the replication client could check on an adress table etc... In any case there will be a dialog that can be different across applications. And this is the point, this part is generally the logic of the application or the application that handle the routing policy. A server id can be used as an adress point but some could decide to use other parameters to associate this replication id to a node. Maybe instead of relying on a fixed node id, we could however introduce an arbitrary remote address id fixed on the node that handles the replication. This remote ID will be associated by to an host , port. The layer assigning this address id to the host/port could be switchable, so the application or user could introduce easily its own routing policy. Which could be relying on a server id or not. Eventually if there is an interrest like we do for the auth handlers or uuid generators, we could have different implementations shipped with couchdb. One for example relying on a server id. btw your example with couchbase mobile is generally solved by using the replication in pull mode only. So here it is relying on a fixed address to replicate. A central node is then used as a meeting point for all the data coming from other nodes and the replication is filtered per nodes.
          Hide
          Jens Alfke added a comment -

          > btw your example with couchbase mobile is generally solved by using the replication in pull mode only. So here it is relying on a fixed address to replicate.

          sigh No, that is exactly the situation I was describing. The mobile client is the only one initiating replication; it pulls from the central (fixed-address) server, and pushes changes to it. So the mobile device's IP address and port are irrelevant, right? Except that the replication state document stored in local has an ID based on several things _including the local server's address and port number. So the effect is that, every time the app launches, all the replication state gets lost/invalidated, and it has to start over again the next time it replicates.

          TouchDB doesn't have this problem because I didn't write it with this design flaw Instead every local database has a UUID as suggested here, and that's used as part of the key.

          Show
          Jens Alfke added a comment - > btw your example with couchbase mobile is generally solved by using the replication in pull mode only. So here it is relying on a fixed address to replicate. sigh No, that is exactly the situation I was describing. The mobile client is the only one initiating replication; it pulls from the central (fixed-address) server, and pushes changes to it. So the mobile device's IP address and port are irrelevant, right? Except that the replication state document stored in local has an ID based on several things _including the local server's address and port number. So the effect is that, every time the app launches, all the replication state gets lost/invalidated, and it has to start over again the next time it replicates. TouchDB doesn't have this problem because I didn't write it with this design flaw Instead every local database has a UUID as suggested here, and that's used as part of the key.
          Hide
          Benoit Chesneau added a comment -

          Except if you are using either the same port on each devices (which is generally what does an application) or the ephemeral port "0" which is also the same then for reach replication. Also in that case you will also have to handle the full address change and security implications. Relying on a unique ids to continue the replication may be a design flaw or at least a non expected/wanted behaviour. What will prevents an hostile node to connect back to your node with the same id? How do you invalidate it?

          I am pretty sure anyway we should let to the application or final user the choice of the routing policy. I will have a patch for that later in the day.

          Show
          Benoit Chesneau added a comment - Except if you are using either the same port on each devices (which is generally what does an application) or the ephemeral port "0" which is also the same then for reach replication. Also in that case you will also have to handle the full address change and security implications. Relying on a unique ids to continue the replication may be a design flaw or at least a non expected/wanted behaviour. What will prevents an hostile node to connect back to your node with the same id? How do you invalidate it? I am pretty sure anyway we should let to the application or final user the choice of the routing policy. I will have a patch for that later in the day.
          Hide
          Jens Alfke added a comment -

          It seems like overkill to get the IANA to assign a fixed port number to an app that doesn't even listen on any external interfaces! The only use of that port is (was) over the loopback interface to let the application communicate with CouchDB.

          Passing zero for the port in the config file didn't make the problem go away. Apparently the replicator bases the ID on the actual random port number in use, not on the fixed 0 from the config.

          > What will prevents an hostile node to connect back to your node with the same id?

          Hello, are you listening at all to what I'm writing? I've already said several times that the app does not accept incoming connections at all. It only makes outgoing connections to replicate.

          And in general: obviously in any real P2P app there would be actual security measures in place to authenticate connections, most likely by using both server and client SSL certs and verifying their public keys. Once the connection is made, then database IDs can be used to restore the state of a replication.

          Show
          Jens Alfke added a comment - It seems like overkill to get the IANA to assign a fixed port number to an app that doesn't even listen on any external interfaces! The only use of that port is (was) over the loopback interface to let the application communicate with CouchDB. Passing zero for the port in the config file didn't make the problem go away. Apparently the replicator bases the ID on the actual random port number in use, not on the fixed 0 from the config. > What will prevents an hostile node to connect back to your node with the same id? Hello, are you listening at all to what I'm writing? I've already said several times that the app does not accept incoming connections at all. It only makes outgoing connections to replicate. And in general: obviously in any real P2P app there would be actual security measures in place to authenticate connections, most likely by using both server and client SSL certs and verifying their public keys. Once the connection is made, then database IDs can be used to restore the state of a replication.
          Hide
          Benoit Chesneau added a comment -

          Hello.... are you understanding that this isn't only about your application? Some may have different uses. And different routing policy. And this is not the role of couchdb to fix them.

          Show
          Benoit Chesneau added a comment - Hello.... are you understanding that this isn't only about your application? Some may have different uses. And different routing policy. And this is not the role of couchdb to fix them.
          Hide
          Dustin Sallings added a comment -

          If this is true, it explains why someone pointed me to this bug for the two issues I've been experiencing with replication:

          1. My permanent replication sessions die almost every single day and come back after I restart CouchDB.
          2. Sometimes I end up with more than one replication running for the same configured replica.

          Note: My local address changes at least twice a day when I go into the office. The other end of my replication doesn't. It's hard to argue that this behavior is desirable.

          Show
          Dustin Sallings added a comment - If this is true, it explains why someone pointed me to this bug for the two issues I've been experiencing with replication: 1. My permanent replication sessions die almost every single day and come back after I restart CouchDB. 2. Sometimes I end up with more than one replication running for the same configured replica. Note: My local address changes at least twice a day when I go into the office. The other end of my replication doesn't. It's hard to argue that this behavior is desirable.
          Hide
          Robert Newson added a comment -

          Benoit, are you vetoing this change? If so, please include a reason why improving the hit rate for replication checkpoints should not be included in our next release.

          Everyone else, please comment on the patch itself, I want feedback on 1) the quality and correctness of the patch and 2) whether, when using UUID, whether it is correct to use UUID instead of the

          {Scheme, UserInfo, UUID, Path}

          tuple that I've chosen as I would prefer to simply use the UUID if it's correct to do so for clarity.

          https://github.com/apache/couchdb/pull/35

          Show
          Robert Newson added a comment - Benoit, are you vetoing this change? If so, please include a reason why improving the hit rate for replication checkpoints should not be included in our next release. Everyone else, please comment on the patch itself, I want feedback on 1) the quality and correctness of the patch and 2) whether, when using UUID, whether it is correct to use UUID instead of the {Scheme, UserInfo, UUID, Path} tuple that I've chosen as I would prefer to simply use the UUID if it's correct to do so for clarity. https://github.com/apache/couchdb/pull/35
          Hide
          Benoit Chesneau added a comment -

          @rnewson I'm confused. How does it improve the hit rate fro replications checkpoints ????

          I am -1 on this patch for above reason. Which are likely the first the one i gave when saying won't fix. Changing a port or an IP isn't an innocent event. It has many implications.

          And such things like using a fixed replication id remove any of its implication and will make some app work difficult. What if the node id is for ex anonymized on each restart ? I think such behaviour should be configurable.

          Anyway I don't think this ticket should be a blocker. Let discuss it quitely for the 1.4 .

          Show
          Benoit Chesneau added a comment - @rnewson I'm confused. How does it improve the hit rate fro replications checkpoints ???? I am -1 on this patch for above reason. Which are likely the first the one i gave when saying won't fix. Changing a port or an IP isn't an innocent event. It has many implications. And such things like using a fixed replication id remove any of its implication and will make some app work difficult. What if the node id is for ex anonymized on each restart ? I think such behaviour should be configurable. Anyway I don't think this ticket should be a blocker. Let discuss it quitely for the 1.4 .
          Hide
          Robert Newson added a comment -

          Benoit, the only intention of the patch is to improve the hit rate for replication checkpoint documents. If the port of a participant changes, the current replicator will replicate from update_seq 0 because it won't find the checkpoint from the previous port. With this change, the replicator can negotiate a stable value (the UUID) to use in place of the unstable value (port), and thus find a valid checkpoint document. For large databases, this can be hugely valuable. If you are saying that this breaks replication or eventual consistency, please say so and explain how. The only thing this patch should do is prevent needless and time-consuming replication when a shortcut is available. If you feel it doesn't do that, please help me to see why.

          This ticket is over a year old, I do not want to bump it to 1.4 without a good reason.

          Show
          Robert Newson added a comment - Benoit, the only intention of the patch is to improve the hit rate for replication checkpoint documents. If the port of a participant changes, the current replicator will replicate from update_seq 0 because it won't find the checkpoint from the previous port. With this change, the replicator can negotiate a stable value (the UUID) to use in place of the unstable value (port), and thus find a valid checkpoint document. For large databases, this can be hugely valuable. If you are saying that this breaks replication or eventual consistency, please say so and explain how. The only thing this patch should do is prevent needless and time-consuming replication when a shortcut is available. If you feel it doesn't do that, please help me to see why. This ticket is over a year old, I do not want to bump it to 1.4 without a good reason.
          Hide
          Benoit Chesneau added a comment -

          So i will repeat myself here:

          • changing a port is not an innocent event. With this patch the replication is just ignoring it. And i currently asking myself who take the responsibility to make sure that this replication can still happen with this environment change. Changing a port changes the condition in which your node is acting.

          This patch can only work in a trusted environment. Which can be ok but *must* be optional imo.

          Show
          Benoit Chesneau added a comment - So i will repeat myself here: changing a port is not an innocent event. With this patch the replication is just ignoring it. And i currently asking myself who take the responsibility to make sure that this replication can still happen with this environment change. Changing a port changes the condition in which your node is acting. This patch can only work in a trusted environment. Which can be ok but * must * be optional imo.
          Hide
          Robert Newson added a comment -

          Sorry, Benoit, I simply can't see the security problem you are trying to describe at all. Walk me through a process where a server changes ports and security is violated. As far as I can tell, the only difference this patch makes is that we can resume replication from a checkpoint where we previously couldn't. That a server changes port doesn't invalidate the checkpoint's integrity, nor do I trust the server more or less based on its numeric port value. If I want to trust the machine, I need TLS, which is completely orthogonal to this ticket.

          Show
          Robert Newson added a comment - Sorry, Benoit, I simply can't see the security problem you are trying to describe at all. Walk me through a process where a server changes ports and security is violated. As far as I can tell, the only difference this patch makes is that we can resume replication from a checkpoint where we previously couldn't. That a server changes port doesn't invalidate the checkpoint's integrity, nor do I trust the server more or less based on its numeric port value. If I want to trust the machine, I need TLS, which is completely orthogonal to this ticket.
          Hide
          Benoit Chesneau added a comment -

          @rnewson part of the patch that make the checkpoint locally constant is fine for me (but not relying on a remote uuid) though i don't think we need a server id for that but that can probably be changed later.

          Anyway following the discussion (and that will be post 1.3) I think that what we should really do here is handling the following scenario:

          1. port, ip changed, port crashed, restared -> a checkpooint is orphelin
          2. node come back
          3. Instead of restarting the replication , ask what to do and mark the replication task as paused

          By "ask" I mean having a switchable middle ware that could handle the scenario or just let the application do what it want. Since that is a braking change I guess it could only happen on 2.0 if some want.

          Show
          Benoit Chesneau added a comment - @rnewson part of the patch that make the checkpoint locally constant is fine for me (but not relying on a remote uuid) though i don't think we need a server id for that but that can probably be changed later. Anyway following the discussion (and that will be post 1.3) I think that what we should really do here is handling the following scenario: 1. port, ip changed, port crashed, restared -> a checkpooint is orphelin 2. node come back 3. Instead of restarting the replication , ask what to do and mark the replication task as paused By "ask" I mean having a switchable middle ware that could handle the scenario or just let the application do what it want. Since that is a braking change I guess it could only happen on 2.0 if some want.
          Hide
          Robert Newson added a comment -

          I'll summarize my reasons for not finding a technical justification in your veto as follows: "We do not trust the host:port pair today, we will not trust the uuid substitute with this patch, no security aspect has changed". Your argument implies that seeing foo:5984 now, and seeing foo:5984 five minutes from now means that you are talking to the same machine. This is obviously false. if we are talking to https://foo:5984, and we trust the certificate now, and five minutes from now, then we are sure we are talking to the same machine.

          All this patch does is give a stable identifier to a couchdb server. To date, we have assumed that full-qualified hostname and port number is a stable identifier. It is, up to a point.

          So, I would like comments on whether the patch does what it claims, and on the separate point about clarifying the meaning of the patch by using UUID in place of Host and Port in the replication_id calculations.

          I will also note that, at your step 3, replication could not resume, since the node could not be contacted on the old port. A new replication, with the new port, would start up, and I see no reason why it should redo from start.

          All this said, I think we need other voices on this ticket now.

          Show
          Robert Newson added a comment - I'll summarize my reasons for not finding a technical justification in your veto as follows: "We do not trust the host:port pair today, we will not trust the uuid substitute with this patch, no security aspect has changed". Your argument implies that seeing foo:5984 now, and seeing foo:5984 five minutes from now means that you are talking to the same machine. This is obviously false. if we are talking to https://foo:5984 , and we trust the certificate now, and five minutes from now, then we are sure we are talking to the same machine. All this patch does is give a stable identifier to a couchdb server. To date, we have assumed that full-qualified hostname and port number is a stable identifier. It is, up to a point. So, I would like comments on whether the patch does what it claims, and on the separate point about clarifying the meaning of the patch by using UUID in place of Host and Port in the replication_id calculations. I will also note that, at your step 3, replication could not resume, since the node could not be contacted on the old port. A new replication, with the new port, would start up, and I see no reason why it should redo from start. All this said, I think we need other voices on this ticket now.
          Hide
          Robert Newson added a comment -

          I think

          {Scheme, UserInfo, UUID, Path}

          is necessary, actually, so ignore my second question. Scheme is justified as we should allow a http and https replication that is otherwise identical to run concurrently (though it's a bit silly). UserInfo is mixed in because different users might be able to write only subsets of the data, and Path ensures that checkpoints vary by database name, in the case that multiple sources replicate into the same target (confusing their checkpoints would be very wrong).

          Show
          Robert Newson added a comment - I think {Scheme, UserInfo, UUID, Path} is necessary, actually, so ignore my second question. Scheme is justified as we should allow a http and https replication that is otherwise identical to run concurrently (though it's a bit silly). UserInfo is mixed in because different users might be able to write only subsets of the data, and Path ensures that checkpoints vary by database name, in the case that multiple sources replicate into the same target (confusing their checkpoints would be very wrong).
          Hide
          Dave Cottlehuber added a comment -

          Jens, will the patch address your issue?

          Overall I'm +1 on this approach for enabling faster restarts of replication – I think it's a huge win.

          I don't see that the behaviour of the new patch changes the security constraints vs today, but I think I see Benoit's point. Today if a replication endpoint changes its ephemeral port # (e.g. expired DHCP lease), the replication will fail and cannot restart until it is deleted & recreated.

          With the patch, the replication could restart in some situations, without requiring active intervention - that's the whole point.

          So if Dr.Evil has captured the UUID, it might be possible to acquire the replication without the source endpoint being aware. I think this should be addressed post 1.3. The proposed functionality could note that securing replication requires using TLS and appropriate SSL cert checking in both directions. Which seems common sense anyway! The Dr Evil scenario however is no different under today's activity - if an IP address is hijacked and SSL is not in use, Dr Evil has your documents.

          Show
          Dave Cottlehuber added a comment - Jens, will the patch address your issue? Overall I'm +1 on this approach for enabling faster restarts of replication – I think it's a huge win. I don't see that the behaviour of the new patch changes the security constraints vs today, but I think I see Benoit's point. Today if a replication endpoint changes its ephemeral port # (e.g. expired DHCP lease), the replication will fail and cannot restart until it is deleted & recreated. With the patch, the replication could restart in some situations, without requiring active intervention - that's the whole point. So if Dr.Evil has captured the UUID, it might be possible to acquire the replication without the source endpoint being aware. I think this should be addressed post 1.3. The proposed functionality could note that securing replication requires using TLS and appropriate SSL cert checking in both directions. Which seems common sense anyway! The Dr Evil scenario however is no different under today's activity - if an IP address is hijacked and SSL is not in use, Dr Evil has your documents.
          Hide
          Robert Newson added a comment -

          Dave, if the port of source or target changes, then, no, the replication cannot restart (because the replication task will be trying to contact a server on a port it is not listening on). If a new task is started pointing at the new port, it will simply pick up a valid checkpoint that it otherwise wouldn't.

          Show
          Robert Newson added a comment - Dave, if the port of source or target changes, then, no, the replication cannot restart (because the replication task will be trying to contact a server on a port it is not listening on). If a new task is started pointing at the new port, it will simply pick up a valid checkpoint that it otherwise wouldn't.
          Hide
          Benoit Chesneau added a comment -

          as an alternive solution I think that we could let the user fix the replication id by reusing the current _replicator id or the one used for the _replicate API.

          The only problem I see with that is when the the user will post 2 tasks doing the same thing.

          Show
          Benoit Chesneau added a comment - as an alternive solution I think that we could let the user fix the replication id by reusing the current _replicator id or the one used for the _replicate API. The only problem I see with that is when the the user will post 2 tasks doing the same thing.
          Hide
          Jens Alfke added a comment -

          I have looked at the patch but I don't really understand what it's doing, both because my Erlang is really weak and because I don't know the internals of CouchDB. So I can't really comment on the code.

          It does sound like what's being suggested goes beyond what I asked for. This bug is about the local server (the one running the replication) having a different IP address or port than the last time. The suggested patches seem to also cover changes to the remote server's URL. That's an interesting issue but IMHO not the same thing.

          The point of this bug is that the URL of the local server running the replication is irrelevant to the replication. If I'm opening connections to another server to replicate with it, it doesn't matter what port or IP address I am listening on, because there aren't any incoming connections happening. They don't affect the replication at all.

          As for Benoit's security issues: Replication has no security. Security applies at a more fundamental level of identifying who is connecting and authenticating that principal. You absolutely cannot make security tests based on IP addresses or port numbers.

          Show
          Jens Alfke added a comment - I have looked at the patch but I don't really understand what it's doing, both because my Erlang is really weak and because I don't know the internals of CouchDB. So I can't really comment on the code. It does sound like what's being suggested goes beyond what I asked for. This bug is about the local server (the one running the replication) having a different IP address or port than the last time. The suggested patches seem to also cover changes to the remote server's URL. That's an interesting issue but IMHO not the same thing. The point of this bug is that the URL of the local server running the replication is irrelevant to the replication. If I'm opening connections to another server to replicate with it, it doesn't matter what port or IP address I am listening on, because there aren't any incoming connections happening. They don't affect the replication at all. As for Benoit's security issues: Replication has no security. Security applies at a more fundamental level of identifying who is connecting and authenticating that principal. You absolutely cannot make security tests based on IP addresses or port numbers.
          Hide
          Robert Newson added a comment -

          Jens, Yes, the patch goes further than the ticket, I said as much in my first comment when I took the ticket. As you note, it has no security implications.

          There are three host:port values in play for any one replication task, it seems only a partial solution to fix the stability issue for one of them (though, I agree, that if we only fixed one, it would be the co-ordinating node).

          It is true that the host:port of the co-ordinating node does not affect the replication per se, as long as you ignore what would go wrong if two processes were doing the same replication. This is also true of the host:port of the "source" and "target" servers too.

          I am happy to solve just the initial problem identified in the ticket if that will allow Benoit to retract his veto, however I felt it important that we were all clear about the security implications here (namely, that there are none) before proceeding. If, as seems the case now, we all agree on the security aspect, I don't see the harm in all 3 participants having a stable identifier allowing replication checkpoints to be used if a machine changes name or port.

          Show
          Robert Newson added a comment - Jens, Yes, the patch goes further than the ticket, I said as much in my first comment when I took the ticket. As you note, it has no security implications. There are three host:port values in play for any one replication task, it seems only a partial solution to fix the stability issue for one of them (though, I agree, that if we only fixed one, it would be the co-ordinating node). It is true that the host:port of the co-ordinating node does not affect the replication per se, as long as you ignore what would go wrong if two processes were doing the same replication. This is also true of the host:port of the "source" and "target" servers too. I am happy to solve just the initial problem identified in the ticket if that will allow Benoit to retract his veto, however I felt it important that we were all clear about the security implications here (namely, that there are none) before proceeding. If, as seems the case now, we all agree on the security aspect, I don't see the harm in all 3 participants having a stable identifier allowing replication checkpoints to be used if a machine changes name or port.
          Hide
          Benoit Chesneau added a comment - - edited

          There is a potential security issue using a remote node id like in the second part of the patch. Local to local there is none.

          I am + 1 for fixing the issue related to the node doing coordination.

          Show
          Benoit Chesneau added a comment - - edited There is a potential security issue using a remote node id like in the second part of the patch. Local to local there is none. I am + 1 for fixing the issue related to the node doing coordination.
          Hide
          Robert Newson added a comment -

          The basic version is committed (where we use UUID instead of the local hostname:port).

          Show
          Robert Newson added a comment - The basic version is committed (where we use UUID instead of the local hostname:port).

            People

            • Assignee:
              Robert Newson
              Reporter:
              Jens Alfke
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development