CouchDB
  1. CouchDB
  2. COUCHDB-465

Produce sequential, but unique, document id's

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.11
    • Component/s: None
    • Labels:
      None

      Description

      Currently, if the client does not specify an id (POST'ing a single document or using _bulk_docs) a random 16 byte value is created. This kind of key is particularly brutal on b+tree updates and the append-only nature of couchdb files.

      Attached is a patch to change this to a two-part identifier. The first part is a random 12 byte value and the remainder is a counter. The random prefix is rerandomized when the counter reaches its maximum. The rollover in the patch is at 16 million but can obviously be changed. The upshot is that the b+tree is updated in a better fashion, which should lead to performance benefits.

      1. uuid_generator.patch
        5 kB
        Robert Newson
      2. couch_uuids.patch
        18 kB
        Paul Joseph Davis
      3. 041-uuid-gen-utc.ini
        0.0 kB
        Bob Dionne
      4. 041-uuid-gen-seq.ini
        0.0 kB
        Bob Dionne
      5. 041-uuid-gen.t
        4 kB
        Bob Dionne

        Activity

        Hide
        Adam Kocoloski added a comment -

        Nice work, Robert! I'm +1 on this patch.

        One concern is the guessability of IDs, but if users are really concerned about that they can always generate their own.

        Show
        Adam Kocoloski added a comment - Nice work, Robert! I'm +1 on this patch. One concern is the guessability of IDs, but if users are really concerned about that they can always generate their own.
        Hide
        Robert Newson added a comment -

        Thanks!

        Guessability is a concern, which means this might need to be switchable. Perhaps couch_seq_generator becomes couch_id_generator and an ini file chooses between the two strategies, defaulting to the safest, but worst-case, new_uuid behavior. To get good keys for b+tree insertion necessarily makes them more guessable as they'd have to be close to existing keys by design.

        I do owe some quantitative benchmarking to support the assertions in the description. I did a 10k insertion test with a small document,

        {content: "hello"}

        , and average insertion rate per document was 2ms with random and 1ms with the patch. This was more to prove that I'd changed something rather than a measure of the actual improvement. I would expect to see improved insertion rates across a lot of scenarios, less difference between uncompacted and compacted size (barring document updates and deletes) as less of the b+tree is rewritten, and a smaller post-compaction size vs random. The exact extent of these improvements should be established by a decent benchmark.

        Show
        Robert Newson added a comment - Thanks! Guessability is a concern, which means this might need to be switchable. Perhaps couch_seq_generator becomes couch_id_generator and an ini file chooses between the two strategies, defaulting to the safest, but worst-case, new_uuid behavior. To get good keys for b+tree insertion necessarily makes them more guessable as they'd have to be close to existing keys by design. I do owe some quantitative benchmarking to support the assertions in the description. I did a 10k insertion test with a small document, {content: "hello"} , and average insertion rate per document was 2ms with random and 1ms with the patch. This was more to prove that I'd changed something rather than a measure of the actual improvement. I would expect to see improved insertion rates across a lot of scenarios, less difference between uncompacted and compacted size (barring document updates and deletes) as less of the b+tree is rewritten, and a smaller post-compaction size vs random. The exact extent of these improvements should be established by a decent benchmark.
        Hide
        Brian Candler added a comment -

        I'd like to suggest an alternative algorithm for consideration.

        • first 48 bits of the UUID is the time, in milliseconds, since 1 Jan 1970
        • remaining 80 bits starts as a random value and increments from there,
          for example when doing a _bulk_docs insert

        I have been using this algorithm for a while, generated client-side - it's
        in my 'couchtiny' ruby client.

        I did it this way so as to get monotonically-increasing doc ids; a view with
        equal keys will sort them in order of insertion into the DB. It also avoids
        having to keep a separate "created_at" timestamp field, because you can just
        get it from the id.

        def created_at
        Time.at(id[0,12].to_i(16) / 1000.0) rescue nil
        end

        Of course, the fact I generate uids like this demonstrates that there's no
        one-size-fits-all solution, but I just thought it was worth mentioning
        because you should get the B-tree insertion boost as a side-effect too.

        Regards,

        Brian.

        It's your choice whether you want to re-randomize this when the next
        millisecond comes along, or just leave it to increment as a serial number.
        Even if you have multiple servers inserting documents into the same
        database, the chances of them using the same serial number within the same
        millisecond are infinitessimal, as long as they all start from an
        independent random point within the 2^80 possibilities.

        Wrapping would be very rare, but what I currently do is re-randomize for
        each bulk insert, and choose a starting random value which is more than 2^32
        away from the ceiling.

        Show
        Brian Candler added a comment - I'd like to suggest an alternative algorithm for consideration. first 48 bits of the UUID is the time, in milliseconds, since 1 Jan 1970 remaining 80 bits starts as a random value and increments from there, for example when doing a _bulk_docs insert I have been using this algorithm for a while, generated client-side - it's in my 'couchtiny' ruby client. I did it this way so as to get monotonically-increasing doc ids; a view with equal keys will sort them in order of insertion into the DB. It also avoids having to keep a separate "created_at" timestamp field, because you can just get it from the id. def created_at Time.at(id [0,12] .to_i(16) / 1000.0) rescue nil end Of course, the fact I generate uids like this demonstrates that there's no one-size-fits-all solution, but I just thought it was worth mentioning because you should get the B-tree insertion boost as a side-effect too. Regards, Brian. It's your choice whether you want to re-randomize this when the next millisecond comes along, or just leave it to increment as a serial number. Even if you have multiple servers inserting documents into the same database, the chances of them using the same serial number within the same millisecond are infinitessimal, as long as they all start from an independent random point within the 2^80 possibilities. Wrapping would be very rare, but what I currently do is re-randomize for each bulk insert, and choose a starting random value which is more than 2^32 away from the ceiling.
        Hide
        Robert Newson added a comment -

        Another interesting algorithm. I could change the patch so there's a couch_id_generator where the algorithm is configurable, defaulting to the current one, if that would move things along?

        Show
        Robert Newson added a comment - Another interesting algorithm. I could change the patch so there's a couch_id_generator where the algorithm is configurable, defaulting to the current one, if that would move things along?
        Hide
        Robert Newson added a comment -

        I renamed couch_seq_generator to couch_uuid_generator. It supports two algorithms; the original random one and the new random+sequential. It defaults to random.

        To configure you need a new ini block;

        [uuid]
        algorithm=(random|sequence)

        Show
        Robert Newson added a comment - I renamed couch_seq_generator to couch_uuid_generator. It supports two algorithms; the original random one and the new random+sequential. It defaults to random. To configure you need a new ini block; [uuid] algorithm=(random|sequence)
        Hide
        Joan Touzet added a comment - - edited

        This is a great patch, and solves the problem of having to do it in client-side logic. +1 from me too!

        It looks like Brian's solution above is intended to allow _all_docs to return all documents in chronological order, thus getting a time-sorted view "for free," i.e. without an extra field per document, extra view to maintain and update, extra view storage on the disk, etc. I admit I did the same for myself but it isn't necessarily a consideration for everyone. For example, in a replication situation, you'd need to be sure your clocks were well synchronized, and that you didn't have collisions in the prefix portion.

        Perhaps providing a mechanism to declare your own algorithm to override one of the two defaults (random, or rnewson's) would indeed be the best way forward, and the wiki could have a HOWTO with a set of small recipes on alternative approaches?

        Show
        Joan Touzet added a comment - - edited This is a great patch, and solves the problem of having to do it in client-side logic. +1 from me too! It looks like Brian's solution above is intended to allow _all_docs to return all documents in chronological order, thus getting a time-sorted view "for free," i.e. without an extra field per document, extra view to maintain and update, extra view storage on the disk, etc. I admit I did the same for myself but it isn't necessarily a consideration for everyone. For example, in a replication situation, you'd need to be sure your clocks were well synchronized, and that you didn't have collisions in the prefix portion. Perhaps providing a mechanism to declare your own algorithm to override one of the two defaults (random, or rnewson's) would indeed be the best way forward, and the wiki could have a HOWTO with a set of small recipes on alternative approaches?
        Hide
        Paul Joseph Davis added a comment -

        Just to throw something out in the interest of complicating things, should we consider a query string override the configured default algorithm as well?

        Show
        Paul Joseph Davis added a comment - Just to throw something out in the interest of complicating things, should we consider a query string override the configured default algorithm as well?
        Hide
        Chris Anderson added a comment -

        we may want to add a node identifier if we want to avoid collisions between replicating nodes.

        this is starting to sound a lot like UUID v1, which handles all these edge cases.

        http://en.wikipedia.org/wiki/Universally_Unique_Identifier#Version_1_.28MAC_address.29

        I'm generally +1 for this patch and the ideas it's generating. I'd say let's not apply until after the 0.10 branch

        Show
        Chris Anderson added a comment - we may want to add a node identifier if we want to avoid collisions between replicating nodes. this is starting to sound a lot like UUID v1, which handles all these edge cases. http://en.wikipedia.org/wiki/Universally_Unique_Identifier#Version_1_.28MAC_address.29 I'm generally +1 for this patch and the ideas it's generating. I'd say let's not apply until after the 0.10 branch
        Hide
        Robert Newson added a comment -

        I think the odds of generating the same random 12 byte value preclude collisions with enough probability. A MAC address is 48 bits whereas this prefix is 96 bits. The issue with UUID v1 is that it includes the time. The precision of the system clock could become the limiting factor in id generation. The exact algorithm for v1 is not well-specified either.

        To say it another way, the 96 random bits at the front are the (temporary) node identifier.

        Show
        Robert Newson added a comment - I think the odds of generating the same random 12 byte value preclude collisions with enough probability. A MAC address is 48 bits whereas this prefix is 96 bits. The issue with UUID v1 is that it includes the time. The precision of the system clock could become the limiting factor in id generation. The exact algorithm for v1 is not well-specified either. To say it another way, the 96 random bits at the front are the (temporary) node identifier.
        Hide
        Robert Newson added a comment -


        Updated to match latest trunk. Also increment between id's is a small random number (rather than always 1) to make identifiers less guessable.

        Show
        Robert Newson added a comment - Updated to match latest trunk. Also increment between id's is a small random number (rather than always 1) to make identifiers less guessable.
        Hide
        Chris Anderson added a comment -

        Good patch, thanks for the work.

        Code feedback:

        I think couch_util:new_uuid should be pulled into the couch_uuid_generator module

        Also, the uuid_generator should use the same config and couch_config:register idioms that are used by other modules. This should be more performant as the couch_config lookup will only happen rarely, instead of on every call.

        The randomness algorithm should be fine. As long at it is constantly ascending or descending (even with largish jumps) we should be fine.

        Show
        Chris Anderson added a comment - Good patch, thanks for the work. Code feedback: I think couch_util:new_uuid should be pulled into the couch_uuid_generator module Also, the uuid_generator should use the same config and couch_config:register idioms that are used by other modules. This should be more performant as the couch_config lookup will only happen rarely, instead of on every call. The randomness algorithm should be fine. As long at it is constantly ascending or descending (even with largish jumps) we should be fine.
        Hide
        Robert Newson added a comment -

        Use couch_config:register and a refactoring of handle_call.

        Show
        Robert Newson added a comment - Use couch_config:register and a refactoring of handle_call.
        Hide
        Robert Newson added a comment -


        Updated to switch the _uuids misc handler to the server-selected uuid generation algorithm.

        Show
        Robert Newson added a comment - Updated to switch the _uuids misc handler to the server-selected uuid generation algorithm.
        Hide
        Robert Newson added a comment -

        changed [uuid] config to [uuid_generation]
        changed sequence to sequential
        made sequential the default
        fixed rollover to use all bits of counter suffix

        Show
        Robert Newson added a comment - changed [uuid] config to [uuid_generation] changed sequence to sequential made sequential the default fixed rollover to use all bits of counter suffix
        Hide
        Paul Joseph Davis added a comment -

        Refactored a bit.
        couch_uuid_generator -> couch_uuids
        sequential -> hybrid (this was mostly playing, thoughts?)
        random is back to default and should stay that way
        call to random:uniform() -> crypto:rand_uniform() (someone forgot to call random:seed(), tsk tsk
        [uuid_generator] -> [uuids] in config
        added short description of choices next to option
        probably an unneccessary optimization to allow for couch_uuids:new(N) to return N ids.
        added simple checks to the uuids.js Futon tests
        Pulled in a contribution by Bob Dionne for etap tests.
        All calls to couch_util:new_uuid() are replaced with couch_uuuids:random()

        We should make the prefix length a configuration parameter I think.

        We might also think about adding Brian's algorithm for more diversity as well.

        Show
        Paul Joseph Davis added a comment - Refactored a bit. couch_uuid_generator -> couch_uuids sequential -> hybrid (this was mostly playing, thoughts?) random is back to default and should stay that way call to random:uniform() -> crypto:rand_uniform() (someone forgot to call random:seed(), tsk tsk [uuid_generator] -> [uuids] in config added short description of choices next to option probably an unneccessary optimization to allow for couch_uuids:new(N) to return N ids. added simple checks to the uuids.js Futon tests Pulled in a contribution by Bob Dionne for etap tests. All calls to couch_util:new_uuid() are replaced with couch_uuuids:random() We should make the prefix length a configuration parameter I think. We might also think about adding Brian's algorithm for more diversity as well.
        Hide
        Paul Joseph Davis added a comment -

        Added the utc_random algorithm.
        Updated tests to use it.
        hybrid -> sequential
        unoptimized the calling to return a list for code cleanliness.

        Show
        Paul Joseph Davis added a comment - Added the utc_random algorithm. Updated tests to use it. hybrid -> sequential unoptimized the calling to return a list for code cleanliness.
        Hide
        Paul Joseph Davis added a comment -

        Used byte numbers when I mean characters in default.ini.tpl.in

        Show
        Paul Joseph Davis added a comment - Used byte numbers when I mean characters in default.ini.tpl.in
        Hide
        Paul Joseph Davis added a comment -

        Chris,

        You're the only person I've seen with any wavering on this going in. Is there a reason you want to wait on it? I did push it back so that the default behavior is still the random 128 bits if that was a worry.

        Paul

        Show
        Paul Joseph Davis added a comment - Chris, You're the only person I've seen with any wavering on this going in. Is there a reason you want to wait on it? I did push it back so that the default behavior is still the random 128 bits if that was a worry. Paul
        Hide
        Antony Blakey added a comment -

        I remember a discussion about assigning a UUID to a database to allow external view providesr to more reliably associate indexing artifacts with a given db instance (given that names can be duplicated over a create/delete/create boundary). I suggested that this could be used in document ids (and revids) and the response was that this was a privacy risk because the originating database's identity would leak into the peer group. It seems that this proposal has the same characteristic because you can identify the common source of a set of documents. Is this no longer a concern? If not, then how about revisiting the idea of assigning each database a UUID on creation, and using that UUID as a document identity prefix?

        Show
        Antony Blakey added a comment - I remember a discussion about assigning a UUID to a database to allow external view providesr to more reliably associate indexing artifacts with a given db instance (given that names can be duplicated over a create/delete/create boundary). I suggested that this could be used in document ids (and revids) and the response was that this was a privacy risk because the originating database's identity would leak into the peer group. It seems that this proposal has the same characteristic because you can identify the common source of a set of documents. Is this no longer a concern? If not, then how about revisiting the idea of assigning each database a UUID on creation, and using that UUID as a document identity prefix?
        Hide
        Paul Joseph Davis added a comment -

        Antony,

        The sequential algorithm uses a 13 byte random prefix coupled with a 3 byte random suffix. Each new uuid is incremented by a small random number. When the three bytes overflows a new random prefix is generated. Now the privacy concern is that two docs could've been created by the same node but its not tied directly to a db file.

        I'd definitely be +1 for having a UUID in the db header for other reasons though.

        Paul

        Show
        Paul Joseph Davis added a comment - Antony, The sequential algorithm uses a 13 byte random prefix coupled with a 3 byte random suffix. Each new uuid is incremented by a small random number. When the three bytes overflows a new random prefix is generated. Now the privacy concern is that two docs could've been created by the same node but its not tied directly to a db file. I'd definitely be +1 for having a UUID in the db header for other reasons though. Paul
        Hide
        Robert Newson added a comment -

        I like most of Paul's changes though I thought we'd agreed on IRC to change the default to sequential and I'd still like to see that happen.

        I would also like to see a way to detect very quick delete/create scenarios, though I don't know if database uuids are the only solution there. If a global _changes feed would emit "deleted" and "created" events for databases in the correct order, then couchdb-lucene could work correctly without database uuids.

        Antony's suggestion of a fourth algorithm, where the prefix is completely static, is simple enough to add. This patch allows the deployer to decide how much he cares about predictability and server origin, so I don't see a reason not to add it. It is distinct from the sequential algorithm, though. The prefix there is only used for around 8000 ids and is then never reused, there is also no correlation between prefix and origin server.

        Show
        Robert Newson added a comment - I like most of Paul's changes though I thought we'd agreed on IRC to change the default to sequential and I'd still like to see that happen. I would also like to see a way to detect very quick delete/create scenarios, though I don't know if database uuids are the only solution there. If a global _changes feed would emit "deleted" and "created" events for databases in the correct order, then couchdb-lucene could work correctly without database uuids. Antony's suggestion of a fourth algorithm, where the prefix is completely static, is simple enough to add. This patch allows the deployer to decide how much he cares about predictability and server origin, so I don't see a reason not to add it. It is distinct from the sequential algorithm, though. The prefix there is only used for around 8000 ids and is then never reused, there is also no correlation between prefix and origin server.
        Hide
        Bob Dionne added a comment -

        I tested this all against a clean checkout of trunk and it looks good. The new algorithm is faster on inserts[1] but interestingly for single inserts is makes for a larger db pre-compact. After compaction the db is smaller by a factor of 3

        I thought we leave the new "sequentially random" as the default? Paul, is the concern just what's advertised to users?

        Algorithms based on the system clock I think can be problematic as they assume all machines have the correct time.

        [1] http://gist.github.com/170982

        Show
        Bob Dionne added a comment - I tested this all against a clean checkout of trunk and it looks good. The new algorithm is faster on inserts [1] but interestingly for single inserts is makes for a larger db pre-compact. After compaction the db is smaller by a factor of 3 I thought we leave the new "sequentially random" as the default? Paul, is the concern just what's advertised to users? Algorithms based on the system clock I think can be problematic as they assume all machines have the correct time. [1] http://gist.github.com/170982
        Hide
        Robert Newson added a comment -


        The utc_random has that issue which is why it's not the default. I don't think it would make sense to choose utc_random unless you had already planned to keep your clocks pretty accurately synced. Brian's point, I'm implying, is that since he can arrange that, then if the ids used the clock time he'll get a nice ordering for identifiers for free. Anyone running couchdb in a data center could feasibly arrange for ntp synchronization, so I think it's a nice option for those that understand the consequences.

        Show
        Robert Newson added a comment - The utc_random has that issue which is why it's not the default. I don't think it would make sense to choose utc_random unless you had already planned to keep your clocks pretty accurately synced. Brian's point, I'm implying, is that since he can arrange that, then if the ids used the clock time he'll get a nice ordering for identifiers for free. Anyone running couchdb in a data center could feasibly arrange for ntp synchronization, so I think it's a nice option for those that understand the consequences.
        Hide
        Robert Newson added a comment -

        I just noticed that Paul's implementation is not as Brian described; the suffix is always random with no guarantee the id N+1 is strictly higher than id N;

        + list_to_binary(Prefix ++ couch_util:to_hex(crypto:rand_bytes(9))).

        In practise it just means that id's generated in the same millisecond are randomly ordered. Does that matter?

        Show
        Robert Newson added a comment - I just noticed that Paul's implementation is not as Brian described; the suffix is always random with no guarantee the id N+1 is strictly higher than id N; + list_to_binary(Prefix ++ couch_util:to_hex(crypto:rand_bytes(9))). In practise it just means that id's generated in the same millisecond are randomly ordered. Does that matter?
        Hide
        Robert Newson added a comment -

        Bob,

        I read your gist and the pre-compact size for sequential is smaller than for random (43,144,791 vs 52,517,614) and post-compaction was smaller for sequential too (2,023,522 vs 2,506,850). Perhaps you read the numbers the wrong way around? Or did I?

        I also note that most of this compaction is from not batching as you'll have a 4k footer block for each write. With ?batch=ok and sequential (and no deletions) I find compaction to be only a marginal reduction of space.

        Show
        Robert Newson added a comment - Bob, I read your gist and the pre-compact size for sequential is smaller than for random (43,144,791 vs 52,517,614) and post-compaction was smaller for sequential too (2,023,522 vs 2,506,850). Perhaps you read the numbers the wrong way around? Or did I? I also note that most of this compaction is from not batching as you'll have a 4k footer block for each write. With ?batch=ok and sequential (and no deletions) I find compaction to be only a marginal reduction of space.
        Hide
        Bob Dionne added a comment -

        Robert,

        sorry for the confusion, there are 4 tests there, tests 2 and 4 use the old random and the new "sequential random" codes respectively. So I'm comparing:

        precompact: 50020852 postcompact: 6099042

        to

        precompact: 52517614 postcompact: 2506850

        That's why I used the phrase "sequentially random". Sequential in these tests (1 and 3) means 1,2,3....

        I agree about the batching, the diffs are almost noise with a batch size of 1000, which is what the compactor uses.

        Cheers,

        Bob

        Show
        Bob Dionne added a comment - Robert, sorry for the confusion, there are 4 tests there, tests 2 and 4 use the old random and the new "sequential random" codes respectively. So I'm comparing: precompact: 50020852 postcompact: 6099042 to precompact: 52517614 postcompact: 2506850 That's why I used the phrase "sequentially random". Sequential in these tests (1 and 3) means 1,2,3.... I agree about the batching, the diffs are almost noise with a batch size of 1000, which is what the compactor uses. Cheers, Bob
        Hide
        Paul Joseph Davis added a comment -

        For Bob's comment on times:

        The documentation for Erlang's now() function guarantees that its monotonically increasing so there's no need for me to force the suffix to be random then +1 as it'll already be ordered and the extra randomness will help prevent spurious conflicts when replicating.

        Yes, clocks can go out of sync but its not critical to have them in sync, and its a non-standard option. And adding a comment in the ini file about time syncing would be just fine.

        Regarding the sequential default, I don't remember anyone convincing me to make it something other than random. Though I may have just forgotten that conversation. I fear that any algorithm beyond completely random is a performance hack and I think that we should force people to consider the consequences when using one.

        Show
        Paul Joseph Davis added a comment - For Bob's comment on times: The documentation for Erlang's now() function guarantees that its monotonically increasing so there's no need for me to force the suffix to be random then +1 as it'll already be ordered and the extra randomness will help prevent spurious conflicts when replicating. Yes, clocks can go out of sync but its not critical to have them in sync, and its a non-standard option. And adding a comment in the ini file about time syncing would be just fine. Regarding the sequential default, I don't remember anyone convincing me to make it something other than random. Though I may have just forgotten that conversation. I fear that any algorithm beyond completely random is a performance hack and I think that we should force people to consider the consequences when using one.
        Hide
        Chris Anderson added a comment -

        I think the work that has been done has allayed my concerns. I'd suggest making the sequentialish-randomy one the default.

        I think db uuids are their own issue, and are needed for lots of reasons.

        Show
        Chris Anderson added a comment - I think the work that has been done has allayed my concerns. I'd suggest making the sequentialish-randomy one the default. I think db uuids are their own issue, and are needed for lots of reasons.
        Hide
        Bob Dionne added a comment -

        etaps to test the new couch_uuids work of rnewson and davisp.

        Show
        Bob Dionne added a comment - etaps to test the new couch_uuids work of rnewson and davisp.
        Hide
        Robert Newson added a comment -

        Just capturing that I agree with Paul that random should be the default; his patch adds sufficient documentation for the user to make an informed choice. random uuids generation is the least surprising of the three behaviors, so it heads off interminable "why do my ids look kinda similar?" discussions.

        Is this going into 0.11?

        Show
        Robert Newson added a comment - Just capturing that I agree with Paul that random should be the default; his patch adds sufficient documentation for the user to make an informed choice. random uuids generation is the least surprising of the three behaviors, so it heads off interminable "why do my ids look kinda similar?" discussions. Is this going into 0.11?
        Hide
        Adam Kocoloski added a comment -

        Yes, this is going into 0.11. It looks like things have stabilized; is there any reason not to commit it today?

        Show
        Adam Kocoloski added a comment - Yes, this is going into 0.11. It looks like things have stabilized; is there any reason not to commit it today?
        Hide
        Adam Kocoloski added a comment -

        Closing it out. Nice patch everybody!

        Show
        Adam Kocoloski added a comment - Closing it out. Nice patch everybody!

          People

          • Assignee:
            Unassigned
            Reporter:
            Robert Newson
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development