Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.7, 6.0
    • Component/s: SolrCloud
    • Labels:
      None

      Description

      This would enable multi level routing as compared to the 2 level routing available as of now. On the usage bit, here's an example:

      Document Id: myapp!dummyuser!doc
      myapp!dummyuser! can be used as the shardkey for searching content for dummyuser.
      myapp! can be used for searching across all users of myapp.

      I am looking at either a 3 (or 4) level routing. The 32 bit hash would then comprise of 8X4 components from each part (in case of 4 level).

      Usage:

      Document Id: myapp!dummyuser!doc

      To query all users for a particular app (default setup), the route key should be: 'myapp/8!'.
      To query a particular user for a specific app, the route key should be: 'myapp!dummyuser!'

      The syntax for querying all users for a particular app is required because this router works at both 2 and 3 level of composite id.
      A route key of 'myapp!' would technically translate to constructing the hash range with 16 bits coming from the key i.e. 2-level composite id.

      1. SOLR-5320.patch
        22 kB
        Anshum Gupta
      2. SOLR-5320.patch
        21 kB
        Anshum Gupta
      3. SOLR-5320.patch
        16 kB
        Anshum Gupta
      4. SOLR-5320.patch
        15 kB
        Anshum Gupta
      5. SOLR-5320.patch
        14 kB
        Anshum Gupta
      6. SOLR-5320.patch
        14 kB
        Noble Paul
      7. SOLR-5320.patch
        12 kB
        Anshum Gupta
      8. SOLR-5320-refactored.patch
        13 kB
        Anshum Gupta

        Activity

        Hide
        Anshum Gupta added a comment -

        A 3 level composite id routing to begin with is what I think would be good.
        I'd use 8 bits each from the first 2 components of the key and 16 bits from the last component.
        Functionally, this should work on similar lines as the current 2-level composite id routing.

        Show
        Anshum Gupta added a comment - A 3 level composite id routing to begin with is what I think would be good. I'd use 8 bits each from the first 2 components of the key and 16 bits from the last component. Functionally, this should work on similar lines as the current 2-level composite id routing.
        Hide
        Anshum Gupta added a comment -

        My initial thought about splitting the 32 bits forced me to have a separate router that extended the composite id router.
        Here's the original thought:

        Document Id: myapp!dummyuser!doc
        This could also look like: myapp/numBits1!dummyuser/numBits2!doc
        where numBits* are less than or equal to 8 (or not present, when it defaults to 8).

        Considering have 8/8/16 bits split between the 3 components of the id.
        The point of concern is, in case numBits = 1 (or any other number), how to split the remaining 7 bits or rather 31.
        I was trying to equally split the remainder among the 2 components (not really cleanly possible in case of odd bits).

        I plan on adding remainder bits to the component that is derived from the docId i.e. 16+7. The new split becomes 1/8/23 bits among the components.

        With that in place, I can just change compositeId router to also cater to 3-level composite id routing.

        Thoughts?

        Show
        Anshum Gupta added a comment - My initial thought about splitting the 32 bits forced me to have a separate router that extended the composite id router. Here's the original thought: Document Id: myapp!dummyuser!doc This could also look like: myapp/numBits1!dummyuser/numBits2!doc where numBits* are less than or equal to 8 (or not present, when it defaults to 8). Considering have 8/8/16 bits split between the 3 components of the id. The point of concern is, in case numBits = 1 (or any other number), how to split the remaining 7 bits or rather 31. I was trying to equally split the remainder among the 2 components (not really cleanly possible in case of odd bits). I plan on adding remainder bits to the component that is derived from the docId i.e. 16+7. The new split becomes 1/8/23 bits among the components. With that in place, I can just change compositeId router to also cater to 3-level composite id routing. Thoughts?
        Hide
        Anshum Gupta added a comment -

        Adding to that, if the user wants to override the default numBits, everything would be borrowed/lent back to the document id component i.e. the 3rd component (c in the example below).
        e.g. : a!b!c

        Show
        Anshum Gupta added a comment - Adding to that, if the user wants to override the default numBits, everything would be borrowed/lent back to the document id component i.e. the 3rd component (c in the example below). e.g. : a!b!c
        Hide
        Anshum Gupta added a comment -

        Working on refactoring and testing this.
        Will be uploading a refactored test later in the day.

        Show
        Anshum Gupta added a comment - Working on refactoring and testing this. Will be uploading a refactored test later in the day.
        Hide
        Noble Paul added a comment -

        The router object should be immutable and threadsafe. you are not supposed to change the state (I am referring to the masks)
        I see repetition of the same code across multiple methods . Can you not put the entire logic in one place and reuse ?

        Show
        Noble Paul added a comment - The router object should be immutable and threadsafe. you are not supposed to change the state (I am referring to the masks) I see repetition of the same code across multiple methods . Can you not put the entire logic in one place and reuse ?
        Hide
        Anshum Gupta added a comment -

        Noble Paul Yes, I already realized that and have a patch almost ready. Will just put that up in some time.

        Show
        Anshum Gupta added a comment - Noble Paul Yes, I already realized that and have a patch almost ready. Will just put that up in some time.
        Hide
        Anshum Gupta added a comment -

        Refactored and fixed thread safety (+other) issues.

        Show
        Anshum Gupta added a comment - Refactored and fixed thread safety (+other) issues.
        Hide
        Noble Paul added a comment -

        refactored and cleaned up

        Show
        Noble Paul added a comment - refactored and cleaned up
        Hide
        Anshum Gupta added a comment -

        Thanks for taking a look at that Noble.

        Show
        Anshum Gupta added a comment - Thanks for taking a look at that Noble.
        Hide
        Anshum Gupta added a comment -

        Some minor changes.

        Show
        Anshum Gupta added a comment - Some minor changes.
        Hide
        Anshum Gupta added a comment -

        Fixed a mask calculation related issue.

        Show
        Anshum Gupta added a comment - Fixed a mask calculation related issue.
        Hide
        Anshum Gupta added a comment -

        Yonik Seeley it'd be great if you could have a look at this and give some feedback.

        Show
        Anshum Gupta added a comment - Yonik Seeley it'd be great if you could have a look at this and give some feedback.
        Hide
        Anshum Gupta added a comment -

        Syncing with trunk.

        Show
        Anshum Gupta added a comment - Syncing with trunk.
        Hide
        Anshum Gupta added a comment -

        Added more test coverage.

        The new test adds documents with tri-level ids and validates that a single route-key maps to a single shard only.

        Show
        Anshum Gupta added a comment - Added more test coverage. The new test adds documents with tri-level ids and validates that a single route-key maps to a single shard only.
        Hide
        Anshum Gupta added a comment -

        Added a test for composite ids with randomised bitMask.

        Show
        Anshum Gupta added a comment - Added a test for composite ids with randomised bitMask.
        Hide
        ASF subversion and git services added a comment -

        Commit 1542272 from shalin@apache.org in branch 'dev/trunk'
        [ https://svn.apache.org/r1542272 ]

        SOLR-5320: Added support for tri-level compositeId routing

        Show
        ASF subversion and git services added a comment - Commit 1542272 from shalin@apache.org in branch 'dev/trunk' [ https://svn.apache.org/r1542272 ] SOLR-5320 : Added support for tri-level compositeId routing
        Hide
        ASF subversion and git services added a comment -

        Commit 1543382 from shalin@apache.org in branch 'dev/branches/branch_4x'
        [ https://svn.apache.org/r1543382 ]

        SOLR-5320: Added support for tri-level compositeId routing. Also merged javadoc fixes and forbidden-api fixes from r1542305, r1542321

        Show
        ASF subversion and git services added a comment - Commit 1543382 from shalin@apache.org in branch 'dev/branches/branch_4x' [ https://svn.apache.org/r1543382 ] SOLR-5320 : Added support for tri-level compositeId routing. Also merged javadoc fixes and forbidden-api fixes from r1542305, r1542321
        Hide
        Shalin Shekhar Mangar added a comment -

        Thanks Anshum!

        Show
        Shalin Shekhar Mangar added a comment - Thanks Anshum!

          People

          • Assignee:
            Shalin Shekhar Mangar
            Reporter:
            Anshum Gupta
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Time Tracking

              Estimated:
              Original Estimate - 24h
              24h
              Remaining:
              Remaining Estimate - 24h
              24h
              Logged:
              Time Spent - Not Specified
              Not Specified

                Development