Details

      Description

      CASSANDRA-2474 and CASSANDRA-3647 add the ability to transpose wide rows differently, for efficiency and functionality secondary index api needs to be altered to allow composite indexes.

      I think this will require the IndexManager api to have a maybeIndex(ByteBuffer column) method that SS can call and implement a PerRowSecondaryIndex per column, break the composite into parts and index specific bits, also including the base rowkey.

      Then a search against a TRANSPOSED row or DOCUMENT will be possible.

        Issue Links

          Activity

          Hide
          Sylvain Lebresne added a comment -

          Committed, thanks.

          Show
          Sylvain Lebresne added a comment - Committed, thanks.
          Hide
          Yuki Morishita added a comment -

          lgtm, +1.

          Show
          Yuki Morishita added a comment - lgtm, +1.
          Hide
          Sylvain Lebresne added a comment -

          My bad, pushed a fixed version at https://github.com/pcmanus/cassandra/commits/3680-4.

          Show
          Sylvain Lebresne added a comment - My bad, pushed a fixed version at https://github.com/pcmanus/cassandra/commits/3680-4 .
          Hide
          Yuki Morishita added a comment -

          hmm, filtering with key and 2I works, but this time I get wrong value for key.

          cqlsh:3680> select * from blogs;
           blog_id | posted_at                | author | content
          ---------+--------------------------+--------+---------
                 1 | 2012-11-11 00:00:00-0600 |    foo |     bar
                 2 | 2012-11-12 00:00:00-0600 |    foo |     baz
                 3 | 2012-11-11 00:00:00-0600 |    qux |    quux
          
          cqlsh:3680> select * from blogs where author='foo' and posted_at = '2012-11-11';
           blog_id | posted_at                | author | content
          ---------+--------------------------+--------+---------
                 2 | 2012-11-11 00:00:00-0600 |    foo |     bar
          

          blog_id should be '1'.

          cqlsh:3680> select * from blogs where author='foo';
           blog_id | posted_at                | author | content
          ---------+--------------------------+--------+---------
                 2 | 2012-11-11 00:00:00-0600 |    foo |     bar
                 2 | 2012-11-12 00:00:00-0600 |    foo |     baz
          

          Here, something is wrong with first row in result set.

          Show
          Yuki Morishita added a comment - hmm, filtering with key and 2I works, but this time I get wrong value for key. cqlsh:3680> select * from blogs; blog_id | posted_at | author | content ---------+--------------------------+--------+--------- 1 | 2012-11-11 00:00:00-0600 | foo | bar 2 | 2012-11-12 00:00:00-0600 | foo | baz 3 | 2012-11-11 00:00:00-0600 | qux | quux cqlsh:3680> select * from blogs where author='foo' and posted_at = '2012-11-11'; blog_id | posted_at | author | content ---------+--------------------------+--------+--------- 2 | 2012-11-11 00:00:00-0600 | foo | bar blog_id should be '1'. cqlsh:3680> select * from blogs where author='foo'; blog_id | posted_at | author | content ---------+--------------------------+--------+--------- 2 | 2012-11-11 00:00:00-0600 | foo | bar 2 | 2012-11-12 00:00:00-0600 | foo | baz Here, something is wrong with first row in result set.
          Hide
          Sylvain Lebresne added a comment -

          You're right. Pushed a rebased and fixed version at https://github.com/pcmanus/cassandra/commits/3680-3.

          Show
          Sylvain Lebresne added a comment - You're right. Pushed a rebased and fixed version at https://github.com/pcmanus/cassandra/commits/3680-3 .
          Hide
          Yuki Morishita added a comment -

          I ran couple of test with above blogs CF(index is created on author).

          First, insert 3 rows below:

          cqlsh:3680> INSERT INTO blogs (blog_id, posted_at, author, content) VALUES (1, '2012-11-11', 'foo', 'bar');
          cqlsh:3680> INSERT INTO blogs (blog_id, posted_at, author, content) VALUES (2, '2012-11-12', 'foo', 'baz');
          cqlsh:3680> INSERT INTO blogs (blog_id, posted_at, author, content) VALUES (3, '2012-11-11', 'gux', 'quux');
          cqlsh:3680> SELECT * FROM blogs;
           blog_id | posted_at                | author | content
          ---------+--------------------------+--------+---------
                 1 | 2012-11-11 00:00:00-0600 |    foo |     bar
                 2 | 2012-11-12 00:00:00-0600 |    foo |     baz
                 3 | 2012-11-11 00:00:00-0600 |    gux |    quux
          

          Select on indexed column works fine:

          cqlsh:3680> SELECT * FROM blogs WHERE author='foo';
           blog_id | posted_at                | author | content
          ---------+--------------------------+--------+---------
                 1 | 2012-11-11 00:00:00-0600 |    foo |     bar
                 2 | 2012-11-12 00:00:00-0600 |    foo |     baz
          

          But, query combined with primary key(2nd query below) is not working as expected:

          cqlsh:3680> SELECT * FROM blogs WHERE posted_at='2012-11-11';
           blog_id | posted_at                | author | content
          ---------+--------------------------+--------+---------
                 1 | 2012-11-11 00:00:00-0600 |    foo |     bar
                 3 | 2012-11-11 00:00:00-0600 |    gux |    quux
          
          cqlsh:3680> SELECT * FROM blogs WHERE posted_at='2012-11-11' AND author='foo';
           blog_id | posted_at                | author | content
          ---------+--------------------------+--------+---------
                 1 | 2012-11-11 00:00:00-0600 |    foo |     bar
                 2 | 2012-11-12 00:00:00-0600 |    foo |     baz
          

          Here, I expected only row with blog_id=1, but both 1 and 2 are returned.

          Show
          Yuki Morishita added a comment - I ran couple of test with above blogs CF(index is created on author ). First, insert 3 rows below: cqlsh:3680> INSERT INTO blogs (blog_id, posted_at, author, content) VALUES (1, '2012-11-11', 'foo', 'bar'); cqlsh:3680> INSERT INTO blogs (blog_id, posted_at, author, content) VALUES (2, '2012-11-12', 'foo', 'baz'); cqlsh:3680> INSERT INTO blogs (blog_id, posted_at, author, content) VALUES (3, '2012-11-11', 'gux', 'quux'); cqlsh:3680> SELECT * FROM blogs; blog_id | posted_at | author | content ---------+--------------------------+--------+--------- 1 | 2012-11-11 00:00:00-0600 | foo | bar 2 | 2012-11-12 00:00:00-0600 | foo | baz 3 | 2012-11-11 00:00:00-0600 | gux | quux Select on indexed column works fine: cqlsh:3680> SELECT * FROM blogs WHERE author='foo'; blog_id | posted_at | author | content ---------+--------------------------+--------+--------- 1 | 2012-11-11 00:00:00-0600 | foo | bar 2 | 2012-11-12 00:00:00-0600 | foo | baz But, query combined with primary key(2nd query below) is not working as expected: cqlsh:3680> SELECT * FROM blogs WHERE posted_at='2012-11-11'; blog_id | posted_at | author | content ---------+--------------------------+--------+--------- 1 | 2012-11-11 00:00:00-0600 | foo | bar 3 | 2012-11-11 00:00:00-0600 | gux | quux cqlsh:3680> SELECT * FROM blogs WHERE posted_at='2012-11-11' AND author='foo'; blog_id | posted_at | author | content ---------+--------------------------+--------+--------- 1 | 2012-11-11 00:00:00-0600 | foo | bar 2 | 2012-11-12 00:00:00-0600 | foo | baz Here, I expected only row with blog_id=1, but both 1 and 2 are returned.
          Hide
          Sylvain Lebresne added a comment -

          I've pushed a rebased version of the patch above at https://github.com/pcmanus/cassandra/commits/3680-2. The previous comments still applies though.

          Show
          Sylvain Lebresne added a comment - I've pushed a rebased version of the patch above at https://github.com/pcmanus/cassandra/commits/3680-2 . The previous comments still applies though.
          Hide
          Sylvain Lebresne added a comment -

          Attaching initial patch to support indexing on composites. This basically supports indexing a slice of columns based on the value of one of these columns. In other words, given definition

          CREATE TABLE blogs (
              blog_id int,
              posted_at timestamp,
              author text,
              content text,
              PRIMARY KEY (blog_id, posted_at)
          )
          

          It allows to create index:

          CREATE INDEX ON blogs(author);
          

          The patch does not however support indexing on PRIMARY KEY part, but I believe this is a different enough problem that this could be left to a separate ticket.

          The indexing itself is a slight variation on our current KeysIndex, but where each index column holds a row key and a column name prefix instead of just the row key. Technically however, there is a few difficulties:

          • The code to query secondary indexes is a bit too wired to one row per index column, which is not the case anymore. The patch hacks around that but this is arguably a bit ugly. I don't have a better solution however.
          • Index locking during index built is row based. With this, large rows can have lots and lots of indexed columns. It follows that index rebuild might have to lock the row for a long time. The patch tries to mitigate that by paging on wide row and releasing the lock between page to let writes make progress but in practice this probably mean that index rebuild will have a more heavy impact on a live node.
          • For the same reason, when doing a row delete, we might have lots of indexed columns mutated and we have to read the whole row to know which one that is. This might make such updates fairly expensive since in that case too the whole row will be locked in the process. The best fix for this is probably CASSANDRA-2897.
          Show
          Sylvain Lebresne added a comment - Attaching initial patch to support indexing on composites. This basically supports indexing a slice of columns based on the value of one of these columns. In other words, given definition CREATE TABLE blogs ( blog_id int, posted_at timestamp, author text, content text, PRIMARY KEY (blog_id, posted_at) ) It allows to create index: CREATE INDEX ON blogs(author); The patch does not however support indexing on PRIMARY KEY part, but I believe this is a different enough problem that this could be left to a separate ticket. The indexing itself is a slight variation on our current KeysIndex, but where each index column holds a row key and a column name prefix instead of just the row key. Technically however, there is a few difficulties: The code to query secondary indexes is a bit too wired to one row per index column, which is not the case anymore. The patch hacks around that but this is arguably a bit ugly. I don't have a better solution however. Index locking during index built is row based. With this, large rows can have lots and lots of indexed columns. It follows that index rebuild might have to lock the row for a long time. The patch tries to mitigate that by paging on wide row and releasing the lock between page to let writes make progress but in practice this probably mean that index rebuild will have a more heavy impact on a live node. For the same reason, when doing a row delete, we might have lots of indexed columns mutated and we have to read the whole row to know which one that is. This might make such updates fairly expensive since in that case too the whole row will be locked in the process. The best fix for this is probably CASSANDRA-2897 .
          Hide
          Sylvain Lebresne added a comment -

          I think there is two things here: the composite parts and the transposed part. I've created CASSANDRA-3782 to handle the transposed part and to keep this one focused on the composite part. Both issues are not completely unrelated but I feel are sufficiently orthogonal to warrant 2 separate tickets.

          I'll note however that for this issue we may want to keep the sparse case in mind. Typically, if I declare:

          CREATE TABLE timeline (
             userid uuid,
             posted_at timestamp,
             body text,
             posted_by text,
             PRIMARY KEY (userid, posted_at)
          );
          

          then we want to be able to create an index on say posted_by. Which means it's really a PerColumnPrefixSecondaryIndex.

          Show
          Sylvain Lebresne added a comment - I think there is two things here: the composite parts and the transposed part. I've created CASSANDRA-3782 to handle the transposed part and to keep this one focused on the composite part. Both issues are not completely unrelated but I feel are sufficiently orthogonal to warrant 2 separate tickets. I'll note however that for this issue we may want to keep the sparse case in mind. Typically, if I declare: CREATE TABLE timeline ( userid uuid, posted_at timestamp, body text, posted_by text, PRIMARY KEY (userid, posted_at) ); then we want to be able to create an index on say posted_by. Which means it's really a PerColumnPrefixSecondaryIndex.

            People

            • Assignee:
              Sylvain Lebresne
              Reporter:
              T Jake Luciani
              Reviewer:
              Yuki Morishita
            • Votes:
              7 Vote for this issue
              Watchers:
              18 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development