Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Fix Version/s: 1.1.0
    • Component/s: Core, Tools
    • Labels:
      None

      Description

      CASSANDRA-1740 allows aborting an index build, but there is no way to re-attempt the build without restarting the server.

      We've also had requests to allow rebuilding an index that has been built, so it would be nice to kill two birds with one stone here.

      1. 0001-CASSANDRA-3583-fix.patch
        0.9 kB
        Vijay
      2. 0001-3583-v2.patch
        8 kB
        Vijay
      3. 0001-3583.patch
        5 kB
        Vijay

        Activity

        Hide
        Vijay added a comment -

        Hi Jonathan,

        Will it be ok to do a full index rebuild always, this will be simpler though.

        Other option is to track the SSTables which failed to build the index (due to crash or due to the user interruption).
        Something like:
        nt rebuild <CFNAME> all
        nt rebuild <CFNAME> failed

        I dont see a restart building a index should we try to check there too? Let me know thanks!

        Show
        Vijay added a comment - Hi Jonathan, Will it be ok to do a full index rebuild always, this will be simpler though. Other option is to track the SSTables which failed to build the index (due to crash or due to the user interruption). Something like: nt rebuild <CFNAME> all nt rebuild <CFNAME> failed I dont see a restart building a index should we try to check there too? Let me know thanks!
        Hide
        Jonathan Ellis added a comment -

        I don't think it's worth trying to support "resuming" a partial index build, if that's what you mean.

        Show
        Jonathan Ellis added a comment - I don't think it's worth trying to support "resuming" a partial index build, if that's what you mean.
        Hide
        Vijay added a comment -

        Simple patch to do index rebuild via

        nt rebuild index <KS> <CF>

        Show
        Vijay added a comment - Simple patch to do index rebuild via nt rebuild index <KS> <CF>
        Hide
        Jonathan Ellis added a comment -

        what future "types" do you anticipate adding?

        Show
        Jonathan Ellis added a comment - what future "types" do you anticipate adding?
        Hide
        Vijay added a comment -

        I was thinking of adding rebuild SSTableIndex or rebuild bloomfilter etc. Yet to seen a case where it is broken may be after CASSANDRA-2392...

        Show
        Vijay added a comment - I was thinking of adding rebuild SSTableIndex or rebuild bloomfilter etc. Yet to seen a case where it is broken may be after CASSANDRA-2392 ...
        Hide
        Jonathan Ellis added a comment -

        Let's allow specifying a single index, as well as all of them [which can be the default].

        I think you also want to clear the Built flag on the index(es) or the rebuild will be incomplete if you cancel or restart partway through.

        Nit: prefer "cfs" for local CFS reference names rather than "store"

        Show
        Jonathan Ellis added a comment - Let's allow specifying a single index, as well as all of them [which can be the default] . I think you also want to clear the Built flag on the index(es) or the rebuild will be incomplete if you cancel or restart partway through. Nit: prefer "cfs" for local CFS reference names rather than "store"
        Hide
        Vijay added a comment -

        Hi Jonathan,

        "I think you also want to clear the Built flag on the index(es) or the rebuild will be incomplete if you cancel or restart partway through."
        in the current patch, we dont need to as the new indexes will be new SST's and if some one stopped in between then it wont be worser than what it was earlier... otherwise we might want to clear the field when we start and reset it in the end that way the clients might notice some additional missing indexes. Agree?

        Show
        Vijay added a comment - Hi Jonathan, "I think you also want to clear the Built flag on the index(es) or the rebuild will be incomplete if you cancel or restart partway through." in the current patch, we dont need to as the new indexes will be new SST's and if some one stopped in between then it wont be worser than what it was earlier... otherwise we might want to clear the field when we start and reset it in the end that way the clients might notice some additional missing indexes. Agree?
        Hide
        Jonathan Ellis added a comment -

        I don't quite understand. I'm saying that a "rebuild index" command can serve two purposes:

        • restart an index build that got cancelled via 1740
        • rebuild an existing index that is suspected to be corrupt or fragmented

        For the second part it's important to clear the built flag to avoid surprises in the restart scenario. (If the server doesn't restart of course it doesn't matter.)

        Show
        Jonathan Ellis added a comment - I don't quite understand. I'm saying that a "rebuild index" command can serve two purposes: restart an index build that got cancelled via 1740 rebuild an existing index that is suspected to be corrupt or fragmented For the second part it's important to clear the built flag to avoid surprises in the restart scenario. (If the server doesn't restart of course it doesn't matter.)
        Hide
        Jonathan Ellis added a comment -

        Checked with Jake. The use case he wants to support is rebuilding a corrupt index. So, we do want to rip out existing sstables, if any, when starting the rebuild. (Jake envisions this being done on a node that has been taken offline temporarily so index queries can continue to be done against other nodes.)

        Show
        Jonathan Ellis added a comment - Checked with Jake. The use case he wants to support is rebuilding a corrupt index. So, we do want to rip out existing sstables, if any, when starting the rebuild. (Jake envisions this being done on a node that has been taken offline temporarily so index queries can continue to be done against other nodes.)
        Hide
        Vijay added a comment -

        Done! Thanks.

        Show
        Vijay added a comment - Done! Thanks.
        Hide
        Vijay added a comment -

        BTW: i changed the rebuild index to just be rebuild, because i am going to do 2392 without saving to filesystem.

        Command: nt rebuild Keyspace2 Standard3 Standard3.IdxName,Standard3.IdxName1

        Show
        Vijay added a comment - BTW: i changed the rebuild index to just be rebuild, because i am going to do 2392 without saving to filesystem. Command: nt rebuild Keyspace2 Standard3 Standard3.IdxName,Standard3.IdxName1
        Hide
        Jonathan Ellis added a comment -

        Rebased, updated the nodetool command to REBUILD_INDEX (because of CASSANDRA-3483), fixed the column name SortedSet to order by comparator, and committed.

        Show
        Jonathan Ellis added a comment - Rebased, updated the nodetool command to REBUILD_INDEX (because of CASSANDRA-3483 ), fixed the column name SortedSet to order by comparator, and committed.
        Hide
        Pavel Yaskevich added a comment - - edited

        SecondaryIndexManager.setIndexRemoved(Collection<ByteBuffer>) method is broken, it calls SecondaryIndex.setIndexBuilt(ByteBuffer) instead of using SecondaryIndex.setIndexRemoved(ByteBuffer).

        Show
        Pavel Yaskevich added a comment - - edited SecondaryIndexManager.setIndexRemoved(Collection<ByteBuffer>) method is broken, it calls SecondaryIndex.setIndexBuilt(ByteBuffer) instead of using SecondaryIndex.setIndexRemoved(ByteBuffer).
        Hide
        Vijay added a comment -

        Hi Pavel, it is not broken per say... We allow the queries flow through during rebuild phase... attached patch which fixes it. Thanks!

        Show
        Vijay added a comment - Hi Pavel, it is not broken per say... We allow the queries flow through during rebuild phase... attached patch which fixes it. Thanks!
        Hide
        Jonathan Ellis added a comment -

        +1

        Show
        Jonathan Ellis added a comment - +1
        Hide
        Vijay added a comment -

        Committed thanks!

        Show
        Vijay added a comment - Committed thanks!

          People

          • Assignee:
            Vijay
            Reporter:
            Jonathan Ellis
            Reviewer:
            Jonathan Ellis
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development