Cassandra
  1. Cassandra
  2. CASSANDRA-2683

Allow writes to bypass the commit log for certain keyspaces

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Fix Version/s: 0.8.1
    • Component/s: Core
    • Labels:
      None

      Description

      Some data in cassandra does not need durability on a single node. Also ec2 users often wish to disable the commitlog since they are using the ephemeral disks.

      This patch let's you set bypass_commitlog=true in the CfMetaData, the effect being RowMutations on ColumnFamilies with this flag will bypass the commit log

        Activity

        Hide
        T Jake Luciani added a comment -

        Seems like thrift generated code changed a bunch, I'm using the thrift-0.6 compiler.

        Show
        T Jake Luciani added a comment - Seems like thrift generated code changed a bunch, I'm using the thrift-0.6 compiler.
        Hide
        Jonathan Ellis added a comment -

        the serializer approach to skipping CL seems excessively complicated. why not just have RMVH and RM.apply pass cfs.metadata.bypass_commitlog as the boolean writeCommitlog parameter?

        (note that RMVH is basically inlining RM.apply in its own Table.apply call, if you fix that then there will be only one code path so even less risk of confusion.)

        Show
        Jonathan Ellis added a comment - the serializer approach to skipping CL seems excessively complicated. why not just have RMVH and RM.apply pass cfs.metadata.bypass_commitlog as the boolean writeCommitlog parameter? (note that RMVH is basically inlining RM.apply in its own Table.apply call, if you fix that then there will be only one code path so even less risk of confusion.)
        Hide
        Jonathan Ellis added a comment -

        Seems like thrift generated code changed a bunch, I'm using the thrift-0.6 compiler.

        go ahead and commit a re-gen of the existing IDL first to keep the history un-muddled.

        Show
        Jonathan Ellis added a comment - Seems like thrift generated code changed a bunch, I'm using the thrift-0.6 compiler. go ahead and commit a re-gen of the existing IDL first to keep the history un-muddled.
        Hide
        Jonathan Ellis added a comment -

        I think there's a further wrinkle here: Table.apply and RM are per-keyspace, but CL-skipping is per-CF. (Is that why you took the serializer approach? If so, I'm a bit slow.)

        I'm still not a fan of making it more complex, though. I'd rather just && the bypass values together and if any CF wants the CL, we throw the whole RM into it. IMO mixing bypassed- and non- in the same batch is not something we should encourage anyway.

        Show
        Jonathan Ellis added a comment - I think there's a further wrinkle here: Table.apply and RM are per-keyspace, but CL-skipping is per-CF. (Is that why you took the serializer approach? If so, I'm a bit slow.) I'm still not a fan of making it more complex, though. I'd rather just && the bypass values together and if any CF wants the CL, we throw the whole RM into it. IMO mixing bypassed- and non- in the same batch is not something we should encourage anyway.
        Hide
        T Jake Luciani added a comment -

        Right, the problem there is the mixed case...

        IMO sending a batch_mutate with a key being used across many CFs in the same mutation is important.

        Show
        T Jake Luciani added a comment - Right, the problem there is the mixed case... IMO sending a batch_mutate with a key being used across many CFs in the same mutation is important.
        Hide
        T Jake Luciani added a comment -

        I'd rather just && the bypass values together and if any CF wants the CL, we throw the whole RM into it

        that would make the CL logic non-deterministic since the flag would be respected depending on what gets sent with it...

        I was going to group the non-cl vs cl applies together but that would require another container copy. My patch tries to make the check as low impact as possible...

        Show
        T Jake Luciani added a comment - I'd rather just && the bypass values together and if any CF wants the CL, we throw the whole RM into it that would make the CL logic non-deterministic since the flag would be respected depending on what gets sent with it... I was going to group the non-cl vs cl applies together but that would require another container copy. My patch tries to make the check as low impact as possible...
        Hide
        T Jake Luciani added a comment -

        Making this a per-KS option would make the logic cleaner in both cases... I'll roll another patch with that change

        Show
        T Jake Luciani added a comment - Making this a per-KS option would make the logic cleaner in both cases... I'll roll another patch with that change
        Hide
        Jonathan Ellis added a comment -

        Looks good.

        Final nit: "don't do something" boolean options are not as easily understood as "do something" – would rather call this "durable_writes" or similar.

        Show
        Jonathan Ellis added a comment - Looks good. Final nit: "don't do something" boolean options are not as easily understood as "do something" – would rather call this "durable_writes" or similar.
        Hide
        T Jake Luciani added a comment -

        done

        Show
        T Jake Luciani added a comment - done
        Hide
        Jonathan Ellis added a comment -

        +1 (please squash for commit)

        Show
        Jonathan Ellis added a comment - +1 (please squash for commit)
        Hide
        T Jake Luciani added a comment -

        committed

        Show
        T Jake Luciani added a comment - committed
        Hide
        Hudson added a comment -

        Integrated in Cassandra-0.8 #135 (See https://builds.apache.org/hudson/job/Cassandra-0.8/135/)
        Allow writes to bypass the commit log for certain keyspaces

        Patch by tjake; reviewed by jbellis for CASSANDRA-2683

        jake : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1127505
        Files :

        • /cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/db/RowMutation.java
        • /cassandra/branches/cassandra-0.8/test/unit/org/apache/cassandra/SchemaLoader.java
        • /cassandra/branches/cassandra-0.8/test/unit/org/apache/cassandra/db/ReadMessageTest.java
        • /cassandra/branches/cassandra-0.8/CHANGES.txt
        • /cassandra/branches/cassandra-0.8/src/avro/internode.genavro
        • /cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/db/RowMutationVerbHandler.java
        • /cassandra/branches/cassandra-0.8/interface/cassandra.thrift
        • /cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/config/KSMetaData.java
        • /cassandra/branches/cassandra-0.8/interface/thrift/gen-java/org/apache/cassandra/thrift/KsDef.java
        Show
        Hudson added a comment - Integrated in Cassandra-0.8 #135 (See https://builds.apache.org/hudson/job/Cassandra-0.8/135/ ) Allow writes to bypass the commit log for certain keyspaces Patch by tjake; reviewed by jbellis for CASSANDRA-2683 jake : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1127505 Files : /cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/db/RowMutation.java /cassandra/branches/cassandra-0.8/test/unit/org/apache/cassandra/SchemaLoader.java /cassandra/branches/cassandra-0.8/test/unit/org/apache/cassandra/db/ReadMessageTest.java /cassandra/branches/cassandra-0.8/CHANGES.txt /cassandra/branches/cassandra-0.8/src/avro/internode.genavro /cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/db/RowMutationVerbHandler.java /cassandra/branches/cassandra-0.8/interface/cassandra.thrift /cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/config/KSMetaData.java /cassandra/branches/cassandra-0.8/interface/thrift/gen-java/org/apache/cassandra/thrift/KsDef.java
        Hide
        Hudson added a comment -

        Integrated in Cassandra-0.8 #136 (See https://builds.apache.org/hudson/job/Cassandra-0.8/136/)
        add cassandra-cli support for durable_writes flag

        Patch by xedin; reviewed by tjake for CASSANDRA-2683

        jake : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1127528
        Files :

        • /cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/cli/CliClient.java
        • /cassandra/branches/cassandra-0.8/src/resources/org/apache/cassandra/cli/CliHelp.yaml
        • /cassandra/branches/cassandra-0.8/test/unit/org/apache/cassandra/cli/CliTest.java
        Show
        Hudson added a comment - Integrated in Cassandra-0.8 #136 (See https://builds.apache.org/hudson/job/Cassandra-0.8/136/ ) add cassandra-cli support for durable_writes flag Patch by xedin; reviewed by tjake for CASSANDRA-2683 jake : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1127528 Files : /cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/cli/CliClient.java /cassandra/branches/cassandra-0.8/src/resources/org/apache/cassandra/cli/CliHelp.yaml /cassandra/branches/cassandra-0.8/test/unit/org/apache/cassandra/cli/CliTest.java

          People

          • Assignee:
            T Jake Luciani
            Reporter:
            T Jake Luciani
            Reviewer:
            Jonathan Ellis
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development