Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-10392

Allow Cassandra to trace to custom tracing implementations

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Fix Version/s: 3.4
    • Component/s: None
    • Labels:
      None

      Description

      It can be possible to use an external tracing solution in Cassandra by abstracting out the writing of tracing to system_traces tables in the tracing package to separate implementation classes and leaving abstract classes in place that define the interface and behaviour otherwise of C* tracing.

      Then via a system property "cassandra.custom_tracing_class" the Tracing class implementation could be swapped out with something third party.

      An example of this is adding Zipkin tracing into Cassandra in the Summit presentation. Code for the implemented Zipkin plugin can be found at https://github.com/thelastpickle/cassandra-zipkin-tracing/

      In addition this patch passes the custom payload through into the tracing session allowing a third party tracing solution like Zipkin to do full-stack tracing from clients through and into Cassandra.

        Activity

        Hide
        michaelsembwever mck added a comment -

        Initial patch version. Like said, there's edges to polish here but i'm after community input if the general idea and direction works.

        Show
        michaelsembwever mck added a comment - Initial patch version. Like said, there's edges to polish here but i'm after community input if the general idea and direction works.
        Hide
        michaelsembwever mck added a comment -

        updated description and patch.

        Show
        michaelsembwever mck added a comment - updated description and patch.
        Hide
        tjake T Jake Luciani added a comment -

        I think this is a very useful change for operators!

        It would be useful to test this across nodes with a cassandra-dtest, is that something you could provide? I'm specifically talking about verifying the message passing makes it across multiple nodes.

        Show
        tjake T Jake Luciani added a comment - I think this is a very useful change for operators! It would be useful to test this across nodes with a cassandra-dtest, is that something you could provide? I'm specifically talking about verifying the message passing makes it across multiple nodes.
        Hide
        michaelsembwever mck added a comment -

        Sounds smart T Jake Luciani. Will do.

        Show
        michaelsembwever mck added a comment - Sounds smart T Jake Luciani . Will do.
        Hide
        jbellis Jonathan Ellis added a comment -

        Nice blog post! (http://thelastpickle.com/blog/2015/12/07/using-zipkin-for-full-stack-tracing-including-cassandra.html)

        Are you looking to get the ball rolling again here?

        Show
        jbellis Jonathan Ellis added a comment - Nice blog post! ( http://thelastpickle.com/blog/2015/12/07/using-zipkin-for-full-stack-tracing-including-cassandra.html ) Are you looking to get the ball rolling again here?
        Hide
        michaelsembwever mck added a comment -

        Yes I am Jonathan Ellis. I've started on the dtest, just dragging my feet on it

        Show
        michaelsembwever mck added a comment - Yes I am Jonathan Ellis . I've started on the dtest, just dragging my feet on it
        Hide
        cburroughs Chris Burroughs added a comment -

        One minor suggestion: It would be nice if multiple tracers could be enabled at the same time. For example, to use 'normal' tracing to debug a problem with a zipkin cluster, or to enable both zipkin and SOME_OTHER_TRACER for comparison.

        Show
        cburroughs Chris Burroughs added a comment - One minor suggestion: It would be nice if multiple tracers could be enabled at the same time. For example, to use 'normal' tracing to debug a problem with a zipkin cluster, or to enable both zipkin and SOME_OTHER_TRACER for comparison.
        Hide
        michaelsembwever mck added a comment -

        In the patch provided there's nothing in the design preventing this as a simple addition.

        Show
        michaelsembwever mck added a comment - In the patch provided there's nothing in the design preventing this as a simple addition.
        Hide
        michaelsembwever mck added a comment - - edited

        added barebones dtest. working on "verifying the message passing makes it across multiple nodes".

        Show
        michaelsembwever mck added a comment - - edited added barebones dtest. working on "verifying the message passing makes it across multiple nodes".
        Hide
        michaelsembwever mck added a comment -

        working on "verifying the message passing makes it across multiple nodes".

        done

        Show
        michaelsembwever mck added a comment - working on "verifying the message passing makes it across multiple nodes". done
        Hide
        tjake T Jake Luciani added a comment -

        Could you rebase the C* patch against trunk? then I can kick off the test runs.

        Show
        tjake T Jake Luciani added a comment - Could you rebase the C* patch against trunk? then I can kick off the test runs.
        Hide
        michaelsembwever mck added a comment -

        > Could you rebase the C* patch against trunk? then I can kick off the test runs.

        done T Jake Luciani.

        Show
        michaelsembwever mck added a comment - > Could you rebase the C* patch against trunk? then I can kick off the test runs. done T Jake Luciani .
        Hide
        tjake T Jake Luciani added a comment -

        I ran the utests and dtests against cassci

        trunk unittests
        trunk dtests

        There is one related failing dtest the materialized view one.

        CassandraDaemon.java:195 - Exception in thread Thread[MessagingService-Incoming-/127.0.0.2,5,main]\njava.lang.AssertionError: null\n\tat org.apache.cassandra.tracing.TraceState.<init>(TraceState.java:68) ~[main/:na]\n\tat org.apache.cassandra.tracing.ExpiredTraceState.<init>(ExpiredTraceState.java:30) ~[main/:na]\n\tat org.apache.cassandra.tracing.Tracing.initializeFromMessage(Tracing.java:244) ~[main/:na]\n\tat org.apache.cassandra.net.MessagingService.receive(MessagingService.java:815) ~[main/:na]\n\tat org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:198) ~[main/:na]\n\tat org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:178) ~[main/:na]\n\tat org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:92) ~[main/:na]']
        
        Show
        tjake T Jake Luciani added a comment - I ran the utests and dtests against cassci trunk unittests trunk dtests There is one related failing dtest the materialized view one. CassandraDaemon.java:195 - Exception in thread Thread [MessagingService-Incoming-/127.0.0.2,5,main]\njava.lang.AssertionError: null \n\tat org.apache.cassandra.tracing.TraceState.<init>(TraceState.java:68) ~[main/:na]\n\tat org.apache.cassandra.tracing.ExpiredTraceState.<init>(ExpiredTraceState.java:30) ~[main/:na]\n\tat org.apache.cassandra.tracing.Tracing.initializeFromMessage(Tracing.java:244) ~[main/:na]\n\tat org.apache.cassandra.net.MessagingService.receive(MessagingService.java:815) ~[main/:na]\n\tat org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:198) ~[main/:na]\n\tat org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:178) ~[main/:na]\n\tat org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:92) ~[main/:na]']
        Hide
        tjake T Jake Luciani added a comment -

        I pushed a fix ( ExpiredTraceState was passing in null)

        Tests run clean now. will commit.

        trunk unittests
        trunk dtests

        Show
        tjake T Jake Luciani added a comment - I pushed a fix ( ExpiredTraceState was passing in null) Tests run clean now. will commit. trunk unittests trunk dtests
        Hide
        tjake T Jake Luciani added a comment - - edited

        committed in bf25e668f416b1c279986d1b23fee2a0192d8022

        Thanks!

        Show
        tjake T Jake Luciani added a comment - - edited committed in bf25e668f416b1c279986d1b23fee2a0192d8022 Thanks!
        Hide
        michaelsembwever mck added a comment -

        sweet, was just running the same patch through tests. Thanks T Jake Luciani!

        Show
        michaelsembwever mck added a comment - sweet, was just running the same patch through tests. Thanks T Jake Luciani !

          People

          • Assignee:
            michaelsembwever mck
            Reporter:
            michaelsembwever mck
            Reviewer:
            T Jake Luciani
            Tester:
            T Jake Luciani
          • Votes:
            9 Vote for this issue
            Watchers:
            20 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development