Cassandra
  1. Cassandra
  2. CASSANDRA-4209

Randomly getting "Invalid bytes remaining after an end-of-component at component1" exceptions

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Cannot Reproduce
    • Fix Version/s: None
    • Component/s: Core
    • Labels:
      None
    • Environment:

      Linux version 2.6.21.7-2.fc8xen (mockbuild@xenbuilder4.fedora.phx.redhat.com)

      Description

      I have a CF with composite column sort:

      ColumnFamily: MyCF
        Key Validation Class: org.apache.cassandra.db.marshal.UTF8Type
        Default column value validator: org.apache.cassandra.db.marshal.UTF8Type
        Columns sorted by: org.apache.cassandra.db.marshal.CompositeType(org.apache.cassandra.db.marshal.AsciiType,org.apache.cassandra.db.marshal.IntegerType)
        Row cache size / save period in seconds / keys to save : 0.0/0/all
        Row Cache Provider: org.apache.cassandra.cache.SerializingCacheProvider
        Key cache size / save period in seconds: 200000.0/14400
        GC grace seconds: 864000
        Compaction min/max thresholds: 4/32
        Read repair chance: 0.1
        Replicate on write: true
        Bloom Filter FP chance: default
        Built indexes: []
        Compaction Strategy: org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy
        Compression Options:
          chunk_length_kb: 64
          sstable_compression: org.apache.cassandra.io.compress.SnappyCompressor
      

      I'm using phpcassa to communicate with my Cassandra cluster.

      I'm getting the following exception from time to time while reading/writing to this CF:

      exception 'cassandra_InvalidRequestException' with message
      'Invalid bytes remaining after an end-of-component at component1' in /phpcassa/thrift/packages/cassandra/Cassandra.php:256\nStack trace:
      #0 /phpcassa/thrift/packages/cassandra/Cassandra.php(256): thrift_protocol_read_binary(Object(TBinaryProtocolAccelerated), 'cassandra_Cassa...', false)
      #1 /phpcassa/thrift/packages/cassandra/Cassandra.php(229): CassandraClient->recv_get_slice()
      #2 [internal function]: CassandraClient->get_slice('6034-28141406', Object(cassandra_ColumnParent), Object(cassandra_SlicePredicate), 2)
      ...
      

      I've thoroughly checked my code, I'm always sending the same parameters to Cassandra, and there's no format error. My column name request is always AsciiType:IntegerType.

      The weird part is that this is completely random. If I run a get() query once, I get the exception, but the following request with the same parameters succeeds.

        Activity

        Alon Peer created issue -
        Hide
        Sylvain Lebresne added a comment -

        I don't want to point fingers but I would open a bug report on the phpcassa side too as this sound a lot like a race in phpcassa packing the bytes for the composite name (especially the random aspect of it).

        Show
        Sylvain Lebresne added a comment - I don't want to point fingers but I would open a bug report on the phpcassa side too as this sound a lot like a race in phpcassa packing the bytes for the composite name (especially the random aspect of it).
        Hide
        Alon Peer added a comment -

        I did: https://github.com/thobbs/phpcassa/issues/66, but their response time isn't that great.

        I'll post your thoughts on this there too. But how can this be reproduced in your opinion? I only saw this happening on our production environment, with thousands of concurrent requests. I wasn't able to reproduce this permanently on dev environment.

        Show
        Alon Peer added a comment - I did: https://github.com/thobbs/phpcassa/issues/66 , but their response time isn't that great. I'll post your thoughts on this there too. But how can this be reproduced in your opinion? I only saw this happening on our production environment, with thousands of concurrent requests. I wasn't able to reproduce this permanently on dev environment.
        Hide
        Alon Peer added a comment -

        BTW, I don't understand how it can be a race, since the process is serial, single thread.

        Show
        Alon Peer added a comment - BTW, I don't understand how it can be a race, since the process is serial, single thread.
        Gavin made changes -
        Field Original Value New Value
        Workflow no-reopen-closed, patch-avail [ 12665131 ] patch-available, re-open possible [ 12749638 ]
        Gavin made changes -
        Workflow patch-available, re-open possible [ 12749638 ] reopen-resolved, no closed status, patch-avail, testing [ 12757147 ]
        Jonathan Ellis made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Resolution Cannot Reproduce [ 5 ]

          People

          • Assignee:
            Unassigned
            Reporter:
            Alon Peer
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development