Cassandra
  1. Cassandra
  2. CASSANDRA-4196

While loading data using BulkOutPutFormat gettting an exception "java.lang.ClassCastException: org.apache.cassandra.utils.Murmur3BloomFilter cannot be cast to org.apache.cassandra.utils.Murmur2BloomFilter"

    Details

      Description

      We are using cassandra-1.1 rc1 for production setup and getting following error while bulkloading data using BulkOutPutFormat.

      WARN 09:04:52,384 Failed closing IndexWriter(/cassandra/production/Data_daily/production-Data_daily-tmp-hc-2692)
      java.lang.ClassCastException: org.apache.cassandra.utils.Murmur3BloomFilter cannot be cast to org.apache.cassandra.utils.Murmur2BloomFilter
              at org.apache.cassandra.utils.FilterFactory.serialize(FilterFactory.java:50)
              at org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.close(SSTableWriter.java:410)
              at org.apache.cassandra.io.util.FileUtils.closeQuietly(FileUtils.java:94)
              at org.apache.cassandra.io.sstable.SSTableWriter.abort(SSTableWriter.java:255)
              at org.apache.cassandra.streaming.IncomingStreamReader.streamIn(IncomingStreamReader.java:154)
              at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:92)
              at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:178)
              at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:74)
       WARN 09:04:52,393 Failed closing IndexWriter(/cassandra/production/Data_daily/production-Data_daily-tmp-hc-2693)
      java.lang.ClassCastException: org.apache.cassandra.utils.Murmur3BloomFilter cannot be cast to org.apache.cassandra.utils.Murmur2BloomFilter
              at org.apache.cassandra.utils.FilterFactory.serialize(FilterFactory.java:50)
              at org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.close(SSTableWriter.java:410)
              at org.apache.cassandra.io.util.FileUtils.closeQuietly(FileUtils.java:94)
              at org.apache.cassandra.io.sstable.SSTableWriter.abort(SSTableWriter.java:255)
              at org.apache.cassandra.streaming.IncomingStreamReader.streamIn(IncomingStreamReader.java:154)
              at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:92)
              at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:178)
              at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:74)
       WARN 09:04:52,544 Failed closing IndexWriter(/cassandra/production/Data_daily/production-Data_daily-tmp-hc-2698)
      java.lang.ClassCastException: org.apache.cassandra.utils.Murmur3BloomFilter cannot be cast to org.apache.cassandra.utils.Murmur2BloomFilter
              at org.apache.cassandra.utils.FilterFactory.serialize(FilterFactory.java:50)
              at org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.close(SSTableWriter.java:410)
              at org.apache.cassandra.io.util.FileUtils.closeQuietly(FileUtils.java:94)
              at org.apache.cassandra.io.sstable.SSTableWriter.abort(SSTableWriter.java:255)
              at org.apache.cassandra.streaming.IncomingStreamReader.streamIn(IncomingStreamReader.java:154)
              at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:92)
              at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:178)
              at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:74)
      ERROR 09:04:52,544 Exception in thread Thread[Thread-39,5,main]
      [3:02:34 PM] Mariusz Dymarek: java.lang.IndexOutOfBoundsException
              at java.nio.Buffer.checkIndex(Buffer.java:520)
              at java.nio.HeapByteBuffer.getShort(HeapByteBuffer.java:289)
              at org.apache.cassandra.db.CounterColumn.create(CounterColumn.java:79)
              at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:102)
              at org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:251)
              at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:271)
              at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:228)
              at edu.stanford.ppl.concurrent.SnapTreeMap.<init>(SnapTreeMap.java:453)
              at org.apache.cassandra.db.AtomicSortedColumns$Holder.<init>(AtomicSortedColumns.java:301)
              at org.apache.cassandra.db.AtomicSortedColumns.<init>(AtomicSortedColumns.java:77)
              at org.apache.cassandra.db.AtomicSortedColumns.<init>(AtomicSortedColumns.java:48)
              at org.apache.cassandra.db.AtomicSortedColumns$1.fromSorted(AtomicSortedColumns.java:61)
              at org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:398)
              at org.apache.cassandra.io.sstable.SSTableWriter.appendFromStream(SSTableWriter.java:211)
              at org.apache.cassandra.streaming.IncomingStreamReader.streamIn(IncomingStreamReader.java:142)
              at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:92)
              at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:178)
              at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:74)
       INFO 09:04:52,548 Streaming of file /opera/log1/hadoop/mapred/local/taskTracker/jobcache/job_201111101207_320989/attempt_201111101207_320989_m_000014_0/work/tmp/production/Data_daily/production-Data_daily-hc-1-Data.db sections=1 progress=0/21544606 - 0% for org.apache.cassandra.streaming.StreamInSession@49bbb2d4 failed: requesting a retry.
      

      I tried to check into the code and it seems like "Murmur3BloomFilter" and "Murmur2BloomFilter" are recently added classes and might be causing this issue.

        Activity

        Hide
        Dave Brosius added a comment - - edited

        SSTAbleWriter.IndexWriter constructor uses

        FilterFactory.getFilter(long numElements, int targetBucketsPerElem)

        which imposes MURMUR3 always, regardless of what descriptor.filterType is set to

        it seems that IndexWriter should use this signature

        static Filter getFilter(long numElements, int targetBucketsPerElem, Type type)

        and pass descriptor.filterType

        alternatively

        FilterFactory.serialize should ignore the type, and just infer the type from the BloomFilter runtime type.

        Show
        Dave Brosius added a comment - - edited SSTAbleWriter.IndexWriter constructor uses FilterFactory.getFilter(long numElements, int targetBucketsPerElem) which imposes MURMUR3 always, regardless of what descriptor.filterType is set to it seems that IndexWriter should use this signature static Filter getFilter(long numElements, int targetBucketsPerElem, Type type) and pass descriptor.filterType alternatively FilterFactory.serialize should ignore the type, and just infer the type from the BloomFilter runtime type.
        Hide
        Vijay added a comment -

        Yeah i was almost close to complete the patch and it affects only trunk and streaming not 1.1.0...
        Alternatively the Streaming can upgrade the version of the file.

        Show
        Vijay added a comment - Yeah i was almost close to complete the patch and it affects only trunk and streaming not 1.1.0... Alternatively the Streaming can upgrade the version of the file.
        Hide
        Dave Brosius added a comment -

        pass down the descriptor.filtertype down to the filter factory when creating the bloom filter.

        against trunk

        Show
        Dave Brosius added a comment - pass down the descriptor.filtertype down to the filter factory when creating the bloom filter. against trunk
        Hide
        Pavel Yaskevich added a comment -

        So we need to reschedule it to the 1.2 instead because we have added murmur3 in trunk, is that right?

        Show
        Pavel Yaskevich added a comment - So we need to reschedule it to the 1.2 instead because we have added murmur3 in trunk, is that right?
        Hide
        Jonathan Ellis added a comment -

        yes, it doesn't look like this affects a pure 1.1 cluster at all

        Show
        Jonathan Ellis added a comment - yes, it doesn't look like this affects a pure 1.1 cluster at all
        Hide
        Vijay added a comment -

        Hi David, we need to do the same on ColumnIndex.Builder too i dont see it in the attached patch.

        Show
        Vijay added a comment - Hi David, we need to do the same on ColumnIndex.Builder too i dont see it in the attached patch.
        Hide
        Pavel Yaskevich added a comment -

        +1 with Vijay, we need to add descriptor.filterType information to the ColumnIndex.Serializer.serialize(RowIndexEntry, DataOutput) method otherwise it just assumes that BF is always Murmur3.

        Show
        Pavel Yaskevich added a comment - +1 with Vijay, we need to add descriptor.filterType information to the ColumnIndex.Serializer.serialize(RowIndexEntry, DataOutput) method otherwise it just assumes that BF is always Murmur3.
        Hide
        Dave Brosius added a comment -

        Ok, agreed.

        I'm not sure how LazilyCompactedRow is supposed to infer the filter type, for the call to new ColumnIndex.Builder.

        Show
        Dave Brosius added a comment - Ok, agreed. I'm not sure how LazilyCompactedRow is supposed to infer the filter type, for the call to new ColumnIndex.Builder.
        Hide
        Pavel Yaskevich added a comment -

        If we can't get descriptor there I think the best option would be to make serializer/deserialize explicitly use Murmur3...

        Show
        Pavel Yaskevich added a comment - If we can't get descriptor there I think the best option would be to make serializer/deserialize explicitly use Murmur3...
        Hide
        Vijay added a comment -

        CASSANDRA-4203 will make sure versions "h" SST's will not be allowed to stream into 1.2.

        IRC discussion: "run upgradesstables after the upgrade" works well so far, so we dont need to worry about fixing/supporting Murmur2 and Murmur3 in SSTW.

        Right version of the bulk-uploader should be used after 1.2 is released.

        Show
        Vijay added a comment - CASSANDRA-4203 will make sure versions "h" SST's will not be allowed to stream into 1.2. IRC discussion: "run upgradesstables after the upgrade" works well so far, so we dont need to worry about fixing/supporting Murmur2 and Murmur3 in SSTW. Right version of the bulk-uploader should be used after 1.2 is released.
        Hide
        Samarth Gahire added a comment -

        So is this issue fixed for cassandra-1.1 rc1 ? Do I need to apply a patch to resolve this? or will it be fixed only for cassandra-0.2?

        Show
        Samarth Gahire added a comment - So is this issue fixed for cassandra-1.1 rc1 ? Do I need to apply a patch to resolve this? or will it be fixed only for cassandra-0.2?
        Hide
        Pavel Yaskevich added a comment -

        This was committed to the 1.2 as Murmur3 changes are related only to that version.

        Show
        Pavel Yaskevich added a comment - This was committed to the 1 .2 as Murmur3 changes are related only to that version.
        Hide
        Sylvain Lebresne added a comment -

        I'm also confused by reading this ticket. I think it could use some on the record clarification.

        From reading this, it seems Samarth reported this on 1.1 rc1 initially (not on trunk unless there has been some off-the-record conversation), and then somehow this turns out to be a 1.2 issue? Did we remove code from 1.1 between rc1 and final that makes this a non-issue for 1.1? Is it something that does affect 1.1 (i.e. is it still possible to get the stack trace of the description) but can be simply fixed by running updadesstables after a 1.1 upgrade? Also, why is this marked as "Won't fix" if something was committed to 1.2?

        Not saying something wrong has been done here, but just want to get the records straight.

        Show
        Sylvain Lebresne added a comment - I'm also confused by reading this ticket. I think it could use some on the record clarification. From reading this, it seems Samarth reported this on 1.1 rc1 initially ( not on trunk unless there has been some off-the-record conversation), and then somehow this turns out to be a 1.2 issue? Did we remove code from 1.1 between rc1 and final that makes this a non-issue for 1.1? Is it something that does affect 1.1 (i.e. is it still possible to get the stack trace of the description) but can be simply fixed by running updadesstables after a 1.1 upgrade? Also, why is this marked as "Won't fix" if something was committed to 1.2? Not saying something wrong has been done here, but just want to get the records straight.
        Hide
        Jonathan Ellis added a comment -

        Murmur3 was only introduced into trunk, so we assume he must have been mixing cluster versions.

        CASSANDRA-4203 is what was committed to 1.2 to fix.

        Show
        Jonathan Ellis added a comment - Murmur3 was only introduced into trunk, so we assume he must have been mixing cluster versions. CASSANDRA-4203 is what was committed to 1.2 to fix.
        Hide
        Samarth Gahire added a comment -

        I Agree with Jonathan.Today After reinstalling Cassandra properly bulk-loading is working fine.We might had mixed the versions while upgrading.

        Show
        Samarth Gahire added a comment - I Agree with Jonathan.Today After reinstalling Cassandra properly bulk-loading is working fine.We might had mixed the versions while upgrading.

          People

          • Assignee:
            Dave Brosius
            Reporter:
            Samarth Gahire
            Reviewer:
            Pavel Yaskevich
          • Votes:
            1 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Time Tracking

              Estimated:
              Original Estimate - 48h
              48h
              Remaining:
              Remaining Estimate - 48h
              48h
              Logged:
              Time Spent - Not Specified
              Not Specified

                Development