Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-4726

Streaming sstables during repair is limited by compression

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Normal
    • Resolution: Duplicate
    • Fix Version/s: 1.2.0
    • Component/s: None
    • Labels:
      None
    • Severity:
      Normal

      Description

      In our benchmarks of node rebuilds, streaming SSTables is limited by sender uncompressing them using snappy and compressing using lzf. The receiver has to reverse this process and is likewise bottlenecked in compression code. We are therefore only able to utilize less than 1/2 of receiver's 1gbps network link in a test with one receiver and two senders.

      If sstable is already compressed using snappy, it should be just sent to receiver as is.

      Here are hprof fragments excluding socket accepts and reads by idle threads:

      Sender:

      4 3.14% 95.51% 6128 302816 com.ning.compress.lzf.ChunkEncoder.tryCompress
      5 1.93% 97.44% 3761 302815 java.net.SocketOutputStream.socketWrite0
      6 1.27% 98.71% 2477 301463 java.util.zip.CRC32.updateBytes
      7 0.41% 99.12% 798 302822 com.ning.compress.lzf.ChunkEncoder.tryCompress
      8 0.31% 99.43% 602 301465 org.xerial.snappy.SnappyNative.rawUncompress
      9 0.18% 99.60% 349 302820 java.net.SocketInputStream.socketRead0
      10 0.14% 99.75% 282 302817 java.net.SocketOutputStream.socketWrite0
      11 0.14% 99.89% 276 301468 java.io.FileInputStream.readBytes
      12 0.03% 99.92% 52 301929 sun.nio.ch.FileDispatcher.preClose0

      Receiver:

      4 7.15% 84.63% 2782 300893 org.xerial.snappy.SnappyNative.rawCompress
      5 3.16% 87.79% 1232 300895 java.util.zip.CRC32.updateBytes
      6 2.83% 90.62% 1101 300896 com.ning.compress.lzf.LZFDecoder.decompressChunk
      7 1.50% 92.12% 583 300916 java.io.DataInputStream.readFully
      8 1.46% 93.57% 568 300921 java.io.FileDescriptor.sync
      9 1.16% 94.73% 451 300910 java.io.RandomAccessFile.writeBytes
      10 0.47% 95.20% 184 300949 sun.nio.ch.NativeThread.current
      11 0.45% 95.66% 177 300926 java.io.RandomAccessFile.getFilePointer
      12 0.31% 95.97% 121 300935 org.apache.cassandra.io.util.SequentialWriter.write

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              okibirev Oleg Kibirev
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: