Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-6665

Batching in CqlRecordWriter

Log workAgile BoardRank to TopRank to BottomAttach filesAttach ScreenshotBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Low
    • Resolution: Fixed
    • 2.0.6
    • None
    • None
    • Cluster of 12 nodes, each node with 256-384 vnodes. RPC threads capped at 2048.

    Description

      We're writing from Pig map tasks, about 20 million records of one integer each.
      For the case of 12 nodes, with 256-384 vnodes per node, we get around 4000 threads per mapper. This obviously overloads the nodes, since the number of RPC threads are capped, and the write fails.
      Also, each transfer is only in the order of a few bytes of payload. Clearly batching is a good solution.

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            ccrolf Christian Rolf Assign to me
            ccrolf Christian Rolf
            Christian Rolf
            Jonathan Ellis
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment