Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Fix Version/s: 2.0.6
    • Component/s: None
    • Labels:
      None
    • Environment:

      Cluster of 12 nodes, each node with 256-384 vnodes. RPC threads capped at 2048.

      Description

      We're writing from Pig map tasks, about 20 million records of one integer each.
      For the case of 12 nodes, with 256-384 vnodes per node, we get around 4000 threads per mapper. This obviously overloads the nodes, since the number of RPC threads are capped, and the write fails.
      Also, each transfer is only in the order of a few bytes of payload. Clearly batching is a good solution.

        Attachments

        1. batchWrite.txt
          7 kB
          Christian Rolf
        2. inet_patch.diff
          2 kB
          Christian Rolf

          Activity

            People

            • Assignee:
              ccrolf Christian Rolf
              Reporter:
              ccrolf Christian Rolf
              Reviewer:
              Jonathan Ellis
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: