Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-683

Add input/output buffering to ThriftEventSource/Sink

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.9.4
    • Fix Version/s: 0.9.5
    • Component/s: Sinks+Sources
    • Labels:
      None

      Description

      Some simple buffering should improve network performance quite a bit. I've so far implemented this only on the sending side (ThriftEventSink) and performance seems to have improved nicely, though I don't have good numbers for this yet. There are two possible solutions:

      1) Use the native Thrift TFastFramedTransport (buffered version of TFramedTransport); lazy socket changes will need to be ported over (for TSaneServerSocket)
      2) Wrap the existing code with a buffered version (see TBufferedTransport in patch)

      I used the latter method for a simple test implementation only on the sending (sink) side.

        Attachments

        1. ASF.LICENSE.NOT.GRANTED--thrift_buffer2.patch
          4 kB
          Disabled imported user
        2. ASF.LICENSE.NOT.GRANTED--thrift_buffer.patch
          3 kB
          Disabled imported user

          Activity

            People

            • Assignee:
              flume_chetan Disabled imported user
              Reporter:
              flume_csarva Disabled imported user
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: