Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-3020

Improve HDFSEventSink Escape Ingestion by more then 10x by not getting InetAddress on every record

Agile BoardAttach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.8.0
    • Component/s: None
    • Labels:
      None

      Description

      If you are use escaping the current code will call InetAddress on every record. Which will result is a huge impact to performance.

      TotalTime,8403,
      totalEventTakeTime,1498,
      totalWriteTime,1981,
      totalWriterSetupTime,65,
      commitTime,201,
      flushTime,18,
      startTrans,7,

      The rest is all InetAddress

        Attachments

        1. flume-3020.patch.2
          2 kB
          Theodore michael Malaska
        2. flume-3020.patch
          2 kB
          Theodore michael Malaska

          Activity
          $i18n.getText('security.level.explanation', $currentSelection) Viewable by All Users
          Cancel

            People

            • Assignee:
              ted.m Theodore michael Malaska
              Reporter:
              ted.m Theodore michael Malaska

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment