Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-3020

Improve HDFSEventSink Escape Ingestion by more then 10x by not getting InetAddress on every record

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 1.8.0
    • None
    • None

    Description

      If you are use escaping the current code will call InetAddress on every record. Which will result is a huge impact to performance.

      TotalTime,8403,
      totalEventTakeTime,1498,
      totalWriteTime,1981,
      totalWriterSetupTime,65,
      commitTime,201,
      flushTime,18,
      startTrans,7,

      The rest is all InetAddress

      Attachments

        1. flume-3020.patch
          2 kB
          Theodore michael Malaska
        2. flume-3020.patch.2
          2 kB
          Theodore michael Malaska

        Activity
          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            ted.m Theodore michael Malaska
            ted.m Theodore michael Malaska
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment