Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-2357

HDFS sink should retry closing files that previously had close errors

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.4.0
    • Fix Version/s: 1.5.0
    • Component/s: Sinks+Sources
    • Labels:
      None

      Description

      When the AbstractHDFSWriter fails to close a file (due to exceeding the callTimeout or other hdfs issues), it will leave the the file open and never try again. The only way to close the open files is to restart the flume agent. There should be a configurable option to allow the sink to retry to close files that had previously failed to close.

        Attachments

        1. FLUME-2357.patch
          55 kB
          Hari Shreedharan
        2. FLUME-2357-1.patch
          62 kB
          Hari Shreedharan
        3. FLUME-2357-2.patch
          62 kB
          Hari Shreedharan
        4. FLUME-2357-3.patch
          62 kB
          Hari Shreedharan

          Activity

            People

            • Assignee:
              hshreedharan Hari Shreedharan
              Reporter:
              pdvorak Patrick Dvorak

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment