Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-1916

HDFS sink should poll for # of active replicas. If less than required, roll the file.

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 1.3.1
    • 1.4.0
    • Sinks+Sources
    • None

    Description

      Add functionality to the HDFS sink which constantly polls the number of replicas for files being written. If the number of replicas drops below 3 (or specified number), it should immediately close the HDFS output file and start a new file that should be able to create a pipeline with the correct number of replicas.

      This is the same behavior taken by HBase for its write-ahead log (WAL) and should help us avoid hitting the more complex corner cases around failed close() calls.

      Attachments

        1. FLUME-1916.patch
          25 kB
          Mike Percy
        2. FLUME-1916-1.patch
          25 kB
          Mike Percy
        3. FLUME-1916-2.patch
          25 kB
          Mike Percy
        4. FLUME-1916-3.patch
          25 kB
          Mike Percy
        5. FLUME-1916-4.patch
          25 kB
          Mike Percy

        Issue Links

          Activity

            People

              mpercy Mike Percy
              mpercy Mike Percy
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: