Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-1916

HDFS sink should poll for # of active replicas. If less than required, roll the file.

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.3.1
    • Fix Version/s: 1.4.0
    • Component/s: Sinks+Sources
    • Labels:
      None

      Description

      Add functionality to the HDFS sink which constantly polls the number of replicas for files being written. If the number of replicas drops below 3 (or specified number), it should immediately close the HDFS output file and start a new file that should be able to create a pipeline with the correct number of replicas.

      This is the same behavior taken by HBase for its write-ahead log (WAL) and should help us avoid hitting the more complex corner cases around failed close() calls.

        Attachments

        1. FLUME-1916.patch
          25 kB
          Mike Percy
        2. FLUME-1916-1.patch
          25 kB
          Mike Percy
        3. FLUME-1916-2.patch
          25 kB
          Mike Percy
        4. FLUME-1916-3.patch
          25 kB
          Mike Percy
        5. FLUME-1916-4.patch
          25 kB
          Mike Percy

          Issue Links

            Activity

              People

              • Assignee:
                mpercy Mike Percy
                Reporter:
                mpercy Mike Percy
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: