Flume
  1. Flume
  2. FLUME-1916

HDFS sink should poll for # of active replicas. If less than required, roll the file.

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: v1.3.1
    • Fix Version/s: v1.4.0
    • Component/s: Sinks+Sources
    • Labels:
      None

      Description

      Add functionality to the HDFS sink which constantly polls the number of replicas for files being written. If the number of replicas drops below 3 (or specified number), it should immediately close the HDFS output file and start a new file that should be able to create a pipeline with the correct number of replicas.

      This is the same behavior taken by HBase for its write-ahead log (WAL) and should help us avoid hitting the more complex corner cases around failed close() calls.

      1. FLUME-1916.patch
        25 kB
        Mike Percy
      2. FLUME-1916-1.patch
        25 kB
        Mike Percy
      3. FLUME-1916-2.patch
        25 kB
        Mike Percy
      4. FLUME-1916-3.patch
        25 kB
        Mike Percy
      5. FLUME-1916-4.patch
        25 kB
        Mike Percy

        Issue Links

          Activity

          No work has yet been logged on this issue.

            People

            • Assignee:
              Mike Percy
              Reporter:
              Mike Percy
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development