Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-1714

Improve handling of HDFS sink .tmp files after crash


    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:


      Currently, the .tmp files left after a system or Flume client crash are never cleaned up, and several users have noted that it would be better if Flume itself took care of this.

      This is actually a complicated issue, with multiple facets. These include:

      1. We would need to persist the in-progress filenames somewhere, probably on the agent's local FS. This is not very hard.
      2. At startup, we would need to handle the files in some way to guarantee at least one of the following:
        • Mark it as a potentially partial file somehow when renaming from .tmp
        • Ensure that the file format is valid before renaming it from .tmp
          • This 2nd option is actually harder than it sounds, since arbitrary serializers may be plugged in. Say it's an XML serializer, then we would need some way to programmatically read (deserialize) the file, throw away any potentially unfinished records at the end (this is OK since the transaction must not have been committed), then re-serialize the file with all the valid records and correct opening/closing tags.
          • General deserialization / recovery APIs would need to be added to support this, and this would need to be very carefully designed and implemented in order to work. In the end, it also seems likely that if this a complex thing (sounds complex) then most people would rely on out-of-the-box implementations (supported file formats) to get this functionality, unless they are building on top of abstract classes (e.g. for XML schema handling) to help accomplish this.




            • Assignee:
              mpercy Mike Percy
            • Votes:
              1 Vote for this issue
              3 Start watching this issue


              • Created: