Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-2648

File Channel restarts can fail if metadata file creation fails during file roll

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • None
    • None

    Description

      If during the creation of a new File in the data dirs, the creation of the metadata file fails - then the log-n file is simply left as an orphan. During replay though, this causes issues - as we try to read the metadata file at this time, causing the replay to fail and the file channel being unable to restart.

      To fix this we must remove the data file if the creation of the metadata file fails. Or we could simply ignore files whose metadata files don't exist (though in this case, user error could cause data loss). If the delete fails, then there would a problem though. So if the metafile is missing during replay, we must still fail - though more gracefully, and ask the operator to remove the corresponding data file. It might actually be possible to figure out if the data file has any content at all - by checking if size == 0 (need to verify this) - in that case we can probably remove/ignore that file.

      I will submit a patch for this soon.

      Attachments

        Activity

          People

            hshreedharan Hari Shreedharan
            hshreedharan Hari Shreedharan
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated: