Flume
  1. Flume
  2. FLUME-502

Corrupted gzip files are being created in HDFS (file attached)

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Critical Critical
    • Resolution: Won't Fix
    • Affects Version/s: v0.9.2
    • Fix Version/s: v0.9.5
    • Component/s: Sinks+Sources
    • Labels:
      None

      Description

      There are corrupted gzip files being created in HDFS. These files appear to be truncated gzip files. We've had lots of datanodes go down, so not sure if this is a contributor.

        Activity

        Hide
        Ashish Paliwal added a comment -

        Won't fix. 0.X branch not maintained anymore

        Show
        Ashish Paliwal added a comment - Won't fix. 0.X branch not maintained anymore
        Hide
        flume_lordkev added a comment -

        We're running into the same issue, although we're not using compression. We are rolling every 10 seconds, and often (but not always) end up with two files for the same time period. One file is the normal size of 65-90MB, and the other is only a few KB.

        We have about nine tailDir sinks, and only one collectorSink that is writing to HDFS.

        So far we haven't been able to find any hints towards what may be causing this behavior, especially only in some time periods but not others.

        Show
        flume_lordkev added a comment - We're running into the same issue, although we're not using compression. We are rolling every 10 seconds, and often (but not always) end up with two files for the same time period. One file is the normal size of 65-90MB, and the other is only a few KB. We have about nine tailDir sinks, and only one collectorSink that is writing to HDFS. So far we haven't been able to find any hints towards what may be causing this behavior, especially only in some time periods but not others.
        Hide
        Jonathan Hsieh added a comment -

        What are your sources and sinks here? How many collectors are going into this dir – just one?

        Show
        Jonathan Hsieh added a comment - What are your sources and sinks here? How many collectors are going into this dir – just one?
        Hide
        flume_waynemak added a comment -

        Hi, I'm a co-worker of Clay. I attached a screenshot of our S3 logs sorted by filename. Seems like the small and large files are alternating here. Not sure if this is related to this bug, but here it is Jonathan. Thanks in advance!

        Wayne

        Show
        flume_waynemak added a comment - Hi, I'm a co-worker of Clay. I attached a screenshot of our S3 logs sorted by filename. Seems like the small and large files are alternating here. Not sure if this is related to this bug, but here it is Jonathan. Thanks in advance! Wayne
        Hide
        Kim Vogt added a comment -

        It's probably related but a lot of the corrupt files we were seeing weren't 0 bytes. This little script will delete the corrupted files:

        https://gist.github.com/1040741

        Which isn't ideal, cause you lose data. You can recover most of the file with http://www.gzip.org/recover.txt, but that was getting old real fast so
        we've stopped compressing them from flume, and instead run another nightly job to compact/compress.

        Show
        Kim Vogt added a comment - It's probably related but a lot of the corrupt files we were seeing weren't 0 bytes. This little script will delete the corrupted files: https://gist.github.com/1040741 Which isn't ideal, cause you lose data. You can recover most of the file with http://www.gzip.org/recover.txt , but that was getting old real fast so we've stopped compressing them from flume, and instead run another nightly job to compact/compress.
        Hide
        Jonathan Hsieh added a comment - - edited
        Show
        Jonathan Hsieh added a comment - - edited This link is likely related, and has a workaround. http://www.justinjworkman.com/hadoop/hadoop-tips-and-tricks/2011/03/dealing-with-0-byte-files-in-hdfs/
        Hide
        Kim Vogt added a comment -

        Still seeing this behavior. Some of the corrupted files are larger unlike the one attached.

        Show
        Kim Vogt added a comment - Still seeing this behavior. Some of the corrupted files are larger unlike the one attached.

          People

          • Assignee:
            Unassigned
            Reporter:
            Kim Vogt
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development