Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-2360

Spooling Source: LineDeserializer incorrectly stops reading events if UTF-8 char occurs at end of buffer

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 1.4.0
    • Fix Version/s: None
    • Component/s: Sinks+Sources
    • Labels:
      None
    • Environment:

      CDH 4.4

      Description

      We are using a Spooling source to read from files and have noticed that there are situations where the source will stop reading in the middle of a file (see attached example file). After much debugging, I have determine that this occurs if the buffer in ResettableFileInputStream happens to fill when in the middle of handling a UTF-8 character. In this case readChar() will return -1 which ends the read lines loop in LineSerializer. Running the spool source with the attached file should result in > 5400 events in the channel but now it will only return ~3300.

        Attachments

        1. FBX_BID.80.log.2014-04-10-00-47.log
          7.85 MB
          Keith Wright

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                keithwrightbos Keith Wright
              • Votes:
                2 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated: