Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-2182

Spooling Directory Source will not ingest data completely when a wide character appears at the edge of a buffer

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 1.4.0
    • Fix Version/s: 1.5.0
    • Component/s: Sinks+Sources
    • Labels:
      None

      Description

      the bug is in ResettableFileInputStream.java: int readChar().
      if the last byte of buf is only a partial of a wide character, readChar() shouldn't return -1(ResettableFileInputStream.java:186). it
      loses the remanent data in a file.

      I fix it such as:
      public synchronized int readChar() throws IOException {
      // if (!buf.hasRemaining()) {
      if(buf.limit()- buf.position < 10)

      { refillBuf(); }

      int start = buf.position();
      charBuf.clear();

      boolean isEndOfInput = false;
      if (position >= fileSize)

      { isEndOfInput = true; }

      CoderResult res = decoder.decode(buf, charBuf, isEndOfInput);
      if (res.isMalformed() || res.isUnmappable())

      { res.throwException(); }

      int delta = buf.position() - start;

      charBuf.flip();
      if (charBuf.hasRemaining())

      { char c = charBuf.get(); // don't increment the persisted location if we are in between a // surrogate pair, otherwise we may never recover if we seek() to this // location! incrPosition(delta, !Character.isHighSurrogate(c)); return c; // there may be a partial character in the decoder buffer }

      else

      { incrPosition(delta, false); return -1; }

      }

      it avoid a partial character, but have new issue. sometime, some lines of a log file have a repeated character.
      eg.
      original file: 123456
      sink file: 1233456

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                svenmeys Sven Meys
                Reporter:
                syntonyliu syntony liu
              • Votes:
                1 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: