Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-2182

Spooling Directory Source will not ingest data completely when a wide character appears at the edge of a buffer

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • 1.4.0
    • 1.5.0
    • Sinks+Sources
    • None

    Description

      the bug is in ResettableFileInputStream.java: int readChar().
      if the last byte of buf is only a partial of a wide character, readChar() shouldn't return -1(ResettableFileInputStream.java:186). it
      loses the remanent data in a file.

      I fix it such as:
      public synchronized int readChar() throws IOException {
      // if (!buf.hasRemaining()) {
      if(buf.limit()- buf.position < 10)

      { refillBuf(); }

      int start = buf.position();
      charBuf.clear();

      boolean isEndOfInput = false;
      if (position >= fileSize)

      { isEndOfInput = true; }

      CoderResult res = decoder.decode(buf, charBuf, isEndOfInput);
      if (res.isMalformed() || res.isUnmappable())

      { res.throwException(); }

      int delta = buf.position() - start;

      charBuf.flip();
      if (charBuf.hasRemaining())

      { char c = charBuf.get(); // don't increment the persisted location if we are in between a // surrogate pair, otherwise we may never recover if we seek() to this // location! incrPosition(delta, !Character.isHighSurrogate(c)); return c; // there may be a partial character in the decoder buffer }

      else

      { incrPosition(delta, false); return -1; }

      }

      it avoid a partial character, but have new issue. sometime, some lines of a log file have a repeated character.
      eg.
      original file: 123456
      sink file: 1233456

      Attachments

        1. ModifiedLineDeserializer.java
          8 kB
          Sven Meys
        2. Fix_for_FLUME-2182.patch
          4 kB
          Sven Meys

        Issue Links

          Activity

            People

              svenmeys Sven Meys
              syntonyliu syntony liu
              Votes:
              1 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: