Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-5583

Add DataInput.skipBytes


    • Bug
    • Status: Closed
    • Blocker
    • Resolution: Fixed
    • 4.8
    • 4.8, 6.0
    • None
    • None
    • New


      I was playing with on-the-fly checksum verification and this made me stumble upon an issue with BufferedChecksumIndexInput.

      I have some code that skips over a DataInput by reading bytes into /dev/null, eg.

        private static final byte[] SKIP_BUFFER = new byte[1024];
        private static void skipBytes(DataInput in, long numBytes) throws IOException {
          assert numBytes >= 0;
          for (long skipped = 0; skipped < numBytes; ) {
            final int toRead = (int) Math.min(numBytes - skipped, SKIP_BUFFER.length);
            in.readBytes(SKIP_BUFFER, 0, toRead);
            skipped += toRead;

      It is fine to read into this static buffer, even from multiple threads, since the content that is read doesn't matter here. However, it breaks with BufferedChecksumIndexInput because of the way that it updates the checksum:

        public void readBytes(byte[] b, int offset, int len)
          throws IOException {
          main.readBytes(b, offset, len);
          digest.update(b, offset, len);

      If you are unlucky enough so that a concurrent call to skipBytes started modifying the content of b before the call to digest.update(b, offset, len) finished, then your checksum will be wrong.

      I think we should make BufferedChecksumIndexInput read into a private buffer first instead of relying on the user-provided buffer.


        1. LUCENE-5583.patch
          7 kB
          Adrien Grand
        2. LUCENE-5583.patch
          7 kB
          Adrien Grand



            jpountz Adrien Grand
            jpountz Adrien Grand
            0 Vote for this issue
            4 Start watching this issue