Hadoop Common
  1. Hadoop Common
  2. HADOOP-6834

TFile.append compares initial key against null lastKey

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.20.1, 0.20.2
    • Fix Version/s: 0.22.0
    • Component/s: io
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The following code in TFile.KeyReigster.close:

      byte[] lastKey = lastKeyBufferOS.getBuffer();
      int lastLen = lastKeyBufferOS.size();
      if (tfileMeta.getComparator().compare(key, 0, len, lastKey, 0,
      lastLen) < 0)

      { throw new IOException("Keys are not added in sorted order"); }

      compares the initial key (passed in via TFile.Writer.append) against a technically NULL lastKey. lastKey is not initialized until after the first call to TFile.Writer.append. The underlying RawComparator interface used for comparisons does not stipulate the proper behavior when either length 1 or length 2 is zero. In the case of LongWritable, its WritableComparator implementation does an unsafe read on the passed in byte arrays b1 and b2. Since TFile pre-allocates the buffer used for storing lastKey, this passes a valid buffer with zero count to LongWritable's comparator, which ignores length and thus produces incorrect results.

        Activity

          People

          • Assignee:
            Hong Tang
            Reporter:
            Ahad Rana
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development