Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Minor Minor
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Labels:
      None
    • Tags:
      rcfile

      Description

      Some potential issues with RCFile

      1. Remove unwanted synchronized modifiers on the methods of RCFile. As per yongqiang he, the class is not meant to be thread-safe (and it is not). Might as well get rid of the confusing and performance-impacting lock acquisitions.

      2. Record Length overstated for compressed files. IIUC, the key compression happens after we have written the record length.

            int keyLength = key.getSize();
            if (keyLength < 0) {
              throw new IOException("negative length keys not allowed: " + key);
            }
      
            out.writeInt(keyLength + valueLength); // total record length
            out.writeInt(keyLength); // key portion length
            if (!isCompressed()) {
              out.writeInt(keyLength);
              key.write(out); // key
            } else {
              keyCompressionBuffer.reset();
              keyDeflateFilter.resetState();
              key.write(keyDeflateOut);
              keyDeflateOut.flush();
              keyDeflateFilter.finish();
              int compressedKeyLen = keyCompressionBuffer.getLength();
              out.writeInt(compressedKeyLen);
              out.write(keyCompressionBuffer.getData(), 0, compressedKeyLen);
            }
      

      3. For sequence file compatibility, the compressed key length should be the next field to record length, not the uncompressed key length.

      1. HIVE.2065.patch.0.txt
        113 kB
        Krishna Kumar
      2. HIVE.2065.patch.1.txt
        90 kB
        Krishna Kumar
      3. proposal.png
        152 kB
        Krishna Kumar
      4. Slide1.png
        121 kB
        Krishna Kumar

        Activity

        No work has yet been logged on this issue.

          People

          • Assignee:
            Krishna Kumar
            Reporter:
            Krishna Kumar
          • Votes:
            1 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:

              Development