Uploaded image for project: 'Thrift'
  1. Thrift
  2. THRIFT-5288

Move Support for ByteBuffer into TTransport

VotersWatch issueWatchersLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Minor
    • Resolution: Fixed
    • None
    • 0.14.0
    • Java - Library
    • None

    Description

      TCompactProtocol.java
        /**
         * Write a byte array, using a varint for the size.
         */
        public void writeBinary(ByteBuffer bin) throws TException {
          int length = bin.limit() - bin.position();
          writeBinary(bin.array(), bin.position() + bin.arrayOffset(), length);
        }
      

      I was working on something with Parquet and this code was causing some issues:

      java.lang.Exception: java.nio.ReadOnlyBufferException
      	at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
      	at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522)
      Caused by: java.nio.ReadOnlyBufferException
      	at java.nio.ByteBuffer.array(ByteBuffer.java:996)
      	at shaded.parquet.org.apache.thrift.protocol.TCompactProtocol.writeBinary(TCompactProtocol.java:375)
      	at org.apache.parquet.format.InterningProtocol.writeBinary(InterningProtocol.java:135)
      	at org.apache.parquet.format.ColumnIndex$ColumnIndexStandardScheme.write(ColumnIndex.java:945)
      	at org.apache.parquet.format.ColumnIndex$ColumnIndexStandardScheme.write(ColumnIndex.java:820)
      	at org.apache.parquet.format.ColumnIndex.write(ColumnIndex.java:728)
      	at org.apache.parquet.format.Util.write(Util.java:372)
      	at org.apache.parquet.format.Util.writeColumnIndex(Util.java:69)
      	at org.apache.parquet.hadoop.ParquetFileWriter.serializeColumnIndexes(ParquetFileWriter.java:1087)
      	at org.apache.parquet.hadoop.ParquetFileWriter.end(ParquetFileWriter.java:1050)
      

      This happens, because not all Buffer allow for direct access to the backing Array,... for example a ByteBuffer tied to a file does not have an Array. Read-only (immutable) ByteBuffer do not allow for this kind of access to the array since it could then be modified.

      There are two approaches here:

      1. Assert and throw Exception if the backing array must be allowed for access
      2. Deal natively with the ByteBuffer

      I propose the latter. However, the initial naive I approach I propose is to "deal natively" with the ByteBuffer by making a copy of the contents.

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            belugabehr David Mollitor
            belugabehr David Mollitor
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 1h 10m
                1h 10m

                Issue deployment