Uploaded image for project: 'Parquet'
  1. Parquet
  2. PARQUET-725

Parquet AVRO tests fail when debug logging is enabled

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • None
    • None
    • None

    Description

      I found that on my machine some of the tests in the parquet-avro fail.

      Tests run: 25, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.073 sec
      Running org.apache.parquet.avro.TestAvroDataSupplier
      Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec
      Running org.apache.parquet.avro.TestReadWrite
      Tests run: 18, Failures: 4, Errors: 0, Skipped: 0, Time elapsed: 0.414 sec <<< FAILURE!
      Running org.apache.parquet.avro.TestBackwardCompatibility
      Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.016 sec
      Running org.apache.parquet.avro.TestReadWriteOldListBehavior
      Tests run: 16, Failures: 4, Errors: 0, Skipped: 0, Time elapsed: 0.148 sec <<< FAILURE!
      Running org.apache.parquet.avro.TestInputOutputFormat
      Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.29 sec
      Running org.apache.parquet.avro.TestReflectLogicalTypes
      Tests run: 13, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.165 sec
      Running org.apache.parquet.avro.TestCircularReferences
      Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0 sec
      
      Results :
      
      Failed tests:   testWriteReflectReadGeneric(org.apache.parquet.avro.TestReflectReadWrite): expected:<{"myboolean": true, "mybyte": 1, "myshort": 1, "myint": 1, "mylong": 2, "myfloat": 3.1, "mydouble": 4.1, "mybytes": {"bytes": "\u0001\u0002\u0003\u0004"}, "mystring": "Hello", "myenum": "A", "mymap": {"a": "1", "b": "2"}, "myshortarray": [1, 2], "myintarray": [1, 2], "mystringarray": ["a", "b"], "mylist": ["a", "b", "c"]}> but was:<{"myboolean": true, "mybyte": 1, "myshort": 1, "myint": 1, "mylong": 2, "myfloat": 3.1, "mydouble": 4.1, "mybytes": {"bytes": ""}, "mystring": "Hello", "myenum": "A", "mymap": {"a": "1", "b": "2"}, "myshortarray": [1, 2], "myintarray": [1, 2], "mystringarray": ["a", "b"], "mylist": ["a", "b", "c"]}>
        testWriteDecimalBytes(org.apache.parquet.avro.TestGenericLogicalTypes): Should read BigDecimals as bytes expected:<[{"dec": {"bytes": "ò\u0096"}}, {"dec": {"bytes": "\u0000²àø"}}]> but was:<[{"dec": {"bytes": ""}}, {"dec": {"bytes": ""}}]>
        testAll[0](org.apache.parquet.avro.TestReadWrite): expected:<java.nio.HeapByteBuffer[pos=0 lim=5 cap=5]> but was:<java.nio.HeapByteBuffer[pos=5 lim=5 cap=5]>
        testAllUsingDefaultAvroSchema[0](org.apache.parquet.avro.TestReadWrite): expected:<java.nio.HeapByteBuffer[pos=0 lim=5 cap=5]> but was:<java.nio.HeapByteBuffer[pos=5 lim=5 cap=5]>
        testAll[1](org.apache.parquet.avro.TestReadWrite): expected:<java.nio.HeapByteBuffer[pos=0 lim=5 cap=5]> but was:<java.nio.HeapByteBuffer[pos=5 lim=5 cap=5]>
        testAllUsingDefaultAvroSchema[1](org.apache.parquet.avro.TestReadWrite): expected:<java.nio.HeapByteBuffer[pos=0 lim=5 cap=5]> but was:<java.nio.HeapByteBuffer[pos=5 lim=5 cap=5]>
        testAll[0](org.apache.parquet.avro.TestReadWriteOldListBehavior): expected:<java.nio.HeapByteBuffer[pos=0 lim=5 cap=5]> but was:<java.nio.HeapByteBuffer[pos=5 lim=5 cap=5]>
        testAllUsingDefaultAvroSchema[0](org.apache.parquet.avro.TestReadWriteOldListBehavior): expected:<java.nio.HeapByteBuffer[pos=0 lim=5 cap=5]> but was:<java.nio.HeapByteBuffer[pos=5 lim=5 cap=5]>
        testAll[1](org.apache.parquet.avro.TestReadWriteOldListBehavior): expected:<java.nio.HeapByteBuffer[pos=0 lim=5 cap=5]> but was:<java.nio.HeapByteBuffer[pos=5 lim=5 cap=5]>
        testAllUsingDefaultAvroSchema[1](org.apache.parquet.avro.TestReadWriteOldListBehavior): expected:<java.nio.HeapByteBuffer[pos=0 lim=5 cap=5]> but was:<java.nio.HeapByteBuffer[pos=5 lim=5 cap=5]>
      
      

      I see two classes of problems:

      1. The json with byte arrays appear different.
      2. Some tests compare the 'toString' of a ByteBuffer. Now for two ByteBuffers that both contain the SAME bytes these tests fail simply because the position field of the ByteBuffer is different. I think these should compare the contents of the ByteBuffer instead.
        <java.nio.HeapByteBuffer[pos=0 lim=5 cap=5]> but was:<java.nio.HeapByteBuffer[pos=5 lim=5 cap=5]>
        

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            nielsbasjes Niels Basjes
            nielsbasjes Niels Basjes
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment