Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-3175

[Java] Upgrade to official FlatBuffers release (Flatbuffers incompatibility)

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.10.0
    • Fix Version/s: 0.11.0
    • Component/s: Java

      Description

      Arrow Java currently uses an unofficial flatbuffers dependency - com.vlkan:flatbuffers:
      https://github.com/apache/arrow/blob/master/java/pom.xml#L481-L485

      The likely motivation here is that previously, no Java flatbuffers implementation was available on maven central.
      https://github.com/vy/flatbuffers
      > Unfortunately, FlatBuffers project does not publish any artifacts to the Maven Central Repository

      However, this is no longer the case:
      https://search.maven.org/search?q=g:com.google.flatbuffers%20AND%20a:flatbuffers-java&core=gav

      The flatbuffers version used in Arrow java is a nearly 3-year-old snapshot, not even a version of an official release: https://github.com/vy/flatbuffers#usage

      The main problem is that this version of flatbuffers is not compatible with the official releases of flatbuffers.
      For example, we use the official flatbuffers releases in ND4J and Deeplearning4j: https://github.com/deeplearning4j/deeplearning4j

      Running Arrow with an official flatbuffers library on the classpath results in issues such as:

      java.lang.NoSuchMethodError: com.google.flatbuffers.FlatBufferBuilder.createString(Ljava/lang/String;)I
       at org.apache.arrow.vector.types.pojo.Field.getField(Field.java:154)
       at org.apache.arrow.vector.types.pojo.Schema.getSchema(Schema.java:145)
       at org.apache.arrow.vector.ipc.message.MessageSerializer.serialize(MessageSerializer.java:124)
       at org.apache.arrow.vector.ipc.ArrowWriter.ensureStarted(ArrowWriter.java:136)
       at org.apache.arrow.vector.ipc.ArrowWriter.start(ArrowWriter.java:97)
       at FlatBuffersDependencyIssue.test(FlatBuffersDependencyIssue.java:56)
      

       

      Simply excluding the com.vlkan:flatbuffers dependency in lieu of an official flatbuffers release is not a solution (same exception as above) and we aren't prepared to downgrade all of our projects to use the flatbuffers version that Arrow currently requires.
      Consequently, this is a major issue that prevents us using Arrow in our libraries.

      I have prepared a simple repository to reproduce this issue, if required: https://github.com/AlexDBlack/arrowflatbufferstest

      Is there a reason for using this particular version of flatbuffers, and if not, can Arrow java use an official release of flatbuffers instead?

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                icexelloss Li Jin
                Reporter:
                AlexDBlack Alex Black
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 7h 50m
                  7h 50m