[ARROW-3175] [Java] Upgrade to official FlatBuffers release (Flatbuffers incompatibility)

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 0.10.0
• Fix Version/s: 0.11.0
• Component/s: Java
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/19521
• Assignee: Li Jin
• Reporter: Alex Black
• Created: 05/Sep/18 05:48
• Updated: 11/Jan/23 07:25
• Resolved: 28/Sep/18 17:40

Description

Arrow Java currently uses an unofficial flatbuffers dependency - com.vlkan:flatbuffers:
https://github.com/apache/arrow/blob/master/java/pom.xml#L481-L485

The likely motivation here is that previously, no Java flatbuffers implementation was available on maven central.
https://github.com/vy/flatbuffers
> Unfortunately, FlatBuffers project does not publish any artifacts to the Maven Central Repository

However, this is no longer the case:
https://search.maven.org/search?q=g:com.google.flatbuffers%20AND%20a:flatbuffers-java&core=gav

The flatbuffers version used in Arrow java is a nearly 3-year-old snapshot, not even a version of an official release: https://github.com/vy/flatbuffers#usage

The main problem is that this version of flatbuffers is not compatible with the official releases of flatbuffers.
For example, we use the official flatbuffers releases in ND4J and Deeplearning4j: https://github.com/deeplearning4j/deeplearning4j

Running Arrow with an official flatbuffers library on the classpath results in issues such as:

java.lang.NoSuchMethodError: com.google.flatbuffers.FlatBufferBuilder.createString(Ljava/lang/String;)I
 at org.apache.arrow.vector.types.pojo.Field.getField(Field.java:154)
 at org.apache.arrow.vector.types.pojo.Schema.getSchema(Schema.java:145)
 at org.apache.arrow.vector.ipc.message.MessageSerializer.serialize(MessageSerializer.java:124)
 at org.apache.arrow.vector.ipc.ArrowWriter.ensureStarted(ArrowWriter.java:136)
 at org.apache.arrow.vector.ipc.ArrowWriter.start(ArrowWriter.java:97)
 at FlatBuffersDependencyIssue.test(FlatBuffersDependencyIssue.java:56)

 

Simply excluding the com.vlkan:flatbuffers dependency in lieu of an official flatbuffers release is not a solution (same exception as above) and we aren't prepared to downgrade all of our projects to use the flatbuffers version that Arrow currently requires.
Consequently, this is a major issue that prevents us using Arrow in our libraries.

I have prepared a simple repository to reproduce this issue, if required: https://github.com/AlexDBlack/arrowflatbufferstest

Is there a reason for using this particular version of flatbuffers, and if not, can Arrow java use an official release of flatbuffers instead?

Issue Links

• links to: GitHub Pull Request #2515