[ARROW-3476] [Java] mvn test in memory fails on a big-endian platform

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.0.0
• Component/s: Java
• External issue URL: https://github.com/apache/arrow/issues/19795
• Assignee: Unassigned
• Reporter: Kazuaki Ishizaki
• Created: 09/Oct/18 18:48
• Updated: 11/Jan/23 07:27
• Resolved: 25/Feb/22 07:25

Description

Apache Arrow is becoming commonplace to exchange data among important emerging analytics frameworks such as Pandas, Numpy, and Spark.

IBM Z is one of platforms to process critical transactions such as bank or credit card. Users of IBM Z want to extract insights from these transactions using the emerging analytics systems on IBM Z Linux. These analytics pipelines can be also fast and effective on IBM Z Linux by using Apache Arrow on memory.

From the technical perspective, since IBM Z Linux uses big-endian data format, it is not possible to use Apache Arrow in this pipeline. If Apache Arrow could support big-endian, the use case would be expanded.

When I ran test case of Apache arrow on a big-endian platform (ppc64be), mvn test in memory causes a failure due to an assertion.
In TestEndianess.testLittleEndian test suite, the assertion occurs during an allocation of a RootAllocator class.

$ uname -a
Linux ppc64be.novalocal 4.5.7-300.fc24.ppc64 #1 SMP Fri Jun 10 20:29:32 UTC 2016 ppc64 ppc64 ppc64 GNU/Linux
$ arch  
ppc64
$ cd java/memory
$ mvn test
[INFO] Scanning for projects...
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building Arrow Memory 0.12.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO] 
...
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.082 s - in org.apache.arrow.memory.TestAccountant
[INFO] Running org.apache.arrow.memory.TestLowCostIdentityHashMap
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.012 s - in org.apache.arrow.memory.TestLowCostIdentityHashMap
[INFO] Running org.apache.arrow.memory.TestBaseAllocator
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.746 s <<< FAILURE! - in org.apache.arrow.memory.TestEndianess
[ERROR] testLittleEndian(org.apache.arrow.memory.TestEndianess)  Time elapsed: 0.313 s  <<< ERROR!
java.lang.ExceptionInInitializerError
	at org.apache.arrow.memory.TestEndianess.testLittleEndian(TestEndianess.java:31)
Caused by: java.lang.IllegalStateException: Arrow only runs on LittleEndian systems.
	at org.apache.arrow.memory.TestEndianess.testLittleEndian(TestEndianess.java:31)

[ERROR] Tests run: 22, Failures: 0, Errors: 21, Skipped: 1, Time elapsed: 0.055 s <<< FAILURE! - in org.apache.arrow.memory.TestBaseAllocator
...