[DRILL-7911] Build faillure on Linux ARM64

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.19.0
• Assignee: Martin Tzvetanov Grigorov
• Reporter: Martin Tzvetanov Grigorov
• Created: 29/Apr/21 08:02
• Updated: 20/May/21 12:30
• Resolved: 20/May/21 12:30

Description

There are few problems when trying to build Drill on Linux ARM64 machine:

 

1) org.apache.drill.exec.record.vector.TestValueVector fails with OutOfMemoryError while trying to allocate native memory.

org.apache.drill.exec.record.vector.TestValueVector.testVariableVectorReallocation  Time elapsed: 0.012 s  <<< ERROR!
java.lang.Exception: Unexpected exception, expected<org.apache.drill.exec.exception.OversizedAllocationException> but was<org.apache.drill.exec.exception.OutOfMemoryException>
        at org.apache.drill.exec.record.vector.TestValueVector.testVariableVectorReallocation(TestValueVector.java:165)
Caused by: io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 2147483647 byte(s) of direct memory (used: 889192448, max: 2621440000)
        at org.apache.drill.exec.record.vector.TestValueVector.testVariableVectorReallocation(TestValueVector.java:165) 

I've tried to debug it by adding io.netty.leakDetection.level=paranoid to maven-surefire-plugin's argLine but instead of reporting the leaks this actually worked around the issue

 

2) Contrib Storage Mongo module tests fail because flapdoodle.mongo dependency does not support Linux ARM64

I've created the following PRs to fix it:

https://github.com/flapdoodle-oss/de.flapdoodle.embed.process/pull/119 - Add support for AARCH64

https://github.com/flapdoodle-oss/de.flapdoodle.embed.process/pull/120 - Add support for comparing versions

https://github.com/flapdoodle-oss/de.flapdoodle.embed.mongo/pull/343 - Add support for Linux AARCH64 and newer Mongodb versions

 

Update: Contrib Storage Mongo module has been migrated to use TestContainers-Mongo

 

3) Contrib Storage JDBC tests fail because com.wix:wix-embedded-mysql does not support Linux ARM64

I've started adding support for ARM64 to Wix but then I realized that the project might be dead. Last commit to https://github.com/wix/wix-embedded-mysql is at Nov 3rd 2019. So I am not sure whether the improvement will be ever accepted and released.

My new approach here is to use TestContainers-MySQL instead of com.wix:wix-embedded-mysql. Apache Drill already uses TestContainers for Storage Cassandra and TestContainers Vault for java-exec module.

 

4) Module "Contrib : Storage : Splunk" fails because there is no ARM64 Docker image for "splunk". For now I cannot find any third party Docker image for Splunk on ARM64. The only solutions I see is to disable the Splunk module tests on Linux ARM64

According to https://docs.splunk.com/Documentation/Splunk/latest/Installation/Systemrequirements#Unix_operating_systems Linux ARM64 is not supported.

I've created https://github.com/splunk/docker-splunk/issues/479 requesting for adding support for ARM64.