[AMBARI-25987] Enable Spark to use hadoop native libraries for better performance

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.9.0
• Component/s: ambari-server
• Assignee: Robin Zuo
• Reporter: Robin Zuo
• Created: 15/Aug/23 07:32
• Updated: 04/Sep/23 01:37
• Resolved: 16/Aug/23 21:47

Description

Currently, when run spark tasks, spark will report cannot load hadoop native libraries.

Also, the hadoop short-circuit read feature cannot be applied. For better performance, add hadoop native libraries support.

Some WARN log as following:

SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
23/08/15 13:43:22 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

 

Screen shot:

...

23/08/15 13:43:25 INFO Client: Requesting a new application from cluster with 3 NodeManagers
23/08/15 13:43:26 WARN DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.
23/08/15 13:43:26 INFO Configuration: resource-types.xml not found

Screen shot:

 

Issue Links

• Blocked: AMBARI-26001 ambari2.8 release
• links to: GitHub PR#3740