Uploaded image for project: 'Ambari'
  1. Ambari
  2. AMBARI-25987

Enable Spark to use hadoop native libraries for better performance

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 2.9.0
    • ambari-server
    • None

    Description

      Currently, when run spark tasks, spark will report cannot load hadoop native libraries.

      Also, the hadoop short-circuit read feature cannot be applied. For better performance, add hadoop native libraries support.

      Some WARN log as following:

      SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
      23/08/15 13:43:22 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

       

      Screen shot:

      ...

      23/08/15 13:43:25 INFO Client: Requesting a new application from cluster with 3 NodeManagers
      23/08/15 13:43:26 WARN DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.
      23/08/15 13:43:26 INFO Configuration: resource-types.xml not found

      Screen shot:

       

      Attachments

        1. image-2023-08-15-15-45-26-664.png
          115 kB
          Robin Zuo
        2. image-2023-08-15-15-46-43-947.png
          154 kB
          Robin Zuo

        Issue Links

          Activity

            People

              rongguo.zuo@gmail.com Robin Zuo
              rongguo.zuo@gmail.com Robin Zuo
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 0.5h
                  0.5h