[YARN-9473] [Umbrella] Support Vector Engine ( a new accelerator hardware) based on pluggable device framework

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.3.0
• Component/s: nodemanager
• Assignee: Peter Bacsko
• Reporter: Zhankun Tang
• Created: 11/Apr/19 14:10
• Updated: 01/Jul/19 19:56
• Resolved: 01/Jul/19 19:56

Description

As the heterogeneous computation trend rises, new acceleration hardware like GPU, FPGA is used to satisfy various requirements.

And a new hardware Vector Engine (VE) which released by NEC company is another example. The VE is like GPU but has different characteristics. It's suitable for machine learning and HPC due to better memory bandwidth and no PCIe bottleneck.

Please Check here for more VE details:

https://www.nextplatform.com/2017/11/22/deep-dive-necs-aurora-vector-engine/

https://www.hotchips.org/hc30/2conf/2.14_NEC_vector_NEC_SXAurora_TSUBASA_HotChips30_finalb.pdf

As we know, YARN-8851 is a pluggable device framework which provides an easy way to develop a plugin for such new accelerators. This JIRA proposes to develop a new VE plugin based on that framework and be implemented as current GPU's "NvidiaGPUPluginForRuntimeV2" plugin.

 

Issue Links

• is related to: YARN-8851 [Umbrella] A pluggable device plugin framework to ease vendor plugin development

Sub-Tasks

• Create basic VE plugin: Resolved
• Create unit tests for VE plugin: Resolved
• Implement VE discovery using libudev: Resolved