[SPARK-12620] Proposal of GPU exploitation for Spark

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Component/s: Spark Core
• External issue ID: SPARK-3875
• Assignee: Unassigned
• Reporter: Kazuaki Ishizaki
• Created: 04/Jan/16 14:01
• Updated: 05/Jan/16 07:45
• Resolved: 04/Jan/16 14:17

Description

I created a new JIRA entry to move from SPARK-3875

Exploiting GPUs can allow us to shorten the execution time of a Spark job and to reduce the number of machines in a cluster. We are working to effectively and easily exploit GPUs on Spark at http://github.com/kiszk/spark-gpu. Our project page is http://kiszk.github.io/spark-gpu/. A design document is here

Our ideas for exploiting GPUs are

1. adding a new format for a partition in an RDD, which is a column-based structure in an array format, in addition to the current Iterator[T] format with Seq[T]
2. generating parallelized GPU native code to access data in the new format from a Spark application program by using an optimizer and code generator (this is similar to Project Tungsten) and pre-compiled library

The motivation of idea 1 is to reduce the overhead of serializing/deserializing partition data for copy between CPU and GPU. The motivation of idea 2 is to avoid writing hardware-dependent code by application programmers. At first, we are working for idea A (For idea B, we need to write CUDA code for now).

This prototype achieved 3.15x performance improvement of logistic regression (SparkGPULR) in examples on a 16-thread IvyBridge box with an NVIDIA K40 GPU card over that with no GPU card

You can download the pre-build binary for x86_64 and ppc64le from here. You can run this on Amazon EC2 by the procedure, too.

Issue Links

• duplicates: SPARK-3785 Support off-loading computations to a GPU