[SPARK-13700] Rdd.mapAsync(): Easily mix Spark and asynchroneous transformation

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Not A Problem
• Component/s: Spark Core
• Labels: async features rdd transform
• Assignee: Unassigned
• Reporter: Paulo Costa
• Created: 05/Mar/16 18:52
• Updated: 07/Mar/16 18:39
• Resolved: 06/Mar/16 19:37

Description

Spark is great for synchronous operations.

But sometimes I need to call a database/web server/etc from my transform, and the Spark pipeline stalls waiting for it.

Avoiding that would be great!

I suggest we add a new method RDD.mapAsync(), which can execute these operations concurrently, avoiding the bottleneck.

I've written a quick'n'dirty implementation of what I have in mind:
https://gist.github.com/paulo-raca/d121cf27905cfb1fafc3

What do you think?

If you agree with this feature, I can work on a pull request.