[OODT-698] Adapt OODT Process Streaming Data

• Type: Umbrella
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.1
• Component/s: file manager, resource manager, workflow manager
• Labels: streaming
• Skill Level: Don't Know (Unsure) - The default level
• Assignee: Michael Starch
• Reporter: Michael Starch
• Created: 28/May/14 21:29
• Updated: 15/Jul/17 04:04
• Resolved: 15/Jul/17 04:04

Description

Currently OODT excels at managing data, however; it is not as capable of processing really large data sets. The major drawbacks of OODT are: file-based data storage, and filesystem based io. Both of these drawbacks can be addressed by combining OODT with new stream-processing and cluster management technologies.

This effort is currently focused on combining OODT with the Berkley Data Analysis Stack to achieve these exact results.

Initial designs are captured in the attached slides. Focus on "Track 2" as this was decided to be the best approach.