[CRUNCH-67] Multiple writes in a pipeline are not performed

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.4.0
• Fix Version/s: 0.4.0
• Component/s: Core, Scrunch
• Assignee: Josh Wills
• Reporter: Kiyan Ahmadizadeh
• Created: 18/Sep/12 23:45
• Updated: 16/Nov/12 20:06
• Resolved: 19/Sep/12 21:38

Description

Consider the following simple PipelineApp (in Scala) that:
1. Reads in a text source.
2. Cleans the text of non-alphabetic characters.
3. Writes the sanitized text to a text file.
4. Computes word counts from the text.
5. Writes the word counts to a text file.

When this code is executed, the write from step 5 is performed successfully, but the write from step 3 is not.

object ShakesMultiWrite extends PipelineApp {

val shakes = read(From.textFile("shakes.txt"))

// Now let's clean-up the text
val cleanShakes = shakes.map

{line => val cleanText = line.replaceAll( """[^A-Za-z\W]""", "").toLowerCase() cleanText }

cleanShakes.write(To.textFile("shakesText/cleanShakes"))

// Count words
val wordCounts = cleanShakes.flatMap

{ line => line .split( """\W+""") // Split the text into words. .filter(w => !w.isEmpty()) // Get rid of any empty words created. }

.count()

wordCounts.write(To.textFile("shakesText/wordCounts"))

// Runs the pipeline
run()
}