Running in mode: translation
[SCHED] planned job for class edu.jhu.thrax.hadoop.jobs.VocabularyJob
[SCHED] planned job for class edu.jhu.thrax.hadoop.jobs.ExtractionJob
[SCHED] planned job for class edu.jhu.thrax.hadoop.features.mapred.TargetPhraseGivenSourceFeature
[SCHED] planned job for class edu.jhu.thrax.hadoop.features.mapred.SourcePhraseGivenTargetFeature
[SCHED] planned job for class edu.jhu.thrax.hadoop.jobs.TargetWordGivenSourceWordProbabilityJob
[SCHED] planned job for class edu.jhu.thrax.hadoop.jobs.SourceWordGivenTargetWordProbabilityJob
[SCHED] planned job for class edu.jhu.thrax.hadoop.features.annotation.AnnotationFeatureJob
[SCHED] planned job for class edu.jhu.thrax.hadoop.jobs.OutputJob
[SCHED] class edu.jhu.thrax.hadoop.jobs.OutputJob in state WAITING
[SCHED] class edu.jhu.thrax.hadoop.features.mapred.SourcePhraseGivenTargetFeature in state WAITING
[SCHED] class edu.jhu.thrax.hadoop.jobs.ExtractionJob in state WAITING
[SCHED] class edu.jhu.thrax.hadoop.jobs.VocabularyJob in state WAITING
[SCHED] class edu.jhu.thrax.hadoop.jobs.VocabularyJob in state READY
[SCHED] class edu.jhu.thrax.hadoop.features.annotation.AnnotationFeatureJob in state WAITING
[SCHED] class edu.jhu.thrax.hadoop.jobs.TargetWordGivenSourceWordProbabilityJob in state WAITING
[SCHED] class edu.jhu.thrax.hadoop.jobs.VocabularyJob in state WAITING
[SCHED] class edu.jhu.thrax.hadoop.jobs.VocabularyJob in state READY
[SCHED] class edu.jhu.thrax.hadoop.jobs.SourceWordGivenTargetWordProbabilityJob in state WAITING
[SCHED] class edu.jhu.thrax.hadoop.jobs.VocabularyJob in state WAITING
[SCHED] class edu.jhu.thrax.hadoop.jobs.VocabularyJob in state READY
[SCHED] class edu.jhu.thrax.hadoop.jobs.ExtractionJob in state WAITING
[SCHED] class edu.jhu.thrax.hadoop.jobs.VocabularyJob in state WAITING
[SCHED] class edu.jhu.thrax.hadoop.jobs.VocabularyJob in state READY
[SCHED] class edu.jhu.thrax.hadoop.features.mapred.TargetPhraseGivenSourceFeature in state WAITING
[SCHED] class edu.jhu.thrax.hadoop.jobs.ExtractionJob in state WAITING
[SCHED] class edu.jhu.thrax.hadoop.jobs.VocabularyJob in state WAITING
[SCHED] class edu.jhu.thrax.hadoop.jobs.VocabularyJob in state READY
[SCHED] class edu.jhu.thrax.hadoop.jobs.VocabularyJob in state RUNNING
16/08/17 21:06:32 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
16/08/17 21:06:32 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
16/08/17 21:06:33 INFO input.FileInputFormat: Total input paths to process : 1
16/08/17 21:06:33 INFO mapreduce.JobSubmitter: number of splits:1
16/08/17 21:06:33 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local1962608113_0001
16/08/17 21:06:33 INFO mapreduce.Job: The url to track the job: http://localhost:8080/
16/08/17 21:06:33 INFO mapred.LocalJobRunner: OutputCommitter set in config null
16/08/17 21:06:33 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
16/08/17 21:06:33 INFO mapred.LocalJobRunner: Waiting for map tasks
16/08/17 21:06:33 INFO mapred.LocalJobRunner: Starting task: attempt_local1962608113_0001_m_000000_0
16/08/17 21:06:33 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:06:33 INFO mapred.MapTask: Processing split: file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/input-file:0+5049122
16/08/17 21:06:33 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
16/08/17 21:06:33 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
16/08/17 21:06:33 INFO mapred.MapTask: soft limit at 83886080
16/08/17 21:06:33 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
16/08/17 21:06:33 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
16/08/17 21:06:33 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
16/08/17 21:06:34 INFO mapred.LocalJobRunner: 
16/08/17 21:06:34 INFO mapred.MapTask: Starting flush of map output
16/08/17 21:06:34 INFO mapred.MapTask: Spilling map output
16/08/17 21:06:34 INFO mapred.MapTask: bufstart = 0; bufend = 2749715; bufvoid = 104857600
16/08/17 21:06:34 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 23751248(95004992); length = 2463149/6553600
16/08/17 21:06:35 INFO mapred.MapTask: Finished spill 0
16/08/17 21:06:35 INFO mapred.Task: Task:attempt_local1962608113_0001_m_000000_0 is done. And is in the process of committing
16/08/17 21:06:35 INFO mapred.LocalJobRunner: map
16/08/17 21:06:35 INFO mapred.Task: Task 'attempt_local1962608113_0001_m_000000_0' done.
16/08/17 21:06:35 INFO mapred.LocalJobRunner: Finishing task: attempt_local1962608113_0001_m_000000_0
16/08/17 21:06:35 INFO mapred.LocalJobRunner: map task executor complete.
16/08/17 21:06:35 INFO mapred.LocalJobRunner: Waiting for reduce tasks
16/08/17 21:06:35 INFO mapred.LocalJobRunner: Starting task: attempt_local1962608113_0001_r_000000_0
16/08/17 21:06:35 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:06:35 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@6d475dbe
16/08/17 21:06:35 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=334338464, maxSingleShuffleLimit=83584616, mergeThreshold=220663392, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:06:35 INFO reduce.EventFetcher: attempt_local1962608113_0001_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:06:36 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local1962608113_0001_m_000000_0 decomp: 10996 len: 11000 to MEMORY
16/08/17 21:06:36 INFO reduce.InMemoryMapOutput: Read 10996 bytes from map-output for attempt_local1962608113_0001_m_000000_0
16/08/17 21:06:36 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 10996, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->10996
16/08/17 21:06:36 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)
	at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
16/08/17 21:06:36 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:06:36 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:36 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:06:36 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:36 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 10992 bytes
16/08/17 21:06:36 INFO reduce.MergeManagerImpl: Merged 1 segments, 10996 bytes to disk to satisfy reduce memory limit
16/08/17 21:06:36 INFO reduce.MergeManagerImpl: Merging 1 files, 11000 bytes from disk
16/08/17 21:06:36 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:06:36 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:36 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 10992 bytes
16/08/17 21:06:36 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:36 INFO Configuration.deprecation: mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords
16/08/17 21:06:36 INFO mapred.Task: Task:attempt_local1962608113_0001_r_000000_0 is done. And is in the process of committing
16/08/17 21:06:36 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:36 INFO mapred.Task: Task attempt_local1962608113_0001_r_000000_0 is allowed to commit now
16/08/17 21:06:36 INFO output.FileOutputCommitter: Saved output of task 'attempt_local1962608113_0001_r_000000_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/vocabulary/_temporary/0/task_local1962608113_0001_r_000000
16/08/17 21:06:36 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:06:36 INFO mapred.Task: Task 'attempt_local1962608113_0001_r_000000_0' done.
16/08/17 21:06:36 INFO mapred.LocalJobRunner: Finishing task: attempt_local1962608113_0001_r_000000_0
16/08/17 21:06:36 INFO mapred.LocalJobRunner: Starting task: attempt_local1962608113_0001_r_000001_0
16/08/17 21:06:36 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:06:36 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@504bc2df
16/08/17 21:06:36 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=334338464, maxSingleShuffleLimit=83584616, mergeThreshold=220663392, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:06:36 INFO reduce.EventFetcher: attempt_local1962608113_0001_r_000001_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:06:36 INFO reduce.LocalFetcher: localfetcher#2 about to shuffle output of map attempt_local1962608113_0001_m_000000_0 decomp: 11057 len: 11061 to MEMORY
16/08/17 21:06:36 INFO reduce.InMemoryMapOutput: Read 11057 bytes from map-output for attempt_local1962608113_0001_m_000000_0
16/08/17 21:06:36 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 11057, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->11057
16/08/17 21:06:36 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:06:36 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)
	at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
16/08/17 21:06:36 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:36 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:06:36 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:36 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 11053 bytes
16/08/17 21:06:36 INFO reduce.MergeManagerImpl: Merged 1 segments, 11057 bytes to disk to satisfy reduce memory limit
16/08/17 21:06:36 INFO reduce.MergeManagerImpl: Merging 1 files, 11061 bytes from disk
16/08/17 21:06:36 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:06:36 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:36 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 11053 bytes
16/08/17 21:06:36 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:36 INFO mapred.Task: Task:attempt_local1962608113_0001_r_000001_0 is done. And is in the process of committing
16/08/17 21:06:36 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:36 INFO mapred.Task: Task attempt_local1962608113_0001_r_000001_0 is allowed to commit now
16/08/17 21:06:36 INFO output.FileOutputCommitter: Saved output of task 'attempt_local1962608113_0001_r_000001_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/vocabulary/_temporary/0/task_local1962608113_0001_r_000001
16/08/17 21:06:36 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:06:36 INFO mapred.Task: Task 'attempt_local1962608113_0001_r_000001_0' done.
16/08/17 21:06:36 INFO mapred.LocalJobRunner: Finishing task: attempt_local1962608113_0001_r_000001_0
16/08/17 21:06:36 INFO mapred.LocalJobRunner: Starting task: attempt_local1962608113_0001_r_000002_0
16/08/17 21:06:36 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:06:36 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@43ec7974
16/08/17 21:06:36 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=334338464, maxSingleShuffleLimit=83584616, mergeThreshold=220663392, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:06:36 INFO reduce.EventFetcher: attempt_local1962608113_0001_r_000002_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:06:36 INFO reduce.LocalFetcher: localfetcher#3 about to shuffle output of map attempt_local1962608113_0001_m_000000_0 decomp: 11120 len: 11124 to MEMORY
16/08/17 21:06:36 INFO reduce.InMemoryMapOutput: Read 11120 bytes from map-output for attempt_local1962608113_0001_m_000000_0
16/08/17 21:06:36 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 11120, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->11120
16/08/17 21:06:36 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)
	at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
16/08/17 21:06:36 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:06:36 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:36 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:06:36 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:36 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 11114 bytes
16/08/17 21:06:36 INFO reduce.MergeManagerImpl: Merged 1 segments, 11120 bytes to disk to satisfy reduce memory limit
16/08/17 21:06:36 INFO reduce.MergeManagerImpl: Merging 1 files, 11124 bytes from disk
16/08/17 21:06:36 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:06:36 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:36 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 11114 bytes
16/08/17 21:06:36 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:36 INFO mapred.Task: Task:attempt_local1962608113_0001_r_000002_0 is done. And is in the process of committing
16/08/17 21:06:36 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:36 INFO mapred.Task: Task attempt_local1962608113_0001_r_000002_0 is allowed to commit now
16/08/17 21:06:36 INFO output.FileOutputCommitter: Saved output of task 'attempt_local1962608113_0001_r_000002_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/vocabulary/_temporary/0/task_local1962608113_0001_r_000002
16/08/17 21:06:36 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:06:36 INFO mapred.Task: Task 'attempt_local1962608113_0001_r_000002_0' done.
16/08/17 21:06:36 INFO mapred.LocalJobRunner: Finishing task: attempt_local1962608113_0001_r_000002_0
16/08/17 21:06:36 INFO mapred.LocalJobRunner: Starting task: attempt_local1962608113_0001_r_000003_0
16/08/17 21:06:36 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:06:36 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@3f23383c
16/08/17 21:06:36 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=334338464, maxSingleShuffleLimit=83584616, mergeThreshold=220663392, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:06:36 INFO reduce.EventFetcher: attempt_local1962608113_0001_r_000003_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:06:36 INFO reduce.LocalFetcher: localfetcher#4 about to shuffle output of map attempt_local1962608113_0001_m_000000_0 decomp: 10913 len: 10917 to MEMORY
16/08/17 21:06:36 INFO reduce.InMemoryMapOutput: Read 10913 bytes from map-output for attempt_local1962608113_0001_m_000000_0
16/08/17 21:06:36 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 10913, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->10913
16/08/17 21:06:36 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:06:36 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:36 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:06:36 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:36 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 10909 bytes
16/08/17 21:06:36 INFO reduce.MergeManagerImpl: Merged 1 segments, 10913 bytes to disk to satisfy reduce memory limit
16/08/17 21:06:36 INFO reduce.MergeManagerImpl: Merging 1 files, 10917 bytes from disk
16/08/17 21:06:36 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:06:36 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:36 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 10909 bytes
16/08/17 21:06:36 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:36 INFO mapred.Task: Task:attempt_local1962608113_0001_r_000003_0 is done. And is in the process of committing
16/08/17 21:06:36 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:36 INFO mapred.Task: Task attempt_local1962608113_0001_r_000003_0 is allowed to commit now
16/08/17 21:06:36 INFO output.FileOutputCommitter: Saved output of task 'attempt_local1962608113_0001_r_000003_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/vocabulary/_temporary/0/task_local1962608113_0001_r_000003
16/08/17 21:06:36 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:06:36 INFO mapred.Task: Task 'attempt_local1962608113_0001_r_000003_0' done.
16/08/17 21:06:36 INFO mapred.LocalJobRunner: Finishing task: attempt_local1962608113_0001_r_000003_0
16/08/17 21:06:36 INFO mapred.LocalJobRunner: Starting task: attempt_local1962608113_0001_r_000004_0
16/08/17 21:06:36 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:06:36 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@60a0caff
16/08/17 21:06:36 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=334338464, maxSingleShuffleLimit=83584616, mergeThreshold=220663392, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:06:36 INFO reduce.EventFetcher: attempt_local1962608113_0001_r_000004_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:06:36 INFO reduce.LocalFetcher: localfetcher#5 about to shuffle output of map attempt_local1962608113_0001_m_000000_0 decomp: 10674 len: 10678 to MEMORY
16/08/17 21:06:36 INFO reduce.InMemoryMapOutput: Read 10674 bytes from map-output for attempt_local1962608113_0001_m_000000_0
16/08/17 21:06:36 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 10674, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->10674
16/08/17 21:06:36 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:06:36 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:36 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:06:36 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:36 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 10667 bytes
16/08/17 21:06:36 INFO reduce.MergeManagerImpl: Merged 1 segments, 10674 bytes to disk to satisfy reduce memory limit
16/08/17 21:06:36 INFO reduce.MergeManagerImpl: Merging 1 files, 10678 bytes from disk
16/08/17 21:06:36 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:06:36 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:36 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 10667 bytes
16/08/17 21:06:36 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:36 INFO mapred.Task: Task:attempt_local1962608113_0001_r_000004_0 is done. And is in the process of committing
16/08/17 21:06:36 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:36 INFO mapred.Task: Task attempt_local1962608113_0001_r_000004_0 is allowed to commit now
16/08/17 21:06:36 INFO output.FileOutputCommitter: Saved output of task 'attempt_local1962608113_0001_r_000004_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/vocabulary/_temporary/0/task_local1962608113_0001_r_000004
16/08/17 21:06:36 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:06:36 INFO mapred.Task: Task 'attempt_local1962608113_0001_r_000004_0' done.
16/08/17 21:06:36 INFO mapred.LocalJobRunner: Finishing task: attempt_local1962608113_0001_r_000004_0
16/08/17 21:06:36 INFO mapred.LocalJobRunner: Starting task: attempt_local1962608113_0001_r_000005_0
16/08/17 21:06:36 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:06:36 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@8cc1c4e
16/08/17 21:06:36 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=334338464, maxSingleShuffleLimit=83584616, mergeThreshold=220663392, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:06:36 INFO reduce.EventFetcher: attempt_local1962608113_0001_r_000005_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:06:36 INFO reduce.LocalFetcher: localfetcher#6 about to shuffle output of map attempt_local1962608113_0001_m_000000_0 decomp: 11218 len: 11222 to MEMORY
16/08/17 21:06:36 INFO reduce.InMemoryMapOutput: Read 11218 bytes from map-output for attempt_local1962608113_0001_m_000000_0
16/08/17 21:06:36 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 11218, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->11218
16/08/17 21:06:36 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:06:36 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:36 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:06:36 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:36 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 11214 bytes
16/08/17 21:06:36 INFO reduce.MergeManagerImpl: Merged 1 segments, 11218 bytes to disk to satisfy reduce memory limit
16/08/17 21:06:36 INFO reduce.MergeManagerImpl: Merging 1 files, 11222 bytes from disk
16/08/17 21:06:36 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:06:36 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:36 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 11214 bytes
16/08/17 21:06:36 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:36 INFO mapred.Task: Task:attempt_local1962608113_0001_r_000005_0 is done. And is in the process of committing
16/08/17 21:06:36 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:36 INFO mapred.Task: Task attempt_local1962608113_0001_r_000005_0 is allowed to commit now
16/08/17 21:06:36 INFO output.FileOutputCommitter: Saved output of task 'attempt_local1962608113_0001_r_000005_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/vocabulary/_temporary/0/task_local1962608113_0001_r_000005
16/08/17 21:06:36 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:06:36 INFO mapred.Task: Task 'attempt_local1962608113_0001_r_000005_0' done.
16/08/17 21:06:36 INFO mapred.LocalJobRunner: Finishing task: attempt_local1962608113_0001_r_000005_0
16/08/17 21:06:36 INFO mapred.LocalJobRunner: Starting task: attempt_local1962608113_0001_r_000006_0
16/08/17 21:06:36 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:06:36 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@97c414a
16/08/17 21:06:36 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=334338464, maxSingleShuffleLimit=83584616, mergeThreshold=220663392, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:06:36 INFO reduce.EventFetcher: attempt_local1962608113_0001_r_000006_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:06:36 INFO reduce.LocalFetcher: localfetcher#7 about to shuffle output of map attempt_local1962608113_0001_m_000000_0 decomp: 12013 len: 12017 to MEMORY
16/08/17 21:06:36 INFO reduce.InMemoryMapOutput: Read 12013 bytes from map-output for attempt_local1962608113_0001_m_000000_0
16/08/17 21:06:36 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 12013, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->12013
16/08/17 21:06:36 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:06:36 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:36 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:06:36 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:36 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 12009 bytes
16/08/17 21:06:36 INFO reduce.MergeManagerImpl: Merged 1 segments, 12013 bytes to disk to satisfy reduce memory limit
16/08/17 21:06:36 INFO reduce.MergeManagerImpl: Merging 1 files, 12017 bytes from disk
16/08/17 21:06:36 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:06:36 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:36 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 12009 bytes
16/08/17 21:06:36 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:37 INFO mapred.Task: Task:attempt_local1962608113_0001_r_000006_0 is done. And is in the process of committing
16/08/17 21:06:37 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:37 INFO mapred.Task: Task attempt_local1962608113_0001_r_000006_0 is allowed to commit now
16/08/17 21:06:37 INFO output.FileOutputCommitter: Saved output of task 'attempt_local1962608113_0001_r_000006_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/vocabulary/_temporary/0/task_local1962608113_0001_r_000006
16/08/17 21:06:37 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:06:37 INFO mapred.Task: Task 'attempt_local1962608113_0001_r_000006_0' done.
16/08/17 21:06:37 INFO mapred.LocalJobRunner: Finishing task: attempt_local1962608113_0001_r_000006_0
16/08/17 21:06:37 INFO mapred.LocalJobRunner: Starting task: attempt_local1962608113_0001_r_000007_0
16/08/17 21:06:37 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:06:37 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@2056ac7c
16/08/17 21:06:37 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=334338464, maxSingleShuffleLimit=83584616, mergeThreshold=220663392, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:06:37 INFO reduce.EventFetcher: attempt_local1962608113_0001_r_000007_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:06:37 INFO reduce.LocalFetcher: localfetcher#8 about to shuffle output of map attempt_local1962608113_0001_m_000000_0 decomp: 11059 len: 11063 to MEMORY
16/08/17 21:06:37 INFO reduce.InMemoryMapOutput: Read 11059 bytes from map-output for attempt_local1962608113_0001_m_000000_0
16/08/17 21:06:37 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 11059, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->11059
16/08/17 21:06:37 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:06:37 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:37 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:06:37 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:37 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 11054 bytes
16/08/17 21:06:37 INFO reduce.MergeManagerImpl: Merged 1 segments, 11059 bytes to disk to satisfy reduce memory limit
16/08/17 21:06:37 INFO reduce.MergeManagerImpl: Merging 1 files, 11063 bytes from disk
16/08/17 21:06:37 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:06:37 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:37 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 11054 bytes
16/08/17 21:06:37 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:37 INFO mapred.Task: Task:attempt_local1962608113_0001_r_000007_0 is done. And is in the process of committing
16/08/17 21:06:37 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:37 INFO mapred.Task: Task attempt_local1962608113_0001_r_000007_0 is allowed to commit now
16/08/17 21:06:37 INFO output.FileOutputCommitter: Saved output of task 'attempt_local1962608113_0001_r_000007_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/vocabulary/_temporary/0/task_local1962608113_0001_r_000007
16/08/17 21:06:37 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:06:37 INFO mapred.Task: Task 'attempt_local1962608113_0001_r_000007_0' done.
16/08/17 21:06:37 INFO mapred.LocalJobRunner: Finishing task: attempt_local1962608113_0001_r_000007_0
16/08/17 21:06:37 INFO mapred.LocalJobRunner: Starting task: attempt_local1962608113_0001_r_000008_0
16/08/17 21:06:37 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:06:37 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@684d7a7c
16/08/17 21:06:37 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=334338464, maxSingleShuffleLimit=83584616, mergeThreshold=220663392, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:06:37 INFO reduce.EventFetcher: attempt_local1962608113_0001_r_000008_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:06:37 INFO reduce.LocalFetcher: localfetcher#9 about to shuffle output of map attempt_local1962608113_0001_m_000000_0 decomp: 11824 len: 11828 to MEMORY
16/08/17 21:06:37 INFO reduce.InMemoryMapOutput: Read 11824 bytes from map-output for attempt_local1962608113_0001_m_000000_0
16/08/17 21:06:37 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 11824, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->11824
16/08/17 21:06:37 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:06:37 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:37 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:06:37 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:37 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 11815 bytes
16/08/17 21:06:37 INFO reduce.MergeManagerImpl: Merged 1 segments, 11824 bytes to disk to satisfy reduce memory limit
16/08/17 21:06:37 INFO reduce.MergeManagerImpl: Merging 1 files, 11828 bytes from disk
16/08/17 21:06:37 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:06:37 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:37 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 11815 bytes
16/08/17 21:06:37 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:37 INFO mapred.Task: Task:attempt_local1962608113_0001_r_000008_0 is done. And is in the process of committing
16/08/17 21:06:37 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:37 INFO mapred.Task: Task attempt_local1962608113_0001_r_000008_0 is allowed to commit now
16/08/17 21:06:37 INFO output.FileOutputCommitter: Saved output of task 'attempt_local1962608113_0001_r_000008_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/vocabulary/_temporary/0/task_local1962608113_0001_r_000008
16/08/17 21:06:37 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:06:37 INFO mapred.Task: Task 'attempt_local1962608113_0001_r_000008_0' done.
16/08/17 21:06:37 INFO mapred.LocalJobRunner: Finishing task: attempt_local1962608113_0001_r_000008_0
16/08/17 21:06:37 INFO mapred.LocalJobRunner: Starting task: attempt_local1962608113_0001_r_000009_0
16/08/17 21:06:37 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:06:37 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@71a6f308
16/08/17 21:06:37 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=334338464, maxSingleShuffleLimit=83584616, mergeThreshold=220663392, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:06:37 INFO reduce.EventFetcher: attempt_local1962608113_0001_r_000009_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:06:37 INFO reduce.LocalFetcher: localfetcher#10 about to shuffle output of map attempt_local1962608113_0001_m_000000_0 decomp: 11226 len: 11230 to MEMORY
16/08/17 21:06:37 INFO reduce.InMemoryMapOutput: Read 11226 bytes from map-output for attempt_local1962608113_0001_m_000000_0
16/08/17 21:06:37 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 11226, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->11226
16/08/17 21:06:37 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:06:37 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:37 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:06:37 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:37 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 11220 bytes
16/08/17 21:06:37 INFO reduce.MergeManagerImpl: Merged 1 segments, 11226 bytes to disk to satisfy reduce memory limit
16/08/17 21:06:37 INFO reduce.MergeManagerImpl: Merging 1 files, 11230 bytes from disk
16/08/17 21:06:37 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:06:37 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:37 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 11220 bytes
16/08/17 21:06:37 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:37 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)
	at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
16/08/17 21:06:37 INFO mapred.Task: Task:attempt_local1962608113_0001_r_000009_0 is done. And is in the process of committing
16/08/17 21:06:37 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:37 INFO mapred.Task: Task attempt_local1962608113_0001_r_000009_0 is allowed to commit now
16/08/17 21:06:37 INFO output.FileOutputCommitter: Saved output of task 'attempt_local1962608113_0001_r_000009_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/vocabulary/_temporary/0/task_local1962608113_0001_r_000009
16/08/17 21:06:37 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:06:37 INFO mapred.Task: Task 'attempt_local1962608113_0001_r_000009_0' done.
16/08/17 21:06:37 INFO mapred.LocalJobRunner: Finishing task: attempt_local1962608113_0001_r_000009_0
16/08/17 21:06:37 INFO mapred.LocalJobRunner: Starting task: attempt_local1962608113_0001_r_000010_0
16/08/17 21:06:37 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:06:37 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@2fbb7161
16/08/17 21:06:37 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=334338464, maxSingleShuffleLimit=83584616, mergeThreshold=220663392, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:06:37 INFO reduce.EventFetcher: attempt_local1962608113_0001_r_000010_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:06:37 INFO reduce.LocalFetcher: localfetcher#11 about to shuffle output of map attempt_local1962608113_0001_m_000000_0 decomp: 11549 len: 11553 to MEMORY
16/08/17 21:06:37 INFO reduce.InMemoryMapOutput: Read 11549 bytes from map-output for attempt_local1962608113_0001_m_000000_0
16/08/17 21:06:37 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 11549, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->11549
16/08/17 21:06:37 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)
	at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
16/08/17 21:06:37 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:06:37 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:37 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:06:37 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:37 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 11544 bytes
16/08/17 21:06:37 INFO reduce.MergeManagerImpl: Merged 1 segments, 11549 bytes to disk to satisfy reduce memory limit
16/08/17 21:06:37 INFO reduce.MergeManagerImpl: Merging 1 files, 11553 bytes from disk
16/08/17 21:06:37 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:06:37 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:37 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 11544 bytes
16/08/17 21:06:37 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:37 INFO mapred.Task: Task:attempt_local1962608113_0001_r_000010_0 is done. And is in the process of committing
16/08/17 21:06:37 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:37 INFO mapred.Task: Task attempt_local1962608113_0001_r_000010_0 is allowed to commit now
16/08/17 21:06:37 INFO output.FileOutputCommitter: Saved output of task 'attempt_local1962608113_0001_r_000010_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/vocabulary/_temporary/0/task_local1962608113_0001_r_000010
16/08/17 21:06:37 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:06:37 INFO mapred.Task: Task 'attempt_local1962608113_0001_r_000010_0' done.
16/08/17 21:06:37 INFO mapred.LocalJobRunner: Finishing task: attempt_local1962608113_0001_r_000010_0
16/08/17 21:06:37 INFO mapred.LocalJobRunner: Starting task: attempt_local1962608113_0001_r_000011_0
16/08/17 21:06:37 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:06:37 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@7c249ad
16/08/17 21:06:37 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=334338464, maxSingleShuffleLimit=83584616, mergeThreshold=220663392, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:06:37 INFO reduce.EventFetcher: attempt_local1962608113_0001_r_000011_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:06:37 INFO reduce.LocalFetcher: localfetcher#12 about to shuffle output of map attempt_local1962608113_0001_m_000000_0 decomp: 11749 len: 11753 to MEMORY
16/08/17 21:06:37 INFO reduce.InMemoryMapOutput: Read 11749 bytes from map-output for attempt_local1962608113_0001_m_000000_0
16/08/17 21:06:37 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 11749, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->11749
16/08/17 21:06:37 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:06:37 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:37 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:06:37 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:37 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 11739 bytes
16/08/17 21:06:37 INFO reduce.MergeManagerImpl: Merged 1 segments, 11749 bytes to disk to satisfy reduce memory limit
16/08/17 21:06:37 INFO reduce.MergeManagerImpl: Merging 1 files, 11753 bytes from disk
16/08/17 21:06:37 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:06:37 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:37 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 11739 bytes
16/08/17 21:06:37 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:37 INFO mapred.Task: Task:attempt_local1962608113_0001_r_000011_0 is done. And is in the process of committing
16/08/17 21:06:37 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:37 INFO mapred.Task: Task attempt_local1962608113_0001_r_000011_0 is allowed to commit now
16/08/17 21:06:37 INFO output.FileOutputCommitter: Saved output of task 'attempt_local1962608113_0001_r_000011_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/vocabulary/_temporary/0/task_local1962608113_0001_r_000011
16/08/17 21:06:37 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:06:37 INFO mapred.Task: Task 'attempt_local1962608113_0001_r_000011_0' done.
16/08/17 21:06:37 INFO mapred.LocalJobRunner: Finishing task: attempt_local1962608113_0001_r_000011_0
16/08/17 21:06:37 INFO mapred.LocalJobRunner: Starting task: attempt_local1962608113_0001_r_000012_0
16/08/17 21:06:37 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:06:37 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@4df7110
16/08/17 21:06:37 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=334338464, maxSingleShuffleLimit=83584616, mergeThreshold=220663392, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:06:37 INFO reduce.EventFetcher: attempt_local1962608113_0001_r_000012_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:06:37 INFO reduce.LocalFetcher: localfetcher#13 about to shuffle output of map attempt_local1962608113_0001_m_000000_0 decomp: 11692 len: 11696 to MEMORY
16/08/17 21:06:37 INFO reduce.InMemoryMapOutput: Read 11692 bytes from map-output for attempt_local1962608113_0001_m_000000_0
16/08/17 21:06:37 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 11692, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->11692
16/08/17 21:06:37 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:06:37 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:37 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:06:37 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:37 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 11687 bytes
16/08/17 21:06:37 INFO reduce.MergeManagerImpl: Merged 1 segments, 11692 bytes to disk to satisfy reduce memory limit
16/08/17 21:06:37 INFO reduce.MergeManagerImpl: Merging 1 files, 11696 bytes from disk
16/08/17 21:06:37 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:06:37 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:37 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 11687 bytes
16/08/17 21:06:37 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:37 INFO mapred.Task: Task:attempt_local1962608113_0001_r_000012_0 is done. And is in the process of committing
16/08/17 21:06:37 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:37 INFO mapred.Task: Task attempt_local1962608113_0001_r_000012_0 is allowed to commit now
16/08/17 21:06:37 INFO output.FileOutputCommitter: Saved output of task 'attempt_local1962608113_0001_r_000012_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/vocabulary/_temporary/0/task_local1962608113_0001_r_000012
16/08/17 21:06:37 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:06:37 INFO mapred.Task: Task 'attempt_local1962608113_0001_r_000012_0' done.
16/08/17 21:06:37 INFO mapred.LocalJobRunner: Finishing task: attempt_local1962608113_0001_r_000012_0
16/08/17 21:06:37 INFO mapred.LocalJobRunner: Starting task: attempt_local1962608113_0001_r_000013_0
16/08/17 21:06:37 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:06:37 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@674007bc
16/08/17 21:06:37 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=334338464, maxSingleShuffleLimit=83584616, mergeThreshold=220663392, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:06:37 INFO reduce.EventFetcher: attempt_local1962608113_0001_r_000013_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:06:37 INFO reduce.LocalFetcher: localfetcher#14 about to shuffle output of map attempt_local1962608113_0001_m_000000_0 decomp: 11506 len: 11510 to MEMORY
16/08/17 21:06:37 INFO reduce.InMemoryMapOutput: Read 11506 bytes from map-output for attempt_local1962608113_0001_m_000000_0
16/08/17 21:06:37 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 11506, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->11506
16/08/17 21:06:37 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:06:37 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:37 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:06:37 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:37 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 11501 bytes
16/08/17 21:06:37 INFO reduce.MergeManagerImpl: Merged 1 segments, 11506 bytes to disk to satisfy reduce memory limit
16/08/17 21:06:37 INFO reduce.MergeManagerImpl: Merging 1 files, 11510 bytes from disk
16/08/17 21:06:37 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:06:37 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:37 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 11501 bytes
16/08/17 21:06:37 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:37 INFO mapred.Task: Task:attempt_local1962608113_0001_r_000013_0 is done. And is in the process of committing
16/08/17 21:06:37 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:37 INFO mapred.Task: Task attempt_local1962608113_0001_r_000013_0 is allowed to commit now
16/08/17 21:06:37 INFO output.FileOutputCommitter: Saved output of task 'attempt_local1962608113_0001_r_000013_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/vocabulary/_temporary/0/task_local1962608113_0001_r_000013
16/08/17 21:06:37 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:06:37 INFO mapred.Task: Task 'attempt_local1962608113_0001_r_000013_0' done.
16/08/17 21:06:37 INFO mapred.LocalJobRunner: Finishing task: attempt_local1962608113_0001_r_000013_0
16/08/17 21:06:37 INFO mapred.LocalJobRunner: Starting task: attempt_local1962608113_0001_r_000014_0
16/08/17 21:06:37 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:06:37 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@578015bb
16/08/17 21:06:37 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=334338464, maxSingleShuffleLimit=83584616, mergeThreshold=220663392, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:06:37 INFO reduce.EventFetcher: attempt_local1962608113_0001_r_000014_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:06:37 INFO reduce.LocalFetcher: localfetcher#15 about to shuffle output of map attempt_local1962608113_0001_m_000000_0 decomp: 10689 len: 10693 to MEMORY
16/08/17 21:06:37 INFO reduce.InMemoryMapOutput: Read 10689 bytes from map-output for attempt_local1962608113_0001_m_000000_0
16/08/17 21:06:37 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 10689, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->10689
16/08/17 21:06:37 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:06:37 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:37 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:06:37 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:37 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 10684 bytes
16/08/17 21:06:37 INFO reduce.MergeManagerImpl: Merged 1 segments, 10689 bytes to disk to satisfy reduce memory limit
16/08/17 21:06:37 INFO reduce.MergeManagerImpl: Merging 1 files, 10693 bytes from disk
16/08/17 21:06:37 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:06:37 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:37 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 10684 bytes
16/08/17 21:06:37 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:37 INFO mapred.Task: Task:attempt_local1962608113_0001_r_000014_0 is done. And is in the process of committing
16/08/17 21:06:37 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:37 INFO mapred.Task: Task attempt_local1962608113_0001_r_000014_0 is allowed to commit now
16/08/17 21:06:37 INFO output.FileOutputCommitter: Saved output of task 'attempt_local1962608113_0001_r_000014_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/vocabulary/_temporary/0/task_local1962608113_0001_r_000014
16/08/17 21:06:37 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:06:37 INFO mapred.Task: Task 'attempt_local1962608113_0001_r_000014_0' done.
16/08/17 21:06:37 INFO mapred.LocalJobRunner: Finishing task: attempt_local1962608113_0001_r_000014_0
16/08/17 21:06:37 INFO mapred.LocalJobRunner: Starting task: attempt_local1962608113_0001_r_000015_0
16/08/17 21:06:37 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:06:37 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@f52d9a4
16/08/17 21:06:37 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=334338464, maxSingleShuffleLimit=83584616, mergeThreshold=220663392, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:06:37 INFO reduce.EventFetcher: attempt_local1962608113_0001_r_000015_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:06:37 INFO reduce.LocalFetcher: localfetcher#16 about to shuffle output of map attempt_local1962608113_0001_m_000000_0 decomp: 10870 len: 10874 to MEMORY
16/08/17 21:06:37 INFO reduce.InMemoryMapOutput: Read 10870 bytes from map-output for attempt_local1962608113_0001_m_000000_0
16/08/17 21:06:37 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 10870, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->10870
16/08/17 21:06:37 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:06:37 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:37 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:06:37 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:37 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 10865 bytes
16/08/17 21:06:37 INFO reduce.MergeManagerImpl: Merged 1 segments, 10870 bytes to disk to satisfy reduce memory limit
16/08/17 21:06:37 INFO reduce.MergeManagerImpl: Merging 1 files, 10874 bytes from disk
16/08/17 21:06:37 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:06:37 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:37 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 10865 bytes
16/08/17 21:06:37 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:37 INFO mapred.Task: Task:attempt_local1962608113_0001_r_000015_0 is done. And is in the process of committing
16/08/17 21:06:37 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:37 INFO mapred.Task: Task attempt_local1962608113_0001_r_000015_0 is allowed to commit now
16/08/17 21:06:37 INFO output.FileOutputCommitter: Saved output of task 'attempt_local1962608113_0001_r_000015_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/vocabulary/_temporary/0/task_local1962608113_0001_r_000015
16/08/17 21:06:37 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:06:37 INFO mapred.Task: Task 'attempt_local1962608113_0001_r_000015_0' done.
16/08/17 21:06:37 INFO mapred.LocalJobRunner: Finishing task: attempt_local1962608113_0001_r_000015_0
16/08/17 21:06:37 INFO mapred.LocalJobRunner: reduce task executor complete.
[SCHED] class edu.jhu.thrax.hadoop.jobs.VocabularyJob in state SUCCESS
[SCHED] class edu.jhu.thrax.hadoop.jobs.TargetWordGivenSourceWordProbabilityJob in state READY
[SCHED] class edu.jhu.thrax.hadoop.jobs.SourceWordGivenTargetWordProbabilityJob in state READY
[SCHED] class edu.jhu.thrax.hadoop.jobs.ExtractionJob in state READY
[SCHED] class edu.jhu.thrax.hadoop.jobs.TargetWordGivenSourceWordProbabilityJob in state RUNNING
[SCHED] class edu.jhu.thrax.hadoop.jobs.SourceWordGivenTargetWordProbabilityJob in state RUNNING
[SCHED] class edu.jhu.thrax.hadoop.jobs.ExtractionJob in state RUNNING
16/08/17 21:06:38 INFO jvm.JvmMetrics: Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
16/08/17 21:06:38 INFO jvm.JvmMetrics: Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
16/08/17 21:06:38 INFO jvm.JvmMetrics: Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
16/08/17 21:06:38 INFO input.FileInputFormat: Total input paths to process : 1
16/08/17 21:06:38 INFO input.FileInputFormat: Total input paths to process : 1
16/08/17 21:06:38 INFO mapreduce.JobSubmitter: number of splits:1
16/08/17 21:06:38 INFO input.FileInputFormat: Total input paths to process : 1
16/08/17 21:06:38 INFO mapreduce.JobSubmitter: number of splits:1
16/08/17 21:06:38 INFO mapreduce.JobSubmitter: number of splits:1
16/08/17 21:06:38 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local583517286_0004
16/08/17 21:06:38 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local2012946159_0003
16/08/17 21:06:38 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local306174234_0002
16/08/17 21:06:38 INFO mapreduce.Job: The url to track the job: http://localhost:8080/
16/08/17 21:06:38 INFO mapred.LocalJobRunner: OutputCommitter set in config null
16/08/17 21:06:38 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
16/08/17 21:06:38 INFO mapred.LocalJobRunner: Waiting for map tasks
16/08/17 21:06:38 INFO mapred.LocalJobRunner: Starting task: attempt_local583517286_0004_m_000000_0
16/08/17 21:06:38 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:06:38 INFO mapred.MapTask: Processing split: file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/input-file:0+5049122
16/08/17 21:06:38 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
16/08/17 21:06:38 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
16/08/17 21:06:38 INFO mapred.MapTask: soft limit at 83886080
16/08/17 21:06:38 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
16/08/17 21:06:38 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
16/08/17 21:06:38 INFO mapreduce.Job: The url to track the job: http://localhost:8080/
16/08/17 21:06:38 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
16/08/17 21:06:38 INFO mapred.LocalJobRunner: OutputCommitter set in config null
16/08/17 21:06:38 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
16/08/17 21:06:38 INFO mapreduce.Job: The url to track the job: http://localhost:8080/
16/08/17 21:06:38 INFO mapred.LocalJobRunner: Waiting for map tasks
16/08/17 21:06:38 INFO mapred.LocalJobRunner: OutputCommitter set in config null
16/08/17 21:06:38 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
16/08/17 21:06:38 INFO mapred.LocalJobRunner: Waiting for map tasks
16/08/17 21:06:38 INFO mapred.LocalJobRunner: Starting task: attempt_local2012946159_0003_m_000000_0
16/08/17 21:06:38 INFO mapred.LocalJobRunner: Starting task: attempt_local306174234_0002_m_000000_0
16/08/17 21:06:38 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:06:38 INFO mapred.MapTask: Processing split: file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/input-file:0+5049122
16/08/17 21:06:38 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:06:38 INFO mapred.MapTask: Processing split: file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/input-file:0+5049122
16/08/17 21:06:39 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
16/08/17 21:06:39 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
16/08/17 21:06:39 INFO mapred.MapTask: soft limit at 83886080
16/08/17 21:06:39 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
16/08/17 21:06:39 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
16/08/17 21:06:39 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
16/08/17 21:06:39 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
16/08/17 21:06:39 INFO mapred.MapTask: soft limit at 83886080
16/08/17 21:06:39 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
16/08/17 21:06:39 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
16/08/17 21:06:39 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
16/08/17 21:06:39 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
16/08/17 21:06:42 INFO mapred.LocalJobRunner: 
16/08/17 21:06:42 INFO mapred.MapTask: Starting flush of map output
16/08/17 21:06:42 INFO mapred.MapTask: Spilling map output
16/08/17 21:06:42 INFO mapred.MapTask: bufstart = 0; bufend = 8623212; bufvoid = 104857600
16/08/17 21:06:42 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 23339996(93359984); length = 2874401/6553600
16/08/17 21:06:42 INFO mapred.LocalJobRunner: 
16/08/17 21:06:42 INFO mapred.MapTask: Starting flush of map output
16/08/17 21:06:42 INFO mapred.MapTask: Spilling map output
16/08/17 21:06:42 INFO mapred.MapTask: bufstart = 0; bufend = 7741752; bufvoid = 104857600
16/08/17 21:06:42 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 23633816(94535264); length = 2580581/6553600
16/08/17 21:06:44 INFO mapred.MapTask: Finished spill 0
16/08/17 21:06:44 INFO mapred.Task: Task:attempt_local306174234_0002_m_000000_0 is done. And is in the process of committing
16/08/17 21:06:44 INFO mapred.LocalJobRunner: map
16/08/17 21:06:44 INFO mapred.Task: Task 'attempt_local306174234_0002_m_000000_0' done.
16/08/17 21:06:44 INFO mapred.LocalJobRunner: Finishing task: attempt_local306174234_0002_m_000000_0
16/08/17 21:06:44 INFO mapred.LocalJobRunner: map task executor complete.
16/08/17 21:06:44 INFO mapred.LocalJobRunner: Waiting for reduce tasks
16/08/17 21:06:44 INFO mapred.LocalJobRunner: Starting task: attempt_local306174234_0002_r_000000_0
16/08/17 21:06:44 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:06:44 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@3607d288
16/08/17 21:06:44 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=369203616, maxSingleShuffleLimit=92300904, mergeThreshold=243674400, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:06:44 INFO mapred.MapTask: Finished spill 0
16/08/17 21:06:44 INFO reduce.EventFetcher: attempt_local306174234_0002_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:06:44 INFO mapred.Task: Task:attempt_local2012946159_0003_m_000000_0 is done. And is in the process of committing
16/08/17 21:06:44 INFO mapred.LocalJobRunner: map
16/08/17 21:06:44 INFO mapred.Task: Task 'attempt_local2012946159_0003_m_000000_0' done.
16/08/17 21:06:44 INFO mapred.LocalJobRunner: Finishing task: attempt_local2012946159_0003_m_000000_0
16/08/17 21:06:44 INFO mapred.LocalJobRunner: map task executor complete.
16/08/17 21:06:44 INFO mapred.LocalJobRunner: Waiting for reduce tasks
16/08/17 21:06:44 INFO reduce.LocalFetcher: localfetcher#17 about to shuffle output of map attempt_local306174234_0002_m_000000_0 decomp: 96728 len: 96732 to MEMORY
16/08/17 21:06:44 INFO reduce.InMemoryMapOutput: Read 96728 bytes from map-output for attempt_local306174234_0002_m_000000_0
16/08/17 21:06:44 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 96728, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->96728
16/08/17 21:06:44 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:06:44 INFO mapred.LocalJobRunner: Starting task: attempt_local2012946159_0003_r_000000_0
16/08/17 21:06:44 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:44 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:06:44 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:44 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 96718 bytes
16/08/17 21:06:44 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:06:44 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@4b1356e3
16/08/17 21:06:44 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=369203616, maxSingleShuffleLimit=92300904, mergeThreshold=243674400, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:06:44 INFO reduce.EventFetcher: attempt_local2012946159_0003_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:06:44 INFO reduce.MergeManagerImpl: Merged 1 segments, 96728 bytes to disk to satisfy reduce memory limit
16/08/17 21:06:44 INFO reduce.MergeManagerImpl: Merging 1 files, 96732 bytes from disk
16/08/17 21:06:44 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:06:44 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:44 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 96718 bytes
16/08/17 21:06:44 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:44 INFO reduce.LocalFetcher: localfetcher#18 about to shuffle output of map attempt_local2012946159_0003_m_000000_0 decomp: 210996 len: 211000 to MEMORY
16/08/17 21:06:44 INFO reduce.InMemoryMapOutput: Read 210996 bytes from map-output for attempt_local2012946159_0003_m_000000_0
16/08/17 21:06:44 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 210996, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->210996
16/08/17 21:06:44 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:06:44 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:44 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:06:44 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:44 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 210986 bytes
16/08/17 21:06:44 INFO reduce.MergeManagerImpl: Merged 1 segments, 210996 bytes to disk to satisfy reduce memory limit
16/08/17 21:06:44 INFO reduce.MergeManagerImpl: Merging 1 files, 211000 bytes from disk
16/08/17 21:06:44 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:06:44 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:44 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 210986 bytes
16/08/17 21:06:44 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:44 INFO mapred.Task: Task:attempt_local306174234_0002_r_000000_0 is done. And is in the process of committing
16/08/17 21:06:44 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:44 INFO mapred.Task: Task attempt_local306174234_0002_r_000000_0 is allowed to commit now
16/08/17 21:06:44 INFO output.FileOutputCommitter: Saved output of task 'attempt_local306174234_0002_r_000000_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/lexprobs_tgs/_temporary/0/task_local306174234_0002_r_000000
16/08/17 21:06:44 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:06:44 INFO mapred.Task: Task 'attempt_local306174234_0002_r_000000_0' done.
16/08/17 21:06:44 INFO mapred.LocalJobRunner: Finishing task: attempt_local306174234_0002_r_000000_0
16/08/17 21:06:44 INFO mapred.LocalJobRunner: Starting task: attempt_local306174234_0002_r_000001_0
16/08/17 21:06:44 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:06:44 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@71e63e26
16/08/17 21:06:44 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=370304608, maxSingleShuffleLimit=92576152, mergeThreshold=244401056, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:06:44 INFO reduce.EventFetcher: attempt_local306174234_0002_r_000001_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:06:44 INFO reduce.LocalFetcher: localfetcher#19 about to shuffle output of map attempt_local306174234_0002_m_000000_0 decomp: 82630 len: 82634 to MEMORY
16/08/17 21:06:44 INFO reduce.InMemoryMapOutput: Read 82630 bytes from map-output for attempt_local306174234_0002_m_000000_0
16/08/17 21:06:44 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 82630, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->82630
16/08/17 21:06:44 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:06:44 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)
	at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
16/08/17 21:06:44 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:44 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:06:44 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:44 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 82620 bytes
16/08/17 21:06:44 INFO reduce.MergeManagerImpl: Merged 1 segments, 82630 bytes to disk to satisfy reduce memory limit
16/08/17 21:06:44 INFO reduce.MergeManagerImpl: Merging 1 files, 82634 bytes from disk
16/08/17 21:06:44 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:06:44 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:44 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 82620 bytes
16/08/17 21:06:44 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:44 INFO mapred.Task: Task:attempt_local2012946159_0003_r_000000_0 is done. And is in the process of committing
16/08/17 21:06:44 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:44 INFO mapred.Task: Task attempt_local2012946159_0003_r_000000_0 is allowed to commit now
16/08/17 21:06:44 INFO output.FileOutputCommitter: Saved output of task 'attempt_local2012946159_0003_r_000000_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/lexprobs_sgt/_temporary/0/task_local2012946159_0003_r_000000
16/08/17 21:06:44 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:06:44 INFO mapred.Task: Task 'attempt_local2012946159_0003_r_000000_0' done.
16/08/17 21:06:44 INFO mapred.LocalJobRunner: Finishing task: attempt_local2012946159_0003_r_000000_0
16/08/17 21:06:44 INFO mapred.LocalJobRunner: Starting task: attempt_local2012946159_0003_r_000001_0
16/08/17 21:06:44 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:06:44 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@4e9bbca
16/08/17 21:06:44 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=370304608, maxSingleShuffleLimit=92576152, mergeThreshold=244401056, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:06:44 INFO mapred.Task: Task:attempt_local306174234_0002_r_000001_0 is done. And is in the process of committing
16/08/17 21:06:44 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:44 INFO mapred.Task: Task attempt_local306174234_0002_r_000001_0 is allowed to commit now
16/08/17 21:06:44 INFO output.FileOutputCommitter: Saved output of task 'attempt_local306174234_0002_r_000001_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/lexprobs_tgs/_temporary/0/task_local306174234_0002_r_000001
16/08/17 21:06:44 INFO reduce.EventFetcher: attempt_local2012946159_0003_r_000001_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:06:44 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:06:44 INFO mapred.Task: Task 'attempt_local306174234_0002_r_000001_0' done.
16/08/17 21:06:44 INFO mapred.LocalJobRunner: Finishing task: attempt_local306174234_0002_r_000001_0
16/08/17 21:06:44 INFO mapred.LocalJobRunner: Starting task: attempt_local306174234_0002_r_000002_0
16/08/17 21:06:44 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:06:44 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@93f9b40
16/08/17 21:06:44 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=370304608, maxSingleShuffleLimit=92576152, mergeThreshold=244401056, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:06:44 INFO reduce.EventFetcher: attempt_local306174234_0002_r_000002_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:06:44 INFO reduce.LocalFetcher: localfetcher#20 about to shuffle output of map attempt_local2012946159_0003_m_000000_0 decomp: 98730 len: 98734 to MEMORY
16/08/17 21:06:44 INFO reduce.InMemoryMapOutput: Read 98730 bytes from map-output for attempt_local2012946159_0003_m_000000_0
16/08/17 21:06:44 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 98730, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->98730
16/08/17 21:06:44 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:06:44 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:44 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:06:44 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:44 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 98720 bytes
16/08/17 21:06:44 INFO reduce.MergeManagerImpl: Merged 1 segments, 98730 bytes to disk to satisfy reduce memory limit
16/08/17 21:06:44 INFO reduce.MergeManagerImpl: Merging 1 files, 98734 bytes from disk
16/08/17 21:06:44 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:06:44 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:44 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 98720 bytes
16/08/17 21:06:44 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:44 INFO reduce.LocalFetcher: localfetcher#21 about to shuffle output of map attempt_local306174234_0002_m_000000_0 decomp: 98338 len: 98342 to MEMORY
16/08/17 21:06:44 INFO reduce.InMemoryMapOutput: Read 98338 bytes from map-output for attempt_local306174234_0002_m_000000_0
16/08/17 21:06:44 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 98338, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->98338
16/08/17 21:06:44 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:06:44 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:44 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:06:44 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:44 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 98328 bytes
16/08/17 21:06:44 INFO reduce.MergeManagerImpl: Merged 1 segments, 98338 bytes to disk to satisfy reduce memory limit
16/08/17 21:06:44 INFO reduce.MergeManagerImpl: Merging 1 files, 98342 bytes from disk
16/08/17 21:06:44 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:06:44 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:44 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 98328 bytes
16/08/17 21:06:44 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:44 INFO mapred.Task: Task:attempt_local2012946159_0003_r_000001_0 is done. And is in the process of committing
16/08/17 21:06:44 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:44 INFO mapred.Task: Task attempt_local2012946159_0003_r_000001_0 is allowed to commit now
16/08/17 21:06:44 INFO output.FileOutputCommitter: Saved output of task 'attempt_local2012946159_0003_r_000001_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/lexprobs_sgt/_temporary/0/task_local2012946159_0003_r_000001
16/08/17 21:06:44 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:06:44 INFO mapred.Task: Task 'attempt_local2012946159_0003_r_000001_0' done.
16/08/17 21:06:44 INFO mapred.LocalJobRunner: Finishing task: attempt_local2012946159_0003_r_000001_0
16/08/17 21:06:44 INFO mapred.LocalJobRunner: Starting task: attempt_local2012946159_0003_r_000002_0
16/08/17 21:06:44 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:06:44 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@35a73991
16/08/17 21:06:44 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=370304608, maxSingleShuffleLimit=92576152, mergeThreshold=244401056, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:06:44 INFO reduce.EventFetcher: attempt_local2012946159_0003_r_000002_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:06:44 INFO reduce.LocalFetcher: localfetcher#22 about to shuffle output of map attempt_local2012946159_0003_m_000000_0 decomp: 76694 len: 76698 to MEMORY
16/08/17 21:06:44 INFO reduce.InMemoryMapOutput: Read 76694 bytes from map-output for attempt_local2012946159_0003_m_000000_0
16/08/17 21:06:44 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 76694, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->76694
16/08/17 21:06:44 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:06:44 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:44 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:06:44 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:44 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 76684 bytes
16/08/17 21:06:44 INFO reduce.MergeManagerImpl: Merged 1 segments, 76694 bytes to disk to satisfy reduce memory limit
16/08/17 21:06:44 INFO reduce.MergeManagerImpl: Merging 1 files, 76698 bytes from disk
16/08/17 21:06:44 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:06:44 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:44 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 76684 bytes
16/08/17 21:06:44 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:44 INFO mapred.LocalJobRunner: map > map
16/08/17 21:06:44 INFO mapred.Task: Task:attempt_local306174234_0002_r_000002_0 is done. And is in the process of committing
16/08/17 21:06:44 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:44 INFO mapred.Task: Task attempt_local306174234_0002_r_000002_0 is allowed to commit now
16/08/17 21:06:44 INFO output.FileOutputCommitter: Saved output of task 'attempt_local306174234_0002_r_000002_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/lexprobs_tgs/_temporary/0/task_local306174234_0002_r_000002
16/08/17 21:06:44 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:06:44 INFO mapred.Task: Task 'attempt_local306174234_0002_r_000002_0' done.
16/08/17 21:06:44 INFO mapred.LocalJobRunner: Finishing task: attempt_local306174234_0002_r_000002_0
16/08/17 21:06:44 INFO mapred.LocalJobRunner: Starting task: attempt_local306174234_0002_r_000003_0
16/08/17 21:06:44 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:06:44 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@5905c55e
16/08/17 21:06:44 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=370304608, maxSingleShuffleLimit=92576152, mergeThreshold=244401056, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:06:44 INFO reduce.EventFetcher: attempt_local306174234_0002_r_000003_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:06:44 INFO reduce.LocalFetcher: localfetcher#23 about to shuffle output of map attempt_local306174234_0002_m_000000_0 decomp: 124126 len: 124130 to MEMORY
16/08/17 21:06:44 INFO reduce.InMemoryMapOutput: Read 124126 bytes from map-output for attempt_local306174234_0002_m_000000_0
16/08/17 21:06:44 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 124126, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->124126
16/08/17 21:06:44 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:06:44 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:44 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:06:44 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:44 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 124116 bytes
16/08/17 21:06:44 INFO reduce.MergeManagerImpl: Merged 1 segments, 124126 bytes to disk to satisfy reduce memory limit
16/08/17 21:06:44 INFO reduce.MergeManagerImpl: Merging 1 files, 124130 bytes from disk
16/08/17 21:06:44 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:06:44 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:44 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 124116 bytes
16/08/17 21:06:44 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:44 INFO mapred.Task: Task:attempt_local2012946159_0003_r_000002_0 is done. And is in the process of committing
16/08/17 21:06:44 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:44 INFO mapred.Task: Task attempt_local2012946159_0003_r_000002_0 is allowed to commit now
16/08/17 21:06:44 INFO output.FileOutputCommitter: Saved output of task 'attempt_local2012946159_0003_r_000002_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/lexprobs_sgt/_temporary/0/task_local2012946159_0003_r_000002
16/08/17 21:06:44 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:06:44 INFO mapred.Task: Task 'attempt_local2012946159_0003_r_000002_0' done.
16/08/17 21:06:44 INFO mapred.LocalJobRunner: Finishing task: attempt_local2012946159_0003_r_000002_0
16/08/17 21:06:44 INFO mapred.LocalJobRunner: Starting task: attempt_local2012946159_0003_r_000003_0
16/08/17 21:06:44 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:06:44 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@64c64485
16/08/17 21:06:44 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=370304608, maxSingleShuffleLimit=92576152, mergeThreshold=244401056, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:06:44 INFO reduce.EventFetcher: attempt_local2012946159_0003_r_000003_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:06:44 INFO reduce.LocalFetcher: localfetcher#24 about to shuffle output of map attempt_local2012946159_0003_m_000000_0 decomp: 49800 len: 49804 to MEMORY
16/08/17 21:06:44 INFO reduce.InMemoryMapOutput: Read 49800 bytes from map-output for attempt_local2012946159_0003_m_000000_0
16/08/17 21:06:44 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 49800, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->49800
16/08/17 21:06:44 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)
	at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
16/08/17 21:06:44 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:06:44 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:44 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:06:44 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:44 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 49790 bytes
16/08/17 21:06:44 INFO reduce.MergeManagerImpl: Merged 1 segments, 49800 bytes to disk to satisfy reduce memory limit
16/08/17 21:06:44 INFO reduce.MergeManagerImpl: Merging 1 files, 49804 bytes from disk
16/08/17 21:06:44 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:06:44 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:44 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 49790 bytes
16/08/17 21:06:44 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:44 INFO mapred.Task: Task:attempt_local306174234_0002_r_000003_0 is done. And is in the process of committing
16/08/17 21:06:44 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:44 INFO mapred.Task: Task attempt_local306174234_0002_r_000003_0 is allowed to commit now
16/08/17 21:06:44 INFO output.FileOutputCommitter: Saved output of task 'attempt_local306174234_0002_r_000003_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/lexprobs_tgs/_temporary/0/task_local306174234_0002_r_000003
16/08/17 21:06:44 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:06:44 INFO mapred.Task: Task 'attempt_local306174234_0002_r_000003_0' done.
16/08/17 21:06:44 INFO mapred.LocalJobRunner: Finishing task: attempt_local306174234_0002_r_000003_0
16/08/17 21:06:44 INFO mapred.LocalJobRunner: Starting task: attempt_local306174234_0002_r_000004_0
16/08/17 21:06:44 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:06:44 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@123b3793
16/08/17 21:06:44 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=370304608, maxSingleShuffleLimit=92576152, mergeThreshold=244401056, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:06:44 INFO reduce.EventFetcher: attempt_local306174234_0002_r_000004_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:06:44 INFO reduce.LocalFetcher: localfetcher#25 about to shuffle output of map attempt_local306174234_0002_m_000000_0 decomp: 112870 len: 112874 to MEMORY
16/08/17 21:06:44 INFO reduce.InMemoryMapOutput: Read 112870 bytes from map-output for attempt_local306174234_0002_m_000000_0
16/08/17 21:06:44 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 112870, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->112870
16/08/17 21:06:44 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:06:44 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:44 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:06:44 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:44 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 112860 bytes
16/08/17 21:06:44 INFO reduce.MergeManagerImpl: Merged 1 segments, 112870 bytes to disk to satisfy reduce memory limit
16/08/17 21:06:44 INFO reduce.MergeManagerImpl: Merging 1 files, 112874 bytes from disk
16/08/17 21:06:44 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:06:44 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:44 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 112860 bytes
16/08/17 21:06:44 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:45 INFO mapred.Task: Task:attempt_local2012946159_0003_r_000003_0 is done. And is in the process of committing
16/08/17 21:06:45 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:45 INFO mapred.Task: Task attempt_local2012946159_0003_r_000003_0 is allowed to commit now
16/08/17 21:06:45 INFO output.FileOutputCommitter: Saved output of task 'attempt_local2012946159_0003_r_000003_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/lexprobs_sgt/_temporary/0/task_local2012946159_0003_r_000003
16/08/17 21:06:45 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:06:45 INFO mapred.Task: Task 'attempt_local2012946159_0003_r_000003_0' done.
16/08/17 21:06:45 INFO mapred.LocalJobRunner: Finishing task: attempt_local2012946159_0003_r_000003_0
16/08/17 21:06:45 INFO mapred.LocalJobRunner: Starting task: attempt_local2012946159_0003_r_000004_0
16/08/17 21:06:45 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:06:45 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@5f059a3a
16/08/17 21:06:45 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=370304608, maxSingleShuffleLimit=92576152, mergeThreshold=244401056, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:06:45 INFO mapred.Task: Task:attempt_local306174234_0002_r_000004_0 is done. And is in the process of committing
16/08/17 21:06:45 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:45 INFO mapred.Task: Task attempt_local306174234_0002_r_000004_0 is allowed to commit now
16/08/17 21:06:45 INFO output.FileOutputCommitter: Saved output of task 'attempt_local306174234_0002_r_000004_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/lexprobs_tgs/_temporary/0/task_local306174234_0002_r_000004
16/08/17 21:06:45 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:06:45 INFO mapred.Task: Task 'attempt_local306174234_0002_r_000004_0' done.
16/08/17 21:06:45 INFO mapred.LocalJobRunner: Finishing task: attempt_local306174234_0002_r_000004_0
16/08/17 21:06:45 INFO mapred.LocalJobRunner: Starting task: attempt_local306174234_0002_r_000005_0
16/08/17 21:06:45 INFO reduce.EventFetcher: attempt_local2012946159_0003_r_000004_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:06:45 INFO reduce.LocalFetcher: localfetcher#26 about to shuffle output of map attempt_local2012946159_0003_m_000000_0 decomp: 76316 len: 76320 to MEMORY
16/08/17 21:06:45 INFO reduce.InMemoryMapOutput: Read 76316 bytes from map-output for attempt_local2012946159_0003_m_000000_0
16/08/17 21:06:45 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 76316, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->76316
16/08/17 21:06:45 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)
	at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
16/08/17 21:06:45 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:06:45 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@6ac42bef
16/08/17 21:06:45 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=370304608, maxSingleShuffleLimit=92576152, mergeThreshold=244401056, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:06:45 INFO reduce.EventFetcher: attempt_local306174234_0002_r_000005_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:06:45 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:06:45 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:45 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:06:45 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:45 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 76306 bytes
16/08/17 21:06:45 INFO reduce.MergeManagerImpl: Merged 1 segments, 76316 bytes to disk to satisfy reduce memory limit
16/08/17 21:06:45 INFO reduce.LocalFetcher: localfetcher#27 about to shuffle output of map attempt_local306174234_0002_m_000000_0 decomp: 156102 len: 156106 to MEMORY
16/08/17 21:06:45 INFO reduce.MergeManagerImpl: Merging 1 files, 76320 bytes from disk
16/08/17 21:06:45 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:06:45 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:45 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 76306 bytes
16/08/17 21:06:45 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:45 INFO reduce.InMemoryMapOutput: Read 156102 bytes from map-output for attempt_local306174234_0002_m_000000_0
16/08/17 21:06:45 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 156102, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->156102
16/08/17 21:06:45 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:06:45 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:45 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:06:45 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:45 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 156092 bytes
16/08/17 21:06:45 INFO reduce.MergeManagerImpl: Merged 1 segments, 156102 bytes to disk to satisfy reduce memory limit
16/08/17 21:06:45 INFO reduce.MergeManagerImpl: Merging 1 files, 156106 bytes from disk
16/08/17 21:06:45 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:06:45 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:45 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 156092 bytes
16/08/17 21:06:45 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:45 INFO mapred.Task: Task:attempt_local2012946159_0003_r_000004_0 is done. And is in the process of committing
16/08/17 21:06:45 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:45 INFO mapred.Task: Task attempt_local2012946159_0003_r_000004_0 is allowed to commit now
16/08/17 21:06:45 INFO output.FileOutputCommitter: Saved output of task 'attempt_local2012946159_0003_r_000004_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/lexprobs_sgt/_temporary/0/task_local2012946159_0003_r_000004
16/08/17 21:06:45 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:06:45 INFO mapred.Task: Task 'attempt_local2012946159_0003_r_000004_0' done.
16/08/17 21:06:45 INFO mapred.LocalJobRunner: Finishing task: attempt_local2012946159_0003_r_000004_0
16/08/17 21:06:45 INFO mapred.LocalJobRunner: Starting task: attempt_local2012946159_0003_r_000005_0
16/08/17 21:06:45 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:06:45 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@6088dce2
16/08/17 21:06:45 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=367001600, maxSingleShuffleLimit=91750400, mergeThreshold=242221072, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:06:45 INFO mapred.Task: Task:attempt_local306174234_0002_r_000005_0 is done. And is in the process of committing
16/08/17 21:06:45 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:45 INFO mapred.Task: Task attempt_local306174234_0002_r_000005_0 is allowed to commit now
16/08/17 21:06:45 INFO output.FileOutputCommitter: Saved output of task 'attempt_local306174234_0002_r_000005_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/lexprobs_tgs/_temporary/0/task_local306174234_0002_r_000005
16/08/17 21:06:45 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:06:45 INFO reduce.EventFetcher: attempt_local2012946159_0003_r_000005_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:06:45 INFO mapred.Task: Task 'attempt_local306174234_0002_r_000005_0' done.
16/08/17 21:06:45 INFO mapred.LocalJobRunner: Finishing task: attempt_local306174234_0002_r_000005_0
16/08/17 21:06:45 INFO mapred.LocalJobRunner: Starting task: attempt_local306174234_0002_r_000006_0
16/08/17 21:06:45 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:06:45 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@1cb3afe4
16/08/17 21:06:45 INFO reduce.LocalFetcher: localfetcher#28 about to shuffle output of map attempt_local2012946159_0003_m_000000_0 decomp: 56604 len: 56608 to MEMORY
16/08/17 21:06:45 INFO reduce.InMemoryMapOutput: Read 56604 bytes from map-output for attempt_local2012946159_0003_m_000000_0
16/08/17 21:06:45 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 56604, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->56604
16/08/17 21:06:45 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=367001600, maxSingleShuffleLimit=91750400, mergeThreshold=242221072, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:06:45 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:06:45 INFO reduce.EventFetcher: attempt_local306174234_0002_r_000006_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:06:45 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:45 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:06:45 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:45 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 56594 bytes
16/08/17 21:06:45 INFO reduce.MergeManagerImpl: Merged 1 segments, 56604 bytes to disk to satisfy reduce memory limit
16/08/17 21:06:45 INFO reduce.MergeManagerImpl: Merging 1 files, 56608 bytes from disk
16/08/17 21:06:45 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:06:45 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:45 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 56594 bytes
16/08/17 21:06:45 INFO reduce.LocalFetcher: localfetcher#29 about to shuffle output of map attempt_local306174234_0002_m_000000_0 decomp: 93578 len: 93582 to MEMORY
16/08/17 21:06:45 INFO reduce.InMemoryMapOutput: Read 93578 bytes from map-output for attempt_local306174234_0002_m_000000_0
16/08/17 21:06:45 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 93578, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->93578
16/08/17 21:06:45 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)
	at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
16/08/17 21:06:45 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:45 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:06:45 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:45 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:06:45 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:45 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 93568 bytes
16/08/17 21:06:45 INFO reduce.MergeManagerImpl: Merged 1 segments, 93578 bytes to disk to satisfy reduce memory limit
16/08/17 21:06:45 INFO reduce.MergeManagerImpl: Merging 1 files, 93582 bytes from disk
16/08/17 21:06:45 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:06:45 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:45 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 93568 bytes
16/08/17 21:06:45 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:45 INFO mapred.Task: Task:attempt_local2012946159_0003_r_000005_0 is done. And is in the process of committing
16/08/17 21:06:45 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:45 INFO mapred.Task: Task attempt_local2012946159_0003_r_000005_0 is allowed to commit now
16/08/17 21:06:45 INFO output.FileOutputCommitter: Saved output of task 'attempt_local2012946159_0003_r_000005_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/lexprobs_sgt/_temporary/0/task_local2012946159_0003_r_000005
16/08/17 21:06:45 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:06:45 INFO mapred.Task: Task 'attempt_local2012946159_0003_r_000005_0' done.
16/08/17 21:06:45 INFO mapred.LocalJobRunner: Finishing task: attempt_local2012946159_0003_r_000005_0
16/08/17 21:06:45 INFO mapred.LocalJobRunner: Starting task: attempt_local2012946159_0003_r_000006_0
16/08/17 21:06:45 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:06:45 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@525f81b0
16/08/17 21:06:45 INFO mapred.Task: Task:attempt_local306174234_0002_r_000006_0 is done. And is in the process of committing
16/08/17 21:06:45 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:45 INFO mapred.Task: Task attempt_local306174234_0002_r_000006_0 is allowed to commit now
16/08/17 21:06:45 INFO output.FileOutputCommitter: Saved output of task 'attempt_local306174234_0002_r_000006_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/lexprobs_tgs/_temporary/0/task_local306174234_0002_r_000006
16/08/17 21:06:45 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=367001600, maxSingleShuffleLimit=91750400, mergeThreshold=242221072, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:06:45 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:06:45 INFO mapred.Task: Task 'attempt_local306174234_0002_r_000006_0' done.
16/08/17 21:06:45 INFO mapred.LocalJobRunner: Finishing task: attempt_local306174234_0002_r_000006_0
16/08/17 21:06:45 INFO mapred.LocalJobRunner: Starting task: attempt_local306174234_0002_r_000007_0
16/08/17 21:06:45 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:06:45 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@2b92ff7b
16/08/17 21:06:45 INFO reduce.EventFetcher: attempt_local2012946159_0003_r_000006_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:06:45 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=367001600, maxSingleShuffleLimit=91750400, mergeThreshold=242221072, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:06:45 INFO reduce.LocalFetcher: localfetcher#30 about to shuffle output of map attempt_local2012946159_0003_m_000000_0 decomp: 147086 len: 147090 to MEMORY
16/08/17 21:06:45 INFO reduce.InMemoryMapOutput: Read 147086 bytes from map-output for attempt_local2012946159_0003_m_000000_0
16/08/17 21:06:45 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 147086, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->147086
16/08/17 21:06:45 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)
	at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
16/08/17 21:06:45 INFO reduce.EventFetcher: attempt_local306174234_0002_r_000007_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:06:45 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:06:45 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:45 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:06:45 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:45 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 147076 bytes
16/08/17 21:06:45 INFO reduce.LocalFetcher: localfetcher#31 about to shuffle output of map attempt_local306174234_0002_m_000000_0 decomp: 85668 len: 85672 to MEMORY
16/08/17 21:06:45 INFO reduce.InMemoryMapOutput: Read 85668 bytes from map-output for attempt_local306174234_0002_m_000000_0
16/08/17 21:06:45 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 85668, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->85668
16/08/17 21:06:45 INFO reduce.MergeManagerImpl: Merged 1 segments, 147086 bytes to disk to satisfy reduce memory limit
16/08/17 21:06:45 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:06:45 INFO reduce.MergeManagerImpl: Merging 1 files, 147090 bytes from disk
16/08/17 21:06:45 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:06:45 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:45 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:45 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:06:45 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 147076 bytes
16/08/17 21:06:45 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:45 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 85658 bytes
16/08/17 21:06:45 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:45 INFO reduce.MergeManagerImpl: Merged 1 segments, 85668 bytes to disk to satisfy reduce memory limit
16/08/17 21:06:45 INFO reduce.MergeManagerImpl: Merging 1 files, 85672 bytes from disk
16/08/17 21:06:45 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:06:45 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:45 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 85658 bytes
16/08/17 21:06:45 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:45 INFO mapred.Task: Task:attempt_local2012946159_0003_r_000006_0 is done. And is in the process of committing
16/08/17 21:06:45 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:45 INFO mapred.Task: Task attempt_local2012946159_0003_r_000006_0 is allowed to commit now
16/08/17 21:06:45 INFO output.FileOutputCommitter: Saved output of task 'attempt_local2012946159_0003_r_000006_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/lexprobs_sgt/_temporary/0/task_local2012946159_0003_r_000006
16/08/17 21:06:45 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:06:45 INFO mapred.Task: Task 'attempt_local2012946159_0003_r_000006_0' done.
16/08/17 21:06:45 INFO mapred.LocalJobRunner: Finishing task: attempt_local2012946159_0003_r_000006_0
16/08/17 21:06:45 INFO mapred.LocalJobRunner: Starting task: attempt_local2012946159_0003_r_000007_0
16/08/17 21:06:45 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:06:45 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@20d8167f
16/08/17 21:06:45 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=367001600, maxSingleShuffleLimit=91750400, mergeThreshold=242221072, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:06:45 INFO mapred.Task: Task:attempt_local306174234_0002_r_000007_0 is done. And is in the process of committing
16/08/17 21:06:45 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:45 INFO mapred.Task: Task attempt_local306174234_0002_r_000007_0 is allowed to commit now
16/08/17 21:06:45 INFO output.FileOutputCommitter: Saved output of task 'attempt_local306174234_0002_r_000007_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/lexprobs_tgs/_temporary/0/task_local306174234_0002_r_000007
16/08/17 21:06:45 INFO reduce.EventFetcher: attempt_local2012946159_0003_r_000007_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:06:45 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:06:45 INFO mapred.Task: Task 'attempt_local306174234_0002_r_000007_0' done.
16/08/17 21:06:45 INFO mapred.LocalJobRunner: Finishing task: attempt_local306174234_0002_r_000007_0
16/08/17 21:06:45 INFO mapred.LocalJobRunner: Starting task: attempt_local306174234_0002_r_000008_0
16/08/17 21:06:45 INFO reduce.LocalFetcher: localfetcher#32 about to shuffle output of map attempt_local2012946159_0003_m_000000_0 decomp: 115628 len: 115632 to MEMORY
16/08/17 21:06:45 INFO reduce.InMemoryMapOutput: Read 115628 bytes from map-output for attempt_local2012946159_0003_m_000000_0
16/08/17 21:06:45 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 115628, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->115628
16/08/17 21:06:45 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:06:45 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:06:45 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@1e20af5
16/08/17 21:06:45 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=367001600, maxSingleShuffleLimit=91750400, mergeThreshold=242221072, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:06:45 INFO reduce.EventFetcher: attempt_local306174234_0002_r_000008_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:06:45 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:45 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:06:45 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:45 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 115618 bytes
16/08/17 21:06:45 INFO reduce.MergeManagerImpl: Merged 1 segments, 115628 bytes to disk to satisfy reduce memory limit
16/08/17 21:06:45 INFO reduce.MergeManagerImpl: Merging 1 files, 115632 bytes from disk
16/08/17 21:06:45 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:06:45 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:45 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 115618 bytes
16/08/17 21:06:45 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:45 INFO reduce.LocalFetcher: localfetcher#33 about to shuffle output of map attempt_local306174234_0002_m_000000_0 decomp: 133212 len: 133216 to MEMORY
16/08/17 21:06:45 INFO reduce.InMemoryMapOutput: Read 133212 bytes from map-output for attempt_local306174234_0002_m_000000_0
16/08/17 21:06:45 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 133212, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->133212
16/08/17 21:06:45 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:06:45 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:45 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:06:45 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:45 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 133202 bytes
16/08/17 21:06:45 INFO reduce.MergeManagerImpl: Merged 1 segments, 133212 bytes to disk to satisfy reduce memory limit
16/08/17 21:06:45 INFO reduce.MergeManagerImpl: Merging 1 files, 133216 bytes from disk
16/08/17 21:06:45 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:06:45 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:45 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 133202 bytes
16/08/17 21:06:45 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:45 INFO mapred.Task: Task:attempt_local2012946159_0003_r_000007_0 is done. And is in the process of committing
16/08/17 21:06:45 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:45 INFO mapred.Task: Task attempt_local2012946159_0003_r_000007_0 is allowed to commit now
16/08/17 21:06:45 INFO output.FileOutputCommitter: Saved output of task 'attempt_local2012946159_0003_r_000007_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/lexprobs_sgt/_temporary/0/task_local2012946159_0003_r_000007
16/08/17 21:06:45 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:06:45 INFO mapred.Task: Task 'attempt_local2012946159_0003_r_000007_0' done.
16/08/17 21:06:45 INFO mapred.LocalJobRunner: Finishing task: attempt_local2012946159_0003_r_000007_0
16/08/17 21:06:45 INFO mapred.LocalJobRunner: Starting task: attempt_local2012946159_0003_r_000008_0
16/08/17 21:06:45 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:06:45 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@645490a8
16/08/17 21:06:45 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=367001600, maxSingleShuffleLimit=91750400, mergeThreshold=242221072, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:06:45 INFO reduce.EventFetcher: attempt_local2012946159_0003_r_000008_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:06:45 INFO reduce.LocalFetcher: localfetcher#34 about to shuffle output of map attempt_local2012946159_0003_m_000000_0 decomp: 128032 len: 128036 to MEMORY
16/08/17 21:06:45 INFO reduce.InMemoryMapOutput: Read 128032 bytes from map-output for attempt_local2012946159_0003_m_000000_0
16/08/17 21:06:45 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 128032, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->128032
16/08/17 21:06:45 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:06:45 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:45 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:06:45 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:45 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 128022 bytes
16/08/17 21:06:45 INFO reduce.MergeManagerImpl: Merged 1 segments, 128032 bytes to disk to satisfy reduce memory limit
16/08/17 21:06:45 INFO reduce.MergeManagerImpl: Merging 1 files, 128036 bytes from disk
16/08/17 21:06:45 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:06:45 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:45 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 128022 bytes
16/08/17 21:06:45 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:45 INFO mapred.Task: Task:attempt_local306174234_0002_r_000008_0 is done. And is in the process of committing
16/08/17 21:06:45 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:45 INFO mapred.Task: Task attempt_local306174234_0002_r_000008_0 is allowed to commit now
16/08/17 21:06:45 INFO output.FileOutputCommitter: Saved output of task 'attempt_local306174234_0002_r_000008_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/lexprobs_tgs/_temporary/0/task_local306174234_0002_r_000008
16/08/17 21:06:45 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:06:45 INFO mapred.Task: Task 'attempt_local306174234_0002_r_000008_0' done.
16/08/17 21:06:45 INFO mapred.LocalJobRunner: Finishing task: attempt_local306174234_0002_r_000008_0
16/08/17 21:06:45 INFO mapred.LocalJobRunner: Starting task: attempt_local306174234_0002_r_000009_0
16/08/17 21:06:45 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:06:45 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@4c5a1508
16/08/17 21:06:45 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=369570592, maxSingleShuffleLimit=92392648, mergeThreshold=243916608, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:06:45 INFO reduce.EventFetcher: attempt_local306174234_0002_r_000009_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:06:45 INFO reduce.LocalFetcher: localfetcher#35 about to shuffle output of map attempt_local306174234_0002_m_000000_0 decomp: 137118 len: 137122 to MEMORY
16/08/17 21:06:45 INFO reduce.InMemoryMapOutput: Read 137118 bytes from map-output for attempt_local306174234_0002_m_000000_0
16/08/17 21:06:45 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 137118, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->137118
16/08/17 21:06:45 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:06:45 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:45 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:06:45 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:45 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 137108 bytes
16/08/17 21:06:45 INFO reduce.MergeManagerImpl: Merged 1 segments, 137118 bytes to disk to satisfy reduce memory limit
16/08/17 21:06:45 INFO reduce.MergeManagerImpl: Merging 1 files, 137122 bytes from disk
16/08/17 21:06:45 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:06:45 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:45 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 137108 bytes
16/08/17 21:06:45 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:45 INFO mapred.Task: Task:attempt_local2012946159_0003_r_000008_0 is done. And is in the process of committing
16/08/17 21:06:45 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:45 INFO mapred.Task: Task attempt_local2012946159_0003_r_000008_0 is allowed to commit now
16/08/17 21:06:45 INFO output.FileOutputCommitter: Saved output of task 'attempt_local2012946159_0003_r_000008_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/lexprobs_sgt/_temporary/0/task_local2012946159_0003_r_000008
16/08/17 21:06:45 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:06:45 INFO mapred.Task: Task 'attempt_local2012946159_0003_r_000008_0' done.
16/08/17 21:06:45 INFO mapred.LocalJobRunner: Finishing task: attempt_local2012946159_0003_r_000008_0
16/08/17 21:06:45 INFO mapred.LocalJobRunner: Starting task: attempt_local2012946159_0003_r_000009_0
16/08/17 21:06:45 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:06:45 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@33413656
16/08/17 21:06:45 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=369570592, maxSingleShuffleLimit=92392648, mergeThreshold=243916608, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:06:45 INFO reduce.EventFetcher: attempt_local2012946159_0003_r_000009_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:06:45 INFO reduce.LocalFetcher: localfetcher#36 about to shuffle output of map attempt_local2012946159_0003_m_000000_0 decomp: 123972 len: 123976 to MEMORY
16/08/17 21:06:45 INFO reduce.InMemoryMapOutput: Read 123972 bytes from map-output for attempt_local2012946159_0003_m_000000_0
16/08/17 21:06:45 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 123972, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->123972
16/08/17 21:06:45 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:06:45 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:45 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:06:45 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:45 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 123962 bytes
16/08/17 21:06:45 INFO reduce.MergeManagerImpl: Merged 1 segments, 123972 bytes to disk to satisfy reduce memory limit
16/08/17 21:06:45 INFO reduce.MergeManagerImpl: Merging 1 files, 123976 bytes from disk
16/08/17 21:06:45 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:06:45 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:45 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 123962 bytes
16/08/17 21:06:45 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:45 INFO mapred.Task: Task:attempt_local306174234_0002_r_000009_0 is done. And is in the process of committing
16/08/17 21:06:45 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:45 INFO mapred.Task: Task attempt_local306174234_0002_r_000009_0 is allowed to commit now
16/08/17 21:06:45 INFO output.FileOutputCommitter: Saved output of task 'attempt_local306174234_0002_r_000009_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/lexprobs_tgs/_temporary/0/task_local306174234_0002_r_000009
16/08/17 21:06:45 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:06:45 INFO mapred.Task: Task 'attempt_local306174234_0002_r_000009_0' done.
16/08/17 21:06:45 INFO mapred.LocalJobRunner: Finishing task: attempt_local306174234_0002_r_000009_0
16/08/17 21:06:45 INFO mapred.LocalJobRunner: Starting task: attempt_local306174234_0002_r_000010_0
16/08/17 21:06:45 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:06:45 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@27677f86
16/08/17 21:06:45 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=369570592, maxSingleShuffleLimit=92392648, mergeThreshold=243916608, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:06:45 INFO mapred.Task: Task:attempt_local2012946159_0003_r_000009_0 is done. And is in the process of committing
16/08/17 21:06:45 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:45 INFO mapred.Task: Task attempt_local2012946159_0003_r_000009_0 is allowed to commit now
16/08/17 21:06:45 INFO output.FileOutputCommitter: Saved output of task 'attempt_local2012946159_0003_r_000009_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/lexprobs_sgt/_temporary/0/task_local2012946159_0003_r_000009
16/08/17 21:06:45 INFO reduce.EventFetcher: attempt_local306174234_0002_r_000010_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:06:45 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:06:45 INFO mapred.Task: Task 'attempt_local2012946159_0003_r_000009_0' done.
16/08/17 21:06:45 INFO mapred.LocalJobRunner: Finishing task: attempt_local2012946159_0003_r_000009_0
16/08/17 21:06:45 INFO mapred.LocalJobRunner: Starting task: attempt_local2012946159_0003_r_000010_0
16/08/17 21:06:45 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:06:45 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@822fce7
16/08/17 21:06:45 INFO reduce.LocalFetcher: localfetcher#37 about to shuffle output of map attempt_local306174234_0002_m_000000_0 decomp: 114326 len: 114330 to MEMORY
16/08/17 21:06:45 INFO reduce.InMemoryMapOutput: Read 114326 bytes from map-output for attempt_local306174234_0002_m_000000_0
16/08/17 21:06:45 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 114326, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->114326
16/08/17 21:06:45 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)
	at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
16/08/17 21:06:45 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=369570592, maxSingleShuffleLimit=92392648, mergeThreshold=243916608, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:06:45 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:06:45 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:45 INFO reduce.EventFetcher: attempt_local2012946159_0003_r_000010_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:06:45 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:06:45 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:45 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 114316 bytes
16/08/17 21:06:45 INFO reduce.LocalFetcher: localfetcher#38 about to shuffle output of map attempt_local2012946159_0003_m_000000_0 decomp: 71094 len: 71098 to MEMORY
16/08/17 21:06:45 INFO reduce.InMemoryMapOutput: Read 71094 bytes from map-output for attempt_local2012946159_0003_m_000000_0
16/08/17 21:06:45 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 71094, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->71094
16/08/17 21:06:45 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)
	at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
16/08/17 21:06:45 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:06:45 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:45 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:06:45 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:45 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 71084 bytes
16/08/17 21:06:45 INFO reduce.MergeManagerImpl: Merged 1 segments, 114326 bytes to disk to satisfy reduce memory limit
16/08/17 21:06:45 INFO reduce.MergeManagerImpl: Merging 1 files, 114330 bytes from disk
16/08/17 21:06:45 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:06:45 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:45 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 114316 bytes
16/08/17 21:06:45 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:45 INFO reduce.MergeManagerImpl: Merged 1 segments, 71094 bytes to disk to satisfy reduce memory limit
16/08/17 21:06:45 INFO reduce.MergeManagerImpl: Merging 1 files, 71098 bytes from disk
16/08/17 21:06:45 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:06:45 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:45 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 71084 bytes
16/08/17 21:06:45 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:45 INFO mapred.Task: Task:attempt_local306174234_0002_r_000010_0 is done. And is in the process of committing
16/08/17 21:06:45 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:45 INFO mapred.Task: Task attempt_local306174234_0002_r_000010_0 is allowed to commit now
16/08/17 21:06:45 INFO output.FileOutputCommitter: Saved output of task 'attempt_local306174234_0002_r_000010_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/lexprobs_tgs/_temporary/0/task_local306174234_0002_r_000010
16/08/17 21:06:45 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:06:45 INFO mapred.Task: Task 'attempt_local306174234_0002_r_000010_0' done.
16/08/17 21:06:45 INFO mapred.LocalJobRunner: Finishing task: attempt_local306174234_0002_r_000010_0
16/08/17 21:06:45 INFO mapred.LocalJobRunner: Starting task: attempt_local306174234_0002_r_000011_0
16/08/17 21:06:45 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:06:45 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@31260136
16/08/17 21:06:45 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=369570592, maxSingleShuffleLimit=92392648, mergeThreshold=243916608, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:06:45 INFO reduce.EventFetcher: attempt_local306174234_0002_r_000011_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:06:45 INFO reduce.LocalFetcher: localfetcher#39 about to shuffle output of map attempt_local306174234_0002_m_000000_0 decomp: 102944 len: 102948 to MEMORY
16/08/17 21:06:45 INFO reduce.InMemoryMapOutput: Read 102944 bytes from map-output for attempt_local306174234_0002_m_000000_0
16/08/17 21:06:45 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 102944, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->102944
16/08/17 21:06:45 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:06:45 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)
	at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
16/08/17 21:06:45 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:45 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:06:45 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:45 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 102934 bytes
16/08/17 21:06:45 INFO reduce.MergeManagerImpl: Merged 1 segments, 102944 bytes to disk to satisfy reduce memory limit
16/08/17 21:06:45 INFO reduce.MergeManagerImpl: Merging 1 files, 102948 bytes from disk
16/08/17 21:06:45 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:06:45 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:45 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 102934 bytes
16/08/17 21:06:45 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:46 INFO mapred.Task: Task:attempt_local2012946159_0003_r_000010_0 is done. And is in the process of committing
16/08/17 21:06:46 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:46 INFO mapred.Task: Task attempt_local2012946159_0003_r_000010_0 is allowed to commit now
16/08/17 21:06:46 INFO output.FileOutputCommitter: Saved output of task 'attempt_local2012946159_0003_r_000010_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/lexprobs_sgt/_temporary/0/task_local2012946159_0003_r_000010
16/08/17 21:06:46 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:06:46 INFO mapred.Task: Task 'attempt_local2012946159_0003_r_000010_0' done.
16/08/17 21:06:46 INFO mapred.LocalJobRunner: Finishing task: attempt_local2012946159_0003_r_000010_0
16/08/17 21:06:46 INFO mapred.LocalJobRunner: Starting task: attempt_local2012946159_0003_r_000011_0
16/08/17 21:06:46 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:06:46 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@4e058588
16/08/17 21:06:46 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=369570592, maxSingleShuffleLimit=92392648, mergeThreshold=243916608, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:06:46 INFO mapred.Task: Task:attempt_local306174234_0002_r_000011_0 is done. And is in the process of committing
16/08/17 21:06:46 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:46 INFO mapred.Task: Task attempt_local306174234_0002_r_000011_0 is allowed to commit now
16/08/17 21:06:46 INFO output.FileOutputCommitter: Saved output of task 'attempt_local306174234_0002_r_000011_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/lexprobs_tgs/_temporary/0/task_local306174234_0002_r_000011
16/08/17 21:06:46 INFO reduce.EventFetcher: attempt_local2012946159_0003_r_000011_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:06:46 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:06:46 INFO mapred.Task: Task 'attempt_local306174234_0002_r_000011_0' done.
16/08/17 21:06:46 INFO mapred.LocalJobRunner: Finishing task: attempt_local306174234_0002_r_000011_0
16/08/17 21:06:46 INFO mapred.LocalJobRunner: Starting task: attempt_local306174234_0002_r_000012_0
16/08/17 21:06:46 INFO reduce.LocalFetcher: localfetcher#40 about to shuffle output of map attempt_local2012946159_0003_m_000000_0 decomp: 59796 len: 59800 to MEMORY
16/08/17 21:06:46 INFO reduce.InMemoryMapOutput: Read 59796 bytes from map-output for attempt_local2012946159_0003_m_000000_0
16/08/17 21:06:46 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 59796, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->59796
16/08/17 21:06:46 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:06:46 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@3f4563
16/08/17 21:06:46 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=369570592, maxSingleShuffleLimit=92392648, mergeThreshold=243916608, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:06:46 INFO reduce.EventFetcher: attempt_local306174234_0002_r_000012_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:06:46 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:06:46 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:46 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:06:46 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:46 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 59786 bytes
16/08/17 21:06:46 INFO reduce.LocalFetcher: localfetcher#41 about to shuffle output of map attempt_local306174234_0002_m_000000_0 decomp: 95370 len: 95374 to MEMORY
16/08/17 21:06:46 INFO reduce.InMemoryMapOutput: Read 95370 bytes from map-output for attempt_local306174234_0002_m_000000_0
16/08/17 21:06:46 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 95370, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->95370
16/08/17 21:06:46 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)
	at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
16/08/17 21:06:46 INFO reduce.MergeManagerImpl: Merged 1 segments, 59796 bytes to disk to satisfy reduce memory limit
16/08/17 21:06:46 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:06:46 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:46 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:06:46 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:46 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 95360 bytes
16/08/17 21:06:46 INFO reduce.MergeManagerImpl: Merged 1 segments, 95370 bytes to disk to satisfy reduce memory limit
16/08/17 21:06:46 INFO reduce.MergeManagerImpl: Merging 1 files, 95374 bytes from disk
16/08/17 21:06:46 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:06:46 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:46 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 95360 bytes
16/08/17 21:06:46 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:46 INFO reduce.MergeManagerImpl: Merging 1 files, 59800 bytes from disk
16/08/17 21:06:46 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:06:46 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:46 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 59786 bytes
16/08/17 21:06:46 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:46 INFO mapred.Task: Task:attempt_local306174234_0002_r_000012_0 is done. And is in the process of committing
16/08/17 21:06:46 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:46 INFO mapred.Task: Task attempt_local306174234_0002_r_000012_0 is allowed to commit now
16/08/17 21:06:46 INFO output.FileOutputCommitter: Saved output of task 'attempt_local306174234_0002_r_000012_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/lexprobs_tgs/_temporary/0/task_local306174234_0002_r_000012
16/08/17 21:06:46 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:06:46 INFO mapred.Task: Task 'attempt_local306174234_0002_r_000012_0' done.
16/08/17 21:06:46 INFO mapred.LocalJobRunner: Finishing task: attempt_local306174234_0002_r_000012_0
16/08/17 21:06:46 INFO mapred.LocalJobRunner: Starting task: attempt_local306174234_0002_r_000013_0
16/08/17 21:06:46 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:06:46 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@692fb302
16/08/17 21:06:46 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=369937600, maxSingleShuffleLimit=92484400, mergeThreshold=244158832, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:06:46 INFO mapred.Task: Task:attempt_local2012946159_0003_r_000011_0 is done. And is in the process of committing
16/08/17 21:06:46 INFO reduce.EventFetcher: attempt_local306174234_0002_r_000013_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:06:46 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:46 INFO mapred.Task: Task attempt_local2012946159_0003_r_000011_0 is allowed to commit now
16/08/17 21:06:46 INFO output.FileOutputCommitter: Saved output of task 'attempt_local2012946159_0003_r_000011_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/lexprobs_sgt/_temporary/0/task_local2012946159_0003_r_000011
16/08/17 21:06:46 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:06:46 INFO mapred.Task: Task 'attempt_local2012946159_0003_r_000011_0' done.
16/08/17 21:06:46 INFO mapred.LocalJobRunner: Finishing task: attempt_local2012946159_0003_r_000011_0
16/08/17 21:06:46 INFO mapred.LocalJobRunner: Starting task: attempt_local2012946159_0003_r_000012_0
16/08/17 21:06:46 INFO reduce.LocalFetcher: localfetcher#42 about to shuffle output of map attempt_local306174234_0002_m_000000_0 decomp: 115530 len: 115534 to MEMORY
16/08/17 21:06:46 INFO reduce.InMemoryMapOutput: Read 115530 bytes from map-output for attempt_local306174234_0002_m_000000_0
16/08/17 21:06:46 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 115530, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->115530
16/08/17 21:06:46 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:06:46 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@69fb5f6
16/08/17 21:06:46 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=369937600, maxSingleShuffleLimit=92484400, mergeThreshold=244158832, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:06:46 INFO reduce.EventFetcher: attempt_local2012946159_0003_r_000012_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:06:46 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:06:46 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:46 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:06:46 INFO reduce.LocalFetcher: localfetcher#43 about to shuffle output of map attempt_local2012946159_0003_m_000000_0 decomp: 101096 len: 101100 to MEMORY
16/08/17 21:06:46 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:46 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 115520 bytes
16/08/17 21:06:46 INFO reduce.InMemoryMapOutput: Read 101096 bytes from map-output for attempt_local2012946159_0003_m_000000_0
16/08/17 21:06:46 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 101096, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->101096
16/08/17 21:06:46 INFO reduce.MergeManagerImpl: Merged 1 segments, 115530 bytes to disk to satisfy reduce memory limit
16/08/17 21:06:46 INFO reduce.MergeManagerImpl: Merging 1 files, 115534 bytes from disk
16/08/17 21:06:46 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:06:46 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:46 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 115520 bytes
16/08/17 21:06:46 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:46 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:06:46 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:46 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:06:46 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:46 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 101086 bytes
16/08/17 21:06:46 INFO reduce.MergeManagerImpl: Merged 1 segments, 101096 bytes to disk to satisfy reduce memory limit
16/08/17 21:06:46 INFO reduce.MergeManagerImpl: Merging 1 files, 101100 bytes from disk
16/08/17 21:06:46 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:06:46 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:46 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 101086 bytes
16/08/17 21:06:46 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:46 INFO mapred.Task: Task:attempt_local306174234_0002_r_000013_0 is done. And is in the process of committing
16/08/17 21:06:46 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:46 INFO mapred.Task: Task attempt_local306174234_0002_r_000013_0 is allowed to commit now
16/08/17 21:06:46 INFO output.FileOutputCommitter: Saved output of task 'attempt_local306174234_0002_r_000013_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/lexprobs_tgs/_temporary/0/task_local306174234_0002_r_000013
16/08/17 21:06:46 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:06:46 INFO mapred.Task: Task 'attempt_local306174234_0002_r_000013_0' done.
16/08/17 21:06:46 INFO mapred.LocalJobRunner: Finishing task: attempt_local306174234_0002_r_000013_0
16/08/17 21:06:46 INFO mapred.LocalJobRunner: Starting task: attempt_local306174234_0002_r_000014_0
16/08/17 21:06:46 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:06:46 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@4bb03275
16/08/17 21:06:46 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=369937600, maxSingleShuffleLimit=92484400, mergeThreshold=244158832, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:06:46 INFO reduce.EventFetcher: attempt_local306174234_0002_r_000014_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:06:46 INFO reduce.LocalFetcher: localfetcher#44 about to shuffle output of map attempt_local306174234_0002_m_000000_0 decomp: 121718 len: 121722 to MEMORY
16/08/17 21:06:46 INFO reduce.InMemoryMapOutput: Read 121718 bytes from map-output for attempt_local306174234_0002_m_000000_0
16/08/17 21:06:46 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 121718, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->121718
16/08/17 21:06:46 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:06:46 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:46 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:06:46 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:46 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 121708 bytes
16/08/17 21:06:46 INFO reduce.MergeManagerImpl: Merged 1 segments, 121718 bytes to disk to satisfy reduce memory limit
16/08/17 21:06:46 INFO reduce.MergeManagerImpl: Merging 1 files, 121722 bytes from disk
16/08/17 21:06:46 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:06:46 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:46 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 121708 bytes
16/08/17 21:06:46 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:46 INFO mapred.Task: Task:attempt_local2012946159_0003_r_000012_0 is done. And is in the process of committing
16/08/17 21:06:46 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:46 INFO mapred.Task: Task attempt_local2012946159_0003_r_000012_0 is allowed to commit now
16/08/17 21:06:46 INFO output.FileOutputCommitter: Saved output of task 'attempt_local2012946159_0003_r_000012_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/lexprobs_sgt/_temporary/0/task_local2012946159_0003_r_000012
16/08/17 21:06:46 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:06:46 INFO mapred.Task: Task 'attempt_local2012946159_0003_r_000012_0' done.
16/08/17 21:06:46 INFO mapred.LocalJobRunner: Finishing task: attempt_local2012946159_0003_r_000012_0
16/08/17 21:06:46 INFO mapred.LocalJobRunner: Starting task: attempt_local2012946159_0003_r_000013_0
16/08/17 21:06:46 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:06:46 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@777d1779
16/08/17 21:06:46 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=369937600, maxSingleShuffleLimit=92484400, mergeThreshold=244158832, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:06:46 INFO reduce.EventFetcher: attempt_local2012946159_0003_r_000013_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:06:46 INFO reduce.LocalFetcher: localfetcher#45 about to shuffle output of map attempt_local2012946159_0003_m_000000_0 decomp: 158132 len: 158136 to MEMORY
16/08/17 21:06:46 INFO reduce.InMemoryMapOutput: Read 158132 bytes from map-output for attempt_local2012946159_0003_m_000000_0
16/08/17 21:06:46 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 158132, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->158132
16/08/17 21:06:46 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:06:46 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:46 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:06:46 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:46 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 158122 bytes
16/08/17 21:06:46 INFO reduce.MergeManagerImpl: Merged 1 segments, 158132 bytes to disk to satisfy reduce memory limit
16/08/17 21:06:46 INFO reduce.MergeManagerImpl: Merging 1 files, 158136 bytes from disk
16/08/17 21:06:46 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:06:46 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:46 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 158122 bytes
16/08/17 21:06:46 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:46 INFO mapred.Task: Task:attempt_local306174234_0002_r_000014_0 is done. And is in the process of committing
16/08/17 21:06:46 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:46 INFO mapred.Task: Task attempt_local306174234_0002_r_000014_0 is allowed to commit now
16/08/17 21:06:46 INFO output.FileOutputCommitter: Saved output of task 'attempt_local306174234_0002_r_000014_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/lexprobs_tgs/_temporary/0/task_local306174234_0002_r_000014
16/08/17 21:06:46 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:06:46 INFO mapred.Task: Task 'attempt_local306174234_0002_r_000014_0' done.
16/08/17 21:06:46 INFO mapred.LocalJobRunner: Finishing task: attempt_local306174234_0002_r_000014_0
16/08/17 21:06:46 INFO mapred.LocalJobRunner: Starting task: attempt_local306174234_0002_r_000015_0
16/08/17 21:06:46 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:06:46 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@3e7727f5
16/08/17 21:06:46 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=369937600, maxSingleShuffleLimit=92484400, mergeThreshold=244158832, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:06:46 INFO reduce.EventFetcher: attempt_local306174234_0002_r_000015_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:06:46 INFO reduce.LocalFetcher: localfetcher#46 about to shuffle output of map attempt_local306174234_0002_m_000000_0 decomp: 27162 len: 27166 to MEMORY
16/08/17 21:06:46 INFO reduce.InMemoryMapOutput: Read 27162 bytes from map-output for attempt_local306174234_0002_m_000000_0
16/08/17 21:06:46 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 27162, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->27162
16/08/17 21:06:46 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)
	at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
16/08/17 21:06:46 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:06:46 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:46 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:06:46 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:46 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 27152 bytes
16/08/17 21:06:46 INFO reduce.MergeManagerImpl: Merged 1 segments, 27162 bytes to disk to satisfy reduce memory limit
16/08/17 21:06:46 INFO reduce.MergeManagerImpl: Merging 1 files, 27166 bytes from disk
16/08/17 21:06:46 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:06:46 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:46 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 27152 bytes
16/08/17 21:06:46 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:46 INFO mapred.Task: Task:attempt_local2012946159_0003_r_000013_0 is done. And is in the process of committing
16/08/17 21:06:46 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:46 INFO mapred.Task: Task attempt_local2012946159_0003_r_000013_0 is allowed to commit now
16/08/17 21:06:46 INFO output.FileOutputCommitter: Saved output of task 'attempt_local2012946159_0003_r_000013_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/lexprobs_sgt/_temporary/0/task_local2012946159_0003_r_000013
16/08/17 21:06:46 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:06:46 INFO mapred.Task: Task 'attempt_local2012946159_0003_r_000013_0' done.
16/08/17 21:06:46 INFO mapred.LocalJobRunner: Finishing task: attempt_local2012946159_0003_r_000013_0
16/08/17 21:06:46 INFO mapred.LocalJobRunner: Starting task: attempt_local2012946159_0003_r_000014_0
16/08/17 21:06:46 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:06:46 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@387881b1
16/08/17 21:06:46 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=369937600, maxSingleShuffleLimit=92484400, mergeThreshold=244158832, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:06:46 INFO mapred.Task: Task:attempt_local306174234_0002_r_000015_0 is done. And is in the process of committing
16/08/17 21:06:46 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:46 INFO mapred.Task: Task attempt_local306174234_0002_r_000015_0 is allowed to commit now
16/08/17 21:06:46 INFO output.FileOutputCommitter: Saved output of task 'attempt_local306174234_0002_r_000015_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/lexprobs_tgs/_temporary/0/task_local306174234_0002_r_000015
16/08/17 21:06:46 INFO reduce.EventFetcher: attempt_local2012946159_0003_r_000014_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:06:46 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:06:46 INFO mapred.Task: Task 'attempt_local306174234_0002_r_000015_0' done.
16/08/17 21:06:46 INFO mapred.LocalJobRunner: Finishing task: attempt_local306174234_0002_r_000015_0
16/08/17 21:06:46 INFO mapred.LocalJobRunner: reduce task executor complete.
16/08/17 21:06:46 INFO reduce.LocalFetcher: localfetcher#47 about to shuffle output of map attempt_local2012946159_0003_m_000000_0 decomp: 163396 len: 163400 to MEMORY
16/08/17 21:06:46 INFO reduce.InMemoryMapOutput: Read 163396 bytes from map-output for attempt_local2012946159_0003_m_000000_0
16/08/17 21:06:46 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 163396, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->163396
16/08/17 21:06:46 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:06:46 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)
	at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
16/08/17 21:06:46 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:46 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:06:46 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:46 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 163386 bytes
16/08/17 21:06:46 INFO reduce.MergeManagerImpl: Merged 1 segments, 163396 bytes to disk to satisfy reduce memory limit
16/08/17 21:06:46 INFO reduce.MergeManagerImpl: Merging 1 files, 163400 bytes from disk
16/08/17 21:06:46 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:06:46 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:46 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 163386 bytes
16/08/17 21:06:46 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:46 INFO mapred.Task: Task:attempt_local2012946159_0003_r_000014_0 is done. And is in the process of committing
16/08/17 21:06:46 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:46 INFO mapred.Task: Task attempt_local2012946159_0003_r_000014_0 is allowed to commit now
16/08/17 21:06:46 INFO output.FileOutputCommitter: Saved output of task 'attempt_local2012946159_0003_r_000014_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/lexprobs_sgt/_temporary/0/task_local2012946159_0003_r_000014
16/08/17 21:06:46 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:06:46 INFO mapred.Task: Task 'attempt_local2012946159_0003_r_000014_0' done.
16/08/17 21:06:46 INFO mapred.LocalJobRunner: Finishing task: attempt_local2012946159_0003_r_000014_0
16/08/17 21:06:46 INFO mapred.LocalJobRunner: Starting task: attempt_local2012946159_0003_r_000015_0
16/08/17 21:06:46 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:06:46 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@5f99fb7
16/08/17 21:06:46 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=369937600, maxSingleShuffleLimit=92484400, mergeThreshold=244158832, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:06:46 INFO reduce.EventFetcher: attempt_local2012946159_0003_r_000015_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:06:46 INFO reduce.LocalFetcher: localfetcher#48 about to shuffle output of map attempt_local2012946159_0003_m_000000_0 decomp: 41540 len: 41544 to MEMORY
16/08/17 21:06:46 INFO reduce.InMemoryMapOutput: Read 41540 bytes from map-output for attempt_local2012946159_0003_m_000000_0
16/08/17 21:06:46 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 41540, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->41540
16/08/17 21:06:46 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:06:46 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:46 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:06:46 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:46 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 41530 bytes
16/08/17 21:06:46 INFO reduce.MergeManagerImpl: Merged 1 segments, 41540 bytes to disk to satisfy reduce memory limit
16/08/17 21:06:46 INFO reduce.MergeManagerImpl: Merging 1 files, 41544 bytes from disk
16/08/17 21:06:46 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:06:46 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:46 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 41530 bytes
16/08/17 21:06:46 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:46 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)
	at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
16/08/17 21:06:46 INFO mapred.Task: Task:attempt_local2012946159_0003_r_000015_0 is done. And is in the process of committing
16/08/17 21:06:46 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:46 INFO mapred.Task: Task attempt_local2012946159_0003_r_000015_0 is allowed to commit now
16/08/17 21:06:46 INFO output.FileOutputCommitter: Saved output of task 'attempt_local2012946159_0003_r_000015_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/lexprobs_sgt/_temporary/0/task_local2012946159_0003_r_000015
16/08/17 21:06:46 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:06:46 INFO mapred.Task: Task 'attempt_local2012946159_0003_r_000015_0' done.
16/08/17 21:06:46 INFO mapred.LocalJobRunner: Finishing task: attempt_local2012946159_0003_r_000015_0
16/08/17 21:06:46 INFO mapred.LocalJobRunner: reduce task executor complete.
16/08/17 21:06:47 INFO mapred.LocalJobRunner: map > map
[SCHED] class edu.jhu.thrax.hadoop.jobs.SourceWordGivenTargetWordProbabilityJob in state SUCCESS
[SCHED] class edu.jhu.thrax.hadoop.jobs.TargetWordGivenSourceWordProbabilityJob in state SUCCESS
16/08/17 21:06:50 INFO mapred.MapTask: Spilling map output
16/08/17 21:06:50 INFO mapred.MapTask: bufstart = 0; bufend = 59149258; bufvoid = 104857600
16/08/17 21:06:50 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 20030188(80120752); length = 6184209/6553600
16/08/17 21:06:50 INFO mapred.MapTask: (EQUATOR) 65363018 kvi 16340748(65362992)
16/08/17 21:06:50 INFO mapred.LocalJobRunner: map > map
16/08/17 21:06:51 INFO mapred.LocalJobRunner: map > map
16/08/17 21:06:51 INFO mapred.MapTask: Starting flush of map output
16/08/17 21:06:53 INFO mapred.LocalJobRunner: map > sort
16/08/17 21:06:55 INFO mapred.MapTask: Finished spill 0
16/08/17 21:06:55 INFO mapred.MapTask: (RESET) equator 65363018 kv 16340748(65362992) kvi 15372944(61491776)
16/08/17 21:06:55 INFO mapred.MapTask: Spilling map output
16/08/17 21:06:55 INFO mapred.MapTask: bufstart = 65363018; bufend = 74548904; bufvoid = 104857600
16/08/17 21:06:55 INFO mapred.MapTask: kvstart = 16340748(65362992); kvend = 15372948(61491792); length = 967801/6553600
16/08/17 21:06:55 INFO mapred.MapTask: Finished spill 1
16/08/17 21:06:55 INFO mapred.Merger: Merging 2 sorted segments
16/08/17 21:06:55 INFO mapred.Merger: Down to the last merge-pass, with 2 segments left of total size: 2151648 bytes
16/08/17 21:06:55 INFO mapred.Merger: Merging 2 sorted segments
16/08/17 21:06:55 INFO mapred.Merger: Down to the last merge-pass, with 2 segments left of total size: 2213882 bytes
16/08/17 21:06:55 INFO mapred.Merger: Merging 2 sorted segments
16/08/17 21:06:55 INFO mapred.Merger: Down to the last merge-pass, with 2 segments left of total size: 2154354 bytes
16/08/17 21:06:55 INFO mapred.Merger: Merging 2 sorted segments
16/08/17 21:06:55 INFO mapred.Merger: Down to the last merge-pass, with 2 segments left of total size: 2211421 bytes
16/08/17 21:06:55 INFO mapred.Merger: Merging 2 sorted segments
16/08/17 21:06:55 INFO mapred.Merger: Down to the last merge-pass, with 2 segments left of total size: 2142781 bytes
16/08/17 21:06:55 INFO mapred.Merger: Merging 2 sorted segments
16/08/17 21:06:55 INFO mapred.Merger: Down to the last merge-pass, with 2 segments left of total size: 2222143 bytes
16/08/17 21:06:55 INFO mapred.Merger: Merging 2 sorted segments
16/08/17 21:06:55 INFO mapred.Merger: Down to the last merge-pass, with 2 segments left of total size: 2136894 bytes
16/08/17 21:06:55 INFO mapred.Merger: Merging 2 sorted segments
16/08/17 21:06:55 INFO mapred.Merger: Down to the last merge-pass, with 2 segments left of total size: 2204224 bytes
16/08/17 21:06:55 INFO mapred.Merger: Merging 2 sorted segments
16/08/17 21:06:55 INFO mapred.Merger: Down to the last merge-pass, with 2 segments left of total size: 2147682 bytes
16/08/17 21:06:56 INFO mapred.Merger: Merging 2 sorted segments
16/08/17 21:06:56 INFO mapred.Merger: Down to the last merge-pass, with 2 segments left of total size: 2216697 bytes
16/08/17 21:06:56 INFO mapred.Merger: Merging 2 sorted segments
16/08/17 21:06:56 INFO mapred.Merger: Down to the last merge-pass, with 2 segments left of total size: 2126238 bytes
16/08/17 21:06:56 INFO mapred.Merger: Merging 2 sorted segments
16/08/17 21:06:56 INFO mapred.Merger: Down to the last merge-pass, with 2 segments left of total size: 2201018 bytes
16/08/17 21:06:56 INFO mapred.Merger: Merging 2 sorted segments
16/08/17 21:06:56 INFO mapred.Merger: Down to the last merge-pass, with 2 segments left of total size: 2149011 bytes
16/08/17 21:06:56 INFO mapred.Merger: Merging 2 sorted segments
16/08/17 21:06:56 INFO mapred.Merger: Down to the last merge-pass, with 2 segments left of total size: 2210500 bytes
16/08/17 21:06:56 INFO mapred.Merger: Merging 2 sorted segments
16/08/17 21:06:56 INFO mapred.Merger: Down to the last merge-pass, with 2 segments left of total size: 2144822 bytes
16/08/17 21:06:56 INFO mapred.Merger: Merging 2 sorted segments
16/08/17 21:06:56 INFO mapred.Merger: Down to the last merge-pass, with 2 segments left of total size: 2204757 bytes
16/08/17 21:06:56 INFO mapred.Task: Task:attempt_local583517286_0004_m_000000_0 is done. And is in the process of committing
16/08/17 21:06:56 INFO mapred.LocalJobRunner: map > sort
16/08/17 21:06:56 INFO mapred.Task: Task 'attempt_local583517286_0004_m_000000_0' done.
16/08/17 21:06:56 INFO mapred.LocalJobRunner: Finishing task: attempt_local583517286_0004_m_000000_0
16/08/17 21:06:56 INFO mapred.LocalJobRunner: map task executor complete.
16/08/17 21:06:56 INFO mapred.LocalJobRunner: Waiting for reduce tasks
16/08/17 21:06:56 INFO mapred.LocalJobRunner: Starting task: attempt_local583517286_0004_r_000000_0
16/08/17 21:06:56 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:06:56 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@642035fc
16/08/17 21:06:56 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=372139616, maxSingleShuffleLimit=93034904, mergeThreshold=245612160, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:06:56 INFO reduce.EventFetcher: attempt_local583517286_0004_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:06:56 INFO reduce.LocalFetcher: localfetcher#49 about to shuffle output of map attempt_local583517286_0004_m_000000_0 decomp: 2151696 len: 2151700 to MEMORY
16/08/17 21:06:56 INFO reduce.InMemoryMapOutput: Read 2151696 bytes from map-output for attempt_local583517286_0004_m_000000_0
16/08/17 21:06:56 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2151696, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->2151696
16/08/17 21:06:56 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:06:56 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:56 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:06:56 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:56 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 2151667 bytes
16/08/17 21:06:56 INFO reduce.MergeManagerImpl: Merged 1 segments, 2151696 bytes to disk to satisfy reduce memory limit
16/08/17 21:06:56 INFO reduce.MergeManagerImpl: Merging 1 files, 2151700 bytes from disk
16/08/17 21:06:56 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:06:56 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:56 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 2151667 bytes
16/08/17 21:06:56 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:56 INFO mapred.Task: Task:attempt_local583517286_0004_r_000000_0 is done. And is in the process of committing
16/08/17 21:06:56 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:56 INFO mapred.Task: Task attempt_local583517286_0004_r_000000_0 is allowed to commit now
16/08/17 21:06:56 INFO output.FileOutputCommitter: Saved output of task 'attempt_local583517286_0004_r_000000_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/rules/_temporary/0/task_local583517286_0004_r_000000
16/08/17 21:06:56 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:06:56 INFO mapred.Task: Task 'attempt_local583517286_0004_r_000000_0' done.
16/08/17 21:06:56 INFO mapred.LocalJobRunner: Finishing task: attempt_local583517286_0004_r_000000_0
16/08/17 21:06:56 INFO mapred.LocalJobRunner: Starting task: attempt_local583517286_0004_r_000001_0
16/08/17 21:06:56 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:06:56 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@363b52cd
16/08/17 21:06:56 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=372139616, maxSingleShuffleLimit=93034904, mergeThreshold=245612160, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:06:56 INFO reduce.EventFetcher: attempt_local583517286_0004_r_000001_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:06:56 INFO reduce.LocalFetcher: localfetcher#50 about to shuffle output of map attempt_local583517286_0004_m_000000_0 decomp: 2213918 len: 2213922 to MEMORY
16/08/17 21:06:56 INFO reduce.InMemoryMapOutput: Read 2213918 bytes from map-output for attempt_local583517286_0004_m_000000_0
16/08/17 21:06:56 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2213918, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->2213918
16/08/17 21:06:56 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:06:56 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:56 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:06:56 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:56 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 2213895 bytes
16/08/17 21:06:56 INFO reduce.MergeManagerImpl: Merged 1 segments, 2213918 bytes to disk to satisfy reduce memory limit
16/08/17 21:06:56 INFO reduce.MergeManagerImpl: Merging 1 files, 2213922 bytes from disk
16/08/17 21:06:56 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:06:56 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:56 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 2213895 bytes
16/08/17 21:06:56 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:56 INFO mapred.Task: Task:attempt_local583517286_0004_r_000001_0 is done. And is in the process of committing
16/08/17 21:06:56 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:56 INFO mapred.Task: Task attempt_local583517286_0004_r_000001_0 is allowed to commit now
16/08/17 21:06:56 INFO output.FileOutputCommitter: Saved output of task 'attempt_local583517286_0004_r_000001_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/rules/_temporary/0/task_local583517286_0004_r_000001
16/08/17 21:06:56 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:06:56 INFO mapred.Task: Task 'attempt_local583517286_0004_r_000001_0' done.
16/08/17 21:06:56 INFO mapred.LocalJobRunner: Finishing task: attempt_local583517286_0004_r_000001_0
16/08/17 21:06:56 INFO mapred.LocalJobRunner: Starting task: attempt_local583517286_0004_r_000002_0
16/08/17 21:06:56 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:06:56 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@2d3fbc29
16/08/17 21:06:56 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=372139616, maxSingleShuffleLimit=93034904, mergeThreshold=245612160, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:06:56 INFO reduce.EventFetcher: attempt_local583517286_0004_r_000002_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:06:56 INFO reduce.LocalFetcher: localfetcher#51 about to shuffle output of map attempt_local583517286_0004_m_000000_0 decomp: 2154392 len: 2154396 to MEMORY
16/08/17 21:06:56 INFO reduce.InMemoryMapOutput: Read 2154392 bytes from map-output for attempt_local583517286_0004_m_000000_0
16/08/17 21:06:56 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2154392, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->2154392
16/08/17 21:06:56 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:06:56 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:56 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:06:56 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:56 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 2154367 bytes
16/08/17 21:06:56 INFO reduce.MergeManagerImpl: Merged 1 segments, 2154392 bytes to disk to satisfy reduce memory limit
16/08/17 21:06:56 INFO reduce.MergeManagerImpl: Merging 1 files, 2154396 bytes from disk
16/08/17 21:06:56 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:06:56 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:56 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 2154367 bytes
16/08/17 21:06:56 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:56 INFO mapred.Task: Task:attempt_local583517286_0004_r_000002_0 is done. And is in the process of committing
16/08/17 21:06:56 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:56 INFO mapred.Task: Task attempt_local583517286_0004_r_000002_0 is allowed to commit now
16/08/17 21:06:56 INFO output.FileOutputCommitter: Saved output of task 'attempt_local583517286_0004_r_000002_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/rules/_temporary/0/task_local583517286_0004_r_000002
16/08/17 21:06:56 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:06:56 INFO mapred.Task: Task 'attempt_local583517286_0004_r_000002_0' done.
16/08/17 21:06:56 INFO mapred.LocalJobRunner: Finishing task: attempt_local583517286_0004_r_000002_0
16/08/17 21:06:56 INFO mapred.LocalJobRunner: Starting task: attempt_local583517286_0004_r_000003_0
16/08/17 21:06:56 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:06:56 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@4e8e33f9
16/08/17 21:06:56 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=372139616, maxSingleShuffleLimit=93034904, mergeThreshold=245612160, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:06:56 INFO reduce.EventFetcher: attempt_local583517286_0004_r_000003_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:06:56 INFO reduce.LocalFetcher: localfetcher#52 about to shuffle output of map attempt_local583517286_0004_m_000000_0 decomp: 2211451 len: 2211455 to MEMORY
16/08/17 21:06:56 INFO reduce.InMemoryMapOutput: Read 2211451 bytes from map-output for attempt_local583517286_0004_m_000000_0
16/08/17 21:06:56 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2211451, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->2211451
16/08/17 21:06:56 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:06:56 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:56 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:06:56 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:56 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 2211434 bytes
16/08/17 21:06:56 INFO reduce.MergeManagerImpl: Merged 1 segments, 2211451 bytes to disk to satisfy reduce memory limit
16/08/17 21:06:56 INFO reduce.MergeManagerImpl: Merging 1 files, 2211455 bytes from disk
16/08/17 21:06:56 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:06:56 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:56 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 2211434 bytes
16/08/17 21:06:56 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:57 INFO mapred.Task: Task:attempt_local583517286_0004_r_000003_0 is done. And is in the process of committing
16/08/17 21:06:57 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:57 INFO mapred.Task: Task attempt_local583517286_0004_r_000003_0 is allowed to commit now
16/08/17 21:06:57 INFO output.FileOutputCommitter: Saved output of task 'attempt_local583517286_0004_r_000003_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/rules/_temporary/0/task_local583517286_0004_r_000003
16/08/17 21:06:57 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:06:57 INFO mapred.Task: Task 'attempt_local583517286_0004_r_000003_0' done.
16/08/17 21:06:57 INFO mapred.LocalJobRunner: Finishing task: attempt_local583517286_0004_r_000003_0
16/08/17 21:06:57 INFO mapred.LocalJobRunner: Starting task: attempt_local583517286_0004_r_000004_0
16/08/17 21:06:57 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:06:57 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@5876f5b8
16/08/17 21:06:57 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=372139616, maxSingleShuffleLimit=93034904, mergeThreshold=245612160, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:06:57 INFO reduce.EventFetcher: attempt_local583517286_0004_r_000004_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:06:57 INFO reduce.LocalFetcher: localfetcher#53 about to shuffle output of map attempt_local583517286_0004_m_000000_0 decomp: 2142851 len: 2142855 to MEMORY
16/08/17 21:06:57 INFO reduce.InMemoryMapOutput: Read 2142851 bytes from map-output for attempt_local583517286_0004_m_000000_0
16/08/17 21:06:57 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2142851, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->2142851
16/08/17 21:06:57 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:06:57 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:57 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:06:57 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:57 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 2142800 bytes
16/08/17 21:06:57 INFO reduce.MergeManagerImpl: Merged 1 segments, 2142851 bytes to disk to satisfy reduce memory limit
16/08/17 21:06:57 INFO reduce.MergeManagerImpl: Merging 1 files, 2142855 bytes from disk
16/08/17 21:06:57 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:06:57 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:57 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 2142800 bytes
16/08/17 21:06:57 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:57 INFO mapred.Task: Task:attempt_local583517286_0004_r_000004_0 is done. And is in the process of committing
16/08/17 21:06:57 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:57 INFO mapred.Task: Task attempt_local583517286_0004_r_000004_0 is allowed to commit now
16/08/17 21:06:57 INFO output.FileOutputCommitter: Saved output of task 'attempt_local583517286_0004_r_000004_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/rules/_temporary/0/task_local583517286_0004_r_000004
16/08/17 21:06:57 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:06:57 INFO mapred.Task: Task 'attempt_local583517286_0004_r_000004_0' done.
16/08/17 21:06:57 INFO mapred.LocalJobRunner: Finishing task: attempt_local583517286_0004_r_000004_0
16/08/17 21:06:57 INFO mapred.LocalJobRunner: Starting task: attempt_local583517286_0004_r_000005_0
16/08/17 21:06:57 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:06:57 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@4088f927
16/08/17 21:06:57 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=372139616, maxSingleShuffleLimit=93034904, mergeThreshold=245612160, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:06:57 INFO reduce.EventFetcher: attempt_local583517286_0004_r_000005_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:06:57 INFO reduce.LocalFetcher: localfetcher#54 about to shuffle output of map attempt_local583517286_0004_m_000000_0 decomp: 2222183 len: 2222187 to MEMORY
16/08/17 21:06:57 INFO reduce.InMemoryMapOutput: Read 2222183 bytes from map-output for attempt_local583517286_0004_m_000000_0
16/08/17 21:06:57 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2222183, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->2222183
16/08/17 21:06:57 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:06:57 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:57 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:06:57 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:57 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 2222166 bytes
16/08/17 21:06:57 INFO reduce.MergeManagerImpl: Merged 1 segments, 2222183 bytes to disk to satisfy reduce memory limit
16/08/17 21:06:57 INFO reduce.MergeManagerImpl: Merging 1 files, 2222187 bytes from disk
16/08/17 21:06:57 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:06:57 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:57 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 2222166 bytes
16/08/17 21:06:57 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:57 INFO mapred.Task: Task:attempt_local583517286_0004_r_000005_0 is done. And is in the process of committing
16/08/17 21:06:57 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:57 INFO mapred.Task: Task attempt_local583517286_0004_r_000005_0 is allowed to commit now
16/08/17 21:06:57 INFO output.FileOutputCommitter: Saved output of task 'attempt_local583517286_0004_r_000005_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/rules/_temporary/0/task_local583517286_0004_r_000005
16/08/17 21:06:57 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:06:57 INFO mapred.Task: Task 'attempt_local583517286_0004_r_000005_0' done.
16/08/17 21:06:57 INFO mapred.LocalJobRunner: Finishing task: attempt_local583517286_0004_r_000005_0
16/08/17 21:06:57 INFO mapred.LocalJobRunner: Starting task: attempt_local583517286_0004_r_000006_0
16/08/17 21:06:57 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:06:57 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@eb0ab6c
16/08/17 21:06:57 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=372139616, maxSingleShuffleLimit=93034904, mergeThreshold=245612160, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:06:57 INFO reduce.EventFetcher: attempt_local583517286_0004_r_000006_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:06:57 INFO reduce.LocalFetcher: localfetcher#55 about to shuffle output of map attempt_local583517286_0004_m_000000_0 decomp: 2136918 len: 2136922 to MEMORY
16/08/17 21:06:57 INFO reduce.InMemoryMapOutput: Read 2136918 bytes from map-output for attempt_local583517286_0004_m_000000_0
16/08/17 21:06:57 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2136918, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->2136918
16/08/17 21:06:57 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:06:57 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:57 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:06:57 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:57 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 2136901 bytes
16/08/17 21:06:57 INFO reduce.MergeManagerImpl: Merged 1 segments, 2136918 bytes to disk to satisfy reduce memory limit
16/08/17 21:06:57 INFO reduce.MergeManagerImpl: Merging 1 files, 2136922 bytes from disk
16/08/17 21:06:57 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:06:57 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:57 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 2136901 bytes
16/08/17 21:06:57 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:57 INFO mapred.Task: Task:attempt_local583517286_0004_r_000006_0 is done. And is in the process of committing
16/08/17 21:06:57 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:57 INFO mapred.Task: Task attempt_local583517286_0004_r_000006_0 is allowed to commit now
16/08/17 21:06:57 INFO output.FileOutputCommitter: Saved output of task 'attempt_local583517286_0004_r_000006_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/rules/_temporary/0/task_local583517286_0004_r_000006
16/08/17 21:06:57 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:06:57 INFO mapred.Task: Task 'attempt_local583517286_0004_r_000006_0' done.
16/08/17 21:06:57 INFO mapred.LocalJobRunner: Finishing task: attempt_local583517286_0004_r_000006_0
16/08/17 21:06:57 INFO mapred.LocalJobRunner: Starting task: attempt_local583517286_0004_r_000007_0
16/08/17 21:06:57 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:06:57 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@6a62046
16/08/17 21:06:57 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=372139616, maxSingleShuffleLimit=93034904, mergeThreshold=245612160, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:06:57 INFO reduce.EventFetcher: attempt_local583517286_0004_r_000007_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:06:57 INFO reduce.LocalFetcher: localfetcher#56 about to shuffle output of map attempt_local583517286_0004_m_000000_0 decomp: 2204272 len: 2204276 to MEMORY
16/08/17 21:06:57 INFO reduce.InMemoryMapOutput: Read 2204272 bytes from map-output for attempt_local583517286_0004_m_000000_0
16/08/17 21:06:57 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2204272, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->2204272
16/08/17 21:06:57 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:06:57 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:57 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:06:57 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:57 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 2204231 bytes
16/08/17 21:06:57 INFO reduce.MergeManagerImpl: Merged 1 segments, 2204272 bytes to disk to satisfy reduce memory limit
16/08/17 21:06:57 INFO reduce.MergeManagerImpl: Merging 1 files, 2204276 bytes from disk
16/08/17 21:06:57 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:06:57 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:57 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 2204231 bytes
16/08/17 21:06:57 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:57 INFO mapred.Task: Task:attempt_local583517286_0004_r_000007_0 is done. And is in the process of committing
16/08/17 21:06:57 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:57 INFO mapred.Task: Task attempt_local583517286_0004_r_000007_0 is allowed to commit now
16/08/17 21:06:57 INFO output.FileOutputCommitter: Saved output of task 'attempt_local583517286_0004_r_000007_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/rules/_temporary/0/task_local583517286_0004_r_000007
16/08/17 21:06:57 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:06:57 INFO mapred.Task: Task 'attempt_local583517286_0004_r_000007_0' done.
16/08/17 21:06:57 INFO mapred.LocalJobRunner: Finishing task: attempt_local583517286_0004_r_000007_0
16/08/17 21:06:57 INFO mapred.LocalJobRunner: Starting task: attempt_local583517286_0004_r_000008_0
16/08/17 21:06:57 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:06:57 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@3392cddb
16/08/17 21:06:57 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=372506624, maxSingleShuffleLimit=93126656, mergeThreshold=245854384, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:06:57 INFO reduce.EventFetcher: attempt_local583517286_0004_r_000008_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:06:57 INFO reduce.LocalFetcher: localfetcher#57 about to shuffle output of map attempt_local583517286_0004_m_000000_0 decomp: 2147750 len: 2147754 to MEMORY
16/08/17 21:06:57 INFO reduce.InMemoryMapOutput: Read 2147750 bytes from map-output for attempt_local583517286_0004_m_000000_0
16/08/17 21:06:57 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2147750, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->2147750
16/08/17 21:06:57 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:06:57 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:57 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:06:57 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:57 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 2147713 bytes
16/08/17 21:06:57 INFO reduce.MergeManagerImpl: Merged 1 segments, 2147750 bytes to disk to satisfy reduce memory limit
16/08/17 21:06:57 INFO reduce.MergeManagerImpl: Merging 1 files, 2147754 bytes from disk
16/08/17 21:06:57 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:06:57 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:57 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 2147713 bytes
16/08/17 21:06:57 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:57 INFO mapred.Task: Task:attempt_local583517286_0004_r_000008_0 is done. And is in the process of committing
16/08/17 21:06:57 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:57 INFO mapred.Task: Task attempt_local583517286_0004_r_000008_0 is allowed to commit now
16/08/17 21:06:57 INFO output.FileOutputCommitter: Saved output of task 'attempt_local583517286_0004_r_000008_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/rules/_temporary/0/task_local583517286_0004_r_000008
16/08/17 21:06:57 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:06:57 INFO mapred.Task: Task 'attempt_local583517286_0004_r_000008_0' done.
16/08/17 21:06:57 INFO mapred.LocalJobRunner: Finishing task: attempt_local583517286_0004_r_000008_0
16/08/17 21:06:57 INFO mapred.LocalJobRunner: Starting task: attempt_local583517286_0004_r_000009_0
16/08/17 21:06:57 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:06:57 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@5a540517
16/08/17 21:06:57 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=372506624, maxSingleShuffleLimit=93126656, mergeThreshold=245854384, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:06:57 INFO reduce.EventFetcher: attempt_local583517286_0004_r_000009_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:06:57 INFO reduce.LocalFetcher: localfetcher#58 about to shuffle output of map attempt_local583517286_0004_m_000000_0 decomp: 2216749 len: 2216753 to MEMORY
16/08/17 21:06:57 INFO reduce.InMemoryMapOutput: Read 2216749 bytes from map-output for attempt_local583517286_0004_m_000000_0
16/08/17 21:06:57 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2216749, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->2216749
16/08/17 21:06:57 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:06:57 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:57 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:06:57 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:57 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 2216710 bytes
16/08/17 21:06:57 INFO reduce.MergeManagerImpl: Merged 1 segments, 2216749 bytes to disk to satisfy reduce memory limit
16/08/17 21:06:57 INFO reduce.MergeManagerImpl: Merging 1 files, 2216753 bytes from disk
16/08/17 21:06:57 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:06:57 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:57 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 2216710 bytes
16/08/17 21:06:57 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:58 INFO mapred.Task: Task:attempt_local583517286_0004_r_000009_0 is done. And is in the process of committing
16/08/17 21:06:58 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:58 INFO mapred.Task: Task attempt_local583517286_0004_r_000009_0 is allowed to commit now
16/08/17 21:06:58 INFO output.FileOutputCommitter: Saved output of task 'attempt_local583517286_0004_r_000009_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/rules/_temporary/0/task_local583517286_0004_r_000009
16/08/17 21:06:58 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:06:58 INFO mapred.Task: Task 'attempt_local583517286_0004_r_000009_0' done.
16/08/17 21:06:58 INFO mapred.LocalJobRunner: Finishing task: attempt_local583517286_0004_r_000009_0
16/08/17 21:06:58 INFO mapred.LocalJobRunner: Starting task: attempt_local583517286_0004_r_000010_0
16/08/17 21:06:58 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:06:58 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@34d9740e
16/08/17 21:06:58 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=372506624, maxSingleShuffleLimit=93126656, mergeThreshold=245854384, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:06:58 INFO reduce.EventFetcher: attempt_local583517286_0004_r_000010_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:06:58 INFO reduce.LocalFetcher: localfetcher#59 about to shuffle output of map attempt_local583517286_0004_m_000000_0 decomp: 2126274 len: 2126278 to MEMORY
16/08/17 21:06:58 INFO reduce.InMemoryMapOutput: Read 2126274 bytes from map-output for attempt_local583517286_0004_m_000000_0
16/08/17 21:06:58 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2126274, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->2126274
16/08/17 21:06:58 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:06:58 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:58 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:06:58 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:58 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 2126251 bytes
16/08/17 21:06:58 INFO reduce.MergeManagerImpl: Merged 1 segments, 2126274 bytes to disk to satisfy reduce memory limit
16/08/17 21:06:58 INFO reduce.MergeManagerImpl: Merging 1 files, 2126278 bytes from disk
16/08/17 21:06:58 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:06:58 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:58 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 2126251 bytes
16/08/17 21:06:58 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:58 INFO mapred.Task: Task:attempt_local583517286_0004_r_000010_0 is done. And is in the process of committing
16/08/17 21:06:58 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:58 INFO mapred.Task: Task attempt_local583517286_0004_r_000010_0 is allowed to commit now
16/08/17 21:06:58 INFO output.FileOutputCommitter: Saved output of task 'attempt_local583517286_0004_r_000010_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/rules/_temporary/0/task_local583517286_0004_r_000010
16/08/17 21:06:58 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:06:58 INFO mapred.Task: Task 'attempt_local583517286_0004_r_000010_0' done.
16/08/17 21:06:58 INFO mapred.LocalJobRunner: Finishing task: attempt_local583517286_0004_r_000010_0
16/08/17 21:06:58 INFO mapred.LocalJobRunner: Starting task: attempt_local583517286_0004_r_000011_0
16/08/17 21:06:58 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:06:58 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@c47bfa0
16/08/17 21:06:58 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=372506624, maxSingleShuffleLimit=93126656, mergeThreshold=245854384, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:06:58 INFO reduce.EventFetcher: attempt_local583517286_0004_r_000011_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:06:58 INFO reduce.LocalFetcher: localfetcher#60 about to shuffle output of map attempt_local583517286_0004_m_000000_0 decomp: 2201066 len: 2201070 to MEMORY
16/08/17 21:06:58 INFO reduce.InMemoryMapOutput: Read 2201066 bytes from map-output for attempt_local583517286_0004_m_000000_0
16/08/17 21:06:58 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2201066, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->2201066
16/08/17 21:06:58 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:06:58 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:58 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:06:58 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:58 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 2201025 bytes
16/08/17 21:06:58 INFO reduce.MergeManagerImpl: Merged 1 segments, 2201066 bytes to disk to satisfy reduce memory limit
16/08/17 21:06:58 INFO reduce.MergeManagerImpl: Merging 1 files, 2201070 bytes from disk
16/08/17 21:06:58 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:06:58 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:58 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 2201025 bytes
16/08/17 21:06:58 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:58 INFO mapred.Task: Task:attempt_local583517286_0004_r_000011_0 is done. And is in the process of committing
16/08/17 21:06:58 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:58 INFO mapred.Task: Task attempt_local583517286_0004_r_000011_0 is allowed to commit now
16/08/17 21:06:58 INFO output.FileOutputCommitter: Saved output of task 'attempt_local583517286_0004_r_000011_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/rules/_temporary/0/task_local583517286_0004_r_000011
16/08/17 21:06:58 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:06:58 INFO mapred.Task: Task 'attempt_local583517286_0004_r_000011_0' done.
16/08/17 21:06:58 INFO mapred.LocalJobRunner: Finishing task: attempt_local583517286_0004_r_000011_0
16/08/17 21:06:58 INFO mapred.LocalJobRunner: Starting task: attempt_local583517286_0004_r_000012_0
16/08/17 21:06:58 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:06:58 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@7f9fed7
16/08/17 21:06:58 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=371772608, maxSingleShuffleLimit=92943152, mergeThreshold=245369936, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:06:58 INFO reduce.EventFetcher: attempt_local583517286_0004_r_000012_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:06:58 INFO reduce.LocalFetcher: localfetcher#61 about to shuffle output of map attempt_local583517286_0004_m_000000_0 decomp: 2149133 len: 2149137 to MEMORY
16/08/17 21:06:58 INFO reduce.InMemoryMapOutput: Read 2149133 bytes from map-output for attempt_local583517286_0004_m_000000_0
16/08/17 21:06:58 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2149133, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->2149133
16/08/17 21:06:58 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:06:58 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:58 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:06:58 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:58 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 2149090 bytes
16/08/17 21:06:58 INFO reduce.MergeManagerImpl: Merged 1 segments, 2149133 bytes to disk to satisfy reduce memory limit
16/08/17 21:06:58 INFO reduce.MergeManagerImpl: Merging 1 files, 2149137 bytes from disk
16/08/17 21:06:58 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:06:58 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:58 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 2149090 bytes
16/08/17 21:06:58 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:58 INFO mapred.Task: Task:attempt_local583517286_0004_r_000012_0 is done. And is in the process of committing
16/08/17 21:06:58 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:58 INFO mapred.Task: Task attempt_local583517286_0004_r_000012_0 is allowed to commit now
16/08/17 21:06:58 INFO output.FileOutputCommitter: Saved output of task 'attempt_local583517286_0004_r_000012_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/rules/_temporary/0/task_local583517286_0004_r_000012
16/08/17 21:06:58 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:06:58 INFO mapred.Task: Task 'attempt_local583517286_0004_r_000012_0' done.
16/08/17 21:06:58 INFO mapred.LocalJobRunner: Finishing task: attempt_local583517286_0004_r_000012_0
16/08/17 21:06:58 INFO mapred.LocalJobRunner: Starting task: attempt_local583517286_0004_r_000013_0
16/08/17 21:06:58 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:06:58 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@3f3c59b2
16/08/17 21:06:58 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=371772608, maxSingleShuffleLimit=92943152, mergeThreshold=245369936, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:06:58 INFO reduce.EventFetcher: attempt_local583517286_0004_r_000013_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:06:58 INFO reduce.LocalFetcher: localfetcher#62 about to shuffle output of map attempt_local583517286_0004_m_000000_0 decomp: 2210562 len: 2210566 to MEMORY
16/08/17 21:06:58 INFO reduce.InMemoryMapOutput: Read 2210562 bytes from map-output for attempt_local583517286_0004_m_000000_0
16/08/17 21:06:58 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2210562, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->2210562
16/08/17 21:06:58 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:06:58 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:58 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:06:58 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:58 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 2210513 bytes
16/08/17 21:06:58 INFO reduce.MergeManagerImpl: Merged 1 segments, 2210562 bytes to disk to satisfy reduce memory limit
16/08/17 21:06:58 INFO reduce.MergeManagerImpl: Merging 1 files, 2210566 bytes from disk
16/08/17 21:06:58 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:06:58 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:58 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 2210513 bytes
16/08/17 21:06:58 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:58 INFO mapred.Task: Task:attempt_local583517286_0004_r_000013_0 is done. And is in the process of committing
16/08/17 21:06:58 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:58 INFO mapred.Task: Task attempt_local583517286_0004_r_000013_0 is allowed to commit now
16/08/17 21:06:58 INFO output.FileOutputCommitter: Saved output of task 'attempt_local583517286_0004_r_000013_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/rules/_temporary/0/task_local583517286_0004_r_000013
16/08/17 21:06:58 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:06:58 INFO mapred.Task: Task 'attempt_local583517286_0004_r_000013_0' done.
16/08/17 21:06:58 INFO mapred.LocalJobRunner: Finishing task: attempt_local583517286_0004_r_000013_0
16/08/17 21:06:58 INFO mapred.LocalJobRunner: Starting task: attempt_local583517286_0004_r_000014_0
16/08/17 21:06:58 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:06:58 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@23dde299
16/08/17 21:06:58 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=371772608, maxSingleShuffleLimit=92943152, mergeThreshold=245369936, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:06:58 INFO reduce.EventFetcher: attempt_local583517286_0004_r_000014_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:06:58 INFO reduce.LocalFetcher: localfetcher#63 about to shuffle output of map attempt_local583517286_0004_m_000000_0 decomp: 2144876 len: 2144880 to MEMORY
16/08/17 21:06:58 INFO reduce.InMemoryMapOutput: Read 2144876 bytes from map-output for attempt_local583517286_0004_m_000000_0
16/08/17 21:06:58 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2144876, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->2144876
16/08/17 21:06:58 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:06:58 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:58 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:06:58 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:58 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 2144845 bytes
16/08/17 21:06:58 INFO reduce.MergeManagerImpl: Merged 1 segments, 2144876 bytes to disk to satisfy reduce memory limit
16/08/17 21:06:58 INFO reduce.MergeManagerImpl: Merging 1 files, 2144880 bytes from disk
16/08/17 21:06:58 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:06:58 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:58 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 2144845 bytes
16/08/17 21:06:58 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:58 INFO mapred.Task: Task:attempt_local583517286_0004_r_000014_0 is done. And is in the process of committing
16/08/17 21:06:58 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:58 INFO mapred.Task: Task attempt_local583517286_0004_r_000014_0 is allowed to commit now
16/08/17 21:06:58 INFO output.FileOutputCommitter: Saved output of task 'attempt_local583517286_0004_r_000014_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/rules/_temporary/0/task_local583517286_0004_r_000014
16/08/17 21:06:58 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:06:58 INFO mapred.Task: Task 'attempt_local583517286_0004_r_000014_0' done.
16/08/17 21:06:58 INFO mapred.LocalJobRunner: Finishing task: attempt_local583517286_0004_r_000014_0
16/08/17 21:06:58 INFO mapred.LocalJobRunner: Starting task: attempt_local583517286_0004_r_000015_0
16/08/17 21:06:58 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:06:58 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@1877620f
16/08/17 21:06:58 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=371772608, maxSingleShuffleLimit=92943152, mergeThreshold=245369936, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:06:58 INFO reduce.EventFetcher: attempt_local583517286_0004_r_000015_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:06:58 INFO reduce.LocalFetcher: localfetcher#64 about to shuffle output of map attempt_local583517286_0004_m_000000_0 decomp: 2204805 len: 2204809 to MEMORY
16/08/17 21:06:58 INFO reduce.InMemoryMapOutput: Read 2204805 bytes from map-output for attempt_local583517286_0004_m_000000_0
16/08/17 21:06:58 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2204805, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->2204805
16/08/17 21:06:58 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:06:58 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:58 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:06:58 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:58 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 2204776 bytes
16/08/17 21:06:58 INFO reduce.MergeManagerImpl: Merged 1 segments, 2204805 bytes to disk to satisfy reduce memory limit
16/08/17 21:06:58 INFO reduce.MergeManagerImpl: Merging 1 files, 2204809 bytes from disk
16/08/17 21:06:58 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:06:58 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:06:58 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 2204776 bytes
16/08/17 21:06:58 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:59 INFO mapred.Task: Task:attempt_local583517286_0004_r_000015_0 is done. And is in the process of committing
16/08/17 21:06:59 INFO mapred.LocalJobRunner: 1 / 1 copied.
16/08/17 21:06:59 INFO mapred.Task: Task attempt_local583517286_0004_r_000015_0 is allowed to commit now
16/08/17 21:06:59 INFO output.FileOutputCommitter: Saved output of task 'attempt_local583517286_0004_r_000015_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/rules/_temporary/0/task_local583517286_0004_r_000015
16/08/17 21:06:59 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:06:59 INFO mapred.Task: Task 'attempt_local583517286_0004_r_000015_0' done.
16/08/17 21:06:59 INFO mapred.LocalJobRunner: Finishing task: attempt_local583517286_0004_r_000015_0
16/08/17 21:06:59 INFO mapred.LocalJobRunner: reduce task executor complete.
[SCHED] class edu.jhu.thrax.hadoop.jobs.ExtractionJob in state SUCCESS
[SCHED] class edu.jhu.thrax.hadoop.features.mapred.SourcePhraseGivenTargetFeature in state READY
[SCHED] class edu.jhu.thrax.hadoop.features.annotation.AnnotationFeatureJob in state READY
[SCHED] class edu.jhu.thrax.hadoop.features.mapred.TargetPhraseGivenSourceFeature in state READY
[SCHED] class edu.jhu.thrax.hadoop.features.mapred.SourcePhraseGivenTargetFeature in state RUNNING
[SCHED] class edu.jhu.thrax.hadoop.features.annotation.AnnotationFeatureJob in state RUNNING
[SCHED] class edu.jhu.thrax.hadoop.features.mapred.TargetPhraseGivenSourceFeature in state RUNNING
16/08/17 21:07:03 INFO jvm.JvmMetrics: Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
16/08/17 21:07:03 INFO jvm.JvmMetrics: Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
16/08/17 21:07:03 INFO jvm.JvmMetrics: Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
16/08/17 21:07:04 INFO input.FileInputFormat: Total input paths to process : 16
16/08/17 21:07:04 INFO input.FileInputFormat: Total input paths to process : 16
16/08/17 21:07:04 INFO mapreduce.JobSubmitter: number of splits:16
16/08/17 21:07:04 INFO mapreduce.JobSubmitter: number of splits:16
16/08/17 21:07:04 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local1944506404_0006
16/08/17 21:07:04 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local518136789_0007
16/08/17 21:07:04 INFO input.FileInputFormat: Total input paths to process : 16
16/08/17 21:07:04 INFO mapreduce.JobSubmitter: number of splits:16
16/08/17 21:07:04 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local1999544224_0005
16/08/17 21:07:04 INFO mapreduce.Job: The url to track the job: http://localhost:8080/
16/08/17 21:07:04 INFO mapred.LocalJobRunner: OutputCommitter set in config null
16/08/17 21:07:04 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
16/08/17 21:07:04 INFO mapred.LocalJobRunner: Waiting for map tasks
16/08/17 21:07:04 INFO mapred.LocalJobRunner: Starting task: attempt_local1944506404_0006_m_000000_0
16/08/17 21:07:04 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:04 INFO mapred.MapTask: Processing split: file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/rules/part-r-00005:0+2463043
16/08/17 21:07:04 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
16/08/17 21:07:04 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
16/08/17 21:07:04 INFO mapred.MapTask: soft limit at 83886080
16/08/17 21:07:04 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
16/08/17 21:07:04 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
16/08/17 21:07:04 INFO mapreduce.Job: The url to track the job: http://localhost:8080/
16/08/17 21:07:04 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
16/08/17 21:07:04 INFO mapreduce.Job: The url to track the job: http://localhost:8080/
16/08/17 21:07:04 INFO mapred.LocalJobRunner: OutputCommitter set in config null
16/08/17 21:07:04 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
16/08/17 21:07:04 INFO mapred.LocalJobRunner: Waiting for map tasks
16/08/17 21:07:04 INFO mapred.LocalJobRunner: OutputCommitter set in config null
16/08/17 21:07:04 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
16/08/17 21:07:04 INFO mapred.LocalJobRunner: Starting task: attempt_local518136789_0007_m_000000_0
16/08/17 21:07:04 INFO mapred.LocalJobRunner: Waiting for map tasks
16/08/17 21:07:04 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:04 INFO mapred.MapTask: Processing split: file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/rules/part-r-00005:0+2463043
16/08/17 21:07:04 INFO mapred.LocalJobRunner: Starting task: attempt_local1999544224_0005_m_000000_0
16/08/17 21:07:04 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:04 INFO mapred.MapTask: Processing split: file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/rules/part-r-00005:0+2463043
16/08/17 21:07:04 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
16/08/17 21:07:04 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
16/08/17 21:07:04 INFO mapred.MapTask: soft limit at 83886080
16/08/17 21:07:04 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
16/08/17 21:07:04 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
16/08/17 21:07:04 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
16/08/17 21:07:04 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
16/08/17 21:07:04 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
16/08/17 21:07:04 INFO mapred.MapTask: soft limit at 83886080
16/08/17 21:07:04 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
16/08/17 21:07:04 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
16/08/17 21:07:04 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
16/08/17 21:07:04 INFO mapred.LocalJobRunner: 
16/08/17 21:07:04 INFO mapred.MapTask: Starting flush of map output
16/08/17 21:07:04 INFO mapred.MapTask: Spilling map output
16/08/17 21:07:04 INFO mapred.MapTask: bufstart = 0; bufend = 2075051; bufvoid = 104857600
16/08/17 21:07:04 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26032480(104129920); length = 181917/6553600
16/08/17 21:07:05 INFO mapred.MapTask: Finished spill 0
16/08/17 21:07:05 INFO mapred.Task: Task:attempt_local1944506404_0006_m_000000_0 is done. And is in the process of committing
16/08/17 21:07:05 INFO mapred.LocalJobRunner: map
16/08/17 21:07:05 INFO mapred.Task: Task 'attempt_local1944506404_0006_m_000000_0' done.
16/08/17 21:07:05 INFO mapred.LocalJobRunner: Finishing task: attempt_local1944506404_0006_m_000000_0
16/08/17 21:07:05 INFO mapred.LocalJobRunner: Starting task: attempt_local1944506404_0006_m_000001_0
16/08/17 21:07:05 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:05 INFO mapred.MapTask: Processing split: file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/rules/part-r-00009:0+2455868
16/08/17 21:07:05 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
16/08/17 21:07:05 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
16/08/17 21:07:05 INFO mapred.MapTask: soft limit at 83886080
16/08/17 21:07:05 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
16/08/17 21:07:05 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
16/08/17 21:07:05 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
16/08/17 21:07:05 INFO mapred.LocalJobRunner: 
16/08/17 21:07:05 INFO mapred.MapTask: Starting flush of map output
16/08/17 21:07:05 INFO mapred.MapTask: Spilling map output
16/08/17 21:07:05 INFO mapred.MapTask: bufstart = 0; bufend = 4774544; bufvoid = 104857600
16/08/17 21:07:05 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 25668640(102674560); length = 545757/6553600
16/08/17 21:07:05 INFO mapred.LocalJobRunner: 
16/08/17 21:07:05 INFO mapred.MapTask: Starting flush of map output
16/08/17 21:07:05 INFO mapred.MapTask: Spilling map output
16/08/17 21:07:05 INFO mapred.MapTask: bufstart = 0; bufend = 4684076; bufvoid = 104857600
16/08/17 21:07:05 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 25668640(102674560); length = 545757/6553600
16/08/17 21:07:05 INFO mapred.LocalJobRunner: 
16/08/17 21:07:05 INFO mapred.MapTask: Starting flush of map output
16/08/17 21:07:05 INFO mapred.MapTask: Spilling map output
16/08/17 21:07:05 INFO mapred.MapTask: bufstart = 0; bufend = 2069488; bufvoid = 104857600
16/08/17 21:07:05 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26033256(104133024); length = 181141/6553600
16/08/17 21:07:05 INFO mapred.MapTask: Finished spill 0
16/08/17 21:07:05 INFO mapred.Task: Task:attempt_local1944506404_0006_m_000001_0 is done. And is in the process of committing
16/08/17 21:07:05 INFO mapred.LocalJobRunner: map
16/08/17 21:07:05 INFO mapred.Task: Task 'attempt_local1944506404_0006_m_000001_0' done.
16/08/17 21:07:05 INFO mapred.LocalJobRunner: Finishing task: attempt_local1944506404_0006_m_000001_0
16/08/17 21:07:05 INFO mapred.LocalJobRunner: Starting task: attempt_local1944506404_0006_m_000002_0
16/08/17 21:07:05 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:05 INFO mapred.MapTask: Processing split: file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/rules/part-r-00001:0+2455395
16/08/17 21:07:06 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
16/08/17 21:07:06 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
16/08/17 21:07:06 INFO mapred.MapTask: soft limit at 83886080
16/08/17 21:07:06 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
16/08/17 21:07:06 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
16/08/17 21:07:06 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
16/08/17 21:07:06 INFO mapred.LocalJobRunner: 
16/08/17 21:07:06 INFO mapred.MapTask: Starting flush of map output
16/08/17 21:07:06 INFO mapred.MapTask: Spilling map output
16/08/17 21:07:06 INFO mapred.MapTask: bufstart = 0; bufend = 2069123; bufvoid = 104857600
16/08/17 21:07:06 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26033300(104133200); length = 181097/6553600
16/08/17 21:07:06 INFO mapred.MapTask: Finished spill 0
16/08/17 21:07:06 INFO mapred.Task: Task:attempt_local1944506404_0006_m_000002_0 is done. And is in the process of committing
16/08/17 21:07:06 INFO mapred.LocalJobRunner: map
16/08/17 21:07:06 INFO mapred.Task: Task 'attempt_local1944506404_0006_m_000002_0' done.
16/08/17 21:07:06 INFO mapred.LocalJobRunner: Finishing task: attempt_local1944506404_0006_m_000002_0
16/08/17 21:07:06 INFO mapred.LocalJobRunner: Starting task: attempt_local1944506404_0006_m_000003_0
16/08/17 21:07:06 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:06 INFO mapred.MapTask: Processing split: file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/rules/part-r-00003:0+2454310
16/08/17 21:07:06 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
16/08/17 21:07:06 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
16/08/17 21:07:06 INFO mapred.MapTask: soft limit at 83886080
16/08/17 21:07:06 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
16/08/17 21:07:06 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
16/08/17 21:07:06 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
16/08/17 21:07:06 INFO mapred.LocalJobRunner: 
16/08/17 21:07:06 INFO mapred.MapTask: Starting flush of map output
16/08/17 21:07:06 INFO mapred.MapTask: Spilling map output
16/08/17 21:07:06 INFO mapred.MapTask: bufstart = 0; bufend = 2068086; bufvoid = 104857600
16/08/17 21:07:06 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26033324(104133296); length = 181073/6553600
16/08/17 21:07:06 INFO mapred.MapTask: Finished spill 0
16/08/17 21:07:06 INFO mapred.Task: Task:attempt_local1999544224_0005_m_000000_0 is done. And is in the process of committing
16/08/17 21:07:06 INFO mapred.LocalJobRunner: map
16/08/17 21:07:06 INFO mapred.Task: Task 'attempt_local1999544224_0005_m_000000_0' done.
16/08/17 21:07:06 INFO mapred.LocalJobRunner: Finishing task: attempt_local1999544224_0005_m_000000_0
16/08/17 21:07:06 INFO mapred.LocalJobRunner: Starting task: attempt_local1999544224_0005_m_000001_0
16/08/17 21:07:06 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:06 INFO mapred.MapTask: Processing split: file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/rules/part-r-00009:0+2455868
16/08/17 21:07:06 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
16/08/17 21:07:06 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
16/08/17 21:07:06 INFO mapred.MapTask: soft limit at 83886080
16/08/17 21:07:06 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
16/08/17 21:07:06 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
16/08/17 21:07:06 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
16/08/17 21:07:07 INFO mapred.MapTask: Finished spill 0
16/08/17 21:07:07 INFO mapred.Task: Task:attempt_local1944506404_0006_m_000003_0 is done. And is in the process of committing
16/08/17 21:07:07 INFO mapred.LocalJobRunner: map
16/08/17 21:07:07 INFO mapred.Task: Task 'attempt_local1944506404_0006_m_000003_0' done.
16/08/17 21:07:07 INFO mapred.LocalJobRunner: Finishing task: attempt_local1944506404_0006_m_000003_0
16/08/17 21:07:07 INFO mapred.LocalJobRunner: Starting task: attempt_local1944506404_0006_m_000004_0
16/08/17 21:07:07 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:07 INFO mapred.MapTask: Processing split: file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/rules/part-r-00013:0+2451169
16/08/17 21:07:07 INFO mapred.MapTask: Finished spill 0
16/08/17 21:07:07 INFO mapred.Task: Task:attempt_local518136789_0007_m_000000_0 is done. And is in the process of committing
16/08/17 21:07:07 INFO mapred.LocalJobRunner: map
16/08/17 21:07:07 INFO mapred.Task: Task 'attempt_local518136789_0007_m_000000_0' done.
16/08/17 21:07:07 INFO mapred.LocalJobRunner: Finishing task: attempt_local518136789_0007_m_000000_0
16/08/17 21:07:07 INFO mapred.LocalJobRunner: Starting task: attempt_local518136789_0007_m_000001_0
16/08/17 21:07:07 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:07 INFO mapred.MapTask: Processing split: file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/rules/part-r-00009:0+2455868
16/08/17 21:07:07 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
16/08/17 21:07:07 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
16/08/17 21:07:07 INFO mapred.MapTask: soft limit at 83886080
16/08/17 21:07:07 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
16/08/17 21:07:07 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
16/08/17 21:07:07 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
16/08/17 21:07:07 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
16/08/17 21:07:07 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
16/08/17 21:07:07 INFO mapred.MapTask: soft limit at 83886080
16/08/17 21:07:07 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
16/08/17 21:07:07 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
16/08/17 21:07:07 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
16/08/17 21:07:07 INFO mapred.LocalJobRunner: 
16/08/17 21:07:07 INFO mapred.MapTask: Starting flush of map output
16/08/17 21:07:07 INFO mapred.MapTask: Spilling map output
16/08/17 21:07:07 INFO mapred.MapTask: bufstart = 0; bufend = 2065353; bufvoid = 104857600
16/08/17 21:07:07 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26033508(104134032); length = 180889/6553600
16/08/17 21:07:07 INFO mapred.LocalJobRunner: 
16/08/17 21:07:07 INFO mapred.MapTask: Starting flush of map output
16/08/17 21:07:07 INFO mapred.MapTask: Spilling map output
16/08/17 21:07:07 INFO mapred.MapTask: bufstart = 0; bufend = 4764392; bufvoid = 104857600
16/08/17 21:07:07 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 25670968(102683872); length = 543429/6553600
16/08/17 21:07:07 INFO mapred.MapTask: Finished spill 0
16/08/17 21:07:07 INFO mapred.Task: Task:attempt_local1944506404_0006_m_000004_0 is done. And is in the process of committing
16/08/17 21:07:07 INFO mapred.LocalJobRunner: map
16/08/17 21:07:07 INFO mapred.Task: Task 'attempt_local1944506404_0006_m_000004_0' done.
16/08/17 21:07:07 INFO mapred.LocalJobRunner: Finishing task: attempt_local1944506404_0006_m_000004_0
16/08/17 21:07:07 INFO mapred.LocalJobRunner: Starting task: attempt_local1944506404_0006_m_000005_0
16/08/17 21:07:07 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:07 INFO mapred.MapTask: Processing split: file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/rules/part-r-00015:0+2444081
16/08/17 21:07:07 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
16/08/17 21:07:07 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
16/08/17 21:07:07 INFO mapred.MapTask: soft limit at 83886080
16/08/17 21:07:07 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
16/08/17 21:07:07 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
16/08/17 21:07:07 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
16/08/17 21:07:07 INFO mapred.LocalJobRunner: 
16/08/17 21:07:07 INFO mapred.MapTask: Starting flush of map output
16/08/17 21:07:07 INFO mapred.MapTask: Spilling map output
16/08/17 21:07:07 INFO mapred.MapTask: bufstart = 0; bufend = 2059357; bufvoid = 104857600
16/08/17 21:07:07 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26034024(104136096); length = 180373/6553600
16/08/17 21:07:07 INFO mapred.LocalJobRunner: 
16/08/17 21:07:07 INFO mapred.MapTask: Starting flush of map output
16/08/17 21:07:07 INFO mapred.MapTask: Spilling map output
16/08/17 21:07:07 INFO mapred.MapTask: bufstart = 0; bufend = 4673056; bufvoid = 104857600
16/08/17 21:07:07 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 25670968(102683872); length = 543429/6553600
16/08/17 21:07:07 INFO mapred.MapTask: Finished spill 0
16/08/17 21:07:07 INFO mapred.Task: Task:attempt_local1944506404_0006_m_000005_0 is done. And is in the process of committing
16/08/17 21:07:07 INFO mapred.LocalJobRunner: map
16/08/17 21:07:07 INFO mapred.Task: Task 'attempt_local1944506404_0006_m_000005_0' done.
16/08/17 21:07:07 INFO mapred.LocalJobRunner: Finishing task: attempt_local1944506404_0006_m_000005_0
16/08/17 21:07:07 INFO mapred.LocalJobRunner: Starting task: attempt_local1944506404_0006_m_000006_0
16/08/17 21:07:07 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:07 INFO mapred.MapTask: Processing split: file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/rules/part-r-00007:0+2442974
16/08/17 21:07:07 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
16/08/17 21:07:07 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
16/08/17 21:07:07 INFO mapred.MapTask: soft limit at 83886080
16/08/17 21:07:07 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
16/08/17 21:07:07 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
16/08/17 21:07:07 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
16/08/17 21:07:08 INFO mapred.LocalJobRunner: 
16/08/17 21:07:08 INFO mapred.MapTask: Starting flush of map output
16/08/17 21:07:08 INFO mapred.MapTask: Spilling map output
16/08/17 21:07:08 INFO mapred.MapTask: bufstart = 0; bufend = 2058602; bufvoid = 104857600
16/08/17 21:07:08 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26034200(104136800); length = 180197/6553600
16/08/17 21:07:08 INFO mapred.MapTask: Finished spill 0
16/08/17 21:07:08 INFO mapred.Task: Task:attempt_local1944506404_0006_m_000006_0 is done. And is in the process of committing
16/08/17 21:07:08 INFO mapred.LocalJobRunner: map
16/08/17 21:07:08 INFO mapred.Task: Task 'attempt_local1944506404_0006_m_000006_0' done.
16/08/17 21:07:08 INFO mapred.LocalJobRunner: Finishing task: attempt_local1944506404_0006_m_000006_0
16/08/17 21:07:08 INFO mapred.LocalJobRunner: Starting task: attempt_local1944506404_0006_m_000007_0
16/08/17 21:07:08 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:08 INFO mapred.MapTask: Processing split: file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/rules/part-r-00011:0+2441386
16/08/17 21:07:08 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
16/08/17 21:07:08 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
16/08/17 21:07:08 INFO mapred.MapTask: soft limit at 83886080
16/08/17 21:07:08 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
16/08/17 21:07:08 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
16/08/17 21:07:08 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
16/08/17 21:07:08 INFO mapred.MapTask: Finished spill 0
16/08/17 21:07:08 INFO mapred.Task: Task:attempt_local1999544224_0005_m_000001_0 is done. And is in the process of committing
16/08/17 21:07:08 INFO mapred.LocalJobRunner: map
16/08/17 21:07:08 INFO mapred.Task: Task 'attempt_local1999544224_0005_m_000001_0' done.
16/08/17 21:07:08 INFO mapred.LocalJobRunner: Finishing task: attempt_local1999544224_0005_m_000001_0
16/08/17 21:07:08 INFO mapred.LocalJobRunner: Starting task: attempt_local1999544224_0005_m_000002_0
16/08/17 21:07:08 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:08 INFO mapred.MapTask: Processing split: file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/rules/part-r-00001:0+2455395
16/08/17 21:07:08 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
16/08/17 21:07:08 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
16/08/17 21:07:08 INFO mapred.MapTask: soft limit at 83886080
16/08/17 21:07:08 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
16/08/17 21:07:08 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
16/08/17 21:07:08 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
16/08/17 21:07:08 INFO mapred.LocalJobRunner: 
16/08/17 21:07:08 INFO mapred.MapTask: Starting flush of map output
16/08/17 21:07:08 INFO mapred.MapTask: Spilling map output
16/08/17 21:07:08 INFO mapred.MapTask: bufstart = 0; bufend = 2058034; bufvoid = 104857600
16/08/17 21:07:08 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26034700(104138800); length = 179697/6553600
16/08/17 21:07:08 INFO mapred.MapTask: Finished spill 0
16/08/17 21:07:08 INFO mapred.Task: Task:attempt_local1944506404_0006_m_000007_0 is done. And is in the process of committing
16/08/17 21:07:08 INFO mapred.LocalJobRunner: map
16/08/17 21:07:08 INFO mapred.Task: Task 'attempt_local1944506404_0006_m_000007_0' done.
16/08/17 21:07:08 INFO mapred.LocalJobRunner: Finishing task: attempt_local1944506404_0006_m_000007_0
16/08/17 21:07:08 INFO mapred.LocalJobRunner: Starting task: attempt_local1944506404_0006_m_000008_0
16/08/17 21:07:08 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:08 INFO mapred.MapTask: Processing split: file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/rules/part-r-00002:0+2395166
16/08/17 21:07:08 INFO mapred.MapTask: Finished spill 0
16/08/17 21:07:08 INFO mapred.Task: Task:attempt_local518136789_0007_m_000001_0 is done. And is in the process of committing
16/08/17 21:07:08 INFO mapred.LocalJobRunner: map
16/08/17 21:07:08 INFO mapred.Task: Task 'attempt_local518136789_0007_m_000001_0' done.
16/08/17 21:07:08 INFO mapred.LocalJobRunner: Finishing task: attempt_local518136789_0007_m_000001_0
16/08/17 21:07:08 INFO mapred.LocalJobRunner: Starting task: attempt_local518136789_0007_m_000002_0
16/08/17 21:07:08 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
16/08/17 21:07:08 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
16/08/17 21:07:08 INFO mapred.MapTask: soft limit at 83886080
16/08/17 21:07:08 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
16/08/17 21:07:08 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
16/08/17 21:07:08 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
16/08/17 21:07:08 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:08 INFO mapred.MapTask: Processing split: file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/rules/part-r-00001:0+2455395
16/08/17 21:07:08 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
16/08/17 21:07:08 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
16/08/17 21:07:08 INFO mapred.MapTask: soft limit at 83886080
16/08/17 21:07:08 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
16/08/17 21:07:08 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
16/08/17 21:07:08 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
16/08/17 21:07:08 INFO mapred.LocalJobRunner: 
16/08/17 21:07:08 INFO mapred.MapTask: Starting flush of map output
16/08/17 21:07:08 INFO mapred.MapTask: Spilling map output
16/08/17 21:07:08 INFO mapred.MapTask: bufstart = 0; bufend = 2019250; bufvoid = 104857600
16/08/17 21:07:08 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26038188(104152752); length = 176209/6553600
16/08/17 21:07:08 INFO mapred.LocalJobRunner: 
16/08/17 21:07:08 INFO mapred.MapTask: Starting flush of map output
16/08/17 21:07:08 INFO mapred.MapTask: Spilling map output
16/08/17 21:07:08 INFO mapred.MapTask: bufstart = 0; bufend = 4760848; bufvoid = 104857600
16/08/17 21:07:08 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 25671100(102684400); length = 543297/6553600
16/08/17 21:07:08 INFO mapred.MapTask: Finished spill 0
16/08/17 21:07:08 INFO mapred.Task: Task:attempt_local1944506404_0006_m_000008_0 is done. And is in the process of committing
16/08/17 21:07:08 INFO mapred.LocalJobRunner: map
16/08/17 21:07:08 INFO mapred.Task: Task 'attempt_local1944506404_0006_m_000008_0' done.
16/08/17 21:07:08 INFO mapred.LocalJobRunner: Finishing task: attempt_local1944506404_0006_m_000008_0
16/08/17 21:07:08 INFO mapred.LocalJobRunner: Starting task: attempt_local1944506404_0006_m_000009_0
16/08/17 21:07:08 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:08 INFO mapred.MapTask: Processing split: file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/rules/part-r-00012:0+2389851
16/08/17 21:07:08 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
16/08/17 21:07:08 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
16/08/17 21:07:08 INFO mapred.MapTask: soft limit at 83886080
16/08/17 21:07:08 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
16/08/17 21:07:08 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
16/08/17 21:07:08 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
16/08/17 21:07:09 INFO mapred.LocalJobRunner: 
16/08/17 21:07:09 INFO mapred.MapTask: Starting flush of map output
16/08/17 21:07:09 INFO mapred.MapTask: Spilling map output
16/08/17 21:07:09 INFO mapred.MapTask: bufstart = 0; bufend = 4669032; bufvoid = 104857600
16/08/17 21:07:09 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 25671100(102684400); length = 543297/6553600
16/08/17 21:07:09 INFO mapred.LocalJobRunner: 
16/08/17 21:07:09 INFO mapred.MapTask: Starting flush of map output
16/08/17 21:07:09 INFO mapred.MapTask: Spilling map output
16/08/17 21:07:09 INFO mapred.MapTask: bufstart = 0; bufend = 2015803; bufvoid = 104857600
16/08/17 21:07:09 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26039092(104156368); length = 175305/6553600
16/08/17 21:07:09 INFO mapred.MapTask: Finished spill 0
16/08/17 21:07:09 INFO mapred.Task: Task:attempt_local1944506404_0006_m_000009_0 is done. And is in the process of committing
16/08/17 21:07:09 INFO mapred.LocalJobRunner: map
16/08/17 21:07:09 INFO mapred.Task: Task 'attempt_local1944506404_0006_m_000009_0' done.
16/08/17 21:07:09 INFO mapred.LocalJobRunner: Finishing task: attempt_local1944506404_0006_m_000009_0
16/08/17 21:07:09 INFO mapred.LocalJobRunner: Starting task: attempt_local1944506404_0006_m_000010_0
16/08/17 21:07:09 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:09 INFO mapred.MapTask: Processing split: file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/rules/part-r-00000:0+2389069
16/08/17 21:07:09 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
16/08/17 21:07:09 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
16/08/17 21:07:09 INFO mapred.MapTask: soft limit at 83886080
16/08/17 21:07:09 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
16/08/17 21:07:09 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
16/08/17 21:07:09 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
16/08/17 21:07:09 INFO mapred.LocalJobRunner: 
16/08/17 21:07:09 INFO mapred.MapTask: Starting flush of map output
16/08/17 21:07:09 INFO mapred.MapTask: Spilling map output
16/08/17 21:07:09 INFO mapred.MapTask: bufstart = 0; bufend = 2014581; bufvoid = 104857600
16/08/17 21:07:09 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26038872(104155488); length = 175525/6553600
16/08/17 21:07:09 INFO mapred.MapTask: Finished spill 0
16/08/17 21:07:09 INFO mapred.Task: Task:attempt_local518136789_0007_m_000002_0 is done. And is in the process of committing
16/08/17 21:07:09 INFO mapred.LocalJobRunner: map
16/08/17 21:07:09 INFO mapred.Task: Task 'attempt_local518136789_0007_m_000002_0' done.
16/08/17 21:07:09 INFO mapred.LocalJobRunner: Finishing task: attempt_local518136789_0007_m_000002_0
16/08/17 21:07:09 INFO mapred.LocalJobRunner: Starting task: attempt_local518136789_0007_m_000003_0
16/08/17 21:07:09 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:09 INFO mapred.MapTask: Processing split: file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/rules/part-r-00003:0+2454310
16/08/17 21:07:09 INFO mapred.MapTask: Finished spill 0
16/08/17 21:07:09 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
16/08/17 21:07:09 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
16/08/17 21:07:09 INFO mapred.MapTask: soft limit at 83886080
16/08/17 21:07:09 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
16/08/17 21:07:09 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
16/08/17 21:07:09 INFO mapred.Task: Task:attempt_local1944506404_0006_m_000010_0 is done. And is in the process of committing
16/08/17 21:07:09 INFO mapred.LocalJobRunner: map
16/08/17 21:07:09 INFO mapred.Task: Task 'attempt_local1944506404_0006_m_000010_0' done.
16/08/17 21:07:09 INFO mapred.LocalJobRunner: Finishing task: attempt_local1944506404_0006_m_000010_0
16/08/17 21:07:09 INFO mapred.LocalJobRunner: Starting task: attempt_local1944506404_0006_m_000011_0
16/08/17 21:07:09 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:09 INFO mapred.MapTask: Processing split: file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/rules/part-r-00008:0+2387786
16/08/17 21:07:09 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
16/08/17 21:07:09 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
16/08/17 21:07:09 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
16/08/17 21:07:09 INFO mapred.MapTask: soft limit at 83886080
16/08/17 21:07:09 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
16/08/17 21:07:09 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
16/08/17 21:07:09 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
16/08/17 21:07:09 INFO mapred.MapTask: Finished spill 0
16/08/17 21:07:09 INFO mapred.Task: Task:attempt_local1999544224_0005_m_000002_0 is done. And is in the process of committing
16/08/17 21:07:09 INFO mapred.LocalJobRunner: map
16/08/17 21:07:09 INFO mapred.Task: Task 'attempt_local1999544224_0005_m_000002_0' done.
16/08/17 21:07:09 INFO mapred.LocalJobRunner: Finishing task: attempt_local1999544224_0005_m_000002_0
16/08/17 21:07:09 INFO mapred.LocalJobRunner: Starting task: attempt_local1999544224_0005_m_000003_0
16/08/17 21:07:09 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:09 INFO mapred.MapTask: Processing split: file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/rules/part-r-00003:0+2454310
16/08/17 21:07:10 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
16/08/17 21:07:10 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
16/08/17 21:07:10 INFO mapred.MapTask: soft limit at 83886080
16/08/17 21:07:10 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
16/08/17 21:07:10 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
16/08/17 21:07:10 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
16/08/17 21:07:10 INFO mapred.LocalJobRunner: 
16/08/17 21:07:10 INFO mapred.MapTask: Starting flush of map output
16/08/17 21:07:10 INFO mapred.MapTask: Spilling map output
16/08/17 21:07:10 INFO mapred.MapTask: bufstart = 0; bufend = 2013310; bufvoid = 104857600
16/08/17 21:07:10 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26038868(104155472); length = 175529/6553600
16/08/17 21:07:10 INFO mapred.MapTask: Finished spill 0
16/08/17 21:07:10 INFO mapred.Task: Task:attempt_local1944506404_0006_m_000011_0 is done. And is in the process of committing
16/08/17 21:07:10 INFO mapred.LocalJobRunner: map
16/08/17 21:07:10 INFO mapred.Task: Task 'attempt_local1944506404_0006_m_000011_0' done.
16/08/17 21:07:10 INFO mapred.LocalJobRunner: Finishing task: attempt_local1944506404_0006_m_000011_0
16/08/17 21:07:10 INFO mapred.LocalJobRunner: Starting task: attempt_local1944506404_0006_m_000012_0
16/08/17 21:07:10 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:10 INFO mapred.MapTask: Processing split: file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/rules/part-r-00014:0+2382869
16/08/17 21:07:10 INFO mapred.LocalJobRunner: 
16/08/17 21:07:10 INFO mapred.MapTask: Starting flush of map output
16/08/17 21:07:10 INFO mapred.MapTask: Spilling map output
16/08/17 21:07:10 INFO mapred.MapTask: bufstart = 0; bufend = 4667264; bufvoid = 104857600
16/08/17 21:07:10 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 25671172(102684688); length = 543225/6553600
16/08/17 21:07:10 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
16/08/17 21:07:10 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
16/08/17 21:07:10 INFO mapred.MapTask: soft limit at 83886080
16/08/17 21:07:10 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
16/08/17 21:07:10 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
16/08/17 21:07:10 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
16/08/17 21:07:10 INFO mapred.LocalJobRunner: 
16/08/17 21:07:10 INFO mapred.MapTask: Starting flush of map output
16/08/17 21:07:10 INFO mapred.MapTask: Spilling map output
16/08/17 21:07:10 INFO mapred.MapTask: bufstart = 0; bufend = 4758428; bufvoid = 104857600
16/08/17 21:07:10 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 25671172(102684688); length = 543225/6553600
16/08/17 21:07:10 INFO mapred.LocalJobRunner: 
16/08/17 21:07:10 INFO mapred.MapTask: Starting flush of map output
16/08/17 21:07:10 INFO mapred.MapTask: Spilling map output
16/08/17 21:07:10 INFO mapred.MapTask: bufstart = 0; bufend = 2009421; bufvoid = 104857600
16/08/17 21:07:10 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26039352(104157408); length = 175045/6553600
16/08/17 21:07:10 INFO mapred.MapTask: Finished spill 0
16/08/17 21:07:10 INFO mapred.Task: Task:attempt_local1944506404_0006_m_000012_0 is done. And is in the process of committing
16/08/17 21:07:10 INFO mapred.LocalJobRunner: map
16/08/17 21:07:10 INFO mapred.Task: Task 'attempt_local1944506404_0006_m_000012_0' done.
16/08/17 21:07:10 INFO mapred.LocalJobRunner: Finishing task: attempt_local1944506404_0006_m_000012_0
16/08/17 21:07:10 INFO mapred.LocalJobRunner: Starting task: attempt_local1944506404_0006_m_000013_0
16/08/17 21:07:10 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:10 INFO mapred.MapTask: Processing split: file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/rules/part-r-00004:0+2379925
16/08/17 21:07:10 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
16/08/17 21:07:10 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
16/08/17 21:07:10 INFO mapred.MapTask: soft limit at 83886080
16/08/17 21:07:10 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
16/08/17 21:07:10 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
16/08/17 21:07:10 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
16/08/17 21:07:10 INFO mapred.MapTask: Finished spill 0
16/08/17 21:07:10 INFO mapred.Task: Task:attempt_local518136789_0007_m_000003_0 is done. And is in the process of committing
16/08/17 21:07:10 INFO mapred.LocalJobRunner: map
16/08/17 21:07:10 INFO mapred.Task: Task 'attempt_local518136789_0007_m_000003_0' done.
16/08/17 21:07:10 INFO mapred.LocalJobRunner: Finishing task: attempt_local518136789_0007_m_000003_0
16/08/17 21:07:10 INFO mapred.LocalJobRunner: Starting task: attempt_local518136789_0007_m_000004_0
16/08/17 21:07:10 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:10 INFO mapred.MapTask: Processing split: file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/rules/part-r-00013:0+2451169
16/08/17 21:07:11 INFO mapred.LocalJobRunner: 
16/08/17 21:07:11 INFO mapred.MapTask: Starting flush of map output
16/08/17 21:07:11 INFO mapred.MapTask: Spilling map output
16/08/17 21:07:11 INFO mapred.MapTask: bufstart = 0; bufend = 2006893; bufvoid = 104857600
16/08/17 21:07:11 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26039560(104158240); length = 174837/6553600
16/08/17 21:07:11 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
16/08/17 21:07:11 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
16/08/17 21:07:11 INFO mapred.MapTask: soft limit at 83886080
16/08/17 21:07:11 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
16/08/17 21:07:11 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
16/08/17 21:07:11 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
16/08/17 21:07:11 INFO mapred.MapTask: Finished spill 0
16/08/17 21:07:11 INFO mapred.Task: Task:attempt_local1944506404_0006_m_000013_0 is done. And is in the process of committing
16/08/17 21:07:11 INFO mapred.LocalJobRunner: map
16/08/17 21:07:11 INFO mapred.Task: Task 'attempt_local1944506404_0006_m_000013_0' done.
16/08/17 21:07:11 INFO mapred.LocalJobRunner: Finishing task: attempt_local1944506404_0006_m_000013_0
16/08/17 21:07:11 INFO mapred.LocalJobRunner: Starting task: attempt_local1944506404_0006_m_000014_0
16/08/17 21:07:11 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:11 INFO mapred.MapTask: Processing split: file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/rules/part-r-00006:0+2375925
16/08/17 21:07:11 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
16/08/17 21:07:11 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
16/08/17 21:07:11 INFO mapred.MapTask: soft limit at 83886080
16/08/17 21:07:11 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
16/08/17 21:07:11 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
16/08/17 21:07:11 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
16/08/17 21:07:11 INFO mapred.MapTask: Finished spill 0
16/08/17 21:07:11 INFO mapred.LocalJobRunner: 
16/08/17 21:07:11 INFO mapred.MapTask: Starting flush of map output
16/08/17 21:07:11 INFO mapred.MapTask: Spilling map output
16/08/17 21:07:11 INFO mapred.MapTask: bufstart = 0; bufend = 2002709; bufvoid = 104857600
16/08/17 21:07:11 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26039448(104157792); length = 174949/6553600
16/08/17 21:07:11 INFO mapred.Task: Task:attempt_local1999544224_0005_m_000003_0 is done. And is in the process of committing
16/08/17 21:07:11 INFO mapred.LocalJobRunner: map
16/08/17 21:07:11 INFO mapred.Task: Task 'attempt_local1999544224_0005_m_000003_0' done.
16/08/17 21:07:11 INFO mapred.LocalJobRunner: Finishing task: attempt_local1999544224_0005_m_000003_0
16/08/17 21:07:11 INFO mapred.LocalJobRunner: Starting task: attempt_local1999544224_0005_m_000004_0
16/08/17 21:07:11 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:11 INFO mapred.MapTask: Processing split: file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/rules/part-r-00013:0+2451169
16/08/17 21:07:11 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
16/08/17 21:07:11 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
16/08/17 21:07:11 INFO mapred.MapTask: soft limit at 83886080
16/08/17 21:07:11 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
16/08/17 21:07:11 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
16/08/17 21:07:11 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
16/08/17 21:07:11 INFO mapred.LocalJobRunner: 
16/08/17 21:07:11 INFO mapred.MapTask: Starting flush of map output
16/08/17 21:07:11 INFO mapred.MapTask: Spilling map output
16/08/17 21:07:11 INFO mapred.MapTask: bufstart = 0; bufend = 4663220; bufvoid = 104857600
16/08/17 21:07:11 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 25671724(102686896); length = 542673/6553600
16/08/17 21:07:11 INFO mapred.MapTask: Finished spill 0
16/08/17 21:07:11 INFO mapred.Task: Task:attempt_local1944506404_0006_m_000014_0 is done. And is in the process of committing
16/08/17 21:07:11 INFO mapred.LocalJobRunner: map
16/08/17 21:07:11 INFO mapred.Task: Task 'attempt_local1944506404_0006_m_000014_0' done.
16/08/17 21:07:11 INFO mapred.LocalJobRunner: Finishing task: attempt_local1944506404_0006_m_000014_0
16/08/17 21:07:11 INFO mapred.LocalJobRunner: Starting task: attempt_local1944506404_0006_m_000015_0
16/08/17 21:07:11 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:11 INFO mapred.MapTask: Processing split: file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/rules/part-r-00010:0+2364196
16/08/17 21:07:11 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
16/08/17 21:07:11 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
16/08/17 21:07:11 INFO mapred.MapTask: soft limit at 83886080
16/08/17 21:07:11 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
16/08/17 21:07:11 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
16/08/17 21:07:11 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
16/08/17 21:07:11 INFO mapred.LocalJobRunner: 
16/08/17 21:07:11 INFO mapred.MapTask: Starting flush of map output
16/08/17 21:07:11 INFO mapred.MapTask: Spilling map output
16/08/17 21:07:11 INFO mapred.MapTask: bufstart = 0; bufend = 4755044; bufvoid = 104857600
16/08/17 21:07:11 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 25671724(102686896); length = 542673/6553600
16/08/17 21:07:11 INFO mapred.LocalJobRunner: 
16/08/17 21:07:11 INFO mapred.MapTask: Starting flush of map output
16/08/17 21:07:11 INFO mapred.MapTask: Spilling map output
16/08/17 21:07:11 INFO mapred.MapTask: bufstart = 0; bufend = 1993412; bufvoid = 104857600
16/08/17 21:07:11 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26040604(104162416); length = 173793/6553600
16/08/17 21:07:11 INFO mapred.MapTask: Finished spill 0
16/08/17 21:07:11 INFO mapred.Task: Task:attempt_local1944506404_0006_m_000015_0 is done. And is in the process of committing
16/08/17 21:07:11 INFO mapred.LocalJobRunner: map
16/08/17 21:07:11 INFO mapred.Task: Task 'attempt_local1944506404_0006_m_000015_0' done.
16/08/17 21:07:11 INFO mapred.LocalJobRunner: Finishing task: attempt_local1944506404_0006_m_000015_0
16/08/17 21:07:11 INFO mapred.LocalJobRunner: map task executor complete.
16/08/17 21:07:11 INFO mapred.LocalJobRunner: Waiting for reduce tasks
16/08/17 21:07:11 INFO mapred.LocalJobRunner: Starting task: attempt_local1944506404_0006_r_000000_0
16/08/17 21:07:11 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:11 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@253b75da
16/08/17 21:07:11 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=355624544, maxSingleShuffleLimit=88906136, mergeThreshold=234712208, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:07:11 INFO reduce.EventFetcher: attempt_local1944506404_0006_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:07:11 INFO reduce.LocalFetcher: localfetcher#65 about to shuffle output of map attempt_local1944506404_0006_m_000011_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:11 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000011_0
16/08/17 21:07:11 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->2
16/08/17 21:07:11 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)
	at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
16/08/17 21:07:11 INFO reduce.LocalFetcher: localfetcher#65 about to shuffle output of map attempt_local1944506404_0006_m_000010_0 decomp: 2102347 len: 2102351 to MEMORY
16/08/17 21:07:11 INFO reduce.InMemoryMapOutput: Read 2102347 bytes from map-output for attempt_local1944506404_0006_m_000010_0
16/08/17 21:07:11 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2102347, inMemoryMapOutputs.size() -> 2, commitMemory -> 2, usedMemory ->2102349
16/08/17 21:07:11 INFO reduce.LocalFetcher: localfetcher#65 about to shuffle output of map attempt_local1944506404_0006_m_000004_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:11 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000004_0
16/08/17 21:07:11 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 3, commitMemory -> 2102349, usedMemory ->2102351
16/08/17 21:07:11 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)
	at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
16/08/17 21:07:11 INFO reduce.LocalFetcher: localfetcher#65 about to shuffle output of map attempt_local1944506404_0006_m_000009_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:11 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000009_0
16/08/17 21:07:11 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 4, commitMemory -> 2102351, usedMemory ->2102353
16/08/17 21:07:11 INFO reduce.LocalFetcher: localfetcher#65 about to shuffle output of map attempt_local1944506404_0006_m_000003_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:11 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000003_0
16/08/17 21:07:11 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 5, commitMemory -> 2102353, usedMemory ->2102355
16/08/17 21:07:11 INFO reduce.LocalFetcher: localfetcher#65 about to shuffle output of map attempt_local1944506404_0006_m_000002_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:11 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000002_0
16/08/17 21:07:11 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 6, commitMemory -> 2102355, usedMemory ->2102357
16/08/17 21:07:11 INFO reduce.LocalFetcher: localfetcher#65 about to shuffle output of map attempt_local1944506404_0006_m_000015_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:11 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000015_0
16/08/17 21:07:11 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 7, commitMemory -> 2102357, usedMemory ->2102359
16/08/17 21:07:11 INFO reduce.LocalFetcher: localfetcher#65 about to shuffle output of map attempt_local1944506404_0006_m_000014_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:11 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000014_0
16/08/17 21:07:11 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 8, commitMemory -> 2102359, usedMemory ->2102361
16/08/17 21:07:11 INFO reduce.LocalFetcher: localfetcher#65 about to shuffle output of map attempt_local1944506404_0006_m_000001_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:11 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000001_0
16/08/17 21:07:11 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 9, commitMemory -> 2102361, usedMemory ->2102363
16/08/17 21:07:11 INFO reduce.LocalFetcher: localfetcher#65 about to shuffle output of map attempt_local1944506404_0006_m_000008_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:11 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000008_0
16/08/17 21:07:11 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 10, commitMemory -> 2102363, usedMemory ->2102365
16/08/17 21:07:11 INFO reduce.LocalFetcher: localfetcher#65 about to shuffle output of map attempt_local1944506404_0006_m_000013_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:11 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000013_0
16/08/17 21:07:11 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 11, commitMemory -> 2102365, usedMemory ->2102367
16/08/17 21:07:11 INFO reduce.LocalFetcher: localfetcher#65 about to shuffle output of map attempt_local1944506404_0006_m_000007_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:11 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000007_0
16/08/17 21:07:11 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 12, commitMemory -> 2102367, usedMemory ->2102369
16/08/17 21:07:11 INFO reduce.LocalFetcher: localfetcher#65 about to shuffle output of map attempt_local1944506404_0006_m_000006_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:11 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000006_0
16/08/17 21:07:11 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 13, commitMemory -> 2102369, usedMemory ->2102371
16/08/17 21:07:11 INFO reduce.LocalFetcher: localfetcher#65 about to shuffle output of map attempt_local1944506404_0006_m_000000_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:11 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000000_0
16/08/17 21:07:11 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 14, commitMemory -> 2102371, usedMemory ->2102373
16/08/17 21:07:11 INFO reduce.LocalFetcher: localfetcher#65 about to shuffle output of map attempt_local1944506404_0006_m_000005_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:11 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)
	at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
16/08/17 21:07:11 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000005_0
16/08/17 21:07:11 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)
	at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
16/08/17 21:07:11 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 15, commitMemory -> 2102373, usedMemory ->2102375
16/08/17 21:07:11 INFO reduce.LocalFetcher: localfetcher#65 about to shuffle output of map attempt_local1944506404_0006_m_000012_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:11 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000012_0
16/08/17 21:07:11 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 16, commitMemory -> 2102375, usedMemory ->2102377
16/08/17 21:07:11 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:07:11 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:11 INFO reduce.MergeManagerImpl: finalMerge called with 16 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:07:11 INFO mapred.Merger: Merging 16 sorted segments
16/08/17 21:07:11 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 2102325 bytes
16/08/17 21:07:12 INFO reduce.MergeManagerImpl: Merged 16 segments, 2102377 bytes to disk to satisfy reduce memory limit
16/08/17 21:07:12 INFO reduce.MergeManagerImpl: Merging 1 files, 2102351 bytes from disk
16/08/17 21:07:12 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:07:12 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:07:12 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 2102325 bytes
16/08/17 21:07:12 INFO mapred.LocalJobRunner: 16 / 16 copied.
[1000]
[2000]
[3000]
[4000]
[5000]
[6000]
[7000]
[8000]
[9000]
16/08/17 21:07:12 INFO mapred.MapTask: Finished spill 0
16/08/17 21:07:12 INFO mapred.Task: Task:attempt_local518136789_0007_m_000004_0 is done. And is in the process of committing
16/08/17 21:07:12 INFO mapred.LocalJobRunner: map
16/08/17 21:07:12 INFO mapred.Task: Task 'attempt_local518136789_0007_m_000004_0' done.
16/08/17 21:07:12 INFO mapred.LocalJobRunner: Finishing task: attempt_local518136789_0007_m_000004_0
16/08/17 21:07:12 INFO mapred.LocalJobRunner: Starting task: attempt_local518136789_0007_m_000005_0
[10000]
16/08/17 21:07:12 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:12 INFO mapred.LocalJobRunner: Starting task: attempt_local1944506404_0006_r_000001_0
16/08/17 21:07:12 INFO mapred.MapTask: Processing split: file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/rules/part-r-00015:0+2444081
16/08/17 21:07:12 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:12 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@142aa811
16/08/17 21:07:12 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=357459552, maxSingleShuffleLimit=89364888, mergeThreshold=235923312, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:07:12 INFO reduce.EventFetcher: attempt_local1944506404_0006_r_000001_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:07:12 INFO reduce.LocalFetcher: localfetcher#66 about to shuffle output of map attempt_local1944506404_0006_m_000011_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:12 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000011_0
16/08/17 21:07:12 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->2
16/08/17 21:07:12 INFO reduce.LocalFetcher: localfetcher#66 about to shuffle output of map attempt_local1944506404_0006_m_000010_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:12 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000010_0
16/08/17 21:07:12 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 2, commitMemory -> 2, usedMemory ->4
16/08/17 21:07:12 INFO reduce.LocalFetcher: localfetcher#66 about to shuffle output of map attempt_local1944506404_0006_m_000004_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:12 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000004_0
16/08/17 21:07:12 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 3, commitMemory -> 4, usedMemory ->6
16/08/17 21:07:12 INFO reduce.LocalFetcher: localfetcher#66 about to shuffle output of map attempt_local1944506404_0006_m_000009_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:12 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
16/08/17 21:07:12 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
16/08/17 21:07:12 INFO mapred.MapTask: soft limit at 83886080
16/08/17 21:07:12 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
16/08/17 21:07:12 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
16/08/17 21:07:12 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000009_0
16/08/17 21:07:12 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 4, commitMemory -> 6, usedMemory ->8
16/08/17 21:07:12 INFO reduce.LocalFetcher: localfetcher#66 about to shuffle output of map attempt_local1944506404_0006_m_000003_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:12 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000003_0
16/08/17 21:07:12 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 5, commitMemory -> 8, usedMemory ->10
16/08/17 21:07:12 INFO reduce.LocalFetcher: localfetcher#66 about to shuffle output of map attempt_local1944506404_0006_m_000002_0 decomp: 2159675 len: 2159679 to MEMORY
16/08/17 21:07:12 INFO reduce.InMemoryMapOutput: Read 2159675 bytes from map-output for attempt_local1944506404_0006_m_000002_0
16/08/17 21:07:12 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2159675, inMemoryMapOutputs.size() -> 6, commitMemory -> 10, usedMemory ->2159685
16/08/17 21:07:12 INFO reduce.LocalFetcher: localfetcher#66 about to shuffle output of map attempt_local1944506404_0006_m_000015_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:12 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000015_0
16/08/17 21:07:12 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 7, commitMemory -> 2159685, usedMemory ->2159687
16/08/17 21:07:12 INFO reduce.LocalFetcher: localfetcher#66 about to shuffle output of map attempt_local1944506404_0006_m_000014_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:12 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000014_0
16/08/17 21:07:12 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 8, commitMemory -> 2159687, usedMemory ->2159689
16/08/17 21:07:12 INFO reduce.LocalFetcher: localfetcher#66 about to shuffle output of map attempt_local1944506404_0006_m_000001_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:12 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000001_0
16/08/17 21:07:12 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 9, commitMemory -> 2159689, usedMemory ->2159691
16/08/17 21:07:12 INFO reduce.LocalFetcher: localfetcher#66 about to shuffle output of map attempt_local1944506404_0006_m_000008_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:12 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
16/08/17 21:07:12 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000008_0
16/08/17 21:07:12 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 10, commitMemory -> 2159691, usedMemory ->2159693
16/08/17 21:07:12 INFO reduce.LocalFetcher: localfetcher#66 about to shuffle output of map attempt_local1944506404_0006_m_000013_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:12 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000013_0
16/08/17 21:07:12 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 11, commitMemory -> 2159693, usedMemory ->2159695
16/08/17 21:07:12 INFO reduce.LocalFetcher: localfetcher#66 about to shuffle output of map attempt_local1944506404_0006_m_000007_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:12 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000007_0
16/08/17 21:07:12 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 12, commitMemory -> 2159695, usedMemory ->2159697
16/08/17 21:07:12 INFO reduce.LocalFetcher: localfetcher#66 about to shuffle output of map attempt_local1944506404_0006_m_000006_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:12 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000006_0
16/08/17 21:07:12 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 13, commitMemory -> 2159697, usedMemory ->2159699
16/08/17 21:07:12 INFO reduce.LocalFetcher: localfetcher#66 about to shuffle output of map attempt_local1944506404_0006_m_000000_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:12 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000000_0
16/08/17 21:07:12 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 14, commitMemory -> 2159699, usedMemory ->2159701
16/08/17 21:07:12 INFO reduce.LocalFetcher: localfetcher#66 about to shuffle output of map attempt_local1944506404_0006_m_000005_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:12 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000005_0
16/08/17 21:07:12 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 15, commitMemory -> 2159701, usedMemory ->2159703
16/08/17 21:07:12 INFO reduce.LocalFetcher: localfetcher#66 about to shuffle output of map attempt_local1944506404_0006_m_000012_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:12 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000012_0
16/08/17 21:07:12 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 16, commitMemory -> 2159703, usedMemory ->2159705
16/08/17 21:07:12 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:07:12 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:12 INFO reduce.MergeManagerImpl: finalMerge called with 16 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:07:12 INFO mapred.Merger: Merging 16 sorted segments
16/08/17 21:07:12 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 2159657 bytes
16/08/17 21:07:12 INFO reduce.MergeManagerImpl: Merged 16 segments, 2159705 bytes to disk to satisfy reduce memory limit
16/08/17 21:07:12 INFO reduce.MergeManagerImpl: Merging 1 files, 2159679 bytes from disk
16/08/17 21:07:12 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:07:12 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:07:12 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 2159657 bytes
16/08/17 21:07:12 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:12 INFO mapred.MapTask: Finished spill 0
16/08/17 21:07:12 INFO mapred.Task: Task:attempt_local1999544224_0005_m_000004_0 is done. And is in the process of committing
16/08/17 21:07:12 INFO mapred.LocalJobRunner: map
16/08/17 21:07:12 INFO mapred.Task: Task 'attempt_local1999544224_0005_m_000004_0' done.
16/08/17 21:07:12 INFO mapred.LocalJobRunner: Finishing task: attempt_local1999544224_0005_m_000004_0
16/08/17 21:07:12 INFO mapred.LocalJobRunner: Starting task: attempt_local1999544224_0005_m_000005_0
16/08/17 21:07:12 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:12 INFO mapred.MapTask: Processing split: file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/rules/part-r-00015:0+2444081
16/08/17 21:07:12 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
16/08/17 21:07:12 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
16/08/17 21:07:12 INFO mapred.MapTask: soft limit at 83886080
16/08/17 21:07:12 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
16/08/17 21:07:12 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
16/08/17 21:07:12 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
[1000]
[2000]
[3000]
[4000]
[5000]
[6000]
[7000]
[8000]
[9000]
[10000]
16/08/17 21:07:13 INFO mapred.LocalJobRunner: Starting task: attempt_local1944506404_0006_r_000002_0
16/08/17 21:07:13 INFO mapred.LocalJobRunner: 
16/08/17 21:07:13 INFO mapred.MapTask: Starting flush of map output
16/08/17 21:07:13 INFO mapred.MapTask: Spilling map output
16/08/17 21:07:13 INFO mapred.MapTask: bufstart = 0; bufend = 4650056; bufvoid = 104857600
16/08/17 21:07:13 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 25673272(102693088); length = 541125/6553600
16/08/17 21:07:13 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:13 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@6ce03af6
16/08/17 21:07:13 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=361129568, maxSingleShuffleLimit=90282392, mergeThreshold=238345520, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:07:13 INFO reduce.EventFetcher: attempt_local1944506404_0006_r_000002_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:07:13 INFO reduce.LocalFetcher: localfetcher#67 about to shuffle output of map attempt_local1944506404_0006_m_000011_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:13 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000011_0
16/08/17 21:07:13 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->2
16/08/17 21:07:13 INFO reduce.LocalFetcher: localfetcher#67 about to shuffle output of map attempt_local1944506404_0006_m_000010_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:13 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000010_0
16/08/17 21:07:13 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 2, commitMemory -> 2, usedMemory ->4
16/08/17 21:07:13 INFO reduce.LocalFetcher: localfetcher#67 about to shuffle output of map attempt_local1944506404_0006_m_000004_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:13 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000004_0
16/08/17 21:07:13 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 3, commitMemory -> 4, usedMemory ->6
16/08/17 21:07:13 INFO reduce.LocalFetcher: localfetcher#67 about to shuffle output of map attempt_local1944506404_0006_m_000009_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:13 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000009_0
16/08/17 21:07:13 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 4, commitMemory -> 6, usedMemory ->8
16/08/17 21:07:13 INFO reduce.LocalFetcher: localfetcher#67 about to shuffle output of map attempt_local1944506404_0006_m_000003_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:13 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000003_0
16/08/17 21:07:13 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 5, commitMemory -> 8, usedMemory ->10
16/08/17 21:07:13 INFO reduce.LocalFetcher: localfetcher#67 about to shuffle output of map attempt_local1944506404_0006_m_000002_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:13 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000002_0
16/08/17 21:07:13 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 6, commitMemory -> 10, usedMemory ->12
16/08/17 21:07:13 INFO reduce.LocalFetcher: localfetcher#67 about to shuffle output of map attempt_local1944506404_0006_m_000015_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:13 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000015_0
16/08/17 21:07:13 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 7, commitMemory -> 12, usedMemory ->14
16/08/17 21:07:13 INFO reduce.LocalFetcher: localfetcher#67 about to shuffle output of map attempt_local1944506404_0006_m_000014_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:13 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000014_0
16/08/17 21:07:13 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 8, commitMemory -> 14, usedMemory ->16
16/08/17 21:07:13 INFO reduce.LocalFetcher: localfetcher#67 about to shuffle output of map attempt_local1944506404_0006_m_000001_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:13 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000001_0
16/08/17 21:07:13 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 9, commitMemory -> 16, usedMemory ->18
16/08/17 21:07:13 INFO reduce.LocalFetcher: localfetcher#67 about to shuffle output of map attempt_local1944506404_0006_m_000008_0 decomp: 2107358 len: 2107362 to MEMORY
16/08/17 21:07:13 INFO reduce.InMemoryMapOutput: Read 2107358 bytes from map-output for attempt_local1944506404_0006_m_000008_0
16/08/17 21:07:13 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2107358, inMemoryMapOutputs.size() -> 10, commitMemory -> 18, usedMemory ->2107376
16/08/17 21:07:13 INFO reduce.LocalFetcher: localfetcher#67 about to shuffle output of map attempt_local1944506404_0006_m_000013_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:13 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000013_0
16/08/17 21:07:13 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 11, commitMemory -> 2107376, usedMemory ->2107378
16/08/17 21:07:13 INFO reduce.LocalFetcher: localfetcher#67 about to shuffle output of map attempt_local1944506404_0006_m_000007_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:13 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000007_0
16/08/17 21:07:13 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 12, commitMemory -> 2107378, usedMemory ->2107380
16/08/17 21:07:13 INFO reduce.LocalFetcher: localfetcher#67 about to shuffle output of map attempt_local1944506404_0006_m_000006_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:13 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000006_0
16/08/17 21:07:13 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 13, commitMemory -> 2107380, usedMemory ->2107382
16/08/17 21:07:13 INFO reduce.LocalFetcher: localfetcher#67 about to shuffle output of map attempt_local1944506404_0006_m_000000_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:13 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000000_0
16/08/17 21:07:13 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 14, commitMemory -> 2107382, usedMemory ->2107384
16/08/17 21:07:13 INFO reduce.LocalFetcher: localfetcher#67 about to shuffle output of map attempt_local1944506404_0006_m_000005_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:13 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000005_0
16/08/17 21:07:13 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 15, commitMemory -> 2107384, usedMemory ->2107386
16/08/17 21:07:13 INFO reduce.LocalFetcher: localfetcher#67 about to shuffle output of map attempt_local1944506404_0006_m_000012_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:13 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000012_0
16/08/17 21:07:13 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 16, commitMemory -> 2107386, usedMemory ->2107388
16/08/17 21:07:13 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:07:13 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:13 INFO reduce.MergeManagerImpl: finalMerge called with 16 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:07:13 INFO mapred.Merger: Merging 16 sorted segments
16/08/17 21:07:13 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 2107336 bytes
16/08/17 21:07:13 INFO reduce.MergeManagerImpl: Merged 16 segments, 2107388 bytes to disk to satisfy reduce memory limit
16/08/17 21:07:13 INFO reduce.MergeManagerImpl: Merging 1 files, 2107362 bytes from disk
16/08/17 21:07:13 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:07:13 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:07:13 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 2107336 bytes
16/08/17 21:07:13 INFO mapred.LocalJobRunner: 16 / 16 copied.
[1000]
16/08/17 21:07:13 INFO mapred.LocalJobRunner: 
16/08/17 21:07:13 INFO mapred.MapTask: Starting flush of map output
16/08/17 21:07:13 INFO mapred.MapTask: Spilling map output
16/08/17 21:07:13 INFO mapred.MapTask: bufstart = 0; bufend = 4742076; bufvoid = 104857600
16/08/17 21:07:13 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 25673272(102693088); length = 541125/6553600
[2000]
[3000]
[4000]
[5000]
[6000]
[7000]
[8000]
[9000]
[10000]
16/08/17 21:07:13 INFO mapred.LocalJobRunner: Starting task: attempt_local1944506404_0006_r_000003_0
16/08/17 21:07:13 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:13 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@4bc53115
16/08/17 21:07:13 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=361129568, maxSingleShuffleLimit=90282392, mergeThreshold=238345520, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:07:13 INFO reduce.EventFetcher: attempt_local1944506404_0006_r_000003_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:07:13 INFO reduce.LocalFetcher: localfetcher#68 about to shuffle output of map attempt_local1944506404_0006_m_000011_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:13 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000011_0
16/08/17 21:07:13 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->2
16/08/17 21:07:13 INFO reduce.LocalFetcher: localfetcher#68 about to shuffle output of map attempt_local1944506404_0006_m_000010_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:13 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000010_0
16/08/17 21:07:13 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 2, commitMemory -> 2, usedMemory ->4
16/08/17 21:07:13 INFO reduce.LocalFetcher: localfetcher#68 about to shuffle output of map attempt_local1944506404_0006_m_000004_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:13 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000004_0
16/08/17 21:07:13 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 3, commitMemory -> 4, usedMemory ->6
16/08/17 21:07:13 INFO reduce.LocalFetcher: localfetcher#68 about to shuffle output of map attempt_local1944506404_0006_m_000009_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:13 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000009_0
16/08/17 21:07:13 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 4, commitMemory -> 6, usedMemory ->8
16/08/17 21:07:13 INFO reduce.LocalFetcher: localfetcher#68 about to shuffle output of map attempt_local1944506404_0006_m_000003_0 decomp: 2158626 len: 2158630 to MEMORY
16/08/17 21:07:13 INFO reduce.InMemoryMapOutput: Read 2158626 bytes from map-output for attempt_local1944506404_0006_m_000003_0
16/08/17 21:07:13 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2158626, inMemoryMapOutputs.size() -> 5, commitMemory -> 8, usedMemory ->2158634
16/08/17 21:07:13 INFO reduce.LocalFetcher: localfetcher#68 about to shuffle output of map attempt_local1944506404_0006_m_000002_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:13 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000002_0
16/08/17 21:07:13 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 6, commitMemory -> 2158634, usedMemory ->2158636
16/08/17 21:07:13 INFO reduce.LocalFetcher: localfetcher#68 about to shuffle output of map attempt_local1944506404_0006_m_000015_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:13 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000015_0
16/08/17 21:07:13 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 7, commitMemory -> 2158636, usedMemory ->2158638
16/08/17 21:07:13 INFO reduce.LocalFetcher: localfetcher#68 about to shuffle output of map attempt_local1944506404_0006_m_000014_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:13 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000014_0
16/08/17 21:07:13 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 8, commitMemory -> 2158638, usedMemory ->2158640
16/08/17 21:07:13 INFO reduce.LocalFetcher: localfetcher#68 about to shuffle output of map attempt_local1944506404_0006_m_000001_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:13 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000001_0
16/08/17 21:07:13 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 9, commitMemory -> 2158640, usedMemory ->2158642
16/08/17 21:07:13 INFO reduce.LocalFetcher: localfetcher#68 about to shuffle output of map attempt_local1944506404_0006_m_000008_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:13 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000008_0
16/08/17 21:07:13 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 10, commitMemory -> 2158642, usedMemory ->2158644
16/08/17 21:07:13 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)
	at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
16/08/17 21:07:13 INFO reduce.LocalFetcher: localfetcher#68 about to shuffle output of map attempt_local1944506404_0006_m_000013_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:13 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000013_0
16/08/17 21:07:13 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 11, commitMemory -> 2158644, usedMemory ->2158646
16/08/17 21:07:13 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)
	at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
16/08/17 21:07:13 INFO reduce.LocalFetcher: localfetcher#68 about to shuffle output of map attempt_local1944506404_0006_m_000007_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:13 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000007_0
16/08/17 21:07:13 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 12, commitMemory -> 2158646, usedMemory ->2158648
16/08/17 21:07:13 INFO reduce.LocalFetcher: localfetcher#68 about to shuffle output of map attempt_local1944506404_0006_m_000006_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:13 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000006_0
16/08/17 21:07:13 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 13, commitMemory -> 2158648, usedMemory ->2158650
16/08/17 21:07:13 INFO reduce.LocalFetcher: localfetcher#68 about to shuffle output of map attempt_local1944506404_0006_m_000000_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:13 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000000_0
16/08/17 21:07:13 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 14, commitMemory -> 2158650, usedMemory ->2158652
16/08/17 21:07:13 INFO reduce.LocalFetcher: localfetcher#68 about to shuffle output of map attempt_local1944506404_0006_m_000005_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:13 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000005_0
16/08/17 21:07:13 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 15, commitMemory -> 2158652, usedMemory ->2158654
16/08/17 21:07:13 INFO reduce.LocalFetcher: localfetcher#68 about to shuffle output of map attempt_local1944506404_0006_m_000012_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:13 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000012_0
16/08/17 21:07:13 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 16, commitMemory -> 2158654, usedMemory ->2158656
16/08/17 21:07:13 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:07:13 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:13 INFO reduce.MergeManagerImpl: finalMerge called with 16 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:07:13 INFO mapred.Merger: Merging 16 sorted segments
16/08/17 21:07:13 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 2158612 bytes
16/08/17 21:07:13 INFO reduce.MergeManagerImpl: Merged 16 segments, 2158656 bytes to disk to satisfy reduce memory limit
16/08/17 21:07:13 INFO reduce.MergeManagerImpl: Merging 1 files, 2158630 bytes from disk
16/08/17 21:07:13 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:07:13 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:07:13 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 2158612 bytes
16/08/17 21:07:13 INFO mapred.LocalJobRunner: 16 / 16 copied.
[1000]
[2000]
[3000]
[4000]
[5000]
[6000]
[7000]
[8000]
[9000]
[10000]
16/08/17 21:07:13 INFO mapred.LocalJobRunner: Starting task: attempt_local1944506404_0006_r_000004_0
16/08/17 21:07:13 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:13 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@77e49511
16/08/17 21:07:13 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=361129568, maxSingleShuffleLimit=90282392, mergeThreshold=238345520, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:07:13 INFO reduce.EventFetcher: attempt_local1944506404_0006_r_000004_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:07:13 INFO reduce.LocalFetcher: localfetcher#69 about to shuffle output of map attempt_local1944506404_0006_m_000011_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:13 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000011_0
16/08/17 21:07:13 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->2
16/08/17 21:07:13 INFO reduce.LocalFetcher: localfetcher#69 about to shuffle output of map attempt_local1944506404_0006_m_000010_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:13 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000010_0
16/08/17 21:07:13 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 2, commitMemory -> 2, usedMemory ->4
16/08/17 21:07:13 INFO reduce.LocalFetcher: localfetcher#69 about to shuffle output of map attempt_local1944506404_0006_m_000004_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:13 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000004_0
16/08/17 21:07:13 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 3, commitMemory -> 4, usedMemory ->6
16/08/17 21:07:13 INFO reduce.LocalFetcher: localfetcher#69 about to shuffle output of map attempt_local1944506404_0006_m_000009_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:13 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000009_0
16/08/17 21:07:13 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 4, commitMemory -> 6, usedMemory ->8
16/08/17 21:07:13 INFO reduce.LocalFetcher: localfetcher#69 about to shuffle output of map attempt_local1944506404_0006_m_000003_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:13 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000003_0
16/08/17 21:07:13 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 5, commitMemory -> 8, usedMemory ->10
16/08/17 21:07:13 INFO reduce.LocalFetcher: localfetcher#69 about to shuffle output of map attempt_local1944506404_0006_m_000002_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:13 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000002_0
16/08/17 21:07:13 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 6, commitMemory -> 10, usedMemory ->12
16/08/17 21:07:13 INFO reduce.LocalFetcher: localfetcher#69 about to shuffle output of map attempt_local1944506404_0006_m_000015_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:13 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000015_0
16/08/17 21:07:13 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 7, commitMemory -> 12, usedMemory ->14
16/08/17 21:07:13 INFO reduce.LocalFetcher: localfetcher#69 about to shuffle output of map attempt_local1944506404_0006_m_000014_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:13 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000014_0
16/08/17 21:07:13 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 8, commitMemory -> 14, usedMemory ->16
16/08/17 21:07:13 INFO reduce.LocalFetcher: localfetcher#69 about to shuffle output of map attempt_local1944506404_0006_m_000001_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:13 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000001_0
16/08/17 21:07:13 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 9, commitMemory -> 16, usedMemory ->18
16/08/17 21:07:13 INFO reduce.LocalFetcher: localfetcher#69 about to shuffle output of map attempt_local1944506404_0006_m_000008_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:13 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000008_0
16/08/17 21:07:13 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 10, commitMemory -> 18, usedMemory ->20
16/08/17 21:07:13 INFO reduce.LocalFetcher: localfetcher#69 about to shuffle output of map attempt_local1944506404_0006_m_000013_0 decomp: 2094315 len: 2094319 to MEMORY
16/08/17 21:07:13 INFO reduce.InMemoryMapOutput: Read 2094315 bytes from map-output for attempt_local1944506404_0006_m_000013_0
16/08/17 21:07:13 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2094315, inMemoryMapOutputs.size() -> 11, commitMemory -> 20, usedMemory ->2094335
16/08/17 21:07:13 INFO reduce.LocalFetcher: localfetcher#69 about to shuffle output of map attempt_local1944506404_0006_m_000007_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:13 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000007_0
16/08/17 21:07:13 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 12, commitMemory -> 2094335, usedMemory ->2094337
16/08/17 21:07:13 INFO reduce.LocalFetcher: localfetcher#69 about to shuffle output of map attempt_local1944506404_0006_m_000006_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:13 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000006_0
16/08/17 21:07:13 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 13, commitMemory -> 2094337, usedMemory ->2094339
16/08/17 21:07:13 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)
	at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
16/08/17 21:07:13 INFO reduce.LocalFetcher: localfetcher#69 about to shuffle output of map attempt_local1944506404_0006_m_000000_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:13 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000000_0
16/08/17 21:07:13 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 14, commitMemory -> 2094339, usedMemory ->2094341
16/08/17 21:07:13 INFO reduce.LocalFetcher: localfetcher#69 about to shuffle output of map attempt_local1944506404_0006_m_000005_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:13 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000005_0
16/08/17 21:07:13 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 15, commitMemory -> 2094341, usedMemory ->2094343
16/08/17 21:07:13 INFO reduce.LocalFetcher: localfetcher#69 about to shuffle output of map attempt_local1944506404_0006_m_000012_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:13 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000012_0
16/08/17 21:07:13 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 16, commitMemory -> 2094343, usedMemory ->2094345
16/08/17 21:07:13 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:07:13 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:13 INFO reduce.MergeManagerImpl: finalMerge called with 16 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:07:13 INFO mapred.Merger: Merging 16 sorted segments
16/08/17 21:07:13 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 2094277 bytes
16/08/17 21:07:13 INFO mapred.MapTask: Finished spill 0
16/08/17 21:07:13 INFO mapred.Task: Task:attempt_local518136789_0007_m_000005_0 is done. And is in the process of committing
16/08/17 21:07:13 INFO mapred.LocalJobRunner: map
16/08/17 21:07:13 INFO mapred.Task: Task 'attempt_local518136789_0007_m_000005_0' done.
16/08/17 21:07:13 INFO mapred.LocalJobRunner: Finishing task: attempt_local518136789_0007_m_000005_0
16/08/17 21:07:13 INFO mapred.LocalJobRunner: Starting task: attempt_local518136789_0007_m_000006_0
16/08/17 21:07:13 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:13 INFO mapred.MapTask: Processing split: file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/rules/part-r-00007:0+2442974
16/08/17 21:07:13 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
16/08/17 21:07:13 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
16/08/17 21:07:13 INFO mapred.MapTask: soft limit at 83886080
16/08/17 21:07:13 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
16/08/17 21:07:13 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
16/08/17 21:07:13 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
16/08/17 21:07:13 INFO reduce.MergeManagerImpl: Merged 16 segments, 2094345 bytes to disk to satisfy reduce memory limit
16/08/17 21:07:13 INFO reduce.MergeManagerImpl: Merging 1 files, 2094319 bytes from disk
16/08/17 21:07:13 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:07:13 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:07:13 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 2094277 bytes
16/08/17 21:07:13 INFO mapred.LocalJobRunner: 16 / 16 copied.
[1000]
[2000]
[3000]
[4000]
[5000]
[6000]
[7000]
[8000]
[9000]
[10000]
16/08/17 21:07:13 INFO mapred.LocalJobRunner: Starting task: attempt_local1944506404_0006_r_000005_0
16/08/17 21:07:13 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:13 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@6c0f416e
16/08/17 21:07:13 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=361129568, maxSingleShuffleLimit=90282392, mergeThreshold=238345520, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:07:13 INFO reduce.EventFetcher: attempt_local1944506404_0006_r_000005_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:07:13 INFO mapred.LocalJobRunner: 
16/08/17 21:07:13 INFO mapred.MapTask: Starting flush of map output
16/08/17 21:07:13 INFO mapred.MapTask: Spilling map output
16/08/17 21:07:13 INFO mapred.MapTask: bufstart = 0; bufend = 4645764; bufvoid = 104857600
16/08/17 21:07:13 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 25673800(102695200); length = 540597/6553600
16/08/17 21:07:13 INFO reduce.LocalFetcher: localfetcher#70 about to shuffle output of map attempt_local1944506404_0006_m_000011_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:13 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000011_0
16/08/17 21:07:13 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->2
16/08/17 21:07:13 INFO reduce.LocalFetcher: localfetcher#70 about to shuffle output of map attempt_local1944506404_0006_m_000010_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:13 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000010_0
16/08/17 21:07:13 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 2, commitMemory -> 2, usedMemory ->4
16/08/17 21:07:13 INFO reduce.LocalFetcher: localfetcher#70 about to shuffle output of map attempt_local1944506404_0006_m_000004_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:13 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000004_0
16/08/17 21:07:13 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 3, commitMemory -> 4, usedMemory ->6
16/08/17 21:07:13 INFO reduce.LocalFetcher: localfetcher#70 about to shuffle output of map attempt_local1944506404_0006_m_000009_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:13 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000009_0
16/08/17 21:07:13 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 4, commitMemory -> 6, usedMemory ->8
16/08/17 21:07:13 INFO reduce.LocalFetcher: localfetcher#70 about to shuffle output of map attempt_local1944506404_0006_m_000003_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:13 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000003_0
16/08/17 21:07:13 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 5, commitMemory -> 8, usedMemory ->10
16/08/17 21:07:13 INFO reduce.LocalFetcher: localfetcher#70 about to shuffle output of map attempt_local1944506404_0006_m_000002_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:13 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000002_0
16/08/17 21:07:13 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 6, commitMemory -> 10, usedMemory ->12
16/08/17 21:07:13 INFO reduce.LocalFetcher: localfetcher#70 about to shuffle output of map attempt_local1944506404_0006_m_000015_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:13 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000015_0
16/08/17 21:07:13 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 7, commitMemory -> 12, usedMemory ->14
16/08/17 21:07:13 INFO reduce.LocalFetcher: localfetcher#70 about to shuffle output of map attempt_local1944506404_0006_m_000014_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:13 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000014_0
16/08/17 21:07:13 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 8, commitMemory -> 14, usedMemory ->16
16/08/17 21:07:13 INFO reduce.LocalFetcher: localfetcher#70 about to shuffle output of map attempt_local1944506404_0006_m_000001_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:13 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000001_0
16/08/17 21:07:13 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 9, commitMemory -> 16, usedMemory ->18
16/08/17 21:07:13 INFO reduce.LocalFetcher: localfetcher#70 about to shuffle output of map attempt_local1944506404_0006_m_000008_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:13 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000008_0
16/08/17 21:07:13 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 10, commitMemory -> 18, usedMemory ->20
16/08/17 21:07:13 INFO reduce.LocalFetcher: localfetcher#70 about to shuffle output of map attempt_local1944506404_0006_m_000013_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:13 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000013_0
16/08/17 21:07:13 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 11, commitMemory -> 20, usedMemory ->22
16/08/17 21:07:13 INFO reduce.LocalFetcher: localfetcher#70 about to shuffle output of map attempt_local1944506404_0006_m_000007_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:13 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000007_0
16/08/17 21:07:13 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 12, commitMemory -> 22, usedMemory ->24
16/08/17 21:07:13 INFO reduce.LocalFetcher: localfetcher#70 about to shuffle output of map attempt_local1944506404_0006_m_000006_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:13 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000006_0
16/08/17 21:07:13 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 13, commitMemory -> 24, usedMemory ->26
16/08/17 21:07:13 INFO reduce.LocalFetcher: localfetcher#70 about to shuffle output of map attempt_local1944506404_0006_m_000000_0 decomp: 2166013 len: 2166017 to MEMORY
16/08/17 21:07:13 INFO reduce.InMemoryMapOutput: Read 2166013 bytes from map-output for attempt_local1944506404_0006_m_000000_0
16/08/17 21:07:13 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2166013, inMemoryMapOutputs.size() -> 14, commitMemory -> 26, usedMemory ->2166039
16/08/17 21:07:13 INFO reduce.LocalFetcher: localfetcher#70 about to shuffle output of map attempt_local1944506404_0006_m_000005_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:13 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000005_0
16/08/17 21:07:13 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 15, commitMemory -> 2166039, usedMemory ->2166041
16/08/17 21:07:13 INFO reduce.LocalFetcher: localfetcher#70 about to shuffle output of map attempt_local1944506404_0006_m_000012_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:13 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000012_0
16/08/17 21:07:13 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 16, commitMemory -> 2166041, usedMemory ->2166043
16/08/17 21:07:13 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:07:13 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:13 INFO reduce.MergeManagerImpl: finalMerge called with 16 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:07:13 INFO mapred.Merger: Merging 16 sorted segments
16/08/17 21:07:13 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 2165999 bytes
16/08/17 21:07:14 INFO reduce.MergeManagerImpl: Merged 16 segments, 2166043 bytes to disk to satisfy reduce memory limit
16/08/17 21:07:14 INFO reduce.MergeManagerImpl: Merging 1 files, 2166017 bytes from disk
16/08/17 21:07:14 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:07:14 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:07:14 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 2165999 bytes
16/08/17 21:07:14 INFO mapred.LocalJobRunner: 16 / 16 copied.
[1000]
[2000]
[3000]
[4000]
[5000]
[6000]
[7000]
[8000]
[9000]
[10000]
16/08/17 21:07:14 INFO mapred.LocalJobRunner: Starting task: attempt_local1944506404_0006_r_000006_0
16/08/17 21:07:14 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:14 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@541c1142
16/08/17 21:07:14 INFO mapred.MapTask: Finished spill 0
16/08/17 21:07:14 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=361129568, maxSingleShuffleLimit=90282392, mergeThreshold=238345520, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:07:14 INFO mapred.Task: Task:attempt_local1999544224_0005_m_000005_0 is done. And is in the process of committing
16/08/17 21:07:14 INFO reduce.EventFetcher: attempt_local1944506404_0006_r_000006_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:07:14 INFO mapred.LocalJobRunner: map
16/08/17 21:07:14 INFO mapred.Task: Task 'attempt_local1999544224_0005_m_000005_0' done.
16/08/17 21:07:14 INFO mapred.LocalJobRunner: Finishing task: attempt_local1999544224_0005_m_000005_0
16/08/17 21:07:14 INFO mapred.LocalJobRunner: Starting task: attempt_local1999544224_0005_m_000006_0
16/08/17 21:07:14 INFO reduce.LocalFetcher: localfetcher#71 about to shuffle output of map attempt_local1944506404_0006_m_000011_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:14 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000011_0
16/08/17 21:07:14 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->2
16/08/17 21:07:14 INFO reduce.LocalFetcher: localfetcher#71 about to shuffle output of map attempt_local1944506404_0006_m_000010_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:14 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000010_0
16/08/17 21:07:14 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 2, commitMemory -> 2, usedMemory ->4
16/08/17 21:07:14 INFO reduce.LocalFetcher: localfetcher#71 about to shuffle output of map attempt_local1944506404_0006_m_000004_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:14 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000004_0
16/08/17 21:07:14 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 3, commitMemory -> 4, usedMemory ->6
16/08/17 21:07:14 INFO reduce.LocalFetcher: localfetcher#71 about to shuffle output of map attempt_local1944506404_0006_m_000009_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:14 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000009_0
16/08/17 21:07:14 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 4, commitMemory -> 6, usedMemory ->8
16/08/17 21:07:14 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:14 INFO mapred.MapTask: Processing split: file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/rules/part-r-00007:0+2442974
16/08/17 21:07:14 INFO reduce.LocalFetcher: localfetcher#71 about to shuffle output of map attempt_local1944506404_0006_m_000003_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:14 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000003_0
16/08/17 21:07:14 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 5, commitMemory -> 8, usedMemory ->10
16/08/17 21:07:14 INFO reduce.LocalFetcher: localfetcher#71 about to shuffle output of map attempt_local1944506404_0006_m_000002_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:14 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000002_0
16/08/17 21:07:14 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 6, commitMemory -> 10, usedMemory ->12
16/08/17 21:07:14 INFO reduce.LocalFetcher: localfetcher#71 about to shuffle output of map attempt_local1944506404_0006_m_000015_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:14 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000015_0
16/08/17 21:07:14 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 7, commitMemory -> 12, usedMemory ->14
16/08/17 21:07:14 INFO reduce.LocalFetcher: localfetcher#71 about to shuffle output of map attempt_local1944506404_0006_m_000014_0 decomp: 2090187 len: 2090191 to MEMORY
16/08/17 21:07:14 INFO reduce.InMemoryMapOutput: Read 2090187 bytes from map-output for attempt_local1944506404_0006_m_000014_0
16/08/17 21:07:14 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2090187, inMemoryMapOutputs.size() -> 8, commitMemory -> 14, usedMemory ->2090201
16/08/17 21:07:14 INFO reduce.LocalFetcher: localfetcher#71 about to shuffle output of map attempt_local1944506404_0006_m_000001_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:14 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000001_0
16/08/17 21:07:14 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 9, commitMemory -> 2090201, usedMemory ->2090203
16/08/17 21:07:14 INFO reduce.LocalFetcher: localfetcher#71 about to shuffle output of map attempt_local1944506404_0006_m_000008_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:14 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000008_0
16/08/17 21:07:14 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 10, commitMemory -> 2090203, usedMemory ->2090205
16/08/17 21:07:14 INFO reduce.LocalFetcher: localfetcher#71 about to shuffle output of map attempt_local1944506404_0006_m_000013_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:14 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000013_0
16/08/17 21:07:14 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 11, commitMemory -> 2090205, usedMemory ->2090207
16/08/17 21:07:14 INFO reduce.LocalFetcher: localfetcher#71 about to shuffle output of map attempt_local1944506404_0006_m_000007_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:14 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000007_0
16/08/17 21:07:14 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 12, commitMemory -> 2090207, usedMemory ->2090209
16/08/17 21:07:14 INFO reduce.LocalFetcher: localfetcher#71 about to shuffle output of map attempt_local1944506404_0006_m_000006_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:14 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000006_0
16/08/17 21:07:14 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 13, commitMemory -> 2090209, usedMemory ->2090211
16/08/17 21:07:14 INFO reduce.LocalFetcher: localfetcher#71 about to shuffle output of map attempt_local1944506404_0006_m_000000_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:14 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
16/08/17 21:07:14 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
16/08/17 21:07:14 INFO mapred.MapTask: soft limit at 83886080
16/08/17 21:07:14 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
16/08/17 21:07:14 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
16/08/17 21:07:14 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000000_0
16/08/17 21:07:14 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 14, commitMemory -> 2090211, usedMemory ->2090213
16/08/17 21:07:14 INFO reduce.LocalFetcher: localfetcher#71 about to shuffle output of map attempt_local1944506404_0006_m_000005_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:14 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000005_0
16/08/17 21:07:14 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 15, commitMemory -> 2090213, usedMemory ->2090215
16/08/17 21:07:14 INFO reduce.LocalFetcher: localfetcher#71 about to shuffle output of map attempt_local1944506404_0006_m_000012_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:14 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000012_0
16/08/17 21:07:14 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 16, commitMemory -> 2090215, usedMemory ->2090217
16/08/17 21:07:14 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
16/08/17 21:07:14 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:07:14 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:14 INFO reduce.MergeManagerImpl: finalMerge called with 16 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:07:14 INFO mapred.Merger: Merging 16 sorted segments
16/08/17 21:07:14 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 2090173 bytes
16/08/17 21:07:14 INFO reduce.MergeManagerImpl: Merged 16 segments, 2090217 bytes to disk to satisfy reduce memory limit
16/08/17 21:07:14 INFO reduce.MergeManagerImpl: Merging 1 files, 2090191 bytes from disk
16/08/17 21:07:14 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:07:14 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:07:14 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 2090173 bytes
16/08/17 21:07:14 INFO mapred.LocalJobRunner: 16 / 16 copied.
[1000]
[2000]
[3000]
[4000]
[5000]
[6000]
[7000]
[8000]
[9000]
[10000]
16/08/17 21:07:14 INFO mapred.LocalJobRunner: Starting task: attempt_local1944506404_0006_r_000007_0
16/08/17 21:07:14 INFO mapred.LocalJobRunner: 
16/08/17 21:07:14 INFO mapred.MapTask: Starting flush of map output
16/08/17 21:07:14 INFO mapred.MapTask: Spilling map output
16/08/17 21:07:14 INFO mapred.MapTask: bufstart = 0; bufend = 4737736; bufvoid = 104857600
16/08/17 21:07:14 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 25673800(102695200); length = 540597/6553600
16/08/17 21:07:14 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:14 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@4885004e
16/08/17 21:07:14 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=362964576, maxSingleShuffleLimit=90741144, mergeThreshold=239556624, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:07:14 INFO reduce.EventFetcher: attempt_local1944506404_0006_r_000007_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:07:14 INFO reduce.LocalFetcher: localfetcher#72 about to shuffle output of map attempt_local1944506404_0006_m_000011_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:14 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000011_0
16/08/17 21:07:14 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->2
16/08/17 21:07:14 INFO reduce.LocalFetcher: localfetcher#72 about to shuffle output of map attempt_local1944506404_0006_m_000010_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:14 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000010_0
16/08/17 21:07:14 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 2, commitMemory -> 2, usedMemory ->4
16/08/17 21:07:14 INFO reduce.LocalFetcher: localfetcher#72 about to shuffle output of map attempt_local1944506404_0006_m_000004_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:14 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000004_0
16/08/17 21:07:14 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 3, commitMemory -> 4, usedMemory ->6
16/08/17 21:07:14 INFO reduce.LocalFetcher: localfetcher#72 about to shuffle output of map attempt_local1944506404_0006_m_000009_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:14 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000009_0
16/08/17 21:07:14 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 4, commitMemory -> 6, usedMemory ->8
16/08/17 21:07:14 INFO reduce.LocalFetcher: localfetcher#72 about to shuffle output of map attempt_local1944506404_0006_m_000003_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:14 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000003_0
16/08/17 21:07:14 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 5, commitMemory -> 8, usedMemory ->10
16/08/17 21:07:14 INFO reduce.LocalFetcher: localfetcher#72 about to shuffle output of map attempt_local1944506404_0006_m_000002_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:14 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000002_0
16/08/17 21:07:14 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 6, commitMemory -> 10, usedMemory ->12
16/08/17 21:07:14 INFO reduce.LocalFetcher: localfetcher#72 about to shuffle output of map attempt_local1944506404_0006_m_000015_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:14 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000015_0
16/08/17 21:07:14 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 7, commitMemory -> 12, usedMemory ->14
16/08/17 21:07:14 INFO reduce.LocalFetcher: localfetcher#72 about to shuffle output of map attempt_local1944506404_0006_m_000014_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:14 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000014_0
16/08/17 21:07:14 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 8, commitMemory -> 14, usedMemory ->16
16/08/17 21:07:14 INFO reduce.LocalFetcher: localfetcher#72 about to shuffle output of map attempt_local1944506404_0006_m_000001_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:14 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000001_0
16/08/17 21:07:14 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 9, commitMemory -> 16, usedMemory ->18
16/08/17 21:07:14 INFO reduce.LocalFetcher: localfetcher#72 about to shuffle output of map attempt_local1944506404_0006_m_000008_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:14 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000008_0
16/08/17 21:07:14 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 10, commitMemory -> 18, usedMemory ->20
16/08/17 21:07:14 INFO reduce.LocalFetcher: localfetcher#72 about to shuffle output of map attempt_local1944506404_0006_m_000013_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:14 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000013_0
16/08/17 21:07:14 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 11, commitMemory -> 20, usedMemory ->22
16/08/17 21:07:14 INFO reduce.LocalFetcher: localfetcher#72 about to shuffle output of map attempt_local1944506404_0006_m_000007_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:14 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000007_0
16/08/17 21:07:14 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 12, commitMemory -> 22, usedMemory ->24
16/08/17 21:07:14 INFO reduce.LocalFetcher: localfetcher#72 about to shuffle output of map attempt_local1944506404_0006_m_000006_0 decomp: 2148704 len: 2148708 to MEMORY
16/08/17 21:07:14 INFO reduce.InMemoryMapOutput: Read 2148704 bytes from map-output for attempt_local1944506404_0006_m_000006_0
16/08/17 21:07:14 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2148704, inMemoryMapOutputs.size() -> 13, commitMemory -> 24, usedMemory ->2148728
16/08/17 21:07:14 INFO reduce.LocalFetcher: localfetcher#72 about to shuffle output of map attempt_local1944506404_0006_m_000000_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:14 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000000_0
16/08/17 21:07:14 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 14, commitMemory -> 2148728, usedMemory ->2148730
16/08/17 21:07:14 INFO reduce.LocalFetcher: localfetcher#72 about to shuffle output of map attempt_local1944506404_0006_m_000005_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:14 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000005_0
16/08/17 21:07:14 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 15, commitMemory -> 2148730, usedMemory ->2148732
16/08/17 21:07:14 INFO reduce.LocalFetcher: localfetcher#72 about to shuffle output of map attempt_local1944506404_0006_m_000012_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:14 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000012_0
16/08/17 21:07:14 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 16, commitMemory -> 2148732, usedMemory ->2148734
16/08/17 21:07:14 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:07:14 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:14 INFO reduce.MergeManagerImpl: finalMerge called with 16 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:07:14 INFO mapred.Merger: Merging 16 sorted segments
16/08/17 21:07:14 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 2148674 bytes
16/08/17 21:07:14 INFO mapred.MapTask: Finished spill 0
16/08/17 21:07:14 INFO mapred.Task: Task:attempt_local518136789_0007_m_000006_0 is done. And is in the process of committing
16/08/17 21:07:14 INFO mapred.LocalJobRunner: map
16/08/17 21:07:14 INFO mapred.Task: Task 'attempt_local518136789_0007_m_000006_0' done.
16/08/17 21:07:14 INFO mapred.LocalJobRunner: Finishing task: attempt_local518136789_0007_m_000006_0
16/08/17 21:07:14 INFO mapred.LocalJobRunner: Starting task: attempt_local518136789_0007_m_000007_0
16/08/17 21:07:14 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:14 INFO reduce.MergeManagerImpl: Merged 16 segments, 2148734 bytes to disk to satisfy reduce memory limit
16/08/17 21:07:14 INFO mapred.MapTask: Processing split: file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/rules/part-r-00011:0+2441386
16/08/17 21:07:14 INFO reduce.MergeManagerImpl: Merging 1 files, 2148708 bytes from disk
16/08/17 21:07:14 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:07:14 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:07:14 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 2148674 bytes
16/08/17 21:07:14 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:14 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
16/08/17 21:07:14 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
16/08/17 21:07:14 INFO mapred.MapTask: soft limit at 83886080
16/08/17 21:07:14 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
16/08/17 21:07:14 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
16/08/17 21:07:14 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
[1000]
[2000]
[3000]
[4000]
[5000]
[6000]
[7000]
[8000]
[9000]
[10000]
16/08/17 21:07:15 INFO mapred.LocalJobRunner: Starting task: attempt_local1944506404_0006_r_000008_0
16/08/17 21:07:15 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:15 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@61bca2b0
16/08/17 21:07:15 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=365166592, maxSingleShuffleLimit=91291648, mergeThreshold=241009968, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:07:15 INFO reduce.EventFetcher: attempt_local1944506404_0006_r_000008_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:07:15 INFO reduce.LocalFetcher: localfetcher#73 about to shuffle output of map attempt_local1944506404_0006_m_000011_0 decomp: 2101078 len: 2101082 to MEMORY
16/08/17 21:07:15 INFO reduce.InMemoryMapOutput: Read 2101078 bytes from map-output for attempt_local1944506404_0006_m_000011_0
16/08/17 21:07:15 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2101078, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->2101078
16/08/17 21:07:15 INFO reduce.LocalFetcher: localfetcher#73 about to shuffle output of map attempt_local1944506404_0006_m_000010_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:15 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000010_0
16/08/17 21:07:15 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 2, commitMemory -> 2101078, usedMemory ->2101080
16/08/17 21:07:15 INFO reduce.LocalFetcher: localfetcher#73 about to shuffle output of map attempt_local1944506404_0006_m_000004_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:15 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000004_0
16/08/17 21:07:15 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 3, commitMemory -> 2101080, usedMemory ->2101082
16/08/17 21:07:15 INFO reduce.LocalFetcher: localfetcher#73 about to shuffle output of map attempt_local1944506404_0006_m_000009_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:15 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000009_0
16/08/17 21:07:15 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 4, commitMemory -> 2101082, usedMemory ->2101084
16/08/17 21:07:15 INFO reduce.LocalFetcher: localfetcher#73 about to shuffle output of map attempt_local1944506404_0006_m_000003_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:15 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000003_0
16/08/17 21:07:15 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 5, commitMemory -> 2101084, usedMemory ->2101086
16/08/17 21:07:15 INFO reduce.LocalFetcher: localfetcher#73 about to shuffle output of map attempt_local1944506404_0006_m_000002_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:15 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000002_0
16/08/17 21:07:15 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 6, commitMemory -> 2101086, usedMemory ->2101088
16/08/17 21:07:15 INFO reduce.LocalFetcher: localfetcher#73 about to shuffle output of map attempt_local1944506404_0006_m_000015_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:15 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000015_0
16/08/17 21:07:15 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 7, commitMemory -> 2101088, usedMemory ->2101090
16/08/17 21:07:15 INFO reduce.LocalFetcher: localfetcher#73 about to shuffle output of map attempt_local1944506404_0006_m_000014_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:15 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000014_0
16/08/17 21:07:15 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 8, commitMemory -> 2101090, usedMemory ->2101092
16/08/17 21:07:15 INFO reduce.LocalFetcher: localfetcher#73 about to shuffle output of map attempt_local1944506404_0006_m_000001_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:15 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000001_0
16/08/17 21:07:15 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 9, commitMemory -> 2101092, usedMemory ->2101094
16/08/17 21:07:15 INFO reduce.LocalFetcher: localfetcher#73 about to shuffle output of map attempt_local1944506404_0006_m_000008_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:15 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000008_0
16/08/17 21:07:15 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 10, commitMemory -> 2101094, usedMemory ->2101096
16/08/17 21:07:15 INFO reduce.LocalFetcher: localfetcher#73 about to shuffle output of map attempt_local1944506404_0006_m_000013_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:15 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000013_0
16/08/17 21:07:15 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 11, commitMemory -> 2101096, usedMemory ->2101098
16/08/17 21:07:15 INFO reduce.LocalFetcher: localfetcher#73 about to shuffle output of map attempt_local1944506404_0006_m_000007_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:15 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000007_0
16/08/17 21:07:15 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 12, commitMemory -> 2101098, usedMemory ->2101100
16/08/17 21:07:15 INFO reduce.LocalFetcher: localfetcher#73 about to shuffle output of map attempt_local1944506404_0006_m_000006_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:15 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000006_0
16/08/17 21:07:15 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 13, commitMemory -> 2101100, usedMemory ->2101102
16/08/17 21:07:15 INFO reduce.LocalFetcher: localfetcher#73 about to shuffle output of map attempt_local1944506404_0006_m_000000_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:15 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000000_0
16/08/17 21:07:15 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 14, commitMemory -> 2101102, usedMemory ->2101104
16/08/17 21:07:15 INFO reduce.LocalFetcher: localfetcher#73 about to shuffle output of map attempt_local1944506404_0006_m_000005_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:15 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000005_0
16/08/17 21:07:15 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 15, commitMemory -> 2101104, usedMemory ->2101106
16/08/17 21:07:15 INFO reduce.LocalFetcher: localfetcher#73 about to shuffle output of map attempt_local1944506404_0006_m_000012_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:15 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000012_0
16/08/17 21:07:15 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 16, commitMemory -> 2101106, usedMemory ->2101108
16/08/17 21:07:15 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:07:15 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:15 INFO reduce.MergeManagerImpl: finalMerge called with 16 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:07:15 INFO mapred.Merger: Merging 16 sorted segments
16/08/17 21:07:15 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 2101048 bytes
16/08/17 21:07:15 INFO reduce.MergeManagerImpl: Merged 16 segments, 2101108 bytes to disk to satisfy reduce memory limit
16/08/17 21:07:15 INFO reduce.MergeManagerImpl: Merging 1 files, 2101082 bytes from disk
16/08/17 21:07:15 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:07:15 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:07:15 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 2101048 bytes
16/08/17 21:07:15 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:15 INFO mapred.LocalJobRunner: 
16/08/17 21:07:15 INFO mapred.MapTask: Starting flush of map output
16/08/17 21:07:15 INFO mapred.MapTask: Spilling map output
16/08/17 21:07:15 INFO mapred.MapTask: bufstart = 0; bufend = 4644376; bufvoid = 104857600
16/08/17 21:07:15 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 25675300(102701200); length = 539097/6553600
[1000]
[2000]
[3000]
[4000]
[5000]
[6000]
[7000]
[8000]
[9000]
[10000]
16/08/17 21:07:15 INFO mapred.LocalJobRunner: Starting task: attempt_local1944506404_0006_r_000009_0
16/08/17 21:07:15 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:15 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@665d8e0b
16/08/17 21:07:15 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=365166592, maxSingleShuffleLimit=91291648, mergeThreshold=241009968, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:07:15 INFO reduce.EventFetcher: attempt_local1944506404_0006_r_000009_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:07:15 INFO reduce.LocalFetcher: localfetcher#74 about to shuffle output of map attempt_local1944506404_0006_m_000011_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:15 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000011_0
16/08/17 21:07:15 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->2
16/08/17 21:07:15 INFO reduce.LocalFetcher: localfetcher#74 about to shuffle output of map attempt_local1944506404_0006_m_000010_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:15 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000010_0
16/08/17 21:07:15 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 2, commitMemory -> 2, usedMemory ->4
16/08/17 21:07:15 INFO reduce.LocalFetcher: localfetcher#74 about to shuffle output of map attempt_local1944506404_0006_m_000004_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:15 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000004_0
16/08/17 21:07:15 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 3, commitMemory -> 4, usedMemory ->6
16/08/17 21:07:15 INFO reduce.LocalFetcher: localfetcher#74 about to shuffle output of map attempt_local1944506404_0006_m_000009_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:15 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000009_0
16/08/17 21:07:15 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 4, commitMemory -> 6, usedMemory ->8
16/08/17 21:07:15 INFO reduce.LocalFetcher: localfetcher#74 about to shuffle output of map attempt_local1944506404_0006_m_000003_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:15 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000003_0
16/08/17 21:07:15 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 5, commitMemory -> 8, usedMemory ->10
16/08/17 21:07:15 INFO reduce.LocalFetcher: localfetcher#74 about to shuffle output of map attempt_local1944506404_0006_m_000002_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:15 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000002_0
16/08/17 21:07:15 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 6, commitMemory -> 10, usedMemory ->12
16/08/17 21:07:15 INFO reduce.LocalFetcher: localfetcher#74 about to shuffle output of map attempt_local1944506404_0006_m_000015_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:15 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000015_0
16/08/17 21:07:15 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 7, commitMemory -> 12, usedMemory ->14
16/08/17 21:07:15 INFO reduce.LocalFetcher: localfetcher#74 about to shuffle output of map attempt_local1944506404_0006_m_000014_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:15 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000014_0
16/08/17 21:07:15 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 8, commitMemory -> 14, usedMemory ->16
16/08/17 21:07:15 INFO reduce.LocalFetcher: localfetcher#74 about to shuffle output of map attempt_local1944506404_0006_m_000001_0 decomp: 2160062 len: 2160066 to MEMORY
16/08/17 21:07:15 INFO reduce.InMemoryMapOutput: Read 2160062 bytes from map-output for attempt_local1944506404_0006_m_000001_0
16/08/17 21:07:15 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2160062, inMemoryMapOutputs.size() -> 9, commitMemory -> 16, usedMemory ->2160078
16/08/17 21:07:15 INFO reduce.LocalFetcher: localfetcher#74 about to shuffle output of map attempt_local1944506404_0006_m_000008_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:15 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000008_0
16/08/17 21:07:15 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 10, commitMemory -> 2160078, usedMemory ->2160080
16/08/17 21:07:15 INFO reduce.LocalFetcher: localfetcher#74 about to shuffle output of map attempt_local1944506404_0006_m_000013_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:15 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000013_0
16/08/17 21:07:15 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 11, commitMemory -> 2160080, usedMemory ->2160082
16/08/17 21:07:15 INFO reduce.LocalFetcher: localfetcher#74 about to shuffle output of map attempt_local1944506404_0006_m_000007_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:15 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000007_0
16/08/17 21:07:15 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 12, commitMemory -> 2160082, usedMemory ->2160084
16/08/17 21:07:15 INFO reduce.LocalFetcher: localfetcher#74 about to shuffle output of map attempt_local1944506404_0006_m_000006_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:15 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000006_0
16/08/17 21:07:15 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 13, commitMemory -> 2160084, usedMemory ->2160086
16/08/17 21:07:15 INFO reduce.LocalFetcher: localfetcher#74 about to shuffle output of map attempt_local1944506404_0006_m_000000_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:15 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000000_0
16/08/17 21:07:15 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 14, commitMemory -> 2160086, usedMemory ->2160088
16/08/17 21:07:15 INFO reduce.LocalFetcher: localfetcher#74 about to shuffle output of map attempt_local1944506404_0006_m_000005_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:15 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000005_0
16/08/17 21:07:15 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 15, commitMemory -> 2160088, usedMemory ->2160090
16/08/17 21:07:15 INFO reduce.LocalFetcher: localfetcher#74 about to shuffle output of map attempt_local1944506404_0006_m_000012_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:15 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000012_0
16/08/17 21:07:15 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 16, commitMemory -> 2160090, usedMemory ->2160092
16/08/17 21:07:15 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:07:15 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:15 INFO reduce.MergeManagerImpl: finalMerge called with 16 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:07:15 INFO mapred.Merger: Merging 16 sorted segments
16/08/17 21:07:15 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 2160032 bytes
16/08/17 21:07:15 INFO reduce.MergeManagerImpl: Merged 16 segments, 2160092 bytes to disk to satisfy reduce memory limit
16/08/17 21:07:15 INFO reduce.MergeManagerImpl: Merging 1 files, 2160066 bytes from disk
16/08/17 21:07:15 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:07:15 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:07:15 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 2160032 bytes
16/08/17 21:07:15 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:15 INFO mapred.MapTask: Finished spill 0
16/08/17 21:07:15 INFO mapred.Task: Task:attempt_local1999544224_0005_m_000006_0 is done. And is in the process of committing
16/08/17 21:07:15 INFO mapred.LocalJobRunner: map
16/08/17 21:07:15 INFO mapred.Task: Task 'attempt_local1999544224_0005_m_000006_0' done.
16/08/17 21:07:15 INFO mapred.LocalJobRunner: Finishing task: attempt_local1999544224_0005_m_000006_0
16/08/17 21:07:15 INFO mapred.LocalJobRunner: Starting task: attempt_local1999544224_0005_m_000007_0
16/08/17 21:07:15 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:15 INFO mapred.MapTask: Processing split: file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/rules/part-r-00011:0+2441386
16/08/17 21:07:15 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
16/08/17 21:07:15 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
16/08/17 21:07:15 INFO mapred.MapTask: soft limit at 83886080
16/08/17 21:07:15 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
16/08/17 21:07:15 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
16/08/17 21:07:15 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
[1000]
[2000]
[3000]
[4000]
[5000]
[6000]
[7000]
[8000]
[9000]
[10000]
16/08/17 21:07:16 INFO mapred.LocalJobRunner: Starting task: attempt_local1944506404_0006_r_000010_0
16/08/17 21:07:16 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:16 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@46304afe
16/08/17 21:07:16 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=365166592, maxSingleShuffleLimit=91291648, mergeThreshold=241009968, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:07:16 INFO reduce.EventFetcher: attempt_local1944506404_0006_r_000010_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:07:16 INFO reduce.LocalFetcher: localfetcher#75 about to shuffle output of map attempt_local1944506404_0006_m_000011_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:16 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000011_0
16/08/17 21:07:16 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->2
16/08/17 21:07:16 INFO reduce.LocalFetcher: localfetcher#75 about to shuffle output of map attempt_local1944506404_0006_m_000010_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:16 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000010_0
16/08/17 21:07:16 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 2, commitMemory -> 2, usedMemory ->4
16/08/17 21:07:16 INFO reduce.LocalFetcher: localfetcher#75 about to shuffle output of map attempt_local1944506404_0006_m_000004_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:16 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000004_0
16/08/17 21:07:16 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 3, commitMemory -> 4, usedMemory ->6
16/08/17 21:07:16 INFO reduce.LocalFetcher: localfetcher#75 about to shuffle output of map attempt_local1944506404_0006_m_000009_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:16 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000009_0
16/08/17 21:07:16 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 4, commitMemory -> 6, usedMemory ->8
16/08/17 21:07:16 INFO reduce.LocalFetcher: localfetcher#75 about to shuffle output of map attempt_local1944506404_0006_m_000003_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:16 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000003_0
16/08/17 21:07:16 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 5, commitMemory -> 8, usedMemory ->10
16/08/17 21:07:16 INFO reduce.LocalFetcher: localfetcher#75 about to shuffle output of map attempt_local1944506404_0006_m_000002_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:16 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000002_0
16/08/17 21:07:16 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 6, commitMemory -> 10, usedMemory ->12
16/08/17 21:07:16 INFO reduce.LocalFetcher: localfetcher#75 about to shuffle output of map attempt_local1944506404_0006_m_000015_0 decomp: 2080312 len: 2080316 to MEMORY
16/08/17 21:07:16 INFO reduce.InMemoryMapOutput: Read 2080312 bytes from map-output for attempt_local1944506404_0006_m_000015_0
16/08/17 21:07:16 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2080312, inMemoryMapOutputs.size() -> 7, commitMemory -> 12, usedMemory ->2080324
16/08/17 21:07:16 INFO reduce.LocalFetcher: localfetcher#75 about to shuffle output of map attempt_local1944506404_0006_m_000014_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:16 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000014_0
16/08/17 21:07:16 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 8, commitMemory -> 2080324, usedMemory ->2080326
16/08/17 21:07:16 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)
	at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
16/08/17 21:07:16 INFO reduce.LocalFetcher: localfetcher#75 about to shuffle output of map attempt_local1944506404_0006_m_000001_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:16 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000001_0
16/08/17 21:07:16 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 9, commitMemory -> 2080326, usedMemory ->2080328
16/08/17 21:07:16 INFO reduce.LocalFetcher: localfetcher#75 about to shuffle output of map attempt_local1944506404_0006_m_000008_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:16 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000008_0
16/08/17 21:07:16 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 10, commitMemory -> 2080328, usedMemory ->2080330
16/08/17 21:07:16 INFO reduce.LocalFetcher: localfetcher#75 about to shuffle output of map attempt_local1944506404_0006_m_000013_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:16 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000013_0
16/08/17 21:07:16 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 11, commitMemory -> 2080330, usedMemory ->2080332
16/08/17 21:07:16 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)
	at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
16/08/17 21:07:16 INFO reduce.LocalFetcher: localfetcher#75 about to shuffle output of map attempt_local1944506404_0006_m_000007_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:16 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000007_0
16/08/17 21:07:16 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 12, commitMemory -> 2080332, usedMemory ->2080334
16/08/17 21:07:16 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)
	at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
16/08/17 21:07:16 INFO reduce.LocalFetcher: localfetcher#75 about to shuffle output of map attempt_local1944506404_0006_m_000006_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:16 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000006_0
16/08/17 21:07:16 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 13, commitMemory -> 2080334, usedMemory ->2080336
16/08/17 21:07:16 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)
	at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
16/08/17 21:07:16 INFO reduce.LocalFetcher: localfetcher#75 about to shuffle output of map attempt_local1944506404_0006_m_000000_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:16 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000000_0
16/08/17 21:07:16 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 14, commitMemory -> 2080336, usedMemory ->2080338
16/08/17 21:07:16 INFO reduce.LocalFetcher: localfetcher#75 about to shuffle output of map attempt_local1944506404_0006_m_000005_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:16 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000005_0
16/08/17 21:07:16 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 15, commitMemory -> 2080338, usedMemory ->2080340
16/08/17 21:07:16 INFO reduce.LocalFetcher: localfetcher#75 about to shuffle output of map attempt_local1944506404_0006_m_000012_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:16 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000012_0
16/08/17 21:07:16 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 16, commitMemory -> 2080340, usedMemory ->2080342
16/08/17 21:07:16 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:07:16 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:16 INFO reduce.MergeManagerImpl: finalMerge called with 16 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:07:16 INFO mapred.Merger: Merging 16 sorted segments
16/08/17 21:07:16 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 2080294 bytes
16/08/17 21:07:16 INFO reduce.MergeManagerImpl: Merged 16 segments, 2080342 bytes to disk to satisfy reduce memory limit
16/08/17 21:07:16 INFO reduce.MergeManagerImpl: Merging 1 files, 2080316 bytes from disk
16/08/17 21:07:16 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:07:16 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:07:16 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 2080294 bytes
16/08/17 21:07:16 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:16 INFO mapred.LocalJobRunner: 
16/08/17 21:07:16 INFO mapred.MapTask: Starting flush of map output
16/08/17 21:07:16 INFO mapred.MapTask: Spilling map output
16/08/17 21:07:16 INFO mapred.MapTask: bufstart = 0; bufend = 4734624; bufvoid = 104857600
16/08/17 21:07:16 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 25675300(102701200); length = 539097/6553600
[1000]
[2000]
[3000]
[4000]
16/08/17 21:07:16 INFO mapred.MapTask: Finished spill 0
[5000]
16/08/17 21:07:16 INFO mapred.Task: Task:attempt_local518136789_0007_m_000007_0 is done. And is in the process of committing
16/08/17 21:07:16 INFO mapred.LocalJobRunner: map
16/08/17 21:07:16 INFO mapred.Task: Task 'attempt_local518136789_0007_m_000007_0' done.
16/08/17 21:07:16 INFO mapred.LocalJobRunner: Finishing task: attempt_local518136789_0007_m_000007_0
16/08/17 21:07:16 INFO mapred.LocalJobRunner: Starting task: attempt_local518136789_0007_m_000008_0
[6000]
16/08/17 21:07:16 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:16 INFO mapred.MapTask: Processing split: file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/rules/part-r-00002:0+2395166
16/08/17 21:07:16 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
16/08/17 21:07:16 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
16/08/17 21:07:16 INFO mapred.MapTask: soft limit at 83886080
16/08/17 21:07:16 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
16/08/17 21:07:16 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
[7000]
[8000]
[9000]
16/08/17 21:07:16 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
[10000]
16/08/17 21:07:16 INFO mapred.LocalJobRunner: Starting task: attempt_local1944506404_0006_r_000011_0
16/08/17 21:07:16 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:16 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@1937b71c
16/08/17 21:07:16 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=370304608, maxSingleShuffleLimit=92576152, mergeThreshold=244401056, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:07:16 INFO reduce.EventFetcher: attempt_local1944506404_0006_r_000011_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:07:16 INFO reduce.LocalFetcher: localfetcher#76 about to shuffle output of map attempt_local1944506404_0006_m_000011_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:16 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000011_0
16/08/17 21:07:16 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->2
16/08/17 21:07:16 INFO reduce.LocalFetcher: localfetcher#76 about to shuffle output of map attempt_local1944506404_0006_m_000010_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:16 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000010_0
16/08/17 21:07:16 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 2, commitMemory -> 2, usedMemory ->4
16/08/17 21:07:16 INFO reduce.LocalFetcher: localfetcher#76 about to shuffle output of map attempt_local1944506404_0006_m_000004_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:16 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000004_0
16/08/17 21:07:16 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 3, commitMemory -> 4, usedMemory ->6
16/08/17 21:07:16 INFO reduce.LocalFetcher: localfetcher#76 about to shuffle output of map attempt_local1944506404_0006_m_000009_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:16 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000009_0
16/08/17 21:07:16 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 4, commitMemory -> 6, usedMemory ->8
16/08/17 21:07:16 INFO reduce.LocalFetcher: localfetcher#76 about to shuffle output of map attempt_local1944506404_0006_m_000003_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:16 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000003_0
16/08/17 21:07:16 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 5, commitMemory -> 8, usedMemory ->10
16/08/17 21:07:16 INFO reduce.LocalFetcher: localfetcher#76 about to shuffle output of map attempt_local1944506404_0006_m_000002_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:16 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000002_0
16/08/17 21:07:16 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 6, commitMemory -> 10, usedMemory ->12
16/08/17 21:07:16 INFO reduce.LocalFetcher: localfetcher#76 about to shuffle output of map attempt_local1944506404_0006_m_000015_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:16 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000015_0
16/08/17 21:07:16 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 7, commitMemory -> 12, usedMemory ->14
16/08/17 21:07:16 INFO reduce.LocalFetcher: localfetcher#76 about to shuffle output of map attempt_local1944506404_0006_m_000014_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:16 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000014_0
16/08/17 21:07:16 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 8, commitMemory -> 14, usedMemory ->16
16/08/17 21:07:16 INFO reduce.LocalFetcher: localfetcher#76 about to shuffle output of map attempt_local1944506404_0006_m_000001_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:16 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000001_0
16/08/17 21:07:16 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 9, commitMemory -> 16, usedMemory ->18
16/08/17 21:07:16 INFO reduce.LocalFetcher: localfetcher#76 about to shuffle output of map attempt_local1944506404_0006_m_000008_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:16 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000008_0
16/08/17 21:07:16 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 10, commitMemory -> 18, usedMemory ->20
16/08/17 21:07:16 INFO reduce.LocalFetcher: localfetcher#76 about to shuffle output of map attempt_local1944506404_0006_m_000013_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:16 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000013_0
16/08/17 21:07:16 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 11, commitMemory -> 20, usedMemory ->22
16/08/17 21:07:16 INFO reduce.LocalFetcher: localfetcher#76 about to shuffle output of map attempt_local1944506404_0006_m_000007_0 decomp: 2147886 len: 2147890 to MEMORY
16/08/17 21:07:16 INFO reduce.InMemoryMapOutput: Read 2147886 bytes from map-output for attempt_local1944506404_0006_m_000007_0
16/08/17 21:07:16 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2147886, inMemoryMapOutputs.size() -> 12, commitMemory -> 22, usedMemory ->2147908
16/08/17 21:07:16 INFO reduce.LocalFetcher: localfetcher#76 about to shuffle output of map attempt_local1944506404_0006_m_000006_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:16 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000006_0
16/08/17 21:07:16 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 13, commitMemory -> 2147908, usedMemory ->2147910
16/08/17 21:07:16 INFO reduce.LocalFetcher: localfetcher#76 about to shuffle output of map attempt_local1944506404_0006_m_000000_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:16 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000000_0
16/08/17 21:07:16 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 14, commitMemory -> 2147910, usedMemory ->2147912
16/08/17 21:07:16 INFO reduce.LocalFetcher: localfetcher#76 about to shuffle output of map attempt_local1944506404_0006_m_000005_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:16 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000005_0
16/08/17 21:07:16 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 15, commitMemory -> 2147912, usedMemory ->2147914
16/08/17 21:07:16 INFO reduce.LocalFetcher: localfetcher#76 about to shuffle output of map attempt_local1944506404_0006_m_000012_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:16 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000012_0
16/08/17 21:07:16 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 16, commitMemory -> 2147914, usedMemory ->2147916
16/08/17 21:07:16 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:07:16 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:16 INFO reduce.MergeManagerImpl: finalMerge called with 16 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:07:16 INFO mapred.Merger: Merging 16 sorted segments
16/08/17 21:07:16 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 2147856 bytes
16/08/17 21:07:16 INFO reduce.MergeManagerImpl: Merged 16 segments, 2147916 bytes to disk to satisfy reduce memory limit
16/08/17 21:07:16 INFO reduce.MergeManagerImpl: Merging 1 files, 2147890 bytes from disk
16/08/17 21:07:16 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:07:16 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:07:16 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 2147856 bytes
16/08/17 21:07:16 INFO mapred.LocalJobRunner: 16 / 16 copied.
[1000]
[2000]
[3000]
[4000]
[5000]
[6000]
[7000]
[8000]
[9000]
[10000]
16/08/17 21:07:16 INFO mapred.LocalJobRunner: Starting task: attempt_local1944506404_0006_r_000012_0
16/08/17 21:07:16 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:16 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@716c1a49
16/08/17 21:07:16 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=370304608, maxSingleShuffleLimit=92576152, mergeThreshold=244401056, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:07:16 INFO reduce.EventFetcher: attempt_local1944506404_0006_r_000012_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:07:16 INFO reduce.LocalFetcher: localfetcher#77 about to shuffle output of map attempt_local1944506404_0006_m_000011_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:16 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000011_0
16/08/17 21:07:16 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->2
16/08/17 21:07:16 INFO reduce.LocalFetcher: localfetcher#77 about to shuffle output of map attempt_local1944506404_0006_m_000010_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:16 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000010_0
16/08/17 21:07:16 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 2, commitMemory -> 2, usedMemory ->4
16/08/17 21:07:16 INFO reduce.LocalFetcher: localfetcher#77 about to shuffle output of map attempt_local1944506404_0006_m_000004_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:16 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000004_0
16/08/17 21:07:16 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 3, commitMemory -> 4, usedMemory ->6
16/08/17 21:07:16 INFO reduce.LocalFetcher: localfetcher#77 about to shuffle output of map attempt_local1944506404_0006_m_000009_0 decomp: 2103459 len: 2103463 to MEMORY
16/08/17 21:07:16 INFO reduce.InMemoryMapOutput: Read 2103459 bytes from map-output for attempt_local1944506404_0006_m_000009_0
16/08/17 21:07:16 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2103459, inMemoryMapOutputs.size() -> 4, commitMemory -> 6, usedMemory ->2103465
16/08/17 21:07:16 INFO reduce.LocalFetcher: localfetcher#77 about to shuffle output of map attempt_local1944506404_0006_m_000003_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:16 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000003_0
16/08/17 21:07:16 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 5, commitMemory -> 2103465, usedMemory ->2103467
16/08/17 21:07:16 INFO reduce.LocalFetcher: localfetcher#77 about to shuffle output of map attempt_local1944506404_0006_m_000002_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:16 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000002_0
16/08/17 21:07:16 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 6, commitMemory -> 2103467, usedMemory ->2103469
16/08/17 21:07:16 INFO reduce.LocalFetcher: localfetcher#77 about to shuffle output of map attempt_local1944506404_0006_m_000015_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:16 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000015_0
16/08/17 21:07:16 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 7, commitMemory -> 2103469, usedMemory ->2103471
16/08/17 21:07:16 INFO reduce.LocalFetcher: localfetcher#77 about to shuffle output of map attempt_local1944506404_0006_m_000014_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:16 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000014_0
16/08/17 21:07:16 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 8, commitMemory -> 2103471, usedMemory ->2103473
16/08/17 21:07:16 INFO reduce.LocalFetcher: localfetcher#77 about to shuffle output of map attempt_local1944506404_0006_m_000001_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:16 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000001_0
16/08/17 21:07:16 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 9, commitMemory -> 2103473, usedMemory ->2103475
16/08/17 21:07:16 INFO reduce.LocalFetcher: localfetcher#77 about to shuffle output of map attempt_local1944506404_0006_m_000008_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:16 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000008_0
16/08/17 21:07:16 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 10, commitMemory -> 2103475, usedMemory ->2103477
16/08/17 21:07:16 INFO reduce.LocalFetcher: localfetcher#77 about to shuffle output of map attempt_local1944506404_0006_m_000013_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:16 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000013_0
16/08/17 21:07:16 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 11, commitMemory -> 2103477, usedMemory ->2103479
16/08/17 21:07:16 INFO reduce.LocalFetcher: localfetcher#77 about to shuffle output of map attempt_local1944506404_0006_m_000007_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:16 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000007_0
16/08/17 21:07:16 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 12, commitMemory -> 2103479, usedMemory ->2103481
16/08/17 21:07:16 INFO reduce.LocalFetcher: localfetcher#77 about to shuffle output of map attempt_local1944506404_0006_m_000006_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:16 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000006_0
16/08/17 21:07:16 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 13, commitMemory -> 2103481, usedMemory ->2103483
16/08/17 21:07:16 INFO reduce.LocalFetcher: localfetcher#77 about to shuffle output of map attempt_local1944506404_0006_m_000000_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:16 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000000_0
16/08/17 21:07:16 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 14, commitMemory -> 2103483, usedMemory ->2103485
16/08/17 21:07:16 INFO reduce.LocalFetcher: localfetcher#77 about to shuffle output of map attempt_local1944506404_0006_m_000005_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:16 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000005_0
16/08/17 21:07:16 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 15, commitMemory -> 2103485, usedMemory ->2103487
16/08/17 21:07:16 INFO reduce.LocalFetcher: localfetcher#77 about to shuffle output of map attempt_local1944506404_0006_m_000012_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:16 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000012_0
16/08/17 21:07:16 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 16, commitMemory -> 2103487, usedMemory ->2103489
16/08/17 21:07:16 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:07:16 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:16 INFO reduce.MergeManagerImpl: finalMerge called with 16 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:07:16 INFO mapred.Merger: Merging 16 sorted segments
16/08/17 21:07:16 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 2103425 bytes
16/08/17 21:07:16 INFO reduce.MergeManagerImpl: Merged 16 segments, 2103489 bytes to disk to satisfy reduce memory limit
16/08/17 21:07:16 INFO reduce.MergeManagerImpl: Merging 1 files, 2103463 bytes from disk
16/08/17 21:07:16 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:07:16 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:07:16 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 2103425 bytes
16/08/17 21:07:16 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:17 INFO mapred.LocalJobRunner: 
16/08/17 21:07:17 INFO mapred.MapTask: Starting flush of map output
16/08/17 21:07:17 INFO mapred.MapTask: Spilling map output
16/08/17 21:07:17 INFO mapred.MapTask: bufstart = 0; bufend = 4555488; bufvoid = 104857600
16/08/17 21:07:17 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 25685764(102743056); length = 528633/6553600
[1000]
[2000]
[3000]
[4000]
[5000]
[6000]
[7000]
[8000]
[9000]
[10000]
16/08/17 21:07:17 INFO mapred.LocalJobRunner: Starting task: attempt_local1944506404_0006_r_000013_0
16/08/17 21:07:17 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:17 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@44336359
16/08/17 21:07:17 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=367001600, maxSingleShuffleLimit=91750400, mergeThreshold=242221072, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:07:17 INFO reduce.EventFetcher: attempt_local1944506404_0006_r_000013_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:07:17 INFO reduce.LocalFetcher: localfetcher#78 about to shuffle output of map attempt_local1944506404_0006_m_000011_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:17 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000011_0
16/08/17 21:07:17 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->2
16/08/17 21:07:17 INFO reduce.LocalFetcher: localfetcher#78 about to shuffle output of map attempt_local1944506404_0006_m_000010_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:17 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000010_0
16/08/17 21:07:17 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 2, commitMemory -> 2, usedMemory ->4
16/08/17 21:07:17 INFO reduce.LocalFetcher: localfetcher#78 about to shuffle output of map attempt_local1944506404_0006_m_000004_0 decomp: 2155801 len: 2155805 to MEMORY
16/08/17 21:07:17 INFO mapred.MapTask: Finished spill 0
16/08/17 21:07:17 INFO reduce.InMemoryMapOutput: Read 2155801 bytes from map-output for attempt_local1944506404_0006_m_000004_0
16/08/17 21:07:17 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2155801, inMemoryMapOutputs.size() -> 3, commitMemory -> 4, usedMemory ->2155805
16/08/17 21:07:17 INFO reduce.LocalFetcher: localfetcher#78 about to shuffle output of map attempt_local1944506404_0006_m_000009_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:17 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000009_0
16/08/17 21:07:17 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 4, commitMemory -> 2155805, usedMemory ->2155807
16/08/17 21:07:17 INFO reduce.LocalFetcher: localfetcher#78 about to shuffle output of map attempt_local1944506404_0006_m_000003_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:17 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000003_0
16/08/17 21:07:17 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 5, commitMemory -> 2155807, usedMemory ->2155809
16/08/17 21:07:17 INFO reduce.LocalFetcher: localfetcher#78 about to shuffle output of map attempt_local1944506404_0006_m_000002_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:17 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000002_0
16/08/17 21:07:17 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 6, commitMemory -> 2155809, usedMemory ->2155811
16/08/17 21:07:17 INFO reduce.LocalFetcher: localfetcher#78 about to shuffle output of map attempt_local1944506404_0006_m_000015_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:17 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000015_0
16/08/17 21:07:17 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 7, commitMemory -> 2155811, usedMemory ->2155813
16/08/17 21:07:17 INFO reduce.LocalFetcher: localfetcher#78 about to shuffle output of map attempt_local1944506404_0006_m_000014_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:17 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000014_0
16/08/17 21:07:17 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 8, commitMemory -> 2155813, usedMemory ->2155815
16/08/17 21:07:17 INFO reduce.LocalFetcher: localfetcher#78 about to shuffle output of map attempt_local1944506404_0006_m_000001_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:17 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000001_0
16/08/17 21:07:17 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 9, commitMemory -> 2155815, usedMemory ->2155817
16/08/17 21:07:17 INFO mapred.Task: Task:attempt_local1999544224_0005_m_000007_0 is done. And is in the process of committing
16/08/17 21:07:17 INFO mapred.LocalJobRunner: map
16/08/17 21:07:17 INFO mapred.Task: Task 'attempt_local1999544224_0005_m_000007_0' done.
16/08/17 21:07:17 INFO mapred.LocalJobRunner: Finishing task: attempt_local1999544224_0005_m_000007_0
16/08/17 21:07:17 INFO mapred.LocalJobRunner: Starting task: attempt_local1999544224_0005_m_000008_0
16/08/17 21:07:17 INFO reduce.LocalFetcher: localfetcher#78 about to shuffle output of map attempt_local1944506404_0006_m_000008_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:17 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000008_0
16/08/17 21:07:17 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 10, commitMemory -> 2155817, usedMemory ->2155819
16/08/17 21:07:17 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)
	at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
16/08/17 21:07:17 INFO reduce.LocalFetcher: localfetcher#78 about to shuffle output of map attempt_local1944506404_0006_m_000013_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:17 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000013_0
16/08/17 21:07:17 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 11, commitMemory -> 2155819, usedMemory ->2155821
16/08/17 21:07:17 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)
	at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
16/08/17 21:07:17 INFO reduce.LocalFetcher: localfetcher#78 about to shuffle output of map attempt_local1944506404_0006_m_000007_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:17 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000007_0
16/08/17 21:07:17 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 12, commitMemory -> 2155821, usedMemory ->2155823
16/08/17 21:07:17 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)
	at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
16/08/17 21:07:17 INFO reduce.LocalFetcher: localfetcher#78 about to shuffle output of map attempt_local1944506404_0006_m_000006_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:17 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000006_0
16/08/17 21:07:17 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 13, commitMemory -> 2155823, usedMemory ->2155825
16/08/17 21:07:17 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)
	at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
16/08/17 21:07:17 INFO reduce.LocalFetcher: localfetcher#78 about to shuffle output of map attempt_local1944506404_0006_m_000000_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:17 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000000_0
16/08/17 21:07:17 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 14, commitMemory -> 2155825, usedMemory ->2155827
16/08/17 21:07:17 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)
	at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
16/08/17 21:07:17 INFO reduce.LocalFetcher: localfetcher#78 about to shuffle output of map attempt_local1944506404_0006_m_000005_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:17 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000005_0
16/08/17 21:07:17 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 15, commitMemory -> 2155827, usedMemory ->2155829
16/08/17 21:07:17 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)
	at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
16/08/17 21:07:17 INFO reduce.LocalFetcher: localfetcher#78 about to shuffle output of map attempt_local1944506404_0006_m_000012_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:17 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000012_0
16/08/17 21:07:17 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 16, commitMemory -> 2155829, usedMemory ->2155831
16/08/17 21:07:17 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:17 INFO mapred.MapTask: Processing split: file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/rules/part-r-00002:0+2395166
16/08/17 21:07:17 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:07:17 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:17 INFO reduce.MergeManagerImpl: finalMerge called with 16 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:07:17 INFO mapred.Merger: Merging 16 sorted segments
16/08/17 21:07:17 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 2155763 bytes
16/08/17 21:07:17 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
16/08/17 21:07:17 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
16/08/17 21:07:17 INFO mapred.MapTask: soft limit at 83886080
16/08/17 21:07:17 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
16/08/17 21:07:17 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
16/08/17 21:07:17 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
16/08/17 21:07:17 INFO reduce.MergeManagerImpl: Merged 16 segments, 2155831 bytes to disk to satisfy reduce memory limit
16/08/17 21:07:17 INFO reduce.MergeManagerImpl: Merging 1 files, 2155805 bytes from disk
16/08/17 21:07:17 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:07:17 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:07:17 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 2155763 bytes
16/08/17 21:07:17 INFO mapred.LocalJobRunner: 16 / 16 copied.
[1000]
[2000]
[3000]
[4000]
[5000]
[6000]
[7000]
[8000]
[9000]
[10000]
16/08/17 21:07:17 INFO mapred.LocalJobRunner: Starting task: attempt_local1944506404_0006_r_000014_0
16/08/17 21:07:17 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:17 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@1e9525e6
16/08/17 21:07:17 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=367368608, maxSingleShuffleLimit=91842152, mergeThreshold=242463296, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:07:17 INFO reduce.EventFetcher: attempt_local1944506404_0006_r_000014_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:07:17 INFO reduce.LocalFetcher: localfetcher#79 about to shuffle output of map attempt_local1944506404_0006_m_000011_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:17 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000011_0
16/08/17 21:07:17 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->2
16/08/17 21:07:17 INFO reduce.LocalFetcher: localfetcher#79 about to shuffle output of map attempt_local1944506404_0006_m_000010_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:17 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000010_0
16/08/17 21:07:17 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 2, commitMemory -> 2, usedMemory ->4
16/08/17 21:07:17 INFO reduce.LocalFetcher: localfetcher#79 about to shuffle output of map attempt_local1944506404_0006_m_000004_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:17 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000004_0
16/08/17 21:07:17 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 3, commitMemory -> 4, usedMemory ->6
16/08/17 21:07:17 INFO reduce.LocalFetcher: localfetcher#79 about to shuffle output of map attempt_local1944506404_0006_m_000009_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:17 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000009_0
16/08/17 21:07:17 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 4, commitMemory -> 6, usedMemory ->8
16/08/17 21:07:17 INFO reduce.LocalFetcher: localfetcher#79 about to shuffle output of map attempt_local1944506404_0006_m_000003_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:17 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000003_0
16/08/17 21:07:17 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 5, commitMemory -> 8, usedMemory ->10
16/08/17 21:07:17 INFO reduce.LocalFetcher: localfetcher#79 about to shuffle output of map attempt_local1944506404_0006_m_000002_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:17 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000002_0
16/08/17 21:07:17 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 6, commitMemory -> 10, usedMemory ->12
16/08/17 21:07:17 INFO reduce.LocalFetcher: localfetcher#79 about to shuffle output of map attempt_local1944506404_0006_m_000015_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:17 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000015_0
16/08/17 21:07:17 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 7, commitMemory -> 12, usedMemory ->14
16/08/17 21:07:17 INFO reduce.LocalFetcher: localfetcher#79 about to shuffle output of map attempt_local1944506404_0006_m_000014_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:17 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000014_0
16/08/17 21:07:17 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 8, commitMemory -> 14, usedMemory ->16
16/08/17 21:07:17 INFO reduce.LocalFetcher: localfetcher#79 about to shuffle output of map attempt_local1944506404_0006_m_000001_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:17 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000001_0
16/08/17 21:07:17 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 9, commitMemory -> 16, usedMemory ->18
16/08/17 21:07:17 INFO reduce.LocalFetcher: localfetcher#79 about to shuffle output of map attempt_local1944506404_0006_m_000008_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:17 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000008_0
16/08/17 21:07:17 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 10, commitMemory -> 18, usedMemory ->20
16/08/17 21:07:17 INFO reduce.LocalFetcher: localfetcher#79 about to shuffle output of map attempt_local1944506404_0006_m_000013_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:17 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000013_0
16/08/17 21:07:17 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 11, commitMemory -> 20, usedMemory ->22
16/08/17 21:07:17 INFO reduce.LocalFetcher: localfetcher#79 about to shuffle output of map attempt_local1944506404_0006_m_000007_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:17 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000007_0
16/08/17 21:07:17 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 12, commitMemory -> 22, usedMemory ->24
16/08/17 21:07:17 INFO reduce.LocalFetcher: localfetcher#79 about to shuffle output of map attempt_local1944506404_0006_m_000006_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:17 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000006_0
16/08/17 21:07:17 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 13, commitMemory -> 24, usedMemory ->26
16/08/17 21:07:17 INFO reduce.LocalFetcher: localfetcher#79 about to shuffle output of map attempt_local1944506404_0006_m_000000_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:17 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000000_0
16/08/17 21:07:17 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 14, commitMemory -> 26, usedMemory ->28
16/08/17 21:07:17 INFO reduce.LocalFetcher: localfetcher#79 about to shuffle output of map attempt_local1944506404_0006_m_000005_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:17 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000005_0
16/08/17 21:07:17 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 15, commitMemory -> 28, usedMemory ->30
16/08/17 21:07:17 INFO reduce.LocalFetcher: localfetcher#79 about to shuffle output of map attempt_local1944506404_0006_m_000012_0 decomp: 2096947 len: 2096951 to MEMORY
16/08/17 21:07:17 INFO reduce.InMemoryMapOutput: Read 2096947 bytes from map-output for attempt_local1944506404_0006_m_000012_0
16/08/17 21:07:17 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2096947, inMemoryMapOutputs.size() -> 16, commitMemory -> 30, usedMemory ->2096977
16/08/17 21:07:17 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:07:17 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:17 INFO reduce.MergeManagerImpl: finalMerge called with 16 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:07:17 INFO mapred.Merger: Merging 16 sorted segments
16/08/17 21:07:17 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 2096921 bytes
16/08/17 21:07:17 INFO reduce.MergeManagerImpl: Merged 16 segments, 2096977 bytes to disk to satisfy reduce memory limit
16/08/17 21:07:17 INFO reduce.MergeManagerImpl: Merging 1 files, 2096951 bytes from disk
16/08/17 21:07:17 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:07:17 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:07:17 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 2096921 bytes
16/08/17 21:07:17 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:17 INFO mapred.LocalJobRunner: 
16/08/17 21:07:17 INFO mapred.MapTask: Starting flush of map output
16/08/17 21:07:17 INFO mapred.MapTask: Spilling map output
16/08/17 21:07:17 INFO mapred.MapTask: bufstart = 0; bufend = 4644176; bufvoid = 104857600
16/08/17 21:07:17 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 25685764(102743056); length = 528633/6553600
16/08/17 21:07:17 INFO mapred.LocalJobRunner: reduce > reduce
[1000]
[2000]
[3000]
[4000]
[5000]
[6000]
[7000]
[8000]
[9000]
[10000]
16/08/17 21:07:18 INFO mapred.LocalJobRunner: Starting task: attempt_local1944506404_0006_r_000015_0
16/08/17 21:07:18 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:18 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@6466ac8a
16/08/17 21:07:18 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=367368608, maxSingleShuffleLimit=91842152, mergeThreshold=242463296, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:07:18 INFO reduce.EventFetcher: attempt_local1944506404_0006_r_000015_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:07:18 INFO reduce.LocalFetcher: localfetcher#80 about to shuffle output of map attempt_local1944506404_0006_m_000011_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:18 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000011_0
16/08/17 21:07:18 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->2
16/08/17 21:07:18 INFO reduce.LocalFetcher: localfetcher#80 about to shuffle output of map attempt_local1944506404_0006_m_000010_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:18 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000010_0
16/08/17 21:07:18 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 2, commitMemory -> 2, usedMemory ->4
16/08/17 21:07:18 INFO reduce.LocalFetcher: localfetcher#80 about to shuffle output of map attempt_local1944506404_0006_m_000004_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:18 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000004_0
16/08/17 21:07:18 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 3, commitMemory -> 4, usedMemory ->6
16/08/17 21:07:18 INFO reduce.LocalFetcher: localfetcher#80 about to shuffle output of map attempt_local1944506404_0006_m_000009_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:18 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000009_0
16/08/17 21:07:18 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 4, commitMemory -> 6, usedMemory ->8
16/08/17 21:07:18 INFO reduce.LocalFetcher: localfetcher#80 about to shuffle output of map attempt_local1944506404_0006_m_000003_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:18 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000003_0
16/08/17 21:07:18 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 5, commitMemory -> 8, usedMemory ->10
16/08/17 21:07:18 INFO reduce.LocalFetcher: localfetcher#80 about to shuffle output of map attempt_local1944506404_0006_m_000002_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:18 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000002_0
16/08/17 21:07:18 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 6, commitMemory -> 10, usedMemory ->12
16/08/17 21:07:18 INFO reduce.LocalFetcher: localfetcher#80 about to shuffle output of map attempt_local1944506404_0006_m_000015_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:18 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000015_0
16/08/17 21:07:18 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 7, commitMemory -> 12, usedMemory ->14
16/08/17 21:07:18 INFO reduce.LocalFetcher: localfetcher#80 about to shuffle output of map attempt_local1944506404_0006_m_000014_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:18 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000014_0
16/08/17 21:07:18 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 8, commitMemory -> 14, usedMemory ->16
16/08/17 21:07:18 INFO reduce.LocalFetcher: localfetcher#80 about to shuffle output of map attempt_local1944506404_0006_m_000001_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:18 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000001_0
16/08/17 21:07:18 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 9, commitMemory -> 16, usedMemory ->18
16/08/17 21:07:18 INFO reduce.LocalFetcher: localfetcher#80 about to shuffle output of map attempt_local1944506404_0006_m_000008_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:18 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000008_0
16/08/17 21:07:18 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 10, commitMemory -> 18, usedMemory ->20
16/08/17 21:07:18 INFO reduce.LocalFetcher: localfetcher#80 about to shuffle output of map attempt_local1944506404_0006_m_000013_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:18 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000013_0
16/08/17 21:07:18 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 11, commitMemory -> 20, usedMemory ->22
16/08/17 21:07:18 INFO reduce.LocalFetcher: localfetcher#80 about to shuffle output of map attempt_local1944506404_0006_m_000007_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:18 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000007_0
16/08/17 21:07:18 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 12, commitMemory -> 22, usedMemory ->24
16/08/17 21:07:18 INFO reduce.LocalFetcher: localfetcher#80 about to shuffle output of map attempt_local1944506404_0006_m_000006_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:18 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000006_0
16/08/17 21:07:18 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 13, commitMemory -> 24, usedMemory ->26
16/08/17 21:07:18 INFO reduce.LocalFetcher: localfetcher#80 about to shuffle output of map attempt_local1944506404_0006_m_000000_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:18 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000000_0
16/08/17 21:07:18 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 14, commitMemory -> 26, usedMemory ->28
16/08/17 21:07:18 INFO reduce.LocalFetcher: localfetcher#80 about to shuffle output of map attempt_local1944506404_0006_m_000005_0 decomp: 2149547 len: 2149551 to MEMORY
16/08/17 21:07:18 INFO reduce.InMemoryMapOutput: Read 2149547 bytes from map-output for attempt_local1944506404_0006_m_000005_0
16/08/17 21:07:18 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2149547, inMemoryMapOutputs.size() -> 15, commitMemory -> 28, usedMemory ->2149575
16/08/17 21:07:18 INFO reduce.LocalFetcher: localfetcher#80 about to shuffle output of map attempt_local1944506404_0006_m_000012_0 decomp: 2 len: 6 to MEMORY
16/08/17 21:07:18 INFO reduce.InMemoryMapOutput: Read 2 bytes from map-output for attempt_local1944506404_0006_m_000012_0
16/08/17 21:07:18 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 2, inMemoryMapOutputs.size() -> 16, commitMemory -> 2149575, usedMemory ->2149577
16/08/17 21:07:18 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:07:18 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:18 INFO reduce.MergeManagerImpl: finalMerge called with 16 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:07:18 INFO mapred.Merger: Merging 16 sorted segments
16/08/17 21:07:18 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 2149525 bytes
16/08/17 21:07:18 INFO reduce.MergeManagerImpl: Merged 16 segments, 2149577 bytes to disk to satisfy reduce memory limit
16/08/17 21:07:18 INFO reduce.MergeManagerImpl: Merging 1 files, 2149551 bytes from disk
16/08/17 21:07:18 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:07:18 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:07:18 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 2149525 bytes
16/08/17 21:07:18 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:18 INFO mapred.MapTask: Finished spill 0
16/08/17 21:07:18 INFO mapred.Task: Task:attempt_local518136789_0007_m_000008_0 is done. And is in the process of committing
16/08/17 21:07:18 INFO mapred.LocalJobRunner: map
16/08/17 21:07:18 INFO mapred.Task: Task 'attempt_local518136789_0007_m_000008_0' done.
16/08/17 21:07:18 INFO mapred.LocalJobRunner: Finishing task: attempt_local518136789_0007_m_000008_0
16/08/17 21:07:18 INFO mapred.LocalJobRunner: Starting task: attempt_local518136789_0007_m_000009_0
16/08/17 21:07:18 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:18 INFO mapred.MapTask: Processing split: file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/rules/part-r-00012:0+2389851
16/08/17 21:07:18 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
16/08/17 21:07:18 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
16/08/17 21:07:18 INFO mapred.MapTask: soft limit at 83886080
16/08/17 21:07:18 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
16/08/17 21:07:18 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
16/08/17 21:07:18 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
[1000]
[2000]
[3000]
[4000]
[5000]
[6000]
[7000]
[8000]
[9000]
[10000]
16/08/17 21:07:18 INFO mapred.LocalJobRunner: reduce task executor complete.
16/08/17 21:07:18 WARN mapred.LocalJobRunner: job_local1944506404_0006
java.lang.Exception: java.lang.RuntimeException: Failed loading TrieLexprobTable. Entries must be sorted ascendingly, but cars[710]=14123 > cars[711]=10593
	at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
	at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:529)
Caused by: java.lang.RuntimeException: Failed loading TrieLexprobTable. Entries must be sorted ascendingly, but cars[710]=14123 > cars[711]=10593
	at edu.jhu.thrax.lexprob.TrieLexprobTable.checkIntegrity(TrieLexprobTable.java:71)
	at edu.jhu.thrax.lexprob.TrieLexprobTable.initialize(TrieLexprobTable.java:63)
	at edu.jhu.thrax.lexprob.TrieLexprobTable.<init>(TrieLexprobTable.java:23)
	at edu.jhu.thrax.hadoop.features.annotation.TargetGivenSourceLexicalProbabilityFeature.init(TargetGivenSourceLexicalProbabilityFeature.java:45)
	at edu.jhu.thrax.hadoop.features.annotation.AnnotationReducer.setup(AnnotationReducer.java:39)
	at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:168)
	at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:627)
	at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
	at org.apache.hadoop.mapred.LocalJobRunner$Job$ReduceTaskRunnable.run(LocalJobRunner.java:319)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
16/08/17 21:07:18 INFO mapred.LocalJobRunner: 
16/08/17 21:07:18 INFO mapred.MapTask: Starting flush of map output
16/08/17 21:07:18 INFO mapred.MapTask: Spilling map output
16/08/17 21:07:18 INFO mapred.MapTask: bufstart = 0; bufend = 4543600; bufvoid = 104857600
16/08/17 21:07:18 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 25688476(102753904); length = 525921/6553600
16/08/17 21:07:18 INFO mapred.MapTask: Finished spill 0
16/08/17 21:07:18 INFO mapred.Task: Task:attempt_local1999544224_0005_m_000008_0 is done. And is in the process of committing
16/08/17 21:07:18 INFO mapred.LocalJobRunner: map
16/08/17 21:07:18 INFO mapred.Task: Task 'attempt_local1999544224_0005_m_000008_0' done.
16/08/17 21:07:18 INFO mapred.LocalJobRunner: Finishing task: attempt_local1999544224_0005_m_000008_0
16/08/17 21:07:18 INFO mapred.LocalJobRunner: Starting task: attempt_local1999544224_0005_m_000009_0
16/08/17 21:07:18 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:18 INFO mapred.MapTask: Processing split: file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/rules/part-r-00012:0+2389851
16/08/17 21:07:18 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
16/08/17 21:07:18 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
16/08/17 21:07:18 INFO mapred.MapTask: soft limit at 83886080
16/08/17 21:07:18 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
16/08/17 21:07:18 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
16/08/17 21:07:18 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
16/08/17 21:07:18 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:07:18 INFO mapred.LocalJobRunner: 
16/08/17 21:07:18 INFO mapred.MapTask: Starting flush of map output
16/08/17 21:07:18 INFO mapred.MapTask: Spilling map output
16/08/17 21:07:18 INFO mapred.MapTask: bufstart = 0; bufend = 4631956; bufvoid = 104857600
16/08/17 21:07:18 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 25688476(102753904); length = 525921/6553600
16/08/17 21:07:19 INFO mapred.MapTask: Finished spill 0
16/08/17 21:07:19 INFO mapred.Task: Task:attempt_local518136789_0007_m_000009_0 is done. And is in the process of committing
16/08/17 21:07:19 INFO mapred.LocalJobRunner: map
16/08/17 21:07:19 INFO mapred.Task: Task 'attempt_local518136789_0007_m_000009_0' done.
16/08/17 21:07:19 INFO mapred.LocalJobRunner: Finishing task: attempt_local518136789_0007_m_000009_0
16/08/17 21:07:19 INFO mapred.LocalJobRunner: Starting task: attempt_local518136789_0007_m_000010_0
16/08/17 21:07:19 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:19 INFO mapred.MapTask: Processing split: file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/rules/part-r-00000:0+2389069
16/08/17 21:07:19 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
16/08/17 21:07:19 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
16/08/17 21:07:19 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:07:19 INFO mapred.MapTask: soft limit at 83886080
16/08/17 21:07:19 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
16/08/17 21:07:19 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
16/08/17 21:07:19 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
[SCHED] class edu.jhu.thrax.hadoop.features.annotation.AnnotationFeatureJob in state FAILED
[SCHED] class edu.jhu.thrax.hadoop.jobs.OutputJob in state PREREQ_FAILED
16/08/17 21:07:19 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:07:19 INFO mapred.MapTask: Finished spill 0
16/08/17 21:07:19 INFO mapred.Task: Task:attempt_local1999544224_0005_m_000009_0 is done. And is in the process of committing
16/08/17 21:07:19 INFO mapred.LocalJobRunner: map
16/08/17 21:07:19 INFO mapred.Task: Task 'attempt_local1999544224_0005_m_000009_0' done.
16/08/17 21:07:19 INFO mapred.LocalJobRunner: Finishing task: attempt_local1999544224_0005_m_000009_0
16/08/17 21:07:19 INFO mapred.LocalJobRunner: Starting task: attempt_local1999544224_0005_m_000010_0
16/08/17 21:07:19 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:19 INFO mapred.MapTask: Processing split: file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/rules/part-r-00000:0+2389069
16/08/17 21:07:19 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
16/08/17 21:07:19 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
16/08/17 21:07:19 INFO mapred.MapTask: soft limit at 83886080
16/08/17 21:07:19 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
16/08/17 21:07:19 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
16/08/17 21:07:19 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
16/08/17 21:07:19 INFO mapred.LocalJobRunner: 
16/08/17 21:07:19 INFO mapred.MapTask: Starting flush of map output
16/08/17 21:07:19 INFO mapred.MapTask: Spilling map output
16/08/17 21:07:19 INFO mapred.MapTask: bufstart = 0; bufend = 4544988; bufvoid = 104857600
16/08/17 21:07:19 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 25687816(102751264); length = 526581/6553600
16/08/17 21:07:19 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:07:19 INFO mapred.LocalJobRunner: 
16/08/17 21:07:19 INFO mapred.MapTask: Starting flush of map output
16/08/17 21:07:19 INFO mapred.MapTask: Spilling map output
16/08/17 21:07:19 INFO mapred.MapTask: bufstart = 0; bufend = 4632828; bufvoid = 104857600
16/08/17 21:07:19 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 25687816(102751264); length = 526581/6553600
16/08/17 21:07:19 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:07:20 INFO mapred.MapTask: Finished spill 0
16/08/17 21:07:20 INFO mapred.Task: Task:attempt_local518136789_0007_m_000010_0 is done. And is in the process of committing
16/08/17 21:07:20 INFO mapred.LocalJobRunner: map
16/08/17 21:07:20 INFO mapred.Task: Task 'attempt_local518136789_0007_m_000010_0' done.
16/08/17 21:07:20 INFO mapred.LocalJobRunner: Finishing task: attempt_local518136789_0007_m_000010_0
16/08/17 21:07:20 INFO mapred.LocalJobRunner: Starting task: attempt_local518136789_0007_m_000011_0
16/08/17 21:07:20 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:20 INFO mapred.MapTask: Processing split: file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/rules/part-r-00008:0+2387786
16/08/17 21:07:20 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
16/08/17 21:07:20 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
16/08/17 21:07:20 INFO mapred.MapTask: soft limit at 83886080
16/08/17 21:07:20 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
16/08/17 21:07:20 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
16/08/17 21:07:20 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
16/08/17 21:07:20 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:07:20 INFO mapred.MapTask: Finished spill 0
16/08/17 21:07:20 INFO mapred.Task: Task:attempt_local1999544224_0005_m_000010_0 is done. And is in the process of committing
16/08/17 21:07:20 INFO mapred.LocalJobRunner: map
16/08/17 21:07:20 INFO mapred.Task: Task 'attempt_local1999544224_0005_m_000010_0' done.
16/08/17 21:07:20 INFO mapred.LocalJobRunner: Finishing task: attempt_local1999544224_0005_m_000010_0
16/08/17 21:07:20 INFO mapred.LocalJobRunner: Starting task: attempt_local1999544224_0005_m_000011_0
16/08/17 21:07:20 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:20 INFO mapred.MapTask: Processing split: file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/rules/part-r-00008:0+2387786
16/08/17 21:07:20 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
16/08/17 21:07:20 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
16/08/17 21:07:20 INFO mapred.MapTask: soft limit at 83886080
16/08/17 21:07:20 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
16/08/17 21:07:20 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
16/08/17 21:07:20 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
16/08/17 21:07:20 INFO mapred.LocalJobRunner: 
16/08/17 21:07:20 INFO mapred.MapTask: Starting flush of map output
16/08/17 21:07:20 INFO mapred.MapTask: Spilling map output
16/08/17 21:07:20 INFO mapred.MapTask: bufstart = 0; bufend = 4539892; bufvoid = 104857600
16/08/17 21:07:20 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 25687804(102751216); length = 526593/6553600
16/08/17 21:07:20 INFO mapred.LocalJobRunner: 
16/08/17 21:07:20 INFO mapred.MapTask: Starting flush of map output
16/08/17 21:07:20 INFO mapred.MapTask: Spilling map output
16/08/17 21:07:20 INFO mapred.MapTask: bufstart = 0; bufend = 4628392; bufvoid = 104857600
16/08/17 21:07:20 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 25687804(102751216); length = 526593/6553600
16/08/17 21:07:20 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:07:20 INFO mapred.MapTask: Finished spill 0
16/08/17 21:07:20 INFO mapred.Task: Task:attempt_local518136789_0007_m_000011_0 is done. And is in the process of committing
16/08/17 21:07:20 INFO mapred.LocalJobRunner: map
16/08/17 21:07:20 INFO mapred.Task: Task 'attempt_local518136789_0007_m_000011_0' done.
16/08/17 21:07:20 INFO mapred.LocalJobRunner: Finishing task: attempt_local518136789_0007_m_000011_0
16/08/17 21:07:20 INFO mapred.LocalJobRunner: Starting task: attempt_local518136789_0007_m_000012_0
16/08/17 21:07:20 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:20 INFO mapred.MapTask: Processing split: file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/rules/part-r-00014:0+2382869
16/08/17 21:07:20 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
16/08/17 21:07:20 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
16/08/17 21:07:20 INFO mapred.MapTask: soft limit at 83886080
16/08/17 21:07:20 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
16/08/17 21:07:20 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
16/08/17 21:07:20 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
16/08/17 21:07:21 INFO mapred.MapTask: Finished spill 0
16/08/17 21:07:21 INFO mapred.Task: Task:attempt_local1999544224_0005_m_000011_0 is done. And is in the process of committing
16/08/17 21:07:21 INFO mapred.LocalJobRunner: map
16/08/17 21:07:21 INFO mapred.Task: Task 'attempt_local1999544224_0005_m_000011_0' done.
16/08/17 21:07:21 INFO mapred.LocalJobRunner: Finishing task: attempt_local1999544224_0005_m_000011_0
16/08/17 21:07:21 INFO mapred.LocalJobRunner: Starting task: attempt_local1999544224_0005_m_000012_0
16/08/17 21:07:21 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:21 INFO mapred.MapTask: Processing split: file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/rules/part-r-00014:0+2382869
16/08/17 21:07:21 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
16/08/17 21:07:21 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
16/08/17 21:07:21 INFO mapred.MapTask: soft limit at 83886080
16/08/17 21:07:21 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
16/08/17 21:07:21 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
16/08/17 21:07:21 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
16/08/17 21:07:21 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:07:21 INFO mapred.LocalJobRunner: 
16/08/17 21:07:21 INFO mapred.MapTask: Starting flush of map output
16/08/17 21:07:21 INFO mapred.MapTask: Spilling map output
16/08/17 21:07:21 INFO mapred.MapTask: bufstart = 0; bufend = 4531592; bufvoid = 104857600
16/08/17 21:07:21 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 25689256(102757024); length = 525141/6553600
16/08/17 21:07:21 INFO mapred.LocalJobRunner: 
16/08/17 21:07:21 INFO mapred.MapTask: Starting flush of map output
16/08/17 21:07:21 INFO mapred.MapTask: Spilling map output
16/08/17 21:07:21 INFO mapred.MapTask: bufstart = 0; bufend = 4618524; bufvoid = 104857600
16/08/17 21:07:21 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 25689256(102757024); length = 525141/6553600
16/08/17 21:07:21 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:07:21 INFO mapred.MapTask: Finished spill 0
16/08/17 21:07:21 INFO mapred.Task: Task:attempt_local518136789_0007_m_000012_0 is done. And is in the process of committing
16/08/17 21:07:21 INFO mapred.LocalJobRunner: map
16/08/17 21:07:21 INFO mapred.Task: Task 'attempt_local518136789_0007_m_000012_0' done.
16/08/17 21:07:21 INFO mapred.LocalJobRunner: Finishing task: attempt_local518136789_0007_m_000012_0
16/08/17 21:07:21 INFO mapred.LocalJobRunner: Starting task: attempt_local518136789_0007_m_000013_0
16/08/17 21:07:21 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:21 INFO mapred.MapTask: Processing split: file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/rules/part-r-00004:0+2379925
16/08/17 21:07:21 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
16/08/17 21:07:21 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
16/08/17 21:07:21 INFO mapred.MapTask: soft limit at 83886080
16/08/17 21:07:21 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
16/08/17 21:07:21 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
16/08/17 21:07:21 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
16/08/17 21:07:22 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:07:22 INFO mapred.MapTask: Finished spill 0
16/08/17 21:07:22 INFO mapred.Task: Task:attempt_local1999544224_0005_m_000012_0 is done. And is in the process of committing
16/08/17 21:07:22 INFO mapred.LocalJobRunner: map
16/08/17 21:07:22 INFO mapred.Task: Task 'attempt_local1999544224_0005_m_000012_0' done.
16/08/17 21:07:22 INFO mapred.LocalJobRunner: Finishing task: attempt_local1999544224_0005_m_000012_0
16/08/17 21:07:22 INFO mapred.LocalJobRunner: Starting task: attempt_local1999544224_0005_m_000013_0
16/08/17 21:07:22 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:22 INFO mapred.MapTask: Processing split: file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/rules/part-r-00004:0+2379925
16/08/17 21:07:22 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
16/08/17 21:07:22 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
16/08/17 21:07:22 INFO mapred.MapTask: soft limit at 83886080
16/08/17 21:07:22 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
16/08/17 21:07:22 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
16/08/17 21:07:22 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
16/08/17 21:07:22 INFO mapred.LocalJobRunner: 
16/08/17 21:07:22 INFO mapred.MapTask: Starting flush of map output
16/08/17 21:07:22 INFO mapred.MapTask: Spilling map output
16/08/17 21:07:22 INFO mapred.MapTask: bufstart = 0; bufend = 4525288; bufvoid = 104857600
16/08/17 21:07:22 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 25689880(102759520); length = 524517/6553600
16/08/17 21:07:22 INFO mapred.LocalJobRunner: 
16/08/17 21:07:22 INFO mapred.MapTask: Starting flush of map output
16/08/17 21:07:22 INFO mapred.MapTask: Spilling map output
16/08/17 21:07:22 INFO mapred.MapTask: bufstart = 0; bufend = 4613572; bufvoid = 104857600
16/08/17 21:07:22 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 25689880(102759520); length = 524517/6553600
16/08/17 21:07:22 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:07:22 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:07:22 INFO mapred.MapTask: Finished spill 0
16/08/17 21:07:22 INFO mapred.Task: Task:attempt_local518136789_0007_m_000013_0 is done. And is in the process of committing
16/08/17 21:07:22 INFO mapred.LocalJobRunner: map
16/08/17 21:07:22 INFO mapred.Task: Task 'attempt_local518136789_0007_m_000013_0' done.
16/08/17 21:07:22 INFO mapred.LocalJobRunner: Finishing task: attempt_local518136789_0007_m_000013_0
16/08/17 21:07:22 INFO mapred.LocalJobRunner: Starting task: attempt_local518136789_0007_m_000014_0
16/08/17 21:07:22 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:22 INFO mapred.MapTask: Processing split: file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/rules/part-r-00006:0+2375925
16/08/17 21:07:22 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
16/08/17 21:07:22 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
16/08/17 21:07:22 INFO mapred.MapTask: soft limit at 83886080
16/08/17 21:07:22 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
16/08/17 21:07:22 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
16/08/17 21:07:22 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
16/08/17 21:07:22 INFO mapred.MapTask: Finished spill 0
16/08/17 21:07:22 INFO mapred.Task: Task:attempt_local1999544224_0005_m_000013_0 is done. And is in the process of committing
16/08/17 21:07:22 INFO mapred.LocalJobRunner: map
16/08/17 21:07:22 INFO mapred.Task: Task 'attempt_local1999544224_0005_m_000013_0' done.
16/08/17 21:07:22 INFO mapred.LocalJobRunner: Finishing task: attempt_local1999544224_0005_m_000013_0
16/08/17 21:07:22 INFO mapred.LocalJobRunner: Starting task: attempt_local1999544224_0005_m_000014_0
16/08/17 21:07:22 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:22 INFO mapred.MapTask: Processing split: file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/rules/part-r-00006:0+2375925
16/08/17 21:07:23 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
16/08/17 21:07:23 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
16/08/17 21:07:23 INFO mapred.MapTask: soft limit at 83886080
16/08/17 21:07:23 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
16/08/17 21:07:23 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
16/08/17 21:07:23 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
16/08/17 21:07:23 INFO mapred.LocalJobRunner: 
16/08/17 21:07:23 INFO mapred.MapTask: Starting flush of map output
16/08/17 21:07:23 INFO mapred.MapTask: Spilling map output
16/08/17 21:07:23 INFO mapred.MapTask: bufstart = 0; bufend = 4519588; bufvoid = 104857600
16/08/17 21:07:23 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 25689544(102758176); length = 524853/6553600
16/08/17 21:07:23 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:07:23 INFO mapred.LocalJobRunner: 
16/08/17 21:07:23 INFO mapred.MapTask: Starting flush of map output
16/08/17 21:07:23 INFO mapred.MapTask: Spilling map output
16/08/17 21:07:23 INFO mapred.MapTask: bufstart = 0; bufend = 4606116; bufvoid = 104857600
16/08/17 21:07:23 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 25689544(102758176); length = 524853/6553600
16/08/17 21:07:23 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:07:23 INFO mapred.MapTask: Finished spill 0
16/08/17 21:07:23 INFO mapred.Task: Task:attempt_local518136789_0007_m_000014_0 is done. And is in the process of committing
16/08/17 21:07:23 INFO mapred.LocalJobRunner: map
16/08/17 21:07:23 INFO mapred.Task: Task 'attempt_local518136789_0007_m_000014_0' done.
16/08/17 21:07:23 INFO mapred.LocalJobRunner: Finishing task: attempt_local518136789_0007_m_000014_0
16/08/17 21:07:23 INFO mapred.LocalJobRunner: Starting task: attempt_local518136789_0007_m_000015_0
16/08/17 21:07:23 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:23 INFO mapred.MapTask: Processing split: file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/rules/part-r-00010:0+2364196
16/08/17 21:07:23 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
16/08/17 21:07:23 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
16/08/17 21:07:23 INFO mapred.MapTask: soft limit at 83886080
16/08/17 21:07:23 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
16/08/17 21:07:23 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
16/08/17 21:07:23 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
16/08/17 21:07:23 INFO mapred.MapTask: Finished spill 0
16/08/17 21:07:23 INFO mapred.Task: Task:attempt_local1999544224_0005_m_000014_0 is done. And is in the process of committing
16/08/17 21:07:23 INFO mapred.LocalJobRunner: map
16/08/17 21:07:23 INFO mapred.Task: Task 'attempt_local1999544224_0005_m_000014_0' done.
16/08/17 21:07:23 INFO mapred.LocalJobRunner: Finishing task: attempt_local1999544224_0005_m_000014_0
16/08/17 21:07:23 INFO mapred.LocalJobRunner: Starting task: attempt_local1999544224_0005_m_000015_0
16/08/17 21:07:23 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:23 INFO mapred.MapTask: Processing split: file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/rules/part-r-00010:0+2364196
16/08/17 21:07:23 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
16/08/17 21:07:23 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
16/08/17 21:07:23 INFO mapred.MapTask: soft limit at 83886080
16/08/17 21:07:23 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
16/08/17 21:07:23 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
16/08/17 21:07:23 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
16/08/17 21:07:24 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:07:24 INFO mapred.LocalJobRunner: 
16/08/17 21:07:24 INFO mapred.MapTask: Starting flush of map output
16/08/17 21:07:24 INFO mapred.MapTask: Spilling map output
16/08/17 21:07:24 INFO mapred.MapTask: bufstart = 0; bufend = 4495804; bufvoid = 104857600
16/08/17 21:07:24 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 25693012(102772048); length = 521385/6553600
16/08/17 21:07:24 INFO mapred.LocalJobRunner: 
16/08/17 21:07:24 INFO mapred.MapTask: Starting flush of map output
16/08/17 21:07:24 INFO mapred.MapTask: Spilling map output
16/08/17 21:07:24 INFO mapred.MapTask: bufstart = 0; bufend = 4582128; bufvoid = 104857600
16/08/17 21:07:24 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 25693012(102772048); length = 521385/6553600
16/08/17 21:07:24 INFO mapred.MapTask: Finished spill 0
16/08/17 21:07:24 INFO mapred.Task: Task:attempt_local518136789_0007_m_000015_0 is done. And is in the process of committing
16/08/17 21:07:24 INFO mapred.LocalJobRunner: map
16/08/17 21:07:24 INFO mapred.Task: Task 'attempt_local518136789_0007_m_000015_0' done.
16/08/17 21:07:24 INFO mapred.LocalJobRunner: Finishing task: attempt_local518136789_0007_m_000015_0
16/08/17 21:07:24 INFO mapred.LocalJobRunner: map task executor complete.
16/08/17 21:07:24 INFO mapred.LocalJobRunner: Waiting for reduce tasks
16/08/17 21:07:24 INFO mapred.LocalJobRunner: Starting task: attempt_local518136789_0007_r_000000_0
16/08/17 21:07:24 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:24 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@5fcf307e
16/08/17 21:07:24 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=371038624, maxSingleShuffleLimit=92759656, mergeThreshold=244885504, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:07:24 INFO reduce.EventFetcher: attempt_local518136789_0007_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:07:24 INFO reduce.LocalFetcher: localfetcher#81 about to shuffle output of map attempt_local518136789_0007_m_000011_0 decomp: 294716 len: 294720 to MEMORY
16/08/17 21:07:24 INFO reduce.InMemoryMapOutput: Read 294716 bytes from map-output for attempt_local518136789_0007_m_000011_0
16/08/17 21:07:24 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 294716, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->294716
16/08/17 21:07:24 INFO reduce.LocalFetcher: localfetcher#81 about to shuffle output of map attempt_local518136789_0007_m_000004_0 decomp: 309332 len: 309336 to MEMORY
16/08/17 21:07:24 INFO reduce.InMemoryMapOutput: Read 309332 bytes from map-output for attempt_local518136789_0007_m_000004_0
16/08/17 21:07:24 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 309332, inMemoryMapOutputs.size() -> 2, commitMemory -> 294716, usedMemory ->604048
16/08/17 21:07:24 INFO reduce.LocalFetcher: localfetcher#81 about to shuffle output of map attempt_local518136789_0007_m_000010_0 decomp: 299516 len: 299520 to MEMORY
16/08/17 21:07:24 INFO reduce.InMemoryMapOutput: Read 299516 bytes from map-output for attempt_local518136789_0007_m_000010_0
16/08/17 21:07:24 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 299516, inMemoryMapOutputs.size() -> 3, commitMemory -> 604048, usedMemory ->903564
16/08/17 21:07:24 INFO reduce.LocalFetcher: localfetcher#81 about to shuffle output of map attempt_local518136789_0007_m_000000_0 decomp: 298108 len: 298112 to MEMORY
16/08/17 21:07:24 INFO reduce.InMemoryMapOutput: Read 298108 bytes from map-output for attempt_local518136789_0007_m_000000_0
16/08/17 21:07:24 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 298108, inMemoryMapOutputs.size() -> 4, commitMemory -> 903564, usedMemory ->1201672
16/08/17 21:07:24 INFO reduce.LocalFetcher: localfetcher#81 about to shuffle output of map attempt_local518136789_0007_m_000006_0 decomp: 306262 len: 306266 to MEMORY
16/08/17 21:07:24 INFO reduce.InMemoryMapOutput: Read 306262 bytes from map-output for attempt_local518136789_0007_m_000006_0
16/08/17 21:07:24 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 306262, inMemoryMapOutputs.size() -> 5, commitMemory -> 1201672, usedMemory ->1507934
16/08/17 21:07:24 INFO reduce.LocalFetcher: localfetcher#81 about to shuffle output of map attempt_local518136789_0007_m_000012_0 decomp: 293786 len: 293790 to MEMORY
16/08/17 21:07:24 INFO reduce.InMemoryMapOutput: Read 293786 bytes from map-output for attempt_local518136789_0007_m_000012_0
16/08/17 21:07:24 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 293786, inMemoryMapOutputs.size() -> 6, commitMemory -> 1507934, usedMemory ->1801720
16/08/17 21:07:24 INFO reduce.LocalFetcher: localfetcher#81 about to shuffle output of map attempt_local518136789_0007_m_000005_0 decomp: 306810 len: 306814 to MEMORY
16/08/17 21:07:24 INFO reduce.InMemoryMapOutput: Read 306810 bytes from map-output for attempt_local518136789_0007_m_000005_0
16/08/17 21:07:24 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 306810, inMemoryMapOutputs.size() -> 7, commitMemory -> 1801720, usedMemory ->2108530
16/08/17 21:07:24 INFO reduce.LocalFetcher: localfetcher#81 about to shuffle output of map attempt_local518136789_0007_m_000008_0 decomp: 293664 len: 293668 to MEMORY
16/08/17 21:07:24 INFO reduce.InMemoryMapOutput: Read 293664 bytes from map-output for attempt_local518136789_0007_m_000008_0
16/08/17 21:07:24 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 293664, inMemoryMapOutputs.size() -> 8, commitMemory -> 2108530, usedMemory ->2402194
16/08/17 21:07:24 INFO reduce.LocalFetcher: localfetcher#81 about to shuffle output of map attempt_local518136789_0007_m_000014_0 decomp: 295260 len: 295264 to MEMORY
16/08/17 21:07:24 INFO reduce.InMemoryMapOutput: Read 295260 bytes from map-output for attempt_local518136789_0007_m_000014_0
16/08/17 21:07:24 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 295260, inMemoryMapOutputs.size() -> 9, commitMemory -> 2402194, usedMemory ->2697454
16/08/17 21:07:24 INFO reduce.LocalFetcher: localfetcher#81 about to shuffle output of map attempt_local518136789_0007_m_000001_0 decomp: 306862 len: 306866 to MEMORY
16/08/17 21:07:24 INFO reduce.InMemoryMapOutput: Read 306862 bytes from map-output for attempt_local518136789_0007_m_000001_0
16/08/17 21:07:24 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 306862, inMemoryMapOutputs.size() -> 10, commitMemory -> 2697454, usedMemory ->3004316
16/08/17 21:07:24 INFO reduce.LocalFetcher: localfetcher#81 about to shuffle output of map attempt_local518136789_0007_m_000007_0 decomp: 300588 len: 300592 to MEMORY
16/08/17 21:07:24 INFO reduce.InMemoryMapOutput: Read 300588 bytes from map-output for attempt_local518136789_0007_m_000007_0
16/08/17 21:07:24 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 300588, inMemoryMapOutputs.size() -> 11, commitMemory -> 3004316, usedMemory ->3304904
16/08/17 21:07:24 INFO reduce.LocalFetcher: localfetcher#81 about to shuffle output of map attempt_local518136789_0007_m_000013_0 decomp: 282292 len: 282296 to MEMORY
16/08/17 21:07:24 INFO reduce.InMemoryMapOutput: Read 282292 bytes from map-output for attempt_local518136789_0007_m_000013_0
16/08/17 21:07:24 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 282292, inMemoryMapOutputs.size() -> 12, commitMemory -> 3304904, usedMemory ->3587196
16/08/17 21:07:24 INFO reduce.LocalFetcher: localfetcher#81 about to shuffle output of map attempt_local518136789_0007_m_000003_0 decomp: 313036 len: 313040 to MEMORY
16/08/17 21:07:24 INFO reduce.InMemoryMapOutput: Read 313036 bytes from map-output for attempt_local518136789_0007_m_000003_0
16/08/17 21:07:24 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 313036, inMemoryMapOutputs.size() -> 13, commitMemory -> 3587196, usedMemory ->3900232
16/08/17 21:07:24 INFO reduce.LocalFetcher: localfetcher#81 about to shuffle output of map attempt_local518136789_0007_m_000009_0 decomp: 294042 len: 294046 to MEMORY
16/08/17 21:07:24 INFO reduce.InMemoryMapOutput: Read 294042 bytes from map-output for attempt_local518136789_0007_m_000009_0
16/08/17 21:07:24 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 294042, inMemoryMapOutputs.size() -> 14, commitMemory -> 3900232, usedMemory ->4194274
16/08/17 21:07:24 INFO reduce.LocalFetcher: localfetcher#81 about to shuffle output of map attempt_local518136789_0007_m_000015_0 decomp: 293084 len: 293088 to MEMORY
16/08/17 21:07:24 INFO reduce.InMemoryMapOutput: Read 293084 bytes from map-output for attempt_local518136789_0007_m_000015_0
16/08/17 21:07:24 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 293084, inMemoryMapOutputs.size() -> 15, commitMemory -> 4194274, usedMemory ->4487358
16/08/17 21:07:24 INFO reduce.LocalFetcher: localfetcher#81 about to shuffle output of map attempt_local518136789_0007_m_000002_0 decomp: 298120 len: 298124 to MEMORY
16/08/17 21:07:24 INFO reduce.InMemoryMapOutput: Read 298120 bytes from map-output for attempt_local518136789_0007_m_000002_0
16/08/17 21:07:24 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 298120, inMemoryMapOutputs.size() -> 16, commitMemory -> 4487358, usedMemory ->4785478
16/08/17 21:07:24 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:07:24 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:24 INFO reduce.MergeManagerImpl: finalMerge called with 16 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:07:24 INFO mapred.Merger: Merging 16 sorted segments
16/08/17 21:07:24 INFO mapred.Merger: Down to the last merge-pass, with 16 segments left of total size: 4785274 bytes
16/08/17 21:07:24 INFO mapred.MapTask: Finished spill 0
16/08/17 21:07:24 INFO mapred.Task: Task:attempt_local1999544224_0005_m_000015_0 is done. And is in the process of committing
16/08/17 21:07:24 INFO mapred.LocalJobRunner: map
16/08/17 21:07:24 INFO mapred.Task: Task 'attempt_local1999544224_0005_m_000015_0' done.
16/08/17 21:07:24 INFO mapred.LocalJobRunner: Finishing task: attempt_local1999544224_0005_m_000015_0
16/08/17 21:07:24 INFO mapred.LocalJobRunner: map task executor complete.
16/08/17 21:07:24 INFO mapred.LocalJobRunner: Waiting for reduce tasks
16/08/17 21:07:24 INFO mapred.LocalJobRunner: Starting task: attempt_local1999544224_0005_r_000000_0
16/08/17 21:07:24 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:24 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@5dec6311
16/08/17 21:07:24 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=371038624, maxSingleShuffleLimit=92759656, mergeThreshold=244885504, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:07:24 INFO reduce.EventFetcher: attempt_local1999544224_0005_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:07:24 INFO reduce.LocalFetcher: localfetcher#82 about to shuffle output of map attempt_local1999544224_0005_m_000004_0 decomp: 310502 len: 310506 to MEMORY
16/08/17 21:07:24 INFO reduce.InMemoryMapOutput: Read 310502 bytes from map-output for attempt_local1999544224_0005_m_000004_0
16/08/17 21:07:24 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 310502, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->310502
16/08/17 21:07:24 INFO reduce.LocalFetcher: localfetcher#82 about to shuffle output of map attempt_local1999544224_0005_m_000010_0 decomp: 303732 len: 303736 to MEMORY
16/08/17 21:07:24 INFO reduce.InMemoryMapOutput: Read 303732 bytes from map-output for attempt_local1999544224_0005_m_000010_0
16/08/17 21:07:24 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 303732, inMemoryMapOutputs.size() -> 2, commitMemory -> 310502, usedMemory ->614234
16/08/17 21:07:24 INFO reduce.LocalFetcher: localfetcher#82 about to shuffle output of map attempt_local1999544224_0005_m_000003_0 decomp: 314744 len: 314748 to MEMORY
16/08/17 21:07:24 INFO reduce.InMemoryMapOutput: Read 314744 bytes from map-output for attempt_local1999544224_0005_m_000003_0
16/08/17 21:07:24 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 314744, inMemoryMapOutputs.size() -> 3, commitMemory -> 614234, usedMemory ->928978
16/08/17 21:07:24 INFO reduce.LocalFetcher: localfetcher#82 about to shuffle output of map attempt_local1999544224_0005_m_000012_0 decomp: 308780 len: 308784 to MEMORY
16/08/17 21:07:24 INFO reduce.InMemoryMapOutput: Read 308780 bytes from map-output for attempt_local1999544224_0005_m_000012_0
16/08/17 21:07:24 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 308780, inMemoryMapOutputs.size() -> 4, commitMemory -> 928978, usedMemory ->1237758
16/08/17 21:07:24 INFO reduce.LocalFetcher: localfetcher#82 about to shuffle output of map attempt_local1999544224_0005_m_000005_0 decomp: 314886 len: 314890 to MEMORY
16/08/17 21:07:24 INFO reduce.InMemoryMapOutput: Read 314886 bytes from map-output for attempt_local1999544224_0005_m_000005_0
16/08/17 21:07:24 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 314886, inMemoryMapOutputs.size() -> 5, commitMemory -> 1237758, usedMemory ->1552644
16/08/17 21:07:24 INFO reduce.LocalFetcher: localfetcher#82 about to shuffle output of map attempt_local1999544224_0005_m_000011_0 decomp: 291504 len: 291508 to MEMORY
16/08/17 21:07:24 INFO reduce.InMemoryMapOutput: Read 291504 bytes from map-output for attempt_local1999544224_0005_m_000011_0
16/08/17 21:07:24 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 291504, inMemoryMapOutputs.size() -> 6, commitMemory -> 1552644, usedMemory ->1844148
16/08/17 21:07:24 INFO reduce.LocalFetcher: localfetcher#82 about to shuffle output of map attempt_local1999544224_0005_m_000001_0 decomp: 318124 len: 318128 to MEMORY
16/08/17 21:07:24 INFO reduce.InMemoryMapOutput: Read 318124 bytes from map-output for attempt_local1999544224_0005_m_000001_0
16/08/17 21:07:24 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 318124, inMemoryMapOutputs.size() -> 7, commitMemory -> 1844148, usedMemory ->2162272
16/08/17 21:07:24 INFO reduce.LocalFetcher: localfetcher#82 about to shuffle output of map attempt_local1999544224_0005_m_000007_0 decomp: 305246 len: 305250 to MEMORY
16/08/17 21:07:24 INFO reduce.InMemoryMapOutput: Read 305246 bytes from map-output for attempt_local1999544224_0005_m_000007_0
16/08/17 21:07:24 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 305246, inMemoryMapOutputs.size() -> 8, commitMemory -> 2162272, usedMemory ->2467518
16/08/17 21:07:24 INFO reduce.LocalFetcher: localfetcher#82 about to shuffle output of map attempt_local1999544224_0005_m_000013_0 decomp: 306390 len: 306394 to MEMORY
16/08/17 21:07:24 INFO reduce.InMemoryMapOutput: Read 306390 bytes from map-output for attempt_local1999544224_0005_m_000013_0
16/08/17 21:07:24 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 306390, inMemoryMapOutputs.size() -> 9, commitMemory -> 2467518, usedMemory ->2773908
16/08/17 21:07:24 INFO reduce.LocalFetcher: localfetcher#82 about to shuffle output of map attempt_local1999544224_0005_m_000000_0 decomp: 318658 len: 318662 to MEMORY
16/08/17 21:07:24 INFO reduce.InMemoryMapOutput: Read 318658 bytes from map-output for attempt_local1999544224_0005_m_000000_0
16/08/17 21:07:24 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 318658, inMemoryMapOutputs.size() -> 10, commitMemory -> 2773908, usedMemory ->3092566
16/08/17 21:07:24 INFO reduce.LocalFetcher: localfetcher#82 about to shuffle output of map attempt_local1999544224_0005_m_000006_0 decomp: 303640 len: 303644 to MEMORY
16/08/17 21:07:24 INFO reduce.InMemoryMapOutput: Read 303640 bytes from map-output for attempt_local1999544224_0005_m_000006_0
16/08/17 21:07:24 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 303640, inMemoryMapOutputs.size() -> 11, commitMemory -> 3092566, usedMemory ->3396206
16/08/17 21:07:24 INFO reduce.LocalFetcher: localfetcher#82 about to shuffle output of map attempt_local1999544224_0005_m_000009_0 decomp: 297052 len: 297056 to MEMORY
16/08/17 21:07:24 INFO reduce.InMemoryMapOutput: Read 297052 bytes from map-output for attempt_local1999544224_0005_m_000009_0
16/08/17 21:07:24 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 297052, inMemoryMapOutputs.size() -> 12, commitMemory -> 3396206, usedMemory ->3693258
16/08/17 21:07:24 INFO reduce.LocalFetcher: localfetcher#82 about to shuffle output of map attempt_local1999544224_0005_m_000015_0 decomp: 299088 len: 299092 to MEMORY
16/08/17 21:07:24 INFO reduce.InMemoryMapOutput: Read 299088 bytes from map-output for attempt_local1999544224_0005_m_000015_0
16/08/17 21:07:24 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 299088, inMemoryMapOutputs.size() -> 13, commitMemory -> 3693258, usedMemory ->3992346
16/08/17 21:07:24 INFO reduce.LocalFetcher: localfetcher#82 about to shuffle output of map attempt_local1999544224_0005_m_000002_0 decomp: 303676 len: 303680 to MEMORY
16/08/17 21:07:24 INFO reduce.InMemoryMapOutput: Read 303676 bytes from map-output for attempt_local1999544224_0005_m_000002_0
16/08/17 21:07:24 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 303676, inMemoryMapOutputs.size() -> 14, commitMemory -> 3992346, usedMemory ->4296022
16/08/17 21:07:24 INFO reduce.LocalFetcher: localfetcher#82 about to shuffle output of map attempt_local1999544224_0005_m_000008_0 decomp: 304268 len: 304272 to MEMORY
16/08/17 21:07:24 INFO reduce.InMemoryMapOutput: Read 304268 bytes from map-output for attempt_local1999544224_0005_m_000008_0
16/08/17 21:07:24 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 304268, inMemoryMapOutputs.size() -> 15, commitMemory -> 4296022, usedMemory ->4600290
16/08/17 21:07:24 INFO reduce.LocalFetcher: localfetcher#82 about to shuffle output of map attempt_local1999544224_0005_m_000014_0 decomp: 291488 len: 291492 to MEMORY
16/08/17 21:07:24 INFO reduce.InMemoryMapOutput: Read 291488 bytes from map-output for attempt_local1999544224_0005_m_000014_0
16/08/17 21:07:24 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 291488, inMemoryMapOutputs.size() -> 16, commitMemory -> 4600290, usedMemory ->4891778
16/08/17 21:07:24 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:07:24 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:24 INFO reduce.MergeManagerImpl: finalMerge called with 16 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:07:24 INFO mapred.Merger: Merging 16 sorted segments
16/08/17 21:07:24 INFO mapred.Merger: Down to the last merge-pass, with 16 segments left of total size: 4891458 bytes
16/08/17 21:07:24 INFO reduce.MergeManagerImpl: Merged 16 segments, 4785478 bytes to disk to satisfy reduce memory limit
16/08/17 21:07:24 INFO reduce.MergeManagerImpl: Merging 1 files, 4785452 bytes from disk
16/08/17 21:07:24 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:07:24 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:07:24 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 4785430 bytes
16/08/17 21:07:24 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:24 INFO reduce.MergeManagerImpl: Merged 16 segments, 4891778 bytes to disk to satisfy reduce memory limit
16/08/17 21:07:24 INFO reduce.MergeManagerImpl: Merging 1 files, 4891752 bytes from disk
16/08/17 21:07:24 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:07:24 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:07:24 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 4891738 bytes
16/08/17 21:07:24 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:25 INFO mapred.Task: Task:attempt_local1999544224_0005_r_000000_0 is done. And is in the process of committing
16/08/17 21:07:25 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:25 INFO mapred.Task: Task attempt_local1999544224_0005_r_000000_0 is allowed to commit now
16/08/17 21:07:25 INFO output.FileOutputCommitter: Saved output of task 'attempt_local1999544224_0005_r_000000_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/f_given_e_phrase/_temporary/0/task_local1999544224_0005_r_000000
16/08/17 21:07:25 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:07:25 INFO mapred.Task: Task 'attempt_local1999544224_0005_r_000000_0' done.
16/08/17 21:07:25 INFO mapred.LocalJobRunner: Finishing task: attempt_local1999544224_0005_r_000000_0
16/08/17 21:07:25 INFO mapred.LocalJobRunner: Starting task: attempt_local1999544224_0005_r_000001_0
16/08/17 21:07:25 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:25 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@54eebc66
16/08/17 21:07:25 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=371038624, maxSingleShuffleLimit=92759656, mergeThreshold=244885504, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:07:25 INFO reduce.EventFetcher: attempt_local1999544224_0005_r_000001_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:07:25 INFO reduce.LocalFetcher: localfetcher#83 about to shuffle output of map attempt_local1999544224_0005_m_000004_0 decomp: 306200 len: 306204 to MEMORY
16/08/17 21:07:25 INFO reduce.InMemoryMapOutput: Read 306200 bytes from map-output for attempt_local1999544224_0005_m_000004_0
16/08/17 21:07:25 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 306200, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->306200
16/08/17 21:07:25 INFO reduce.LocalFetcher: localfetcher#83 about to shuffle output of map attempt_local1999544224_0005_m_000010_0 decomp: 306618 len: 306622 to MEMORY
16/08/17 21:07:25 INFO reduce.InMemoryMapOutput: Read 306618 bytes from map-output for attempt_local1999544224_0005_m_000010_0
16/08/17 21:07:25 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 306618, inMemoryMapOutputs.size() -> 2, commitMemory -> 306200, usedMemory ->612818
16/08/17 21:07:25 INFO reduce.LocalFetcher: localfetcher#83 about to shuffle output of map attempt_local1999544224_0005_m_000003_0 decomp: 298598 len: 298602 to MEMORY
16/08/17 21:07:25 INFO reduce.InMemoryMapOutput: Read 298598 bytes from map-output for attempt_local1999544224_0005_m_000003_0
16/08/17 21:07:25 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 298598, inMemoryMapOutputs.size() -> 3, commitMemory -> 612818, usedMemory ->911416
16/08/17 21:07:25 INFO reduce.LocalFetcher: localfetcher#83 about to shuffle output of map attempt_local1999544224_0005_m_000012_0 decomp: 296918 len: 296922 to MEMORY
16/08/17 21:07:25 INFO reduce.InMemoryMapOutput: Read 296918 bytes from map-output for attempt_local1999544224_0005_m_000012_0
16/08/17 21:07:25 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 296918, inMemoryMapOutputs.size() -> 4, commitMemory -> 911416, usedMemory ->1208334
16/08/17 21:07:25 INFO reduce.LocalFetcher: localfetcher#83 about to shuffle output of map attempt_local1999544224_0005_m_000005_0 decomp: 307356 len: 307360 to MEMORY
16/08/17 21:07:25 INFO reduce.InMemoryMapOutput: Read 307356 bytes from map-output for attempt_local1999544224_0005_m_000005_0
16/08/17 21:07:25 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 307356, inMemoryMapOutputs.size() -> 5, commitMemory -> 1208334, usedMemory ->1515690
16/08/17 21:07:25 INFO reduce.LocalFetcher: localfetcher#83 about to shuffle output of map attempt_local1999544224_0005_m_000011_0 decomp: 303562 len: 303566 to MEMORY
16/08/17 21:07:25 INFO reduce.InMemoryMapOutput: Read 303562 bytes from map-output for attempt_local1999544224_0005_m_000011_0
16/08/17 21:07:25 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 303562, inMemoryMapOutputs.size() -> 6, commitMemory -> 1515690, usedMemory ->1819252
16/08/17 21:07:25 INFO reduce.LocalFetcher: localfetcher#83 about to shuffle output of map attempt_local1999544224_0005_m_000001_0 decomp: 323406 len: 323410 to MEMORY
16/08/17 21:07:25 INFO reduce.InMemoryMapOutput: Read 323406 bytes from map-output for attempt_local1999544224_0005_m_000001_0
16/08/17 21:07:25 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 323406, inMemoryMapOutputs.size() -> 7, commitMemory -> 1819252, usedMemory ->2142658
16/08/17 21:07:25 INFO reduce.LocalFetcher: localfetcher#83 about to shuffle output of map attempt_local1999544224_0005_m_000007_0 decomp: 314364 len: 314368 to MEMORY
16/08/17 21:07:25 INFO reduce.InMemoryMapOutput: Read 314364 bytes from map-output for attempt_local1999544224_0005_m_000007_0
16/08/17 21:07:25 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 314364, inMemoryMapOutputs.size() -> 8, commitMemory -> 2142658, usedMemory ->2457022
16/08/17 21:07:25 INFO reduce.LocalFetcher: localfetcher#83 about to shuffle output of map attempt_local1999544224_0005_m_000013_0 decomp: 308622 len: 308626 to MEMORY
16/08/17 21:07:25 INFO reduce.InMemoryMapOutput: Read 308622 bytes from map-output for attempt_local1999544224_0005_m_000013_0
16/08/17 21:07:25 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 308622, inMemoryMapOutputs.size() -> 9, commitMemory -> 2457022, usedMemory ->2765644
16/08/17 21:07:25 INFO reduce.LocalFetcher: localfetcher#83 about to shuffle output of map attempt_local1999544224_0005_m_000000_0 decomp: 348246 len: 348250 to MEMORY
16/08/17 21:07:25 INFO reduce.InMemoryMapOutput: Read 348246 bytes from map-output for attempt_local1999544224_0005_m_000000_0
16/08/17 21:07:25 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 348246, inMemoryMapOutputs.size() -> 10, commitMemory -> 2765644, usedMemory ->3113890
16/08/17 21:07:25 INFO reduce.LocalFetcher: localfetcher#83 about to shuffle output of map attempt_local1999544224_0005_m_000006_0 decomp: 302692 len: 302696 to MEMORY
16/08/17 21:07:25 INFO reduce.InMemoryMapOutput: Read 302692 bytes from map-output for attempt_local1999544224_0005_m_000006_0
16/08/17 21:07:25 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 302692, inMemoryMapOutputs.size() -> 11, commitMemory -> 3113890, usedMemory ->3416582
16/08/17 21:07:25 INFO reduce.LocalFetcher: localfetcher#83 about to shuffle output of map attempt_local1999544224_0005_m_000009_0 decomp: 297866 len: 297870 to MEMORY
16/08/17 21:07:25 INFO reduce.InMemoryMapOutput: Read 297866 bytes from map-output for attempt_local1999544224_0005_m_000009_0
16/08/17 21:07:25 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 297866, inMemoryMapOutputs.size() -> 12, commitMemory -> 3416582, usedMemory ->3714448
16/08/17 21:07:25 INFO reduce.LocalFetcher: localfetcher#83 about to shuffle output of map attempt_local1999544224_0005_m_000015_0 decomp: 303196 len: 303200 to MEMORY
16/08/17 21:07:25 INFO reduce.InMemoryMapOutput: Read 303196 bytes from map-output for attempt_local1999544224_0005_m_000015_0
16/08/17 21:07:25 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 303196, inMemoryMapOutputs.size() -> 13, commitMemory -> 3714448, usedMemory ->4017644
16/08/17 21:07:25 INFO reduce.LocalFetcher: localfetcher#83 about to shuffle output of map attempt_local1999544224_0005_m_000002_0 decomp: 316484 len: 316488 to MEMORY
16/08/17 21:07:25 INFO reduce.InMemoryMapOutput: Read 316484 bytes from map-output for attempt_local1999544224_0005_m_000002_0
16/08/17 21:07:25 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 316484, inMemoryMapOutputs.size() -> 14, commitMemory -> 4017644, usedMemory ->4334128
16/08/17 21:07:25 INFO reduce.LocalFetcher: localfetcher#83 about to shuffle output of map attempt_local1999544224_0005_m_000008_0 decomp: 298790 len: 298794 to MEMORY
16/08/17 21:07:25 INFO reduce.InMemoryMapOutput: Read 298790 bytes from map-output for attempt_local1999544224_0005_m_000008_0
16/08/17 21:07:25 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 298790, inMemoryMapOutputs.size() -> 15, commitMemory -> 4334128, usedMemory ->4632918
16/08/17 21:07:25 INFO reduce.LocalFetcher: localfetcher#83 about to shuffle output of map attempt_local1999544224_0005_m_000014_0 decomp: 310130 len: 310134 to MEMORY
16/08/17 21:07:25 INFO reduce.InMemoryMapOutput: Read 310130 bytes from map-output for attempt_local1999544224_0005_m_000014_0
16/08/17 21:07:25 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 310130, inMemoryMapOutputs.size() -> 16, commitMemory -> 4632918, usedMemory ->4943048
16/08/17 21:07:25 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:07:25 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:25 INFO reduce.MergeManagerImpl: finalMerge called with 16 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:07:25 INFO mapred.Merger: Merging 16 sorted segments
16/08/17 21:07:25 INFO mapred.Merger: Down to the last merge-pass, with 16 segments left of total size: 4942652 bytes
16/08/17 21:07:25 INFO mapred.Task: Task:attempt_local518136789_0007_r_000000_0 is done. And is in the process of committing
16/08/17 21:07:25 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:25 INFO mapred.Task: Task attempt_local518136789_0007_r_000000_0 is allowed to commit now
16/08/17 21:07:25 INFO output.FileOutputCommitter: Saved output of task 'attempt_local518136789_0007_r_000000_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/e_given_f_phrase/_temporary/0/task_local518136789_0007_r_000000
16/08/17 21:07:25 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:07:25 INFO mapred.Task: Task 'attempt_local518136789_0007_r_000000_0' done.
16/08/17 21:07:25 INFO mapred.LocalJobRunner: Finishing task: attempt_local518136789_0007_r_000000_0
16/08/17 21:07:25 INFO mapred.LocalJobRunner: Starting task: attempt_local518136789_0007_r_000001_0
16/08/17 21:07:25 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:25 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@3c821b05
16/08/17 21:07:25 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=371038624, maxSingleShuffleLimit=92759656, mergeThreshold=244885504, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:07:25 INFO reduce.EventFetcher: attempt_local518136789_0007_r_000001_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:07:25 INFO reduce.LocalFetcher: localfetcher#84 about to shuffle output of map attempt_local518136789_0007_m_000011_0 decomp: 308426 len: 308430 to MEMORY
16/08/17 21:07:25 INFO reduce.InMemoryMapOutput: Read 308426 bytes from map-output for attempt_local518136789_0007_m_000011_0
16/08/17 21:07:25 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 308426, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->308426
16/08/17 21:07:25 INFO reduce.LocalFetcher: localfetcher#84 about to shuffle output of map attempt_local518136789_0007_m_000004_0 decomp: 301356 len: 301360 to MEMORY
16/08/17 21:07:25 INFO reduce.InMemoryMapOutput: Read 301356 bytes from map-output for attempt_local518136789_0007_m_000004_0
16/08/17 21:07:25 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 301356, inMemoryMapOutputs.size() -> 2, commitMemory -> 308426, usedMemory ->609782
16/08/17 21:07:25 INFO reduce.LocalFetcher: localfetcher#84 about to shuffle output of map attempt_local518136789_0007_m_000010_0 decomp: 296148 len: 296152 to MEMORY
16/08/17 21:07:25 INFO reduce.InMemoryMapOutput: Read 296148 bytes from map-output for attempt_local518136789_0007_m_000010_0
16/08/17 21:07:25 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 296148, inMemoryMapOutputs.size() -> 3, commitMemory -> 609782, usedMemory ->905930
16/08/17 21:07:25 INFO reduce.LocalFetcher: localfetcher#84 about to shuffle output of map attempt_local518136789_0007_m_000000_0 decomp: 337902 len: 337906 to MEMORY
16/08/17 21:07:25 INFO reduce.InMemoryMapOutput: Read 337902 bytes from map-output for attempt_local518136789_0007_m_000000_0
16/08/17 21:07:25 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 337902, inMemoryMapOutputs.size() -> 4, commitMemory -> 905930, usedMemory ->1243832
16/08/17 21:07:25 INFO reduce.LocalFetcher: localfetcher#84 about to shuffle output of map attempt_local518136789_0007_m_000006_0 decomp: 314912 len: 314916 to MEMORY
16/08/17 21:07:25 INFO reduce.InMemoryMapOutput: Read 314912 bytes from map-output for attempt_local518136789_0007_m_000006_0
16/08/17 21:07:25 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 314912, inMemoryMapOutputs.size() -> 5, commitMemory -> 1243832, usedMemory ->1558744
16/08/17 21:07:25 INFO reduce.LocalFetcher: localfetcher#84 about to shuffle output of map attempt_local518136789_0007_m_000012_0 decomp: 304720 len: 304724 to MEMORY
16/08/17 21:07:25 INFO reduce.InMemoryMapOutput: Read 304720 bytes from map-output for attempt_local518136789_0007_m_000012_0
16/08/17 21:07:25 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 304720, inMemoryMapOutputs.size() -> 6, commitMemory -> 1558744, usedMemory ->1863464
16/08/17 21:07:25 INFO reduce.LocalFetcher: localfetcher#84 about to shuffle output of map attempt_local518136789_0007_m_000005_0 decomp: 321080 len: 321084 to MEMORY
16/08/17 21:07:25 INFO reduce.InMemoryMapOutput: Read 321080 bytes from map-output for attempt_local518136789_0007_m_000005_0
16/08/17 21:07:25 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 321080, inMemoryMapOutputs.size() -> 7, commitMemory -> 1863464, usedMemory ->2184544
16/08/17 21:07:25 INFO reduce.LocalFetcher: localfetcher#84 about to shuffle output of map attempt_local518136789_0007_m_000008_0 decomp: 302698 len: 302702 to MEMORY
16/08/17 21:07:25 INFO reduce.InMemoryMapOutput: Read 302698 bytes from map-output for attempt_local518136789_0007_m_000008_0
16/08/17 21:07:25 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 302698, inMemoryMapOutputs.size() -> 8, commitMemory -> 2184544, usedMemory ->2487242
16/08/17 21:07:25 INFO reduce.LocalFetcher: localfetcher#84 about to shuffle output of map attempt_local518136789_0007_m_000014_0 decomp: 293556 len: 293560 to MEMORY
16/08/17 21:07:25 INFO reduce.InMemoryMapOutput: Read 293556 bytes from map-output for attempt_local518136789_0007_m_000014_0
16/08/17 21:07:25 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 293556, inMemoryMapOutputs.size() -> 9, commitMemory -> 2487242, usedMemory ->2780798
16/08/17 21:07:25 INFO reduce.LocalFetcher: localfetcher#84 about to shuffle output of map attempt_local518136789_0007_m_000001_0 decomp: 306304 len: 306308 to MEMORY
16/08/17 21:07:25 INFO reduce.InMemoryMapOutput: Read 306304 bytes from map-output for attempt_local518136789_0007_m_000001_0
16/08/17 21:07:25 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 306304, inMemoryMapOutputs.size() -> 10, commitMemory -> 2780798, usedMemory ->3087102
16/08/17 21:07:25 INFO reduce.LocalFetcher: localfetcher#84 about to shuffle output of map attempt_local518136789_0007_m_000007_0 decomp: 320548 len: 320552 to MEMORY
16/08/17 21:07:25 INFO reduce.InMemoryMapOutput: Read 320548 bytes from map-output for attempt_local518136789_0007_m_000007_0
16/08/17 21:07:25 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 320548, inMemoryMapOutputs.size() -> 11, commitMemory -> 3087102, usedMemory ->3407650
16/08/17 21:07:25 INFO reduce.LocalFetcher: localfetcher#84 about to shuffle output of map attempt_local518136789_0007_m_000013_0 decomp: 298998 len: 299002 to MEMORY
16/08/17 21:07:25 INFO reduce.InMemoryMapOutput: Read 298998 bytes from map-output for attempt_local518136789_0007_m_000013_0
16/08/17 21:07:25 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 298998, inMemoryMapOutputs.size() -> 12, commitMemory -> 3407650, usedMemory ->3706648
16/08/17 21:07:25 INFO reduce.LocalFetcher: localfetcher#84 about to shuffle output of map attempt_local518136789_0007_m_000003_0 decomp: 286878 len: 286882 to MEMORY
16/08/17 21:07:25 INFO reduce.InMemoryMapOutput: Read 286878 bytes from map-output for attempt_local518136789_0007_m_000003_0
16/08/17 21:07:25 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 286878, inMemoryMapOutputs.size() -> 13, commitMemory -> 3706648, usedMemory ->3993526
16/08/17 21:07:25 INFO reduce.LocalFetcher: localfetcher#84 about to shuffle output of map attempt_local518136789_0007_m_000009_0 decomp: 292598 len: 292602 to MEMORY
16/08/17 21:07:25 INFO reduce.InMemoryMapOutput: Read 292598 bytes from map-output for attempt_local518136789_0007_m_000009_0
16/08/17 21:07:25 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 292598, inMemoryMapOutputs.size() -> 14, commitMemory -> 3993526, usedMemory ->4286124
16/08/17 21:07:25 INFO reduce.LocalFetcher: localfetcher#84 about to shuffle output of map attempt_local518136789_0007_m_000015_0 decomp: 301154 len: 301158 to MEMORY
16/08/17 21:07:25 INFO reduce.InMemoryMapOutput: Read 301154 bytes from map-output for attempt_local518136789_0007_m_000015_0
16/08/17 21:07:25 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 301154, inMemoryMapOutputs.size() -> 15, commitMemory -> 4286124, usedMemory ->4587278
16/08/17 21:07:25 INFO reduce.LocalFetcher: localfetcher#84 about to shuffle output of map attempt_local518136789_0007_m_000002_0 decomp: 316980 len: 316984 to MEMORY
16/08/17 21:07:25 INFO reduce.InMemoryMapOutput: Read 316980 bytes from map-output for attempt_local518136789_0007_m_000002_0
16/08/17 21:07:25 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 316980, inMemoryMapOutputs.size() -> 16, commitMemory -> 4587278, usedMemory ->4904258
16/08/17 21:07:25 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:07:25 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:25 INFO reduce.MergeManagerImpl: finalMerge called with 16 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:07:25 INFO mapred.Merger: Merging 16 sorted segments
16/08/17 21:07:25 INFO mapred.Merger: Down to the last merge-pass, with 16 segments left of total size: 4903958 bytes
16/08/17 21:07:25 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)
	at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
16/08/17 21:07:25 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)
	at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
16/08/17 21:07:25 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)
	at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
16/08/17 21:07:25 INFO reduce.MergeManagerImpl: Merged 16 segments, 4904258 bytes to disk to satisfy reduce memory limit
16/08/17 21:07:25 INFO reduce.MergeManagerImpl: Merging 1 files, 4904232 bytes from disk
16/08/17 21:07:25 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:07:25 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:07:25 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 4904210 bytes
16/08/17 21:07:25 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:25 INFO reduce.MergeManagerImpl: Merged 16 segments, 4943048 bytes to disk to satisfy reduce memory limit
16/08/17 21:07:25 INFO reduce.MergeManagerImpl: Merging 1 files, 4943022 bytes from disk
16/08/17 21:07:25 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:07:25 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:07:25 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 4942992 bytes
16/08/17 21:07:25 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:25 INFO mapred.Task: Task:attempt_local518136789_0007_r_000001_0 is done. And is in the process of committing
16/08/17 21:07:25 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:25 INFO mapred.Task: Task attempt_local518136789_0007_r_000001_0 is allowed to commit now
16/08/17 21:07:25 INFO output.FileOutputCommitter: Saved output of task 'attempt_local518136789_0007_r_000001_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/e_given_f_phrase/_temporary/0/task_local518136789_0007_r_000001
16/08/17 21:07:25 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:07:25 INFO mapred.Task: Task 'attempt_local518136789_0007_r_000001_0' done.
16/08/17 21:07:25 INFO mapred.LocalJobRunner: Finishing task: attempt_local518136789_0007_r_000001_0
16/08/17 21:07:25 INFO mapred.LocalJobRunner: Starting task: attempt_local518136789_0007_r_000002_0
16/08/17 21:07:25 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:25 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@58225b8c
16/08/17 21:07:25 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=369570592, maxSingleShuffleLimit=92392648, mergeThreshold=243916608, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:07:25 INFO reduce.EventFetcher: attempt_local518136789_0007_r_000002_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:07:25 INFO reduce.LocalFetcher: localfetcher#85 about to shuffle output of map attempt_local518136789_0007_m_000011_0 decomp: 292804 len: 292808 to MEMORY
16/08/17 21:07:25 INFO reduce.InMemoryMapOutput: Read 292804 bytes from map-output for attempt_local518136789_0007_m_000011_0
16/08/17 21:07:25 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 292804, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->292804
16/08/17 21:07:25 INFO reduce.LocalFetcher: localfetcher#85 about to shuffle output of map attempt_local518136789_0007_m_000004_0 decomp: 304412 len: 304416 to MEMORY
16/08/17 21:07:25 INFO reduce.InMemoryMapOutput: Read 304412 bytes from map-output for attempt_local518136789_0007_m_000004_0
16/08/17 21:07:25 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 304412, inMemoryMapOutputs.size() -> 2, commitMemory -> 292804, usedMemory ->597216
16/08/17 21:07:25 INFO reduce.LocalFetcher: localfetcher#85 about to shuffle output of map attempt_local518136789_0007_m_000010_0 decomp: 297676 len: 297680 to MEMORY
16/08/17 21:07:25 INFO reduce.InMemoryMapOutput: Read 297676 bytes from map-output for attempt_local518136789_0007_m_000010_0
16/08/17 21:07:25 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 297676, inMemoryMapOutputs.size() -> 3, commitMemory -> 597216, usedMemory ->894892
16/08/17 21:07:25 INFO reduce.LocalFetcher: localfetcher#85 about to shuffle output of map attempt_local518136789_0007_m_000000_0 decomp: 307150 len: 307154 to MEMORY
16/08/17 21:07:25 INFO reduce.InMemoryMapOutput: Read 307150 bytes from map-output for attempt_local518136789_0007_m_000000_0
16/08/17 21:07:25 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 307150, inMemoryMapOutputs.size() -> 4, commitMemory -> 894892, usedMemory ->1202042
16/08/17 21:07:25 INFO reduce.LocalFetcher: localfetcher#85 about to shuffle output of map attempt_local518136789_0007_m_000006_0 decomp: 317044 len: 317048 to MEMORY
16/08/17 21:07:25 INFO reduce.InMemoryMapOutput: Read 317044 bytes from map-output for attempt_local518136789_0007_m_000006_0
16/08/17 21:07:25 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 317044, inMemoryMapOutputs.size() -> 5, commitMemory -> 1202042, usedMemory ->1519086
16/08/17 21:07:25 INFO reduce.LocalFetcher: localfetcher#85 about to shuffle output of map attempt_local518136789_0007_m_000012_0 decomp: 291522 len: 291526 to MEMORY
16/08/17 21:07:25 INFO reduce.InMemoryMapOutput: Read 291522 bytes from map-output for attempt_local518136789_0007_m_000012_0
16/08/17 21:07:25 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 291522, inMemoryMapOutputs.size() -> 6, commitMemory -> 1519086, usedMemory ->1810608
16/08/17 21:07:25 INFO reduce.LocalFetcher: localfetcher#85 about to shuffle output of map attempt_local518136789_0007_m_000005_0 decomp: 304724 len: 304728 to MEMORY
16/08/17 21:07:25 INFO reduce.InMemoryMapOutput: Read 304724 bytes from map-output for attempt_local518136789_0007_m_000005_0
16/08/17 21:07:25 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 304724, inMemoryMapOutputs.size() -> 7, commitMemory -> 1810608, usedMemory ->2115332
16/08/17 21:07:25 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)
	at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
16/08/17 21:07:25 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)
	at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
16/08/17 21:07:25 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)
	at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
16/08/17 21:07:25 INFO reduce.LocalFetcher: localfetcher#85 about to shuffle output of map attempt_local518136789_0007_m_000008_0 decomp: 294358 len: 294362 to MEMORY
16/08/17 21:07:25 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)
	at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
16/08/17 21:07:25 INFO reduce.InMemoryMapOutput: Read 294358 bytes from map-output for attempt_local518136789_0007_m_000008_0
16/08/17 21:07:25 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 294358, inMemoryMapOutputs.size() -> 8, commitMemory -> 2115332, usedMemory ->2409690
16/08/17 21:07:25 INFO reduce.LocalFetcher: localfetcher#85 about to shuffle output of map attempt_local518136789_0007_m_000014_0 decomp: 297814 len: 297818 to MEMORY
16/08/17 21:07:25 INFO reduce.InMemoryMapOutput: Read 297814 bytes from map-output for attempt_local518136789_0007_m_000014_0
16/08/17 21:07:25 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 297814, inMemoryMapOutputs.size() -> 9, commitMemory -> 2409690, usedMemory ->2707504
16/08/17 21:07:25 INFO reduce.LocalFetcher: localfetcher#85 about to shuffle output of map attempt_local518136789_0007_m_000001_0 decomp: 309346 len: 309350 to MEMORY
16/08/17 21:07:25 INFO reduce.InMemoryMapOutput: Read 309346 bytes from map-output for attempt_local518136789_0007_m_000001_0
16/08/17 21:07:25 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 309346, inMemoryMapOutputs.size() -> 10, commitMemory -> 2707504, usedMemory ->3016850
16/08/17 21:07:25 INFO reduce.LocalFetcher: localfetcher#85 about to shuffle output of map attempt_local518136789_0007_m_000007_0 decomp: 311656 len: 311660 to MEMORY
16/08/17 21:07:25 INFO reduce.InMemoryMapOutput: Read 311656 bytes from map-output for attempt_local518136789_0007_m_000007_0
16/08/17 21:07:25 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 311656, inMemoryMapOutputs.size() -> 11, commitMemory -> 3016850, usedMemory ->3328506
16/08/17 21:07:25 INFO reduce.LocalFetcher: localfetcher#85 about to shuffle output of map attempt_local518136789_0007_m_000013_0 decomp: 299028 len: 299032 to MEMORY
16/08/17 21:07:25 INFO reduce.InMemoryMapOutput: Read 299028 bytes from map-output for attempt_local518136789_0007_m_000013_0
16/08/17 21:07:25 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 299028, inMemoryMapOutputs.size() -> 12, commitMemory -> 3328506, usedMemory ->3627534
16/08/17 21:07:25 INFO reduce.LocalFetcher: localfetcher#85 about to shuffle output of map attempt_local518136789_0007_m_000003_0 decomp: 300772 len: 300776 to MEMORY
16/08/17 21:07:25 INFO reduce.InMemoryMapOutput: Read 300772 bytes from map-output for attempt_local518136789_0007_m_000003_0
16/08/17 21:07:25 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 300772, inMemoryMapOutputs.size() -> 13, commitMemory -> 3627534, usedMemory ->3928306
16/08/17 21:07:25 INFO reduce.LocalFetcher: localfetcher#85 about to shuffle output of map attempt_local518136789_0007_m_000009_0 decomp: 297032 len: 297036 to MEMORY
16/08/17 21:07:25 INFO reduce.InMemoryMapOutput: Read 297032 bytes from map-output for attempt_local518136789_0007_m_000009_0
16/08/17 21:07:25 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 297032, inMemoryMapOutputs.size() -> 14, commitMemory -> 3928306, usedMemory ->4225338
16/08/17 21:07:25 INFO reduce.LocalFetcher: localfetcher#85 about to shuffle output of map attempt_local518136789_0007_m_000015_0 decomp: 295276 len: 295280 to MEMORY
16/08/17 21:07:25 INFO reduce.InMemoryMapOutput: Read 295276 bytes from map-output for attempt_local518136789_0007_m_000015_0
16/08/17 21:07:25 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 295276, inMemoryMapOutputs.size() -> 15, commitMemory -> 4225338, usedMemory ->4520614
16/08/17 21:07:25 INFO reduce.LocalFetcher: localfetcher#85 about to shuffle output of map attempt_local518136789_0007_m_000002_0 decomp: 303360 len: 303364 to MEMORY
16/08/17 21:07:25 INFO reduce.InMemoryMapOutput: Read 303360 bytes from map-output for attempt_local518136789_0007_m_000002_0
16/08/17 21:07:25 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 303360, inMemoryMapOutputs.size() -> 16, commitMemory -> 4520614, usedMemory ->4823974
16/08/17 21:07:25 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:07:25 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:25 INFO mapred.Task: Task:attempt_local1999544224_0005_r_000001_0 is done. And is in the process of committing
16/08/17 21:07:25 INFO reduce.MergeManagerImpl: finalMerge called with 16 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:07:25 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:25 INFO mapred.Task: Task attempt_local1999544224_0005_r_000001_0 is allowed to commit now
16/08/17 21:07:25 INFO output.FileOutputCommitter: Saved output of task 'attempt_local1999544224_0005_r_000001_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/f_given_e_phrase/_temporary/0/task_local1999544224_0005_r_000001
16/08/17 21:07:25 INFO mapred.Merger: Merging 16 sorted segments
16/08/17 21:07:25 INFO mapred.Merger: Down to the last merge-pass, with 16 segments left of total size: 4823602 bytes
16/08/17 21:07:25 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:07:25 INFO mapred.Task: Task 'attempt_local1999544224_0005_r_000001_0' done.
16/08/17 21:07:25 INFO mapred.LocalJobRunner: Finishing task: attempt_local1999544224_0005_r_000001_0
16/08/17 21:07:25 INFO mapred.LocalJobRunner: Starting task: attempt_local1999544224_0005_r_000002_0
16/08/17 21:07:25 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:25 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@4fea735e
16/08/17 21:07:25 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=369570592, maxSingleShuffleLimit=92392648, mergeThreshold=243916608, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:07:25 INFO reduce.EventFetcher: attempt_local1999544224_0005_r_000002_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:07:25 INFO reduce.LocalFetcher: localfetcher#86 about to shuffle output of map attempt_local1999544224_0005_m_000004_0 decomp: 320830 len: 320834 to MEMORY
16/08/17 21:07:25 INFO reduce.InMemoryMapOutput: Read 320830 bytes from map-output for attempt_local1999544224_0005_m_000004_0
16/08/17 21:07:25 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 320830, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->320830
16/08/17 21:07:25 INFO reduce.LocalFetcher: localfetcher#86 about to shuffle output of map attempt_local1999544224_0005_m_000010_0 decomp: 297242 len: 297246 to MEMORY
16/08/17 21:07:25 INFO reduce.InMemoryMapOutput: Read 297242 bytes from map-output for attempt_local1999544224_0005_m_000010_0
16/08/17 21:07:25 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 297242, inMemoryMapOutputs.size() -> 2, commitMemory -> 320830, usedMemory ->618072
16/08/17 21:07:25 INFO reduce.LocalFetcher: localfetcher#86 about to shuffle output of map attempt_local1999544224_0005_m_000003_0 decomp: 297444 len: 297448 to MEMORY
16/08/17 21:07:25 INFO reduce.InMemoryMapOutput: Read 297444 bytes from map-output for attempt_local1999544224_0005_m_000003_0
16/08/17 21:07:25 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 297444, inMemoryMapOutputs.size() -> 3, commitMemory -> 618072, usedMemory ->915516
16/08/17 21:07:25 INFO reduce.LocalFetcher: localfetcher#86 about to shuffle output of map attempt_local1999544224_0005_m_000012_0 decomp: 296438 len: 296442 to MEMORY
16/08/17 21:07:25 INFO reduce.InMemoryMapOutput: Read 296438 bytes from map-output for attempt_local1999544224_0005_m_000012_0
16/08/17 21:07:25 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 296438, inMemoryMapOutputs.size() -> 4, commitMemory -> 915516, usedMemory ->1211954
16/08/17 21:07:25 INFO reduce.LocalFetcher: localfetcher#86 about to shuffle output of map attempt_local1999544224_0005_m_000005_0 decomp: 309228 len: 309232 to MEMORY
16/08/17 21:07:25 INFO reduce.InMemoryMapOutput: Read 309228 bytes from map-output for attempt_local1999544224_0005_m_000005_0
16/08/17 21:07:25 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 309228, inMemoryMapOutputs.size() -> 5, commitMemory -> 1211954, usedMemory ->1521182
16/08/17 21:07:25 INFO reduce.LocalFetcher: localfetcher#86 about to shuffle output of map attempt_local1999544224_0005_m_000011_0 decomp: 299156 len: 299160 to MEMORY
16/08/17 21:07:25 INFO reduce.InMemoryMapOutput: Read 299156 bytes from map-output for attempt_local1999544224_0005_m_000011_0
16/08/17 21:07:25 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 299156, inMemoryMapOutputs.size() -> 6, commitMemory -> 1521182, usedMemory ->1820338
16/08/17 21:07:25 INFO reduce.LocalFetcher: localfetcher#86 about to shuffle output of map attempt_local1999544224_0005_m_000001_0 decomp: 297898 len: 297902 to MEMORY
16/08/17 21:07:25 INFO reduce.InMemoryMapOutput: Read 297898 bytes from map-output for attempt_local1999544224_0005_m_000001_0
16/08/17 21:07:25 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 297898, inMemoryMapOutputs.size() -> 7, commitMemory -> 1820338, usedMemory ->2118236
16/08/17 21:07:25 INFO reduce.LocalFetcher: localfetcher#86 about to shuffle output of map attempt_local1999544224_0005_m_000007_0 decomp: 319724 len: 319728 to MEMORY
16/08/17 21:07:25 INFO reduce.InMemoryMapOutput: Read 319724 bytes from map-output for attempt_local1999544224_0005_m_000007_0
16/08/17 21:07:25 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 319724, inMemoryMapOutputs.size() -> 8, commitMemory -> 2118236, usedMemory ->2437960
16/08/17 21:07:25 INFO reduce.LocalFetcher: localfetcher#86 about to shuffle output of map attempt_local1999544224_0005_m_000013_0 decomp: 300664 len: 300668 to MEMORY
16/08/17 21:07:25 INFO reduce.InMemoryMapOutput: Read 300664 bytes from map-output for attempt_local1999544224_0005_m_000013_0
16/08/17 21:07:25 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 300664, inMemoryMapOutputs.size() -> 9, commitMemory -> 2437960, usedMemory ->2738624
16/08/17 21:07:25 INFO reduce.LocalFetcher: localfetcher#86 about to shuffle output of map attempt_local1999544224_0005_m_000000_0 decomp: 300556 len: 300560 to MEMORY
16/08/17 21:07:25 INFO reduce.InMemoryMapOutput: Read 300556 bytes from map-output for attempt_local1999544224_0005_m_000000_0
16/08/17 21:07:25 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 300556, inMemoryMapOutputs.size() -> 10, commitMemory -> 2738624, usedMemory ->3039180
16/08/17 21:07:25 INFO reduce.LocalFetcher: localfetcher#86 about to shuffle output of map attempt_local1999544224_0005_m_000006_0 decomp: 303040 len: 303044 to MEMORY
16/08/17 21:07:25 INFO reduce.InMemoryMapOutput: Read 303040 bytes from map-output for attempt_local1999544224_0005_m_000006_0
16/08/17 21:07:25 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 303040, inMemoryMapOutputs.size() -> 11, commitMemory -> 3039180, usedMemory ->3342220
16/08/17 21:07:25 INFO reduce.LocalFetcher: localfetcher#86 about to shuffle output of map attempt_local1999544224_0005_m_000009_0 decomp: 296902 len: 296906 to MEMORY
16/08/17 21:07:25 INFO reduce.InMemoryMapOutput: Read 296902 bytes from map-output for attempt_local1999544224_0005_m_000009_0
16/08/17 21:07:25 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 296902, inMemoryMapOutputs.size() -> 12, commitMemory -> 3342220, usedMemory ->3639122
16/08/17 21:07:25 INFO reduce.LocalFetcher: localfetcher#86 about to shuffle output of map attempt_local1999544224_0005_m_000015_0 decomp: 298786 len: 298790 to MEMORY
16/08/17 21:07:25 INFO reduce.InMemoryMapOutput: Read 298786 bytes from map-output for attempt_local1999544224_0005_m_000015_0
16/08/17 21:07:25 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 298786, inMemoryMapOutputs.size() -> 13, commitMemory -> 3639122, usedMemory ->3937908
16/08/17 21:07:25 INFO reduce.LocalFetcher: localfetcher#86 about to shuffle output of map attempt_local1999544224_0005_m_000002_0 decomp: 305012 len: 305016 to MEMORY
16/08/17 21:07:25 INFO reduce.InMemoryMapOutput: Read 305012 bytes from map-output for attempt_local1999544224_0005_m_000002_0
16/08/17 21:07:25 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 305012, inMemoryMapOutputs.size() -> 14, commitMemory -> 3937908, usedMemory ->4242920
16/08/17 21:07:25 INFO reduce.LocalFetcher: localfetcher#86 about to shuffle output of map attempt_local1999544224_0005_m_000008_0 decomp: 300648 len: 300652 to MEMORY
16/08/17 21:07:25 INFO reduce.InMemoryMapOutput: Read 300648 bytes from map-output for attempt_local1999544224_0005_m_000008_0
16/08/17 21:07:25 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 300648, inMemoryMapOutputs.size() -> 15, commitMemory -> 4242920, usedMemory ->4543568
16/08/17 21:07:25 INFO reduce.LocalFetcher: localfetcher#86 about to shuffle output of map attempt_local1999544224_0005_m_000014_0 decomp: 300744 len: 300748 to MEMORY
16/08/17 21:07:25 INFO reduce.InMemoryMapOutput: Read 300744 bytes from map-output for attempt_local1999544224_0005_m_000014_0
16/08/17 21:07:25 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 300744, inMemoryMapOutputs.size() -> 16, commitMemory -> 4543568, usedMemory ->4844312
16/08/17 21:07:25 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:07:25 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:25 INFO reduce.MergeManagerImpl: finalMerge called with 16 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:07:25 INFO mapred.Merger: Merging 16 sorted segments
16/08/17 21:07:25 INFO mapred.Merger: Down to the last merge-pass, with 16 segments left of total size: 4843924 bytes
16/08/17 21:07:25 INFO reduce.MergeManagerImpl: Merged 16 segments, 4823974 bytes to disk to satisfy reduce memory limit
16/08/17 21:07:25 INFO reduce.MergeManagerImpl: Merging 1 files, 4823948 bytes from disk
16/08/17 21:07:25 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:07:25 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:07:25 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 4823926 bytes
16/08/17 21:07:25 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:25 INFO reduce.MergeManagerImpl: Merged 16 segments, 4844312 bytes to disk to satisfy reduce memory limit
16/08/17 21:07:25 INFO reduce.MergeManagerImpl: Merging 1 files, 4844286 bytes from disk
16/08/17 21:07:25 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:07:25 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:07:25 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 4844244 bytes
16/08/17 21:07:25 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:26 INFO mapred.Task: Task:attempt_local518136789_0007_r_000002_0 is done. And is in the process of committing
16/08/17 21:07:26 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:26 INFO mapred.Task: Task attempt_local518136789_0007_r_000002_0 is allowed to commit now
16/08/17 21:07:26 INFO output.FileOutputCommitter: Saved output of task 'attempt_local518136789_0007_r_000002_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/e_given_f_phrase/_temporary/0/task_local518136789_0007_r_000002
16/08/17 21:07:26 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:07:26 INFO mapred.Task: Task 'attempt_local518136789_0007_r_000002_0' done.
16/08/17 21:07:26 INFO mapred.LocalJobRunner: Finishing task: attempt_local518136789_0007_r_000002_0
16/08/17 21:07:26 INFO mapred.LocalJobRunner: Starting task: attempt_local518136789_0007_r_000003_0
16/08/17 21:07:26 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:26 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@46b8740a
16/08/17 21:07:26 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=369570592, maxSingleShuffleLimit=92392648, mergeThreshold=243916608, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:07:26 INFO reduce.EventFetcher: attempt_local518136789_0007_r_000003_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:07:26 INFO reduce.LocalFetcher: localfetcher#87 about to shuffle output of map attempt_local518136789_0007_m_000011_0 decomp: 286852 len: 286856 to MEMORY
16/08/17 21:07:26 INFO reduce.InMemoryMapOutput: Read 286852 bytes from map-output for attempt_local518136789_0007_m_000011_0
16/08/17 21:07:26 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 286852, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->286852
16/08/17 21:07:26 INFO reduce.LocalFetcher: localfetcher#87 about to shuffle output of map attempt_local518136789_0007_m_000004_0 decomp: 294982 len: 294986 to MEMORY
16/08/17 21:07:26 INFO reduce.InMemoryMapOutput: Read 294982 bytes from map-output for attempt_local518136789_0007_m_000004_0
16/08/17 21:07:26 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 294982, inMemoryMapOutputs.size() -> 2, commitMemory -> 286852, usedMemory ->581834
16/08/17 21:07:26 INFO reduce.LocalFetcher: localfetcher#87 about to shuffle output of map attempt_local518136789_0007_m_000010_0 decomp: 290610 len: 290614 to MEMORY
16/08/17 21:07:26 INFO reduce.InMemoryMapOutput: Read 290610 bytes from map-output for attempt_local518136789_0007_m_000010_0
16/08/17 21:07:26 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 290610, inMemoryMapOutputs.size() -> 3, commitMemory -> 581834, usedMemory ->872444
16/08/17 21:07:26 INFO reduce.LocalFetcher: localfetcher#87 about to shuffle output of map attempt_local518136789_0007_m_000000_0 decomp: 302682 len: 302686 to MEMORY
16/08/17 21:07:26 INFO reduce.InMemoryMapOutput: Read 302682 bytes from map-output for attempt_local518136789_0007_m_000000_0
16/08/17 21:07:26 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 302682, inMemoryMapOutputs.size() -> 4, commitMemory -> 872444, usedMemory ->1175126
16/08/17 21:07:26 INFO reduce.LocalFetcher: localfetcher#87 about to shuffle output of map attempt_local518136789_0007_m_000006_0 decomp: 306736 len: 306740 to MEMORY
16/08/17 21:07:26 INFO reduce.InMemoryMapOutput: Read 306736 bytes from map-output for attempt_local518136789_0007_m_000006_0
16/08/17 21:07:26 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 306736, inMemoryMapOutputs.size() -> 5, commitMemory -> 1175126, usedMemory ->1481862
16/08/17 21:07:26 INFO reduce.LocalFetcher: localfetcher#87 about to shuffle output of map attempt_local518136789_0007_m_000012_0 decomp: 301588 len: 301592 to MEMORY
16/08/17 21:07:26 INFO reduce.InMemoryMapOutput: Read 301588 bytes from map-output for attempt_local518136789_0007_m_000012_0
16/08/17 21:07:26 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 301588, inMemoryMapOutputs.size() -> 6, commitMemory -> 1481862, usedMemory ->1783450
16/08/17 21:07:26 INFO reduce.LocalFetcher: localfetcher#87 about to shuffle output of map attempt_local518136789_0007_m_000005_0 decomp: 299802 len: 299806 to MEMORY
16/08/17 21:07:26 INFO reduce.InMemoryMapOutput: Read 299802 bytes from map-output for attempt_local518136789_0007_m_000005_0
16/08/17 21:07:26 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 299802, inMemoryMapOutputs.size() -> 7, commitMemory -> 1783450, usedMemory ->2083252
16/08/17 21:07:26 INFO reduce.LocalFetcher: localfetcher#87 about to shuffle output of map attempt_local518136789_0007_m_000008_0 decomp: 306850 len: 306854 to MEMORY
16/08/17 21:07:26 INFO reduce.InMemoryMapOutput: Read 306850 bytes from map-output for attempt_local518136789_0007_m_000008_0
16/08/17 21:07:26 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 306850, inMemoryMapOutputs.size() -> 8, commitMemory -> 2083252, usedMemory ->2390102
16/08/17 21:07:26 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)
	at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
16/08/17 21:07:26 INFO reduce.LocalFetcher: localfetcher#87 about to shuffle output of map attempt_local518136789_0007_m_000014_0 decomp: 290610 len: 290614 to MEMORY
16/08/17 21:07:26 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)
	at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
16/08/17 21:07:26 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)
	at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
16/08/17 21:07:26 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)
	at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
16/08/17 21:07:26 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)
	at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
16/08/17 21:07:26 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)
	at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
16/08/17 21:07:26 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)
	at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
16/08/17 21:07:26 INFO reduce.InMemoryMapOutput: Read 290610 bytes from map-output for attempt_local518136789_0007_m_000014_0
16/08/17 21:07:26 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 290610, inMemoryMapOutputs.size() -> 9, commitMemory -> 2390102, usedMemory ->2680712
16/08/17 21:07:26 INFO reduce.LocalFetcher: localfetcher#87 about to shuffle output of map attempt_local518136789_0007_m_000001_0 decomp: 305870 len: 305874 to MEMORY
16/08/17 21:07:26 INFO reduce.InMemoryMapOutput: Read 305870 bytes from map-output for attempt_local518136789_0007_m_000001_0
16/08/17 21:07:26 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 305870, inMemoryMapOutputs.size() -> 10, commitMemory -> 2680712, usedMemory ->2986582
16/08/17 21:07:26 INFO reduce.LocalFetcher: localfetcher#87 about to shuffle output of map attempt_local518136789_0007_m_000007_0 decomp: 305624 len: 305628 to MEMORY
16/08/17 21:07:26 INFO reduce.InMemoryMapOutput: Read 305624 bytes from map-output for attempt_local518136789_0007_m_000007_0
16/08/17 21:07:26 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 305624, inMemoryMapOutputs.size() -> 11, commitMemory -> 2986582, usedMemory ->3292206
16/08/17 21:07:26 INFO reduce.LocalFetcher: localfetcher#87 about to shuffle output of map attempt_local518136789_0007_m_000013_0 decomp: 301148 len: 301152 to MEMORY
16/08/17 21:07:26 INFO reduce.InMemoryMapOutput: Read 301148 bytes from map-output for attempt_local518136789_0007_m_000013_0
16/08/17 21:07:26 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 301148, inMemoryMapOutputs.size() -> 12, commitMemory -> 3292206, usedMemory ->3593354
16/08/17 21:07:26 INFO reduce.LocalFetcher: localfetcher#87 about to shuffle output of map attempt_local518136789_0007_m_000003_0 decomp: 304998 len: 305002 to MEMORY
16/08/17 21:07:26 INFO reduce.InMemoryMapOutput: Read 304998 bytes from map-output for attempt_local518136789_0007_m_000003_0
16/08/17 21:07:26 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 304998, inMemoryMapOutputs.size() -> 13, commitMemory -> 3593354, usedMemory ->3898352
16/08/17 21:07:26 INFO reduce.LocalFetcher: localfetcher#87 about to shuffle output of map attempt_local518136789_0007_m_000009_0 decomp: 300174 len: 300178 to MEMORY
16/08/17 21:07:26 INFO reduce.InMemoryMapOutput: Read 300174 bytes from map-output for attempt_local518136789_0007_m_000009_0
16/08/17 21:07:26 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 300174, inMemoryMapOutputs.size() -> 14, commitMemory -> 3898352, usedMemory ->4198526
16/08/17 21:07:26 INFO reduce.LocalFetcher: localfetcher#87 about to shuffle output of map attempt_local518136789_0007_m_000015_0 decomp: 294946 len: 294950 to MEMORY
16/08/17 21:07:26 INFO reduce.InMemoryMapOutput: Read 294946 bytes from map-output for attempt_local518136789_0007_m_000015_0
16/08/17 21:07:26 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 294946, inMemoryMapOutputs.size() -> 15, commitMemory -> 4198526, usedMemory ->4493472
16/08/17 21:07:26 INFO reduce.LocalFetcher: localfetcher#87 about to shuffle output of map attempt_local518136789_0007_m_000002_0 decomp: 300700 len: 300704 to MEMORY
16/08/17 21:07:26 INFO reduce.InMemoryMapOutput: Read 300700 bytes from map-output for attempt_local518136789_0007_m_000002_0
16/08/17 21:07:26 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 300700, inMemoryMapOutputs.size() -> 16, commitMemory -> 4493472, usedMemory ->4794172
16/08/17 21:07:26 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:07:26 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:26 INFO reduce.MergeManagerImpl: finalMerge called with 16 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:07:26 INFO mapred.Merger: Merging 16 sorted segments
16/08/17 21:07:26 INFO mapred.Merger: Down to the last merge-pass, with 16 segments left of total size: 4793780 bytes
16/08/17 21:07:26 INFO mapred.Task: Task:attempt_local1999544224_0005_r_000002_0 is done. And is in the process of committing
16/08/17 21:07:26 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:26 INFO mapred.Task: Task attempt_local1999544224_0005_r_000002_0 is allowed to commit now
16/08/17 21:07:26 INFO output.FileOutputCommitter: Saved output of task 'attempt_local1999544224_0005_r_000002_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/f_given_e_phrase/_temporary/0/task_local1999544224_0005_r_000002
16/08/17 21:07:26 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:07:26 INFO mapred.Task: Task 'attempt_local1999544224_0005_r_000002_0' done.
16/08/17 21:07:26 INFO mapred.LocalJobRunner: Finishing task: attempt_local1999544224_0005_r_000002_0
16/08/17 21:07:26 INFO mapred.LocalJobRunner: Starting task: attempt_local1999544224_0005_r_000003_0
16/08/17 21:07:26 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:26 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@37ca9237
16/08/17 21:07:26 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=369570592, maxSingleShuffleLimit=92392648, mergeThreshold=243916608, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:07:26 INFO reduce.EventFetcher: attempt_local1999544224_0005_r_000003_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:07:26 INFO reduce.LocalFetcher: localfetcher#88 about to shuffle output of map attempt_local1999544224_0005_m_000004_0 decomp: 308906 len: 308910 to MEMORY
16/08/17 21:07:26 INFO reduce.InMemoryMapOutput: Read 308906 bytes from map-output for attempt_local1999544224_0005_m_000004_0
16/08/17 21:07:26 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 308906, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->308906
16/08/17 21:07:26 INFO reduce.LocalFetcher: localfetcher#88 about to shuffle output of map attempt_local1999544224_0005_m_000010_0 decomp: 302166 len: 302170 to MEMORY
16/08/17 21:07:26 INFO reduce.InMemoryMapOutput: Read 302166 bytes from map-output for attempt_local1999544224_0005_m_000010_0
16/08/17 21:07:26 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 302166, inMemoryMapOutputs.size() -> 2, commitMemory -> 308906, usedMemory ->611072
16/08/17 21:07:26 INFO reduce.LocalFetcher: localfetcher#88 about to shuffle output of map attempt_local1999544224_0005_m_000003_0 decomp: 314232 len: 314236 to MEMORY
16/08/17 21:07:26 INFO reduce.InMemoryMapOutput: Read 314232 bytes from map-output for attempt_local1999544224_0005_m_000003_0
16/08/17 21:07:26 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 314232, inMemoryMapOutputs.size() -> 3, commitMemory -> 611072, usedMemory ->925304
16/08/17 21:07:26 INFO reduce.LocalFetcher: localfetcher#88 about to shuffle output of map attempt_local1999544224_0005_m_000012_0 decomp: 297792 len: 297796 to MEMORY
16/08/17 21:07:26 INFO reduce.InMemoryMapOutput: Read 297792 bytes from map-output for attempt_local1999544224_0005_m_000012_0
16/08/17 21:07:26 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 297792, inMemoryMapOutputs.size() -> 4, commitMemory -> 925304, usedMemory ->1223096
16/08/17 21:07:26 INFO reduce.LocalFetcher: localfetcher#88 about to shuffle output of map attempt_local1999544224_0005_m_000005_0 decomp: 317102 len: 317106 to MEMORY
16/08/17 21:07:26 INFO reduce.InMemoryMapOutput: Read 317102 bytes from map-output for attempt_local1999544224_0005_m_000005_0
16/08/17 21:07:26 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 317102, inMemoryMapOutputs.size() -> 5, commitMemory -> 1223096, usedMemory ->1540198
16/08/17 21:07:26 INFO reduce.LocalFetcher: localfetcher#88 about to shuffle output of map attempt_local1999544224_0005_m_000011_0 decomp: 295714 len: 295718 to MEMORY
16/08/17 21:07:26 INFO reduce.InMemoryMapOutput: Read 295714 bytes from map-output for attempt_local1999544224_0005_m_000011_0
16/08/17 21:07:26 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 295714, inMemoryMapOutputs.size() -> 6, commitMemory -> 1540198, usedMemory ->1835912
16/08/17 21:07:26 INFO reduce.LocalFetcher: localfetcher#88 about to shuffle output of map attempt_local1999544224_0005_m_000001_0 decomp: 307494 len: 307498 to MEMORY
16/08/17 21:07:26 INFO reduce.InMemoryMapOutput: Read 307494 bytes from map-output for attempt_local1999544224_0005_m_000001_0
16/08/17 21:07:26 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 307494, inMemoryMapOutputs.size() -> 7, commitMemory -> 1835912, usedMemory ->2143406
16/08/17 21:07:26 INFO reduce.LocalFetcher: localfetcher#88 about to shuffle output of map attempt_local1999544224_0005_m_000007_0 decomp: 294832 len: 294836 to MEMORY
16/08/17 21:07:26 INFO reduce.InMemoryMapOutput: Read 294832 bytes from map-output for attempt_local1999544224_0005_m_000007_0
16/08/17 21:07:26 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 294832, inMemoryMapOutputs.size() -> 8, commitMemory -> 2143406, usedMemory ->2438238
16/08/17 21:07:26 INFO reduce.LocalFetcher: localfetcher#88 about to shuffle output of map attempt_local1999544224_0005_m_000013_0 decomp: 292898 len: 292902 to MEMORY
16/08/17 21:07:26 INFO reduce.InMemoryMapOutput: Read 292898 bytes from map-output for attempt_local1999544224_0005_m_000013_0
16/08/17 21:07:26 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 292898, inMemoryMapOutputs.size() -> 9, commitMemory -> 2438238, usedMemory ->2731136
16/08/17 21:07:26 INFO reduce.LocalFetcher: localfetcher#88 about to shuffle output of map attempt_local1999544224_0005_m_000000_0 decomp: 308066 len: 308070 to MEMORY
16/08/17 21:07:26 INFO reduce.InMemoryMapOutput: Read 308066 bytes from map-output for attempt_local1999544224_0005_m_000000_0
16/08/17 21:07:26 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 308066, inMemoryMapOutputs.size() -> 10, commitMemory -> 2731136, usedMemory ->3039202
16/08/17 21:07:26 INFO reduce.LocalFetcher: localfetcher#88 about to shuffle output of map attempt_local1999544224_0005_m_000006_0 decomp: 322608 len: 322612 to MEMORY
16/08/17 21:07:26 INFO reduce.InMemoryMapOutput: Read 322608 bytes from map-output for attempt_local1999544224_0005_m_000006_0
16/08/17 21:07:26 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 322608, inMemoryMapOutputs.size() -> 11, commitMemory -> 3039202, usedMemory ->3361810
16/08/17 21:07:26 INFO reduce.LocalFetcher: localfetcher#88 about to shuffle output of map attempt_local1999544224_0005_m_000009_0 decomp: 304374 len: 304378 to MEMORY
16/08/17 21:07:26 INFO reduce.InMemoryMapOutput: Read 304374 bytes from map-output for attempt_local1999544224_0005_m_000009_0
16/08/17 21:07:26 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 304374, inMemoryMapOutputs.size() -> 12, commitMemory -> 3361810, usedMemory ->3666184
16/08/17 21:07:26 INFO reduce.LocalFetcher: localfetcher#88 about to shuffle output of map attempt_local1999544224_0005_m_000015_0 decomp: 292922 len: 292926 to MEMORY
16/08/17 21:07:26 INFO reduce.InMemoryMapOutput: Read 292922 bytes from map-output for attempt_local1999544224_0005_m_000015_0
16/08/17 21:07:26 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 292922, inMemoryMapOutputs.size() -> 13, commitMemory -> 3666184, usedMemory ->3959106
16/08/17 21:07:26 INFO reduce.LocalFetcher: localfetcher#88 about to shuffle output of map attempt_local1999544224_0005_m_000002_0 decomp: 306622 len: 306626 to MEMORY
16/08/17 21:07:26 INFO reduce.InMemoryMapOutput: Read 306622 bytes from map-output for attempt_local1999544224_0005_m_000002_0
16/08/17 21:07:26 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 306622, inMemoryMapOutputs.size() -> 14, commitMemory -> 3959106, usedMemory ->4265728
16/08/17 21:07:26 INFO reduce.LocalFetcher: localfetcher#88 about to shuffle output of map attempt_local1999544224_0005_m_000008_0 decomp: 300708 len: 300712 to MEMORY
16/08/17 21:07:26 INFO reduce.InMemoryMapOutput: Read 300708 bytes from map-output for attempt_local1999544224_0005_m_000008_0
16/08/17 21:07:26 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 300708, inMemoryMapOutputs.size() -> 15, commitMemory -> 4265728, usedMemory ->4566436
16/08/17 21:07:26 INFO reduce.LocalFetcher: localfetcher#88 about to shuffle output of map attempt_local1999544224_0005_m_000014_0 decomp: 304798 len: 304802 to MEMORY
16/08/17 21:07:26 INFO reduce.InMemoryMapOutput: Read 304798 bytes from map-output for attempt_local1999544224_0005_m_000014_0
16/08/17 21:07:26 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 304798, inMemoryMapOutputs.size() -> 16, commitMemory -> 4566436, usedMemory ->4871234
16/08/17 21:07:26 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)
	at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
16/08/17 21:07:26 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)
	at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
16/08/17 21:07:26 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)
	at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
16/08/17 21:07:26 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:07:26 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:26 INFO reduce.MergeManagerImpl: finalMerge called with 16 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:07:26 INFO mapred.Merger: Merging 16 sorted segments
16/08/17 21:07:26 INFO mapred.Merger: Down to the last merge-pass, with 16 segments left of total size: 4870862 bytes
16/08/17 21:07:26 INFO reduce.MergeManagerImpl: Merged 16 segments, 4794172 bytes to disk to satisfy reduce memory limit
16/08/17 21:07:26 INFO reduce.MergeManagerImpl: Merging 1 files, 4794146 bytes from disk
16/08/17 21:07:26 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:07:26 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:07:26 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 4794116 bytes
16/08/17 21:07:26 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:26 INFO reduce.MergeManagerImpl: Merged 16 segments, 4871234 bytes to disk to satisfy reduce memory limit
16/08/17 21:07:26 INFO reduce.MergeManagerImpl: Merging 1 files, 4871208 bytes from disk
16/08/17 21:07:26 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:07:26 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:07:26 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 4871162 bytes
16/08/17 21:07:26 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:26 INFO mapred.Task: Task:attempt_local518136789_0007_r_000003_0 is done. And is in the process of committing
16/08/17 21:07:26 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:26 INFO mapred.Task: Task attempt_local518136789_0007_r_000003_0 is allowed to commit now
16/08/17 21:07:26 INFO output.FileOutputCommitter: Saved output of task 'attempt_local518136789_0007_r_000003_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/e_given_f_phrase/_temporary/0/task_local518136789_0007_r_000003
16/08/17 21:07:26 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:07:26 INFO mapred.Task: Task 'attempt_local518136789_0007_r_000003_0' done.
16/08/17 21:07:26 INFO mapred.LocalJobRunner: Finishing task: attempt_local518136789_0007_r_000003_0
16/08/17 21:07:26 INFO mapred.LocalJobRunner: Starting task: attempt_local518136789_0007_r_000004_0
16/08/17 21:07:26 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:26 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@3d35067d
16/08/17 21:07:26 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=365533600, maxSingleShuffleLimit=91383400, mergeThreshold=241252192, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:07:26 INFO reduce.EventFetcher: attempt_local518136789_0007_r_000004_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:07:26 INFO reduce.LocalFetcher: localfetcher#89 about to shuffle output of map attempt_local518136789_0007_m_000011_0 decomp: 295278 len: 295282 to MEMORY
16/08/17 21:07:26 INFO reduce.InMemoryMapOutput: Read 295278 bytes from map-output for attempt_local518136789_0007_m_000011_0
16/08/17 21:07:26 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 295278, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->295278
16/08/17 21:07:26 INFO reduce.LocalFetcher: localfetcher#89 about to shuffle output of map attempt_local518136789_0007_m_000004_0 decomp: 305026 len: 305030 to MEMORY
16/08/17 21:07:26 INFO reduce.InMemoryMapOutput: Read 305026 bytes from map-output for attempt_local518136789_0007_m_000004_0
16/08/17 21:07:26 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 305026, inMemoryMapOutputs.size() -> 2, commitMemory -> 295278, usedMemory ->600304
16/08/17 21:07:26 INFO reduce.LocalFetcher: localfetcher#89 about to shuffle output of map attempt_local518136789_0007_m_000010_0 decomp: 303372 len: 303376 to MEMORY
16/08/17 21:07:26 INFO reduce.InMemoryMapOutput: Read 303372 bytes from map-output for attempt_local518136789_0007_m_000010_0
16/08/17 21:07:26 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 303372, inMemoryMapOutputs.size() -> 3, commitMemory -> 600304, usedMemory ->903676
16/08/17 21:07:26 INFO reduce.LocalFetcher: localfetcher#89 about to shuffle output of map attempt_local518136789_0007_m_000000_0 decomp: 292478 len: 292482 to MEMORY
16/08/17 21:07:26 INFO reduce.InMemoryMapOutput: Read 292478 bytes from map-output for attempt_local518136789_0007_m_000000_0
16/08/17 21:07:26 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 292478, inMemoryMapOutputs.size() -> 4, commitMemory -> 903676, usedMemory ->1196154
16/08/17 21:07:26 INFO reduce.LocalFetcher: localfetcher#89 about to shuffle output of map attempt_local518136789_0007_m_000006_0 decomp: 311366 len: 311370 to MEMORY
16/08/17 21:07:26 INFO reduce.InMemoryMapOutput: Read 311366 bytes from map-output for attempt_local518136789_0007_m_000006_0
16/08/17 21:07:26 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 311366, inMemoryMapOutputs.size() -> 5, commitMemory -> 1196154, usedMemory ->1507520
16/08/17 21:07:26 INFO reduce.LocalFetcher: localfetcher#89 about to shuffle output of map attempt_local518136789_0007_m_000012_0 decomp: 302768 len: 302772 to MEMORY
16/08/17 21:07:26 INFO reduce.InMemoryMapOutput: Read 302768 bytes from map-output for attempt_local518136789_0007_m_000012_0
16/08/17 21:07:26 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 302768, inMemoryMapOutputs.size() -> 6, commitMemory -> 1507520, usedMemory ->1810288
16/08/17 21:07:26 INFO reduce.LocalFetcher: localfetcher#89 about to shuffle output of map attempt_local518136789_0007_m_000005_0 decomp: 307866 len: 307870 to MEMORY
16/08/17 21:07:26 INFO reduce.InMemoryMapOutput: Read 307866 bytes from map-output for attempt_local518136789_0007_m_000005_0
16/08/17 21:07:26 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 307866, inMemoryMapOutputs.size() -> 7, commitMemory -> 1810288, usedMemory ->2118154
16/08/17 21:07:26 INFO reduce.LocalFetcher: localfetcher#89 about to shuffle output of map attempt_local518136789_0007_m_000008_0 decomp: 293546 len: 293550 to MEMORY
16/08/17 21:07:26 INFO reduce.InMemoryMapOutput: Read 293546 bytes from map-output for attempt_local518136789_0007_m_000008_0
16/08/17 21:07:26 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 293546, inMemoryMapOutputs.size() -> 8, commitMemory -> 2118154, usedMemory ->2411700
16/08/17 21:07:26 INFO reduce.LocalFetcher: localfetcher#89 about to shuffle output of map attempt_local518136789_0007_m_000014_0 decomp: 297090 len: 297094 to MEMORY
16/08/17 21:07:26 INFO reduce.InMemoryMapOutput: Read 297090 bytes from map-output for attempt_local518136789_0007_m_000014_0
16/08/17 21:07:26 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 297090, inMemoryMapOutputs.size() -> 9, commitMemory -> 2411700, usedMemory ->2708790
16/08/17 21:07:26 INFO reduce.LocalFetcher: localfetcher#89 about to shuffle output of map attempt_local518136789_0007_m_000001_0 decomp: 301164 len: 301168 to MEMORY
16/08/17 21:07:26 INFO reduce.InMemoryMapOutput: Read 301164 bytes from map-output for attempt_local518136789_0007_m_000001_0
16/08/17 21:07:26 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 301164, inMemoryMapOutputs.size() -> 10, commitMemory -> 2708790, usedMemory ->3009954
16/08/17 21:07:26 INFO reduce.LocalFetcher: localfetcher#89 about to shuffle output of map attempt_local518136789_0007_m_000007_0 decomp: 304168 len: 304172 to MEMORY
16/08/17 21:07:26 INFO reduce.InMemoryMapOutput: Read 304168 bytes from map-output for attempt_local518136789_0007_m_000007_0
16/08/17 21:07:26 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 304168, inMemoryMapOutputs.size() -> 11, commitMemory -> 3009954, usedMemory ->3314122
16/08/17 21:07:26 INFO reduce.LocalFetcher: localfetcher#89 about to shuffle output of map attempt_local518136789_0007_m_000013_0 decomp: 301770 len: 301774 to MEMORY
16/08/17 21:07:26 INFO reduce.InMemoryMapOutput: Read 301770 bytes from map-output for attempt_local518136789_0007_m_000013_0
16/08/17 21:07:26 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 301770, inMemoryMapOutputs.size() -> 12, commitMemory -> 3314122, usedMemory ->3615892
16/08/17 21:07:26 INFO reduce.LocalFetcher: localfetcher#89 about to shuffle output of map attempt_local518136789_0007_m_000003_0 decomp: 305568 len: 305572 to MEMORY
16/08/17 21:07:26 INFO reduce.InMemoryMapOutput: Read 305568 bytes from map-output for attempt_local518136789_0007_m_000003_0
16/08/17 21:07:26 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 305568, inMemoryMapOutputs.size() -> 13, commitMemory -> 3615892, usedMemory ->3921460
16/08/17 21:07:26 INFO reduce.LocalFetcher: localfetcher#89 about to shuffle output of map attempt_local518136789_0007_m_000009_0 decomp: 301470 len: 301474 to MEMORY
16/08/17 21:07:26 INFO reduce.InMemoryMapOutput: Read 301470 bytes from map-output for attempt_local518136789_0007_m_000009_0
16/08/17 21:07:26 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 301470, inMemoryMapOutputs.size() -> 14, commitMemory -> 3921460, usedMemory ->4222930
16/08/17 21:07:26 INFO reduce.LocalFetcher: localfetcher#89 about to shuffle output of map attempt_local518136789_0007_m_000015_0 decomp: 295780 len: 295784 to MEMORY
16/08/17 21:07:26 INFO reduce.InMemoryMapOutput: Read 295780 bytes from map-output for attempt_local518136789_0007_m_000015_0
16/08/17 21:07:26 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 295780, inMemoryMapOutputs.size() -> 15, commitMemory -> 4222930, usedMemory ->4518710
16/08/17 21:07:26 INFO reduce.LocalFetcher: localfetcher#89 about to shuffle output of map attempt_local518136789_0007_m_000002_0 decomp: 304438 len: 304442 to MEMORY
16/08/17 21:07:26 INFO reduce.InMemoryMapOutput: Read 304438 bytes from map-output for attempt_local518136789_0007_m_000002_0
16/08/17 21:07:26 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 304438, inMemoryMapOutputs.size() -> 16, commitMemory -> 4518710, usedMemory ->4823148
16/08/17 21:07:26 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:07:26 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:26 INFO reduce.MergeManagerImpl: finalMerge called with 16 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:07:26 INFO mapred.Merger: Merging 16 sorted segments
16/08/17 21:07:26 INFO mapred.Merger: Down to the last merge-pass, with 16 segments left of total size: 4822864 bytes
16/08/17 21:07:26 INFO reduce.MergeManagerImpl: Merged 16 segments, 4823148 bytes to disk to satisfy reduce memory limit
16/08/17 21:07:26 INFO reduce.MergeManagerImpl: Merging 1 files, 4823122 bytes from disk
16/08/17 21:07:26 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:07:26 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:07:26 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 4823108 bytes
16/08/17 21:07:26 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:26 INFO mapred.Task: Task:attempt_local1999544224_0005_r_000003_0 is done. And is in the process of committing
16/08/17 21:07:26 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:26 INFO mapred.Task: Task attempt_local1999544224_0005_r_000003_0 is allowed to commit now
16/08/17 21:07:26 INFO output.FileOutputCommitter: Saved output of task 'attempt_local1999544224_0005_r_000003_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/f_given_e_phrase/_temporary/0/task_local1999544224_0005_r_000003
16/08/17 21:07:26 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:07:26 INFO mapred.Task: Task 'attempt_local1999544224_0005_r_000003_0' done.
16/08/17 21:07:26 INFO mapred.LocalJobRunner: Finishing task: attempt_local1999544224_0005_r_000003_0
16/08/17 21:07:26 INFO mapred.LocalJobRunner: Starting task: attempt_local1999544224_0005_r_000004_0
16/08/17 21:07:26 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:26 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@6158bf59
16/08/17 21:07:26 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=365533600, maxSingleShuffleLimit=91383400, mergeThreshold=241252192, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:07:26 INFO reduce.EventFetcher: attempt_local1999544224_0005_r_000004_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:07:26 INFO reduce.LocalFetcher: localfetcher#90 about to shuffle output of map attempt_local1999544224_0005_m_000004_0 decomp: 312252 len: 312256 to MEMORY
16/08/17 21:07:26 INFO reduce.InMemoryMapOutput: Read 312252 bytes from map-output for attempt_local1999544224_0005_m_000004_0
16/08/17 21:07:26 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 312252, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->312252
16/08/17 21:07:26 INFO reduce.LocalFetcher: localfetcher#90 about to shuffle output of map attempt_local1999544224_0005_m_000010_0 decomp: 304446 len: 304450 to MEMORY
16/08/17 21:07:26 INFO reduce.InMemoryMapOutput: Read 304446 bytes from map-output for attempt_local1999544224_0005_m_000010_0
16/08/17 21:07:26 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 304446, inMemoryMapOutputs.size() -> 2, commitMemory -> 312252, usedMemory ->616698
16/08/17 21:07:26 INFO reduce.LocalFetcher: localfetcher#90 about to shuffle output of map attempt_local1999544224_0005_m_000003_0 decomp: 319608 len: 319612 to MEMORY
16/08/17 21:07:26 INFO reduce.InMemoryMapOutput: Read 319608 bytes from map-output for attempt_local1999544224_0005_m_000003_0
16/08/17 21:07:26 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 319608, inMemoryMapOutputs.size() -> 3, commitMemory -> 616698, usedMemory ->936306
16/08/17 21:07:26 INFO reduce.LocalFetcher: localfetcher#90 about to shuffle output of map attempt_local1999544224_0005_m_000012_0 decomp: 294504 len: 294508 to MEMORY
16/08/17 21:07:26 INFO reduce.InMemoryMapOutput: Read 294504 bytes from map-output for attempt_local1999544224_0005_m_000012_0
16/08/17 21:07:26 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 294504, inMemoryMapOutputs.size() -> 4, commitMemory -> 936306, usedMemory ->1230810
16/08/17 21:07:26 INFO reduce.LocalFetcher: localfetcher#90 about to shuffle output of map attempt_local1999544224_0005_m_000005_0 decomp: 306086 len: 306090 to MEMORY
16/08/17 21:07:26 INFO reduce.InMemoryMapOutput: Read 306086 bytes from map-output for attempt_local1999544224_0005_m_000005_0
16/08/17 21:07:26 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 306086, inMemoryMapOutputs.size() -> 5, commitMemory -> 1230810, usedMemory ->1536896
16/08/17 21:07:26 INFO reduce.LocalFetcher: localfetcher#90 about to shuffle output of map attempt_local1999544224_0005_m_000011_0 decomp: 315886 len: 315890 to MEMORY
16/08/17 21:07:26 INFO reduce.InMemoryMapOutput: Read 315886 bytes from map-output for attempt_local1999544224_0005_m_000011_0
16/08/17 21:07:26 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 315886, inMemoryMapOutputs.size() -> 6, commitMemory -> 1536896, usedMemory ->1852782
16/08/17 21:07:26 INFO reduce.LocalFetcher: localfetcher#90 about to shuffle output of map attempt_local1999544224_0005_m_000001_0 decomp: 320398 len: 320402 to MEMORY
16/08/17 21:07:26 INFO reduce.InMemoryMapOutput: Read 320398 bytes from map-output for attempt_local1999544224_0005_m_000001_0
16/08/17 21:07:26 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 320398, inMemoryMapOutputs.size() -> 7, commitMemory -> 1852782, usedMemory ->2173180
16/08/17 21:07:26 INFO reduce.LocalFetcher: localfetcher#90 about to shuffle output of map attempt_local1999544224_0005_m_000007_0 decomp: 301482 len: 301486 to MEMORY
16/08/17 21:07:26 INFO reduce.InMemoryMapOutput: Read 301482 bytes from map-output for attempt_local1999544224_0005_m_000007_0
16/08/17 21:07:26 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 301482, inMemoryMapOutputs.size() -> 8, commitMemory -> 2173180, usedMemory ->2474662
16/08/17 21:07:26 INFO reduce.LocalFetcher: localfetcher#90 about to shuffle output of map attempt_local1999544224_0005_m_000013_0 decomp: 303828 len: 303832 to MEMORY
16/08/17 21:07:26 INFO reduce.InMemoryMapOutput: Read 303828 bytes from map-output for attempt_local1999544224_0005_m_000013_0
16/08/17 21:07:26 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 303828, inMemoryMapOutputs.size() -> 9, commitMemory -> 2474662, usedMemory ->2778490
16/08/17 21:07:26 INFO reduce.LocalFetcher: localfetcher#90 about to shuffle output of map attempt_local1999544224_0005_m_000000_0 decomp: 305122 len: 305126 to MEMORY
16/08/17 21:07:26 INFO reduce.InMemoryMapOutput: Read 305122 bytes from map-output for attempt_local1999544224_0005_m_000000_0
16/08/17 21:07:26 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 305122, inMemoryMapOutputs.size() -> 10, commitMemory -> 2778490, usedMemory ->3083612
16/08/17 21:07:26 INFO reduce.LocalFetcher: localfetcher#90 about to shuffle output of map attempt_local1999544224_0005_m_000006_0 decomp: 319146 len: 319150 to MEMORY
16/08/17 21:07:26 INFO reduce.InMemoryMapOutput: Read 319146 bytes from map-output for attempt_local1999544224_0005_m_000006_0
16/08/17 21:07:26 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 319146, inMemoryMapOutputs.size() -> 11, commitMemory -> 3083612, usedMemory ->3402758
16/08/17 21:07:26 INFO reduce.LocalFetcher: localfetcher#90 about to shuffle output of map attempt_local1999544224_0005_m_000009_0 decomp: 312922 len: 312926 to MEMORY
16/08/17 21:07:26 INFO reduce.InMemoryMapOutput: Read 312922 bytes from map-output for attempt_local1999544224_0005_m_000009_0
16/08/17 21:07:26 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 312922, inMemoryMapOutputs.size() -> 12, commitMemory -> 3402758, usedMemory ->3715680
16/08/17 21:07:26 INFO reduce.LocalFetcher: localfetcher#90 about to shuffle output of map attempt_local1999544224_0005_m_000015_0 decomp: 297680 len: 297684 to MEMORY
16/08/17 21:07:26 INFO reduce.InMemoryMapOutput: Read 297680 bytes from map-output for attempt_local1999544224_0005_m_000015_0
16/08/17 21:07:26 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 297680, inMemoryMapOutputs.size() -> 13, commitMemory -> 3715680, usedMemory ->4013360
16/08/17 21:07:26 INFO reduce.LocalFetcher: localfetcher#90 about to shuffle output of map attempt_local1999544224_0005_m_000002_0 decomp: 302522 len: 302526 to MEMORY
16/08/17 21:07:26 INFO reduce.InMemoryMapOutput: Read 302522 bytes from map-output for attempt_local1999544224_0005_m_000002_0
16/08/17 21:07:26 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 302522, inMemoryMapOutputs.size() -> 14, commitMemory -> 4013360, usedMemory ->4315882
16/08/17 21:07:26 INFO reduce.LocalFetcher: localfetcher#90 about to shuffle output of map attempt_local1999544224_0005_m_000008_0 decomp: 313818 len: 313822 to MEMORY
16/08/17 21:07:26 INFO reduce.InMemoryMapOutput: Read 313818 bytes from map-output for attempt_local1999544224_0005_m_000008_0
16/08/17 21:07:26 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 313818, inMemoryMapOutputs.size() -> 15, commitMemory -> 4315882, usedMemory ->4629700
16/08/17 21:07:26 INFO reduce.LocalFetcher: localfetcher#90 about to shuffle output of map attempt_local1999544224_0005_m_000014_0 decomp: 304842 len: 304846 to MEMORY
16/08/17 21:07:26 INFO reduce.InMemoryMapOutput: Read 304842 bytes from map-output for attempt_local1999544224_0005_m_000014_0
16/08/17 21:07:26 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 304842, inMemoryMapOutputs.size() -> 16, commitMemory -> 4629700, usedMemory ->4934542
16/08/17 21:07:26 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:07:26 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:26 INFO reduce.MergeManagerImpl: finalMerge called with 16 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:07:26 INFO mapred.Merger: Merging 16 sorted segments
16/08/17 21:07:26 INFO mapred.Merger: Down to the last merge-pass, with 16 segments left of total size: 4934186 bytes
16/08/17 21:07:26 INFO reduce.MergeManagerImpl: Merged 16 segments, 4934542 bytes to disk to satisfy reduce memory limit
16/08/17 21:07:26 INFO reduce.MergeManagerImpl: Merging 1 files, 4934516 bytes from disk
16/08/17 21:07:26 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:07:26 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:07:26 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 4934490 bytes
16/08/17 21:07:26 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:27 INFO mapred.Task: Task:attempt_local518136789_0007_r_000004_0 is done. And is in the process of committing
16/08/17 21:07:27 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:27 INFO mapred.Task: Task attempt_local518136789_0007_r_000004_0 is allowed to commit now
16/08/17 21:07:27 INFO output.FileOutputCommitter: Saved output of task 'attempt_local518136789_0007_r_000004_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/e_given_f_phrase/_temporary/0/task_local518136789_0007_r_000004
16/08/17 21:07:27 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:07:27 INFO mapred.Task: Task 'attempt_local518136789_0007_r_000004_0' done.
16/08/17 21:07:27 INFO mapred.LocalJobRunner: Finishing task: attempt_local518136789_0007_r_000004_0
16/08/17 21:07:27 INFO mapred.LocalJobRunner: Starting task: attempt_local518136789_0007_r_000005_0
16/08/17 21:07:27 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:27 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@6c1f9f9e
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=365533600, maxSingleShuffleLimit=91383400, mergeThreshold=241252192, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:07:27 INFO reduce.EventFetcher: attempt_local518136789_0007_r_000005_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:07:27 INFO reduce.LocalFetcher: localfetcher#91 about to shuffle output of map attempt_local518136789_0007_m_000011_0 decomp: 285322 len: 285326 to MEMORY
16/08/17 21:07:27 INFO reduce.InMemoryMapOutput: Read 285322 bytes from map-output for attempt_local518136789_0007_m_000011_0
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 285322, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->285322
16/08/17 21:07:27 INFO reduce.LocalFetcher: localfetcher#91 about to shuffle output of map attempt_local518136789_0007_m_000004_0 decomp: 305298 len: 305302 to MEMORY
16/08/17 21:07:27 INFO reduce.InMemoryMapOutput: Read 305298 bytes from map-output for attempt_local518136789_0007_m_000004_0
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 305298, inMemoryMapOutputs.size() -> 2, commitMemory -> 285322, usedMemory ->590620
16/08/17 21:07:27 INFO reduce.LocalFetcher: localfetcher#91 about to shuffle output of map attempt_local518136789_0007_m_000010_0 decomp: 291822 len: 291826 to MEMORY
16/08/17 21:07:27 INFO reduce.InMemoryMapOutput: Read 291822 bytes from map-output for attempt_local518136789_0007_m_000010_0
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 291822, inMemoryMapOutputs.size() -> 3, commitMemory -> 590620, usedMemory ->882442
16/08/17 21:07:27 INFO reduce.LocalFetcher: localfetcher#91 about to shuffle output of map attempt_local518136789_0007_m_000000_0 decomp: 301222 len: 301226 to MEMORY
16/08/17 21:07:27 INFO reduce.InMemoryMapOutput: Read 301222 bytes from map-output for attempt_local518136789_0007_m_000000_0
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 301222, inMemoryMapOutputs.size() -> 4, commitMemory -> 882442, usedMemory ->1183664
16/08/17 21:07:27 INFO reduce.LocalFetcher: localfetcher#91 about to shuffle output of map attempt_local518136789_0007_m_000006_0 decomp: 295872 len: 295876 to MEMORY
16/08/17 21:07:27 INFO reduce.InMemoryMapOutput: Read 295872 bytes from map-output for attempt_local518136789_0007_m_000006_0
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 295872, inMemoryMapOutputs.size() -> 5, commitMemory -> 1183664, usedMemory ->1479536
16/08/17 21:07:27 INFO reduce.LocalFetcher: localfetcher#91 about to shuffle output of map attempt_local518136789_0007_m_000012_0 decomp: 293286 len: 293290 to MEMORY
16/08/17 21:07:27 INFO reduce.InMemoryMapOutput: Read 293286 bytes from map-output for attempt_local518136789_0007_m_000012_0
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 293286, inMemoryMapOutputs.size() -> 6, commitMemory -> 1479536, usedMemory ->1772822
16/08/17 21:07:27 INFO reduce.LocalFetcher: localfetcher#91 about to shuffle output of map attempt_local518136789_0007_m_000005_0 decomp: 279504 len: 279508 to MEMORY
16/08/17 21:07:27 INFO reduce.InMemoryMapOutput: Read 279504 bytes from map-output for attempt_local518136789_0007_m_000005_0
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 279504, inMemoryMapOutputs.size() -> 7, commitMemory -> 1772822, usedMemory ->2052326
16/08/17 21:07:27 INFO reduce.LocalFetcher: localfetcher#91 about to shuffle output of map attempt_local518136789_0007_m_000008_0 decomp: 299832 len: 299836 to MEMORY
16/08/17 21:07:27 INFO reduce.InMemoryMapOutput: Read 299832 bytes from map-output for attempt_local518136789_0007_m_000008_0
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 299832, inMemoryMapOutputs.size() -> 8, commitMemory -> 2052326, usedMemory ->2352158
16/08/17 21:07:27 INFO reduce.LocalFetcher: localfetcher#91 about to shuffle output of map attempt_local518136789_0007_m_000014_0 decomp: 300448 len: 300452 to MEMORY
16/08/17 21:07:27 INFO reduce.InMemoryMapOutput: Read 300448 bytes from map-output for attempt_local518136789_0007_m_000014_0
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 300448, inMemoryMapOutputs.size() -> 9, commitMemory -> 2352158, usedMemory ->2652606
16/08/17 21:07:27 INFO reduce.LocalFetcher: localfetcher#91 about to shuffle output of map attempt_local518136789_0007_m_000001_0 decomp: 315714 len: 315718 to MEMORY
16/08/17 21:07:27 INFO reduce.InMemoryMapOutput: Read 315714 bytes from map-output for attempt_local518136789_0007_m_000001_0
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 315714, inMemoryMapOutputs.size() -> 10, commitMemory -> 2652606, usedMemory ->2968320
16/08/17 21:07:27 INFO reduce.LocalFetcher: localfetcher#91 about to shuffle output of map attempt_local518136789_0007_m_000007_0 decomp: 288252 len: 288256 to MEMORY
16/08/17 21:07:27 INFO reduce.InMemoryMapOutput: Read 288252 bytes from map-output for attempt_local518136789_0007_m_000007_0
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 288252, inMemoryMapOutputs.size() -> 11, commitMemory -> 2968320, usedMemory ->3256572
16/08/17 21:07:27 INFO reduce.LocalFetcher: localfetcher#91 about to shuffle output of map attempt_local518136789_0007_m_000013_0 decomp: 286780 len: 286784 to MEMORY
16/08/17 21:07:27 INFO reduce.InMemoryMapOutput: Read 286780 bytes from map-output for attempt_local518136789_0007_m_000013_0
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 286780, inMemoryMapOutputs.size() -> 12, commitMemory -> 3256572, usedMemory ->3543352
16/08/17 21:07:27 INFO reduce.LocalFetcher: localfetcher#91 about to shuffle output of map attempt_local518136789_0007_m_000003_0 decomp: 296994 len: 296998 to MEMORY
16/08/17 21:07:27 INFO reduce.InMemoryMapOutput: Read 296994 bytes from map-output for attempt_local518136789_0007_m_000003_0
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 296994, inMemoryMapOutputs.size() -> 13, commitMemory -> 3543352, usedMemory ->3840346
16/08/17 21:07:27 INFO reduce.LocalFetcher: localfetcher#91 about to shuffle output of map attempt_local518136789_0007_m_000009_0 decomp: 306342 len: 306346 to MEMORY
16/08/17 21:07:27 INFO reduce.InMemoryMapOutput: Read 306342 bytes from map-output for attempt_local518136789_0007_m_000009_0
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 306342, inMemoryMapOutputs.size() -> 14, commitMemory -> 3840346, usedMemory ->4146688
16/08/17 21:07:27 INFO reduce.LocalFetcher: localfetcher#91 about to shuffle output of map attempt_local518136789_0007_m_000015_0 decomp: 292470 len: 292474 to MEMORY
16/08/17 21:07:27 INFO reduce.InMemoryMapOutput: Read 292470 bytes from map-output for attempt_local518136789_0007_m_000015_0
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 292470, inMemoryMapOutputs.size() -> 15, commitMemory -> 4146688, usedMemory ->4439158
16/08/17 21:07:27 INFO reduce.LocalFetcher: localfetcher#91 about to shuffle output of map attempt_local518136789_0007_m_000002_0 decomp: 299874 len: 299878 to MEMORY
16/08/17 21:07:27 INFO reduce.InMemoryMapOutput: Read 299874 bytes from map-output for attempt_local518136789_0007_m_000002_0
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 299874, inMemoryMapOutputs.size() -> 16, commitMemory -> 4439158, usedMemory ->4739032
16/08/17 21:07:27 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:07:27 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: finalMerge called with 16 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:07:27 INFO mapred.Merger: Merging 16 sorted segments
16/08/17 21:07:27 INFO mapred.Merger: Down to the last merge-pass, with 16 segments left of total size: 4738688 bytes
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: Merged 16 segments, 4739032 bytes to disk to satisfy reduce memory limit
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: Merging 1 files, 4739006 bytes from disk
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:07:27 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:07:27 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 4738988 bytes
16/08/17 21:07:27 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:27 INFO mapred.Task: Task:attempt_local1999544224_0005_r_000004_0 is done. And is in the process of committing
16/08/17 21:07:27 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:27 INFO mapred.Task: Task attempt_local1999544224_0005_r_000004_0 is allowed to commit now
16/08/17 21:07:27 INFO output.FileOutputCommitter: Saved output of task 'attempt_local1999544224_0005_r_000004_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/f_given_e_phrase/_temporary/0/task_local1999544224_0005_r_000004
16/08/17 21:07:27 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:07:27 INFO mapred.Task: Task 'attempt_local1999544224_0005_r_000004_0' done.
16/08/17 21:07:27 INFO mapred.LocalJobRunner: Finishing task: attempt_local1999544224_0005_r_000004_0
16/08/17 21:07:27 INFO mapred.LocalJobRunner: Starting task: attempt_local1999544224_0005_r_000005_0
16/08/17 21:07:27 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:27 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@1a92b95f
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=366634592, maxSingleShuffleLimit=91658648, mergeThreshold=241978848, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:07:27 INFO reduce.EventFetcher: attempt_local1999544224_0005_r_000005_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:07:27 INFO reduce.LocalFetcher: localfetcher#92 about to shuffle output of map attempt_local1999544224_0005_m_000004_0 decomp: 313284 len: 313288 to MEMORY
16/08/17 21:07:27 INFO reduce.InMemoryMapOutput: Read 313284 bytes from map-output for attempt_local1999544224_0005_m_000004_0
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 313284, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->313284
16/08/17 21:07:27 INFO reduce.LocalFetcher: localfetcher#92 about to shuffle output of map attempt_local1999544224_0005_m_000010_0 decomp: 304436 len: 304440 to MEMORY
16/08/17 21:07:27 INFO reduce.InMemoryMapOutput: Read 304436 bytes from map-output for attempt_local1999544224_0005_m_000010_0
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 304436, inMemoryMapOutputs.size() -> 2, commitMemory -> 313284, usedMemory ->617720
16/08/17 21:07:27 INFO reduce.LocalFetcher: localfetcher#92 about to shuffle output of map attempt_local1999544224_0005_m_000003_0 decomp: 311280 len: 311284 to MEMORY
16/08/17 21:07:27 INFO reduce.InMemoryMapOutput: Read 311280 bytes from map-output for attempt_local1999544224_0005_m_000003_0
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 311280, inMemoryMapOutputs.size() -> 3, commitMemory -> 617720, usedMemory ->929000
16/08/17 21:07:27 INFO reduce.LocalFetcher: localfetcher#92 about to shuffle output of map attempt_local1999544224_0005_m_000012_0 decomp: 299084 len: 299088 to MEMORY
16/08/17 21:07:27 INFO reduce.InMemoryMapOutput: Read 299084 bytes from map-output for attempt_local1999544224_0005_m_000012_0
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 299084, inMemoryMapOutputs.size() -> 4, commitMemory -> 929000, usedMemory ->1228084
16/08/17 21:07:27 INFO reduce.LocalFetcher: localfetcher#92 about to shuffle output of map attempt_local1999544224_0005_m_000005_0 decomp: 316656 len: 316660 to MEMORY
16/08/17 21:07:27 INFO reduce.InMemoryMapOutput: Read 316656 bytes from map-output for attempt_local1999544224_0005_m_000005_0
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 316656, inMemoryMapOutputs.size() -> 5, commitMemory -> 1228084, usedMemory ->1544740
16/08/17 21:07:27 INFO reduce.LocalFetcher: localfetcher#92 about to shuffle output of map attempt_local1999544224_0005_m_000011_0 decomp: 301814 len: 301818 to MEMORY
16/08/17 21:07:27 INFO reduce.InMemoryMapOutput: Read 301814 bytes from map-output for attempt_local1999544224_0005_m_000011_0
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 301814, inMemoryMapOutputs.size() -> 6, commitMemory -> 1544740, usedMemory ->1846554
16/08/17 21:07:27 INFO reduce.LocalFetcher: localfetcher#92 about to shuffle output of map attempt_local1999544224_0005_m_000001_0 decomp: 314160 len: 314164 to MEMORY
16/08/17 21:07:27 INFO reduce.InMemoryMapOutput: Read 314160 bytes from map-output for attempt_local1999544224_0005_m_000001_0
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 314160, inMemoryMapOutputs.size() -> 7, commitMemory -> 1846554, usedMemory ->2160714
16/08/17 21:07:27 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)
	at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
16/08/17 21:07:27 INFO reduce.LocalFetcher: localfetcher#92 about to shuffle output of map attempt_local1999544224_0005_m_000007_0 decomp: 303684 len: 303688 to MEMORY
16/08/17 21:07:27 INFO reduce.InMemoryMapOutput: Read 303684 bytes from map-output for attempt_local1999544224_0005_m_000007_0
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 303684, inMemoryMapOutputs.size() -> 8, commitMemory -> 2160714, usedMemory ->2464398
16/08/17 21:07:27 INFO reduce.LocalFetcher: localfetcher#92 about to shuffle output of map attempt_local1999544224_0005_m_000013_0 decomp: 287998 len: 288002 to MEMORY
16/08/17 21:07:27 INFO reduce.InMemoryMapOutput: Read 287998 bytes from map-output for attempt_local1999544224_0005_m_000013_0
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 287998, inMemoryMapOutputs.size() -> 9, commitMemory -> 2464398, usedMemory ->2752396
16/08/17 21:07:27 INFO reduce.LocalFetcher: localfetcher#92 about to shuffle output of map attempt_local1999544224_0005_m_000000_0 decomp: 303770 len: 303774 to MEMORY
16/08/17 21:07:27 INFO reduce.InMemoryMapOutput: Read 303770 bytes from map-output for attempt_local1999544224_0005_m_000000_0
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 303770, inMemoryMapOutputs.size() -> 10, commitMemory -> 2752396, usedMemory ->3056166
16/08/17 21:07:27 INFO reduce.LocalFetcher: localfetcher#92 about to shuffle output of map attempt_local1999544224_0005_m_000006_0 decomp: 289348 len: 289352 to MEMORY
16/08/17 21:07:27 INFO reduce.InMemoryMapOutput: Read 289348 bytes from map-output for attempt_local1999544224_0005_m_000006_0
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 289348, inMemoryMapOutputs.size() -> 11, commitMemory -> 3056166, usedMemory ->3345514
16/08/17 21:07:27 INFO reduce.LocalFetcher: localfetcher#92 about to shuffle output of map attempt_local1999544224_0005_m_000009_0 decomp: 304930 len: 304934 to MEMORY
16/08/17 21:07:27 INFO reduce.InMemoryMapOutput: Read 304930 bytes from map-output for attempt_local1999544224_0005_m_000009_0
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 304930, inMemoryMapOutputs.size() -> 12, commitMemory -> 3345514, usedMemory ->3650444
16/08/17 21:07:27 INFO reduce.LocalFetcher: localfetcher#92 about to shuffle output of map attempt_local1999544224_0005_m_000015_0 decomp: 303144 len: 303148 to MEMORY
16/08/17 21:07:27 INFO reduce.InMemoryMapOutput: Read 303144 bytes from map-output for attempt_local1999544224_0005_m_000015_0
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 303144, inMemoryMapOutputs.size() -> 13, commitMemory -> 3650444, usedMemory ->3953588
16/08/17 21:07:27 INFO reduce.LocalFetcher: localfetcher#92 about to shuffle output of map attempt_local1999544224_0005_m_000002_0 decomp: 305226 len: 305230 to MEMORY
16/08/17 21:07:27 INFO reduce.InMemoryMapOutput: Read 305226 bytes from map-output for attempt_local1999544224_0005_m_000002_0
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 305226, inMemoryMapOutputs.size() -> 14, commitMemory -> 3953588, usedMemory ->4258814
16/08/17 21:07:27 INFO reduce.LocalFetcher: localfetcher#92 about to shuffle output of map attempt_local1999544224_0005_m_000008_0 decomp: 297970 len: 297974 to MEMORY
16/08/17 21:07:27 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)
	at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
16/08/17 21:07:27 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)
	at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
16/08/17 21:07:27 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)
	at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
16/08/17 21:07:27 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)
	at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
16/08/17 21:07:27 INFO reduce.InMemoryMapOutput: Read 297970 bytes from map-output for attempt_local1999544224_0005_m_000008_0
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 297970, inMemoryMapOutputs.size() -> 15, commitMemory -> 4258814, usedMemory ->4556784
16/08/17 21:07:27 INFO reduce.LocalFetcher: localfetcher#92 about to shuffle output of map attempt_local1999544224_0005_m_000014_0 decomp: 299624 len: 299628 to MEMORY
16/08/17 21:07:27 INFO reduce.InMemoryMapOutput: Read 299624 bytes from map-output for attempt_local1999544224_0005_m_000014_0
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 299624, inMemoryMapOutputs.size() -> 16, commitMemory -> 4556784, usedMemory ->4856408
16/08/17 21:07:27 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:07:27 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: finalMerge called with 16 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:07:27 INFO mapred.Merger: Merging 16 sorted segments
16/08/17 21:07:27 INFO mapred.Merger: Down to the last merge-pass, with 16 segments left of total size: 4856032 bytes
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: Merged 16 segments, 4856408 bytes to disk to satisfy reduce memory limit
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: Merging 1 files, 4856382 bytes from disk
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:07:27 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:07:27 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 4856348 bytes
16/08/17 21:07:27 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:27 INFO mapred.Task: Task:attempt_local518136789_0007_r_000005_0 is done. And is in the process of committing
16/08/17 21:07:27 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:27 INFO mapred.Task: Task attempt_local518136789_0007_r_000005_0 is allowed to commit now
16/08/17 21:07:27 INFO output.FileOutputCommitter: Saved output of task 'attempt_local518136789_0007_r_000005_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/e_given_f_phrase/_temporary/0/task_local518136789_0007_r_000005
16/08/17 21:07:27 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:07:27 INFO mapred.Task: Task 'attempt_local518136789_0007_r_000005_0' done.
16/08/17 21:07:27 INFO mapred.LocalJobRunner: Finishing task: attempt_local518136789_0007_r_000005_0
16/08/17 21:07:27 INFO mapred.LocalJobRunner: Starting task: attempt_local518136789_0007_r_000006_0
16/08/17 21:07:27 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:27 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@27b16d48
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=366634592, maxSingleShuffleLimit=91658648, mergeThreshold=241978848, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:07:27 INFO reduce.EventFetcher: attempt_local518136789_0007_r_000006_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:07:27 INFO reduce.LocalFetcher: localfetcher#93 about to shuffle output of map attempt_local518136789_0007_m_000011_0 decomp: 303080 len: 303084 to MEMORY
16/08/17 21:07:27 INFO reduce.InMemoryMapOutput: Read 303080 bytes from map-output for attempt_local518136789_0007_m_000011_0
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 303080, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->303080
16/08/17 21:07:27 INFO reduce.LocalFetcher: localfetcher#93 about to shuffle output of map attempt_local518136789_0007_m_000004_0 decomp: 302846 len: 302850 to MEMORY
16/08/17 21:07:27 INFO reduce.InMemoryMapOutput: Read 302846 bytes from map-output for attempt_local518136789_0007_m_000004_0
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 302846, inMemoryMapOutputs.size() -> 2, commitMemory -> 303080, usedMemory ->605926
16/08/17 21:07:27 INFO reduce.LocalFetcher: localfetcher#93 about to shuffle output of map attempt_local518136789_0007_m_000010_0 decomp: 295050 len: 295054 to MEMORY
16/08/17 21:07:27 INFO reduce.InMemoryMapOutput: Read 295050 bytes from map-output for attempt_local518136789_0007_m_000010_0
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 295050, inMemoryMapOutputs.size() -> 3, commitMemory -> 605926, usedMemory ->900976
16/08/17 21:07:27 INFO reduce.LocalFetcher: localfetcher#93 about to shuffle output of map attempt_local518136789_0007_m_000000_0 decomp: 301042 len: 301046 to MEMORY
16/08/17 21:07:27 INFO reduce.InMemoryMapOutput: Read 301042 bytes from map-output for attempt_local518136789_0007_m_000000_0
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 301042, inMemoryMapOutputs.size() -> 4, commitMemory -> 900976, usedMemory ->1202018
16/08/17 21:07:27 INFO reduce.LocalFetcher: localfetcher#93 about to shuffle output of map attempt_local518136789_0007_m_000006_0 decomp: 296256 len: 296260 to MEMORY
16/08/17 21:07:27 INFO reduce.InMemoryMapOutput: Read 296256 bytes from map-output for attempt_local518136789_0007_m_000006_0
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 296256, inMemoryMapOutputs.size() -> 5, commitMemory -> 1202018, usedMemory ->1498274
16/08/17 21:07:27 INFO reduce.LocalFetcher: localfetcher#93 about to shuffle output of map attempt_local518136789_0007_m_000012_0 decomp: 290742 len: 290746 to MEMORY
16/08/17 21:07:27 INFO reduce.InMemoryMapOutput: Read 290742 bytes from map-output for attempt_local518136789_0007_m_000012_0
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 290742, inMemoryMapOutputs.size() -> 6, commitMemory -> 1498274, usedMemory ->1789016
16/08/17 21:07:27 INFO reduce.LocalFetcher: localfetcher#93 about to shuffle output of map attempt_local518136789_0007_m_000005_0 decomp: 302486 len: 302490 to MEMORY
16/08/17 21:07:27 INFO reduce.InMemoryMapOutput: Read 302486 bytes from map-output for attempt_local518136789_0007_m_000005_0
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 302486, inMemoryMapOutputs.size() -> 7, commitMemory -> 1789016, usedMemory ->2091502
16/08/17 21:07:27 INFO reduce.LocalFetcher: localfetcher#93 about to shuffle output of map attempt_local518136789_0007_m_000008_0 decomp: 291232 len: 291236 to MEMORY
16/08/17 21:07:27 INFO reduce.InMemoryMapOutput: Read 291232 bytes from map-output for attempt_local518136789_0007_m_000008_0
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 291232, inMemoryMapOutputs.size() -> 8, commitMemory -> 2091502, usedMemory ->2382734
16/08/17 21:07:27 INFO reduce.LocalFetcher: localfetcher#93 about to shuffle output of map attempt_local518136789_0007_m_000014_0 decomp: 301440 len: 301444 to MEMORY
16/08/17 21:07:27 INFO reduce.InMemoryMapOutput: Read 301440 bytes from map-output for attempt_local518136789_0007_m_000014_0
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 301440, inMemoryMapOutputs.size() -> 9, commitMemory -> 2382734, usedMemory ->2684174
16/08/17 21:07:27 INFO reduce.LocalFetcher: localfetcher#93 about to shuffle output of map attempt_local518136789_0007_m_000001_0 decomp: 302848 len: 302852 to MEMORY
16/08/17 21:07:27 INFO reduce.InMemoryMapOutput: Read 302848 bytes from map-output for attempt_local518136789_0007_m_000001_0
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 302848, inMemoryMapOutputs.size() -> 10, commitMemory -> 2684174, usedMemory ->2987022
16/08/17 21:07:27 INFO reduce.LocalFetcher: localfetcher#93 about to shuffle output of map attempt_local518136789_0007_m_000007_0 decomp: 292746 len: 292750 to MEMORY
16/08/17 21:07:27 INFO reduce.InMemoryMapOutput: Read 292746 bytes from map-output for attempt_local518136789_0007_m_000007_0
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 292746, inMemoryMapOutputs.size() -> 11, commitMemory -> 2987022, usedMemory ->3279768
16/08/17 21:07:27 INFO reduce.LocalFetcher: localfetcher#93 about to shuffle output of map attempt_local518136789_0007_m_000013_0 decomp: 287020 len: 287024 to MEMORY
16/08/17 21:07:27 INFO reduce.InMemoryMapOutput: Read 287020 bytes from map-output for attempt_local518136789_0007_m_000013_0
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 287020, inMemoryMapOutputs.size() -> 12, commitMemory -> 3279768, usedMemory ->3566788
16/08/17 21:07:27 INFO reduce.LocalFetcher: localfetcher#93 about to shuffle output of map attempt_local518136789_0007_m_000003_0 decomp: 310952 len: 310956 to MEMORY
16/08/17 21:07:27 INFO reduce.InMemoryMapOutput: Read 310952 bytes from map-output for attempt_local518136789_0007_m_000003_0
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 310952, inMemoryMapOutputs.size() -> 13, commitMemory -> 3566788, usedMemory ->3877740
16/08/17 21:07:27 INFO reduce.LocalFetcher: localfetcher#93 about to shuffle output of map attempt_local518136789_0007_m_000009_0 decomp: 281366 len: 281370 to MEMORY
16/08/17 21:07:27 INFO reduce.InMemoryMapOutput: Read 281366 bytes from map-output for attempt_local518136789_0007_m_000009_0
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 281366, inMemoryMapOutputs.size() -> 14, commitMemory -> 3877740, usedMemory ->4159106
16/08/17 21:07:27 INFO reduce.LocalFetcher: localfetcher#93 about to shuffle output of map attempt_local518136789_0007_m_000015_0 decomp: 283220 len: 283224 to MEMORY
16/08/17 21:07:27 INFO reduce.InMemoryMapOutput: Read 283220 bytes from map-output for attempt_local518136789_0007_m_000015_0
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 283220, inMemoryMapOutputs.size() -> 15, commitMemory -> 4159106, usedMemory ->4442326
16/08/17 21:07:27 INFO reduce.LocalFetcher: localfetcher#93 about to shuffle output of map attempt_local518136789_0007_m_000002_0 decomp: 296296 len: 296300 to MEMORY
16/08/17 21:07:27 INFO reduce.InMemoryMapOutput: Read 296296 bytes from map-output for attempt_local518136789_0007_m_000002_0
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 296296, inMemoryMapOutputs.size() -> 16, commitMemory -> 4442326, usedMemory ->4738622
16/08/17 21:07:27 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:07:27 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: finalMerge called with 16 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:07:27 INFO mapred.Merger: Merging 16 sorted segments
16/08/17 21:07:27 INFO mapred.Merger: Down to the last merge-pass, with 16 segments left of total size: 4738266 bytes
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: Merged 16 segments, 4738622 bytes to disk to satisfy reduce memory limit
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: Merging 1 files, 4738596 bytes from disk
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:07:27 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:07:27 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 4738570 bytes
16/08/17 21:07:27 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:27 INFO mapred.Task: Task:attempt_local1999544224_0005_r_000005_0 is done. And is in the process of committing
16/08/17 21:07:27 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:27 INFO mapred.Task: Task attempt_local1999544224_0005_r_000005_0 is allowed to commit now
16/08/17 21:07:27 INFO output.FileOutputCommitter: Saved output of task 'attempt_local1999544224_0005_r_000005_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/f_given_e_phrase/_temporary/0/task_local1999544224_0005_r_000005
16/08/17 21:07:27 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:07:27 INFO mapred.Task: Task 'attempt_local1999544224_0005_r_000005_0' done.
16/08/17 21:07:27 INFO mapred.LocalJobRunner: Finishing task: attempt_local1999544224_0005_r_000005_0
16/08/17 21:07:27 INFO mapred.LocalJobRunner: Starting task: attempt_local1999544224_0005_r_000006_0
16/08/17 21:07:27 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:27 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@49c0ba9
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=366634592, maxSingleShuffleLimit=91658648, mergeThreshold=241978848, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:07:27 INFO reduce.EventFetcher: attempt_local1999544224_0005_r_000006_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:07:27 INFO reduce.LocalFetcher: localfetcher#94 about to shuffle output of map attempt_local1999544224_0005_m_000004_0 decomp: 303870 len: 303874 to MEMORY
16/08/17 21:07:27 INFO reduce.InMemoryMapOutput: Read 303870 bytes from map-output for attempt_local1999544224_0005_m_000004_0
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 303870, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->303870
16/08/17 21:07:27 INFO reduce.LocalFetcher: localfetcher#94 about to shuffle output of map attempt_local1999544224_0005_m_000010_0 decomp: 294032 len: 294036 to MEMORY
16/08/17 21:07:27 INFO reduce.InMemoryMapOutput: Read 294032 bytes from map-output for attempt_local1999544224_0005_m_000010_0
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 294032, inMemoryMapOutputs.size() -> 2, commitMemory -> 303870, usedMemory ->597902
16/08/17 21:07:27 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)
	at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
16/08/17 21:07:27 INFO reduce.LocalFetcher: localfetcher#94 about to shuffle output of map attempt_local1999544224_0005_m_000003_0 decomp: 319120 len: 319124 to MEMORY
16/08/17 21:07:27 INFO reduce.InMemoryMapOutput: Read 319120 bytes from map-output for attempt_local1999544224_0005_m_000003_0
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 319120, inMemoryMapOutputs.size() -> 3, commitMemory -> 597902, usedMemory ->917022
16/08/17 21:07:27 INFO reduce.LocalFetcher: localfetcher#94 about to shuffle output of map attempt_local1999544224_0005_m_000012_0 decomp: 299834 len: 299838 to MEMORY
16/08/17 21:07:27 INFO reduce.InMemoryMapOutput: Read 299834 bytes from map-output for attempt_local1999544224_0005_m_000012_0
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 299834, inMemoryMapOutputs.size() -> 4, commitMemory -> 917022, usedMemory ->1216856
16/08/17 21:07:27 INFO reduce.LocalFetcher: localfetcher#94 about to shuffle output of map attempt_local1999544224_0005_m_000005_0 decomp: 313174 len: 313178 to MEMORY
16/08/17 21:07:27 INFO reduce.InMemoryMapOutput: Read 313174 bytes from map-output for attempt_local1999544224_0005_m_000005_0
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 313174, inMemoryMapOutputs.size() -> 5, commitMemory -> 1216856, usedMemory ->1530030
16/08/17 21:07:27 INFO reduce.LocalFetcher: localfetcher#94 about to shuffle output of map attempt_local1999544224_0005_m_000011_0 decomp: 297188 len: 297192 to MEMORY
16/08/17 21:07:27 INFO reduce.InMemoryMapOutput: Read 297188 bytes from map-output for attempt_local1999544224_0005_m_000011_0
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 297188, inMemoryMapOutputs.size() -> 6, commitMemory -> 1530030, usedMemory ->1827218
16/08/17 21:07:27 INFO reduce.LocalFetcher: localfetcher#94 about to shuffle output of map attempt_local1999544224_0005_m_000001_0 decomp: 309480 len: 309484 to MEMORY
16/08/17 21:07:27 INFO reduce.InMemoryMapOutput: Read 309480 bytes from map-output for attempt_local1999544224_0005_m_000001_0
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 309480, inMemoryMapOutputs.size() -> 7, commitMemory -> 1827218, usedMemory ->2136698
16/08/17 21:07:27 INFO reduce.LocalFetcher: localfetcher#94 about to shuffle output of map attempt_local1999544224_0005_m_000007_0 decomp: 298218 len: 298222 to MEMORY
16/08/17 21:07:27 INFO reduce.InMemoryMapOutput: Read 298218 bytes from map-output for attempt_local1999544224_0005_m_000007_0
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 298218, inMemoryMapOutputs.size() -> 8, commitMemory -> 2136698, usedMemory ->2434916
16/08/17 21:07:27 INFO reduce.LocalFetcher: localfetcher#94 about to shuffle output of map attempt_local1999544224_0005_m_000013_0 decomp: 308546 len: 308550 to MEMORY
16/08/17 21:07:27 INFO reduce.InMemoryMapOutput: Read 308546 bytes from map-output for attempt_local1999544224_0005_m_000013_0
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 308546, inMemoryMapOutputs.size() -> 9, commitMemory -> 2434916, usedMemory ->2743462
16/08/17 21:07:27 INFO reduce.LocalFetcher: localfetcher#94 about to shuffle output of map attempt_local1999544224_0005_m_000000_0 decomp: 305512 len: 305516 to MEMORY
16/08/17 21:07:27 INFO reduce.InMemoryMapOutput: Read 305512 bytes from map-output for attempt_local1999544224_0005_m_000000_0
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 305512, inMemoryMapOutputs.size() -> 10, commitMemory -> 2743462, usedMemory ->3048974
16/08/17 21:07:27 INFO reduce.LocalFetcher: localfetcher#94 about to shuffle output of map attempt_local1999544224_0005_m_000006_0 decomp: 294782 len: 294786 to MEMORY
16/08/17 21:07:27 INFO reduce.InMemoryMapOutput: Read 294782 bytes from map-output for attempt_local1999544224_0005_m_000006_0
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 294782, inMemoryMapOutputs.size() -> 11, commitMemory -> 3048974, usedMemory ->3343756
16/08/17 21:07:27 INFO reduce.LocalFetcher: localfetcher#94 about to shuffle output of map attempt_local1999544224_0005_m_000009_0 decomp: 306922 len: 306926 to MEMORY
16/08/17 21:07:27 INFO reduce.InMemoryMapOutput: Read 306922 bytes from map-output for attempt_local1999544224_0005_m_000009_0
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 306922, inMemoryMapOutputs.size() -> 12, commitMemory -> 3343756, usedMemory ->3650678
16/08/17 21:07:27 INFO reduce.LocalFetcher: localfetcher#94 about to shuffle output of map attempt_local1999544224_0005_m_000015_0 decomp: 307456 len: 307460 to MEMORY
16/08/17 21:07:27 INFO reduce.InMemoryMapOutput: Read 307456 bytes from map-output for attempt_local1999544224_0005_m_000015_0
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 307456, inMemoryMapOutputs.size() -> 13, commitMemory -> 3650678, usedMemory ->3958134
16/08/17 21:07:27 INFO reduce.LocalFetcher: localfetcher#94 about to shuffle output of map attempt_local1999544224_0005_m_000002_0 decomp: 320244 len: 320248 to MEMORY
16/08/17 21:07:27 INFO reduce.InMemoryMapOutput: Read 320244 bytes from map-output for attempt_local1999544224_0005_m_000002_0
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 320244, inMemoryMapOutputs.size() -> 14, commitMemory -> 3958134, usedMemory ->4278378
16/08/17 21:07:27 INFO reduce.LocalFetcher: localfetcher#94 about to shuffle output of map attempt_local1999544224_0005_m_000008_0 decomp: 297580 len: 297584 to MEMORY
16/08/17 21:07:27 INFO reduce.InMemoryMapOutput: Read 297580 bytes from map-output for attempt_local1999544224_0005_m_000008_0
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 297580, inMemoryMapOutputs.size() -> 15, commitMemory -> 4278378, usedMemory ->4575958
16/08/17 21:07:27 INFO reduce.LocalFetcher: localfetcher#94 about to shuffle output of map attempt_local1999544224_0005_m_000014_0 decomp: 288918 len: 288922 to MEMORY
16/08/17 21:07:27 INFO reduce.InMemoryMapOutput: Read 288918 bytes from map-output for attempt_local1999544224_0005_m_000014_0
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 288918, inMemoryMapOutputs.size() -> 16, commitMemory -> 4575958, usedMemory ->4864876
16/08/17 21:07:27 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:07:27 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: finalMerge called with 16 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:07:27 INFO mapred.Merger: Merging 16 sorted segments
16/08/17 21:07:27 INFO mapred.Merger: Down to the last merge-pass, with 16 segments left of total size: 4864508 bytes
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: Merged 16 segments, 4864876 bytes to disk to satisfy reduce memory limit
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: Merging 1 files, 4864850 bytes from disk
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:07:27 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:07:27 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 4864820 bytes
16/08/17 21:07:27 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:27 INFO mapred.Task: Task:attempt_local518136789_0007_r_000006_0 is done. And is in the process of committing
16/08/17 21:07:27 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:27 INFO mapred.Task: Task attempt_local518136789_0007_r_000006_0 is allowed to commit now
16/08/17 21:07:27 INFO output.FileOutputCommitter: Saved output of task 'attempt_local518136789_0007_r_000006_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/e_given_f_phrase/_temporary/0/task_local518136789_0007_r_000006
16/08/17 21:07:27 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:07:27 INFO mapred.Task: Task 'attempt_local518136789_0007_r_000006_0' done.
16/08/17 21:07:27 INFO mapred.LocalJobRunner: Finishing task: attempt_local518136789_0007_r_000006_0
16/08/17 21:07:27 INFO mapred.LocalJobRunner: Starting task: attempt_local518136789_0007_r_000007_0
16/08/17 21:07:27 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:27 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@3f917433
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=367735584, maxSingleShuffleLimit=91933896, mergeThreshold=242705488, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:07:27 INFO reduce.EventFetcher: attempt_local518136789_0007_r_000007_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:07:27 INFO reduce.LocalFetcher: localfetcher#95 about to shuffle output of map attempt_local518136789_0007_m_000011_0 decomp: 289516 len: 289520 to MEMORY
16/08/17 21:07:27 INFO reduce.InMemoryMapOutput: Read 289516 bytes from map-output for attempt_local518136789_0007_m_000011_0
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 289516, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->289516
16/08/17 21:07:27 INFO reduce.LocalFetcher: localfetcher#95 about to shuffle output of map attempt_local518136789_0007_m_000004_0 decomp: 305872 len: 305876 to MEMORY
16/08/17 21:07:27 INFO reduce.InMemoryMapOutput: Read 305872 bytes from map-output for attempt_local518136789_0007_m_000004_0
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 305872, inMemoryMapOutputs.size() -> 2, commitMemory -> 289516, usedMemory ->595388
16/08/17 21:07:27 INFO reduce.LocalFetcher: localfetcher#95 about to shuffle output of map attempt_local518136789_0007_m_000010_0 decomp: 302652 len: 302656 to MEMORY
16/08/17 21:07:27 INFO reduce.InMemoryMapOutput: Read 302652 bytes from map-output for attempt_local518136789_0007_m_000010_0
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 302652, inMemoryMapOutputs.size() -> 3, commitMemory -> 595388, usedMemory ->898040
16/08/17 21:07:27 INFO reduce.LocalFetcher: localfetcher#95 about to shuffle output of map attempt_local518136789_0007_m_000000_0 decomp: 301356 len: 301360 to MEMORY
16/08/17 21:07:27 INFO reduce.InMemoryMapOutput: Read 301356 bytes from map-output for attempt_local518136789_0007_m_000000_0
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 301356, inMemoryMapOutputs.size() -> 4, commitMemory -> 898040, usedMemory ->1199396
16/08/17 21:07:27 INFO reduce.LocalFetcher: localfetcher#95 about to shuffle output of map attempt_local518136789_0007_m_000006_0 decomp: 282566 len: 282570 to MEMORY
16/08/17 21:07:27 INFO reduce.InMemoryMapOutput: Read 282566 bytes from map-output for attempt_local518136789_0007_m_000006_0
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 282566, inMemoryMapOutputs.size() -> 5, commitMemory -> 1199396, usedMemory ->1481962
16/08/17 21:07:27 INFO reduce.LocalFetcher: localfetcher#95 about to shuffle output of map attempt_local518136789_0007_m_000012_0 decomp: 291118 len: 291122 to MEMORY
16/08/17 21:07:27 INFO reduce.InMemoryMapOutput: Read 291118 bytes from map-output for attempt_local518136789_0007_m_000012_0
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 291118, inMemoryMapOutputs.size() -> 6, commitMemory -> 1481962, usedMemory ->1773080
16/08/17 21:07:27 INFO reduce.LocalFetcher: localfetcher#95 about to shuffle output of map attempt_local518136789_0007_m_000005_0 decomp: 293516 len: 293520 to MEMORY
16/08/17 21:07:27 INFO reduce.InMemoryMapOutput: Read 293516 bytes from map-output for attempt_local518136789_0007_m_000005_0
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 293516, inMemoryMapOutputs.size() -> 7, commitMemory -> 1773080, usedMemory ->2066596
16/08/17 21:07:27 INFO reduce.LocalFetcher: localfetcher#95 about to shuffle output of map attempt_local518136789_0007_m_000008_0 decomp: 297802 len: 297806 to MEMORY
16/08/17 21:07:27 INFO reduce.InMemoryMapOutput: Read 297802 bytes from map-output for attempt_local518136789_0007_m_000008_0
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 297802, inMemoryMapOutputs.size() -> 8, commitMemory -> 2066596, usedMemory ->2364398
16/08/17 21:07:27 INFO reduce.LocalFetcher: localfetcher#95 about to shuffle output of map attempt_local518136789_0007_m_000014_0 decomp: 298886 len: 298890 to MEMORY
16/08/17 21:07:27 INFO reduce.InMemoryMapOutput: Read 298886 bytes from map-output for attempt_local518136789_0007_m_000014_0
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 298886, inMemoryMapOutputs.size() -> 9, commitMemory -> 2364398, usedMemory ->2663284
16/08/17 21:07:27 INFO reduce.LocalFetcher: localfetcher#95 about to shuffle output of map attempt_local518136789_0007_m_000001_0 decomp: 300898 len: 300902 to MEMORY
16/08/17 21:07:27 INFO reduce.InMemoryMapOutput: Read 300898 bytes from map-output for attempt_local518136789_0007_m_000001_0
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 300898, inMemoryMapOutputs.size() -> 10, commitMemory -> 2663284, usedMemory ->2964182
16/08/17 21:07:27 INFO reduce.LocalFetcher: localfetcher#95 about to shuffle output of map attempt_local518136789_0007_m_000007_0 decomp: 302878 len: 302882 to MEMORY
16/08/17 21:07:27 INFO reduce.InMemoryMapOutput: Read 302878 bytes from map-output for attempt_local518136789_0007_m_000007_0
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 302878, inMemoryMapOutputs.size() -> 11, commitMemory -> 2964182, usedMemory ->3267060
16/08/17 21:07:27 INFO reduce.LocalFetcher: localfetcher#95 about to shuffle output of map attempt_local518136789_0007_m_000013_0 decomp: 293960 len: 293964 to MEMORY
16/08/17 21:07:27 INFO reduce.InMemoryMapOutput: Read 293960 bytes from map-output for attempt_local518136789_0007_m_000013_0
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 293960, inMemoryMapOutputs.size() -> 12, commitMemory -> 3267060, usedMemory ->3561020
16/08/17 21:07:27 INFO reduce.LocalFetcher: localfetcher#95 about to shuffle output of map attempt_local518136789_0007_m_000003_0 decomp: 291024 len: 291028 to MEMORY
16/08/17 21:07:27 INFO reduce.InMemoryMapOutput: Read 291024 bytes from map-output for attempt_local518136789_0007_m_000003_0
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 291024, inMemoryMapOutputs.size() -> 13, commitMemory -> 3561020, usedMemory ->3852044
16/08/17 21:07:27 INFO reduce.LocalFetcher: localfetcher#95 about to shuffle output of map attempt_local518136789_0007_m_000009_0 decomp: 287238 len: 287242 to MEMORY
16/08/17 21:07:27 INFO reduce.InMemoryMapOutput: Read 287238 bytes from map-output for attempt_local518136789_0007_m_000009_0
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 287238, inMemoryMapOutputs.size() -> 14, commitMemory -> 3852044, usedMemory ->4139282
16/08/17 21:07:27 INFO reduce.LocalFetcher: localfetcher#95 about to shuffle output of map attempt_local518136789_0007_m_000015_0 decomp: 286586 len: 286590 to MEMORY
16/08/17 21:07:27 INFO reduce.InMemoryMapOutput: Read 286586 bytes from map-output for attempt_local518136789_0007_m_000015_0
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 286586, inMemoryMapOutputs.size() -> 15, commitMemory -> 4139282, usedMemory ->4425868
16/08/17 21:07:27 INFO reduce.LocalFetcher: localfetcher#95 about to shuffle output of map attempt_local518136789_0007_m_000002_0 decomp: 302862 len: 302866 to MEMORY
16/08/17 21:07:27 INFO reduce.InMemoryMapOutput: Read 302862 bytes from map-output for attempt_local518136789_0007_m_000002_0
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 302862, inMemoryMapOutputs.size() -> 16, commitMemory -> 4425868, usedMemory ->4728730
16/08/17 21:07:27 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:07:27 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:27 INFO reduce.MergeManagerImpl: finalMerge called with 16 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:07:27 INFO mapred.Merger: Merging 16 sorted segments
16/08/17 21:07:27 INFO mapred.Merger: Down to the last merge-pass, with 16 segments left of total size: 4728374 bytes
16/08/17 21:07:28 INFO reduce.MergeManagerImpl: Merged 16 segments, 4728730 bytes to disk to satisfy reduce memory limit
16/08/17 21:07:28 INFO reduce.MergeManagerImpl: Merging 1 files, 4728704 bytes from disk
16/08/17 21:07:28 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:07:28 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:07:28 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 4728682 bytes
16/08/17 21:07:28 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:28 INFO mapred.Task: Task:attempt_local1999544224_0005_r_000006_0 is done. And is in the process of committing
16/08/17 21:07:28 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:28 INFO mapred.Task: Task attempt_local1999544224_0005_r_000006_0 is allowed to commit now
16/08/17 21:07:28 INFO output.FileOutputCommitter: Saved output of task 'attempt_local1999544224_0005_r_000006_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/f_given_e_phrase/_temporary/0/task_local1999544224_0005_r_000006
16/08/17 21:07:28 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:07:28 INFO mapred.Task: Task 'attempt_local1999544224_0005_r_000006_0' done.
16/08/17 21:07:28 INFO mapred.LocalJobRunner: Finishing task: attempt_local1999544224_0005_r_000006_0
16/08/17 21:07:28 INFO mapred.LocalJobRunner: Starting task: attempt_local1999544224_0005_r_000007_0
16/08/17 21:07:28 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:28 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@5c3cc9d2
16/08/17 21:07:28 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=367735584, maxSingleShuffleLimit=91933896, mergeThreshold=242705488, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:07:28 INFO reduce.EventFetcher: attempt_local1999544224_0005_r_000007_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:07:28 INFO reduce.LocalFetcher: localfetcher#96 about to shuffle output of map attempt_local1999544224_0005_m_000004_0 decomp: 301810 len: 301814 to MEMORY
16/08/17 21:07:28 INFO reduce.InMemoryMapOutput: Read 301810 bytes from map-output for attempt_local1999544224_0005_m_000004_0
16/08/17 21:07:28 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 301810, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->301810
16/08/17 21:07:28 INFO reduce.LocalFetcher: localfetcher#96 about to shuffle output of map attempt_local1999544224_0005_m_000010_0 decomp: 305704 len: 305708 to MEMORY
16/08/17 21:07:28 INFO reduce.InMemoryMapOutput: Read 305704 bytes from map-output for attempt_local1999544224_0005_m_000010_0
16/08/17 21:07:28 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 305704, inMemoryMapOutputs.size() -> 2, commitMemory -> 301810, usedMemory ->607514
16/08/17 21:07:28 INFO reduce.LocalFetcher: localfetcher#96 about to shuffle output of map attempt_local1999544224_0005_m_000003_0 decomp: 302496 len: 302500 to MEMORY
16/08/17 21:07:28 INFO reduce.InMemoryMapOutput: Read 302496 bytes from map-output for attempt_local1999544224_0005_m_000003_0
16/08/17 21:07:28 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 302496, inMemoryMapOutputs.size() -> 3, commitMemory -> 607514, usedMemory ->910010
16/08/17 21:07:28 INFO reduce.LocalFetcher: localfetcher#96 about to shuffle output of map attempt_local1999544224_0005_m_000012_0 decomp: 291062 len: 291066 to MEMORY
16/08/17 21:07:28 INFO reduce.InMemoryMapOutput: Read 291062 bytes from map-output for attempt_local1999544224_0005_m_000012_0
16/08/17 21:07:28 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 291062, inMemoryMapOutputs.size() -> 4, commitMemory -> 910010, usedMemory ->1201072
16/08/17 21:07:28 INFO reduce.LocalFetcher: localfetcher#96 about to shuffle output of map attempt_local1999544224_0005_m_000005_0 decomp: 286704 len: 286708 to MEMORY
16/08/17 21:07:28 INFO reduce.InMemoryMapOutput: Read 286704 bytes from map-output for attempt_local1999544224_0005_m_000005_0
16/08/17 21:07:28 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 286704, inMemoryMapOutputs.size() -> 5, commitMemory -> 1201072, usedMemory ->1487776
16/08/17 21:07:28 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)
	at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
16/08/17 21:07:28 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)
	at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
16/08/17 21:07:28 INFO reduce.LocalFetcher: localfetcher#96 about to shuffle output of map attempt_local1999544224_0005_m_000011_0 decomp: 296784 len: 296788 to MEMORY
16/08/17 21:07:28 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)
	at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
16/08/17 21:07:28 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)
	at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
16/08/17 21:07:28 INFO reduce.InMemoryMapOutput: Read 296784 bytes from map-output for attempt_local1999544224_0005_m_000011_0
16/08/17 21:07:28 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 296784, inMemoryMapOutputs.size() -> 6, commitMemory -> 1487776, usedMemory ->1784560
16/08/17 21:07:28 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)
	at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
16/08/17 21:07:28 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)
	at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
16/08/17 21:07:28 INFO reduce.LocalFetcher: localfetcher#96 about to shuffle output of map attempt_local1999544224_0005_m_000001_0 decomp: 308204 len: 308208 to MEMORY
16/08/17 21:07:28 INFO reduce.InMemoryMapOutput: Read 308204 bytes from map-output for attempt_local1999544224_0005_m_000001_0
16/08/17 21:07:28 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 308204, inMemoryMapOutputs.size() -> 7, commitMemory -> 1784560, usedMemory ->2092764
16/08/17 21:07:28 INFO reduce.LocalFetcher: localfetcher#96 about to shuffle output of map attempt_local1999544224_0005_m_000007_0 decomp: 328872 len: 328876 to MEMORY
16/08/17 21:07:28 INFO reduce.InMemoryMapOutput: Read 328872 bytes from map-output for attempt_local1999544224_0005_m_000007_0
16/08/17 21:07:28 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 328872, inMemoryMapOutputs.size() -> 8, commitMemory -> 2092764, usedMemory ->2421636
16/08/17 21:07:28 INFO reduce.LocalFetcher: localfetcher#96 about to shuffle output of map attempt_local1999544224_0005_m_000013_0 decomp: 293424 len: 293428 to MEMORY
16/08/17 21:07:28 INFO reduce.InMemoryMapOutput: Read 293424 bytes from map-output for attempt_local1999544224_0005_m_000013_0
16/08/17 21:07:28 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 293424, inMemoryMapOutputs.size() -> 9, commitMemory -> 2421636, usedMemory ->2715060
16/08/17 21:07:28 INFO reduce.LocalFetcher: localfetcher#96 about to shuffle output of map attempt_local1999544224_0005_m_000000_0 decomp: 309150 len: 309154 to MEMORY
16/08/17 21:07:28 INFO reduce.InMemoryMapOutput: Read 309150 bytes from map-output for attempt_local1999544224_0005_m_000000_0
16/08/17 21:07:28 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 309150, inMemoryMapOutputs.size() -> 10, commitMemory -> 2715060, usedMemory ->3024210
16/08/17 21:07:28 INFO reduce.LocalFetcher: localfetcher#96 about to shuffle output of map attempt_local1999544224_0005_m_000006_0 decomp: 316698 len: 316702 to MEMORY
16/08/17 21:07:28 INFO reduce.InMemoryMapOutput: Read 316698 bytes from map-output for attempt_local1999544224_0005_m_000006_0
16/08/17 21:07:28 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 316698, inMemoryMapOutputs.size() -> 11, commitMemory -> 3024210, usedMemory ->3340908
16/08/17 21:07:28 INFO reduce.LocalFetcher: localfetcher#96 about to shuffle output of map attempt_local1999544224_0005_m_000009_0 decomp: 306788 len: 306792 to MEMORY
16/08/17 21:07:28 INFO reduce.InMemoryMapOutput: Read 306788 bytes from map-output for attempt_local1999544224_0005_m_000009_0
16/08/17 21:07:28 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 306788, inMemoryMapOutputs.size() -> 12, commitMemory -> 3340908, usedMemory ->3647696
16/08/17 21:07:28 INFO reduce.LocalFetcher: localfetcher#96 about to shuffle output of map attempt_local1999544224_0005_m_000015_0 decomp: 302314 len: 302318 to MEMORY
16/08/17 21:07:28 INFO reduce.InMemoryMapOutput: Read 302314 bytes from map-output for attempt_local1999544224_0005_m_000015_0
16/08/17 21:07:28 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 302314, inMemoryMapOutputs.size() -> 13, commitMemory -> 3647696, usedMemory ->3950010
16/08/17 21:07:28 INFO reduce.LocalFetcher: localfetcher#96 about to shuffle output of map attempt_local1999544224_0005_m_000002_0 decomp: 305816 len: 305820 to MEMORY
16/08/17 21:07:28 INFO reduce.InMemoryMapOutput: Read 305816 bytes from map-output for attempt_local1999544224_0005_m_000002_0
16/08/17 21:07:28 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 305816, inMemoryMapOutputs.size() -> 14, commitMemory -> 3950010, usedMemory ->4255826
16/08/17 21:07:28 INFO reduce.LocalFetcher: localfetcher#96 about to shuffle output of map attempt_local1999544224_0005_m_000008_0 decomp: 306470 len: 306474 to MEMORY
16/08/17 21:07:28 INFO reduce.InMemoryMapOutput: Read 306470 bytes from map-output for attempt_local1999544224_0005_m_000008_0
16/08/17 21:07:28 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 306470, inMemoryMapOutputs.size() -> 15, commitMemory -> 4255826, usedMemory ->4562296
16/08/17 21:07:28 INFO reduce.LocalFetcher: localfetcher#96 about to shuffle output of map attempt_local1999544224_0005_m_000014_0 decomp: 301812 len: 301816 to MEMORY
16/08/17 21:07:28 INFO reduce.InMemoryMapOutput: Read 301812 bytes from map-output for attempt_local1999544224_0005_m_000014_0
16/08/17 21:07:28 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 301812, inMemoryMapOutputs.size() -> 16, commitMemory -> 4562296, usedMemory ->4864108
16/08/17 21:07:28 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:07:28 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:28 INFO reduce.MergeManagerImpl: finalMerge called with 16 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:07:28 INFO mapred.Merger: Merging 16 sorted segments
16/08/17 21:07:28 INFO mapred.Merger: Down to the last merge-pass, with 16 segments left of total size: 4863856 bytes
16/08/17 21:07:28 INFO reduce.MergeManagerImpl: Merged 16 segments, 4864108 bytes to disk to satisfy reduce memory limit
16/08/17 21:07:28 INFO reduce.MergeManagerImpl: Merging 1 files, 4864082 bytes from disk
16/08/17 21:07:28 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:07:28 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:07:28 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 4864040 bytes
16/08/17 21:07:28 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:28 INFO mapred.Task: Task:attempt_local518136789_0007_r_000007_0 is done. And is in the process of committing
16/08/17 21:07:28 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:28 INFO mapred.Task: Task attempt_local518136789_0007_r_000007_0 is allowed to commit now
16/08/17 21:07:28 INFO output.FileOutputCommitter: Saved output of task 'attempt_local518136789_0007_r_000007_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/e_given_f_phrase/_temporary/0/task_local518136789_0007_r_000007
16/08/17 21:07:28 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:07:28 INFO mapred.Task: Task 'attempt_local518136789_0007_r_000007_0' done.
16/08/17 21:07:28 INFO mapred.LocalJobRunner: Finishing task: attempt_local518136789_0007_r_000007_0
16/08/17 21:07:28 INFO mapred.LocalJobRunner: Starting task: attempt_local518136789_0007_r_000008_0
16/08/17 21:07:28 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:28 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@5112c4f0
16/08/17 21:07:28 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=367735584, maxSingleShuffleLimit=91933896, mergeThreshold=242705488, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:07:28 INFO reduce.EventFetcher: attempt_local518136789_0007_r_000008_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:07:28 INFO reduce.LocalFetcher: localfetcher#97 about to shuffle output of map attempt_local518136789_0007_m_000011_0 decomp: 296508 len: 296512 to MEMORY
16/08/17 21:07:28 INFO reduce.InMemoryMapOutput: Read 296508 bytes from map-output for attempt_local518136789_0007_m_000011_0
16/08/17 21:07:28 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 296508, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->296508
16/08/17 21:07:28 INFO reduce.LocalFetcher: localfetcher#97 about to shuffle output of map attempt_local518136789_0007_m_000004_0 decomp: 297184 len: 297188 to MEMORY
16/08/17 21:07:28 INFO reduce.InMemoryMapOutput: Read 297184 bytes from map-output for attempt_local518136789_0007_m_000004_0
16/08/17 21:07:28 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 297184, inMemoryMapOutputs.size() -> 2, commitMemory -> 296508, usedMemory ->593692
16/08/17 21:07:28 INFO reduce.LocalFetcher: localfetcher#97 about to shuffle output of map attempt_local518136789_0007_m_000010_0 decomp: 293092 len: 293096 to MEMORY
16/08/17 21:07:28 INFO reduce.InMemoryMapOutput: Read 293092 bytes from map-output for attempt_local518136789_0007_m_000010_0
16/08/17 21:07:28 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 293092, inMemoryMapOutputs.size() -> 3, commitMemory -> 593692, usedMemory ->886784
16/08/17 21:07:28 INFO reduce.LocalFetcher: localfetcher#97 about to shuffle output of map attempt_local518136789_0007_m_000000_0 decomp: 294214 len: 294218 to MEMORY
16/08/17 21:07:28 INFO reduce.InMemoryMapOutput: Read 294214 bytes from map-output for attempt_local518136789_0007_m_000000_0
16/08/17 21:07:28 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 294214, inMemoryMapOutputs.size() -> 4, commitMemory -> 886784, usedMemory ->1180998
16/08/17 21:07:28 INFO reduce.LocalFetcher: localfetcher#97 about to shuffle output of map attempt_local518136789_0007_m_000006_0 decomp: 296016 len: 296020 to MEMORY
16/08/17 21:07:28 INFO reduce.InMemoryMapOutput: Read 296016 bytes from map-output for attempt_local518136789_0007_m_000006_0
16/08/17 21:07:28 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 296016, inMemoryMapOutputs.size() -> 5, commitMemory -> 1180998, usedMemory ->1477014
16/08/17 21:07:28 INFO reduce.LocalFetcher: localfetcher#97 about to shuffle output of map attempt_local518136789_0007_m_000012_0 decomp: 285964 len: 285968 to MEMORY
16/08/17 21:07:28 INFO reduce.InMemoryMapOutput: Read 285964 bytes from map-output for attempt_local518136789_0007_m_000012_0
16/08/17 21:07:28 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 285964, inMemoryMapOutputs.size() -> 6, commitMemory -> 1477014, usedMemory ->1762978
16/08/17 21:07:28 INFO reduce.LocalFetcher: localfetcher#97 about to shuffle output of map attempt_local518136789_0007_m_000005_0 decomp: 296642 len: 296646 to MEMORY
16/08/17 21:07:28 INFO reduce.InMemoryMapOutput: Read 296642 bytes from map-output for attempt_local518136789_0007_m_000005_0
16/08/17 21:07:28 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 296642, inMemoryMapOutputs.size() -> 7, commitMemory -> 1762978, usedMemory ->2059620
16/08/17 21:07:28 INFO reduce.LocalFetcher: localfetcher#97 about to shuffle output of map attempt_local518136789_0007_m_000008_0 decomp: 298800 len: 298804 to MEMORY
16/08/17 21:07:28 INFO reduce.InMemoryMapOutput: Read 298800 bytes from map-output for attempt_local518136789_0007_m_000008_0
16/08/17 21:07:28 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 298800, inMemoryMapOutputs.size() -> 8, commitMemory -> 2059620, usedMemory ->2358420
16/08/17 21:07:28 INFO reduce.LocalFetcher: localfetcher#97 about to shuffle output of map attempt_local518136789_0007_m_000014_0 decomp: 289132 len: 289136 to MEMORY
16/08/17 21:07:28 INFO reduce.InMemoryMapOutput: Read 289132 bytes from map-output for attempt_local518136789_0007_m_000014_0
16/08/17 21:07:28 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 289132, inMemoryMapOutputs.size() -> 9, commitMemory -> 2358420, usedMemory ->2647552
16/08/17 21:07:28 INFO reduce.LocalFetcher: localfetcher#97 about to shuffle output of map attempt_local518136789_0007_m_000001_0 decomp: 290742 len: 290746 to MEMORY
16/08/17 21:07:28 INFO reduce.InMemoryMapOutput: Read 290742 bytes from map-output for attempt_local518136789_0007_m_000001_0
16/08/17 21:07:28 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 290742, inMemoryMapOutputs.size() -> 10, commitMemory -> 2647552, usedMemory ->2938294
16/08/17 21:07:28 INFO reduce.LocalFetcher: localfetcher#97 about to shuffle output of map attempt_local518136789_0007_m_000007_0 decomp: 295168 len: 295172 to MEMORY
16/08/17 21:07:28 INFO reduce.InMemoryMapOutput: Read 295168 bytes from map-output for attempt_local518136789_0007_m_000007_0
16/08/17 21:07:28 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 295168, inMemoryMapOutputs.size() -> 11, commitMemory -> 2938294, usedMemory ->3233462
16/08/17 21:07:28 INFO reduce.LocalFetcher: localfetcher#97 about to shuffle output of map attempt_local518136789_0007_m_000013_0 decomp: 292860 len: 292864 to MEMORY
16/08/17 21:07:28 INFO reduce.InMemoryMapOutput: Read 292860 bytes from map-output for attempt_local518136789_0007_m_000013_0
16/08/17 21:07:28 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 292860, inMemoryMapOutputs.size() -> 12, commitMemory -> 3233462, usedMemory ->3526322
16/08/17 21:07:28 INFO reduce.LocalFetcher: localfetcher#97 about to shuffle output of map attempt_local518136789_0007_m_000003_0 decomp: 299184 len: 299188 to MEMORY
16/08/17 21:07:28 INFO reduce.InMemoryMapOutput: Read 299184 bytes from map-output for attempt_local518136789_0007_m_000003_0
16/08/17 21:07:28 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 299184, inMemoryMapOutputs.size() -> 13, commitMemory -> 3526322, usedMemory ->3825506
16/08/17 21:07:28 INFO reduce.LocalFetcher: localfetcher#97 about to shuffle output of map attempt_local518136789_0007_m_000009_0 decomp: 298470 len: 298474 to MEMORY
16/08/17 21:07:28 INFO reduce.InMemoryMapOutput: Read 298470 bytes from map-output for attempt_local518136789_0007_m_000009_0
16/08/17 21:07:28 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 298470, inMemoryMapOutputs.size() -> 14, commitMemory -> 3825506, usedMemory ->4123976
16/08/17 21:07:28 INFO reduce.LocalFetcher: localfetcher#97 about to shuffle output of map attempt_local518136789_0007_m_000015_0 decomp: 286650 len: 286654 to MEMORY
16/08/17 21:07:28 INFO reduce.InMemoryMapOutput: Read 286650 bytes from map-output for attempt_local518136789_0007_m_000015_0
16/08/17 21:07:28 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 286650, inMemoryMapOutputs.size() -> 15, commitMemory -> 4123976, usedMemory ->4410626
16/08/17 21:07:28 INFO reduce.LocalFetcher: localfetcher#97 about to shuffle output of map attempt_local518136789_0007_m_000002_0 decomp: 291604 len: 291608 to MEMORY
16/08/17 21:07:28 INFO reduce.InMemoryMapOutput: Read 291604 bytes from map-output for attempt_local518136789_0007_m_000002_0
16/08/17 21:07:28 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 291604, inMemoryMapOutputs.size() -> 16, commitMemory -> 4410626, usedMemory ->4702230
16/08/17 21:07:28 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:07:28 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:28 INFO reduce.MergeManagerImpl: finalMerge called with 16 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:07:28 INFO mapred.Merger: Merging 16 sorted segments
16/08/17 21:07:28 INFO mapred.Merger: Down to the last merge-pass, with 16 segments left of total size: 4701854 bytes
16/08/17 21:07:28 INFO mapred.Task: Task:attempt_local1999544224_0005_r_000007_0 is done. And is in the process of committing
16/08/17 21:07:28 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:28 INFO mapred.Task: Task attempt_local1999544224_0005_r_000007_0 is allowed to commit now
16/08/17 21:07:28 INFO output.FileOutputCommitter: Saved output of task 'attempt_local1999544224_0005_r_000007_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/f_given_e_phrase/_temporary/0/task_local1999544224_0005_r_000007
16/08/17 21:07:28 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:07:28 INFO mapred.Task: Task 'attempt_local1999544224_0005_r_000007_0' done.
16/08/17 21:07:28 INFO mapred.LocalJobRunner: Finishing task: attempt_local1999544224_0005_r_000007_0
16/08/17 21:07:28 INFO mapred.LocalJobRunner: Starting task: attempt_local1999544224_0005_r_000008_0
16/08/17 21:07:28 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:28 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@b7b4d1f
16/08/17 21:07:28 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=367735584, maxSingleShuffleLimit=91933896, mergeThreshold=242705488, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:07:28 INFO reduce.EventFetcher: attempt_local1999544224_0005_r_000008_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:07:28 INFO reduce.LocalFetcher: localfetcher#98 about to shuffle output of map attempt_local1999544224_0005_m_000004_0 decomp: 321806 len: 321810 to MEMORY
16/08/17 21:07:28 INFO reduce.InMemoryMapOutput: Read 321806 bytes from map-output for attempt_local1999544224_0005_m_000004_0
16/08/17 21:07:28 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 321806, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->321806
16/08/17 21:07:28 INFO reduce.LocalFetcher: localfetcher#98 about to shuffle output of map attempt_local1999544224_0005_m_000010_0 decomp: 297674 len: 297678 to MEMORY
16/08/17 21:07:28 INFO reduce.InMemoryMapOutput: Read 297674 bytes from map-output for attempt_local1999544224_0005_m_000010_0
16/08/17 21:07:28 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 297674, inMemoryMapOutputs.size() -> 2, commitMemory -> 321806, usedMemory ->619480
16/08/17 21:07:28 INFO reduce.LocalFetcher: localfetcher#98 about to shuffle output of map attempt_local1999544224_0005_m_000003_0 decomp: 318254 len: 318258 to MEMORY
16/08/17 21:07:28 INFO reduce.InMemoryMapOutput: Read 318254 bytes from map-output for attempt_local1999544224_0005_m_000003_0
16/08/17 21:07:28 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 318254, inMemoryMapOutputs.size() -> 3, commitMemory -> 619480, usedMemory ->937734
16/08/17 21:07:28 INFO reduce.LocalFetcher: localfetcher#98 about to shuffle output of map attempt_local1999544224_0005_m_000012_0 decomp: 298176 len: 298180 to MEMORY
16/08/17 21:07:28 INFO reduce.InMemoryMapOutput: Read 298176 bytes from map-output for attempt_local1999544224_0005_m_000012_0
16/08/17 21:07:28 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 298176, inMemoryMapOutputs.size() -> 4, commitMemory -> 937734, usedMemory ->1235910
16/08/17 21:07:28 INFO reduce.LocalFetcher: localfetcher#98 about to shuffle output of map attempt_local1999544224_0005_m_000005_0 decomp: 304268 len: 304272 to MEMORY
16/08/17 21:07:28 INFO reduce.InMemoryMapOutput: Read 304268 bytes from map-output for attempt_local1999544224_0005_m_000005_0
16/08/17 21:07:28 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 304268, inMemoryMapOutputs.size() -> 5, commitMemory -> 1235910, usedMemory ->1540178
16/08/17 21:07:28 INFO reduce.LocalFetcher: localfetcher#98 about to shuffle output of map attempt_local1999544224_0005_m_000011_0 decomp: 310310 len: 310314 to MEMORY
16/08/17 21:07:28 INFO reduce.InMemoryMapOutput: Read 310310 bytes from map-output for attempt_local1999544224_0005_m_000011_0
16/08/17 21:07:28 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 310310, inMemoryMapOutputs.size() -> 6, commitMemory -> 1540178, usedMemory ->1850488
16/08/17 21:07:28 INFO reduce.LocalFetcher: localfetcher#98 about to shuffle output of map attempt_local1999544224_0005_m_000001_0 decomp: 300502 len: 300506 to MEMORY
16/08/17 21:07:28 INFO reduce.InMemoryMapOutput: Read 300502 bytes from map-output for attempt_local1999544224_0005_m_000001_0
16/08/17 21:07:28 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 300502, inMemoryMapOutputs.size() -> 7, commitMemory -> 1850488, usedMemory ->2150990
16/08/17 21:07:28 INFO reduce.LocalFetcher: localfetcher#98 about to shuffle output of map attempt_local1999544224_0005_m_000007_0 decomp: 310880 len: 310884 to MEMORY
16/08/17 21:07:28 INFO reduce.InMemoryMapOutput: Read 310880 bytes from map-output for attempt_local1999544224_0005_m_000007_0
16/08/17 21:07:28 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 310880, inMemoryMapOutputs.size() -> 8, commitMemory -> 2150990, usedMemory ->2461870
16/08/17 21:07:28 INFO reduce.LocalFetcher: localfetcher#98 about to shuffle output of map attempt_local1999544224_0005_m_000013_0 decomp: 301808 len: 301812 to MEMORY
16/08/17 21:07:28 INFO reduce.InMemoryMapOutput: Read 301808 bytes from map-output for attempt_local1999544224_0005_m_000013_0
16/08/17 21:07:28 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 301808, inMemoryMapOutputs.size() -> 9, commitMemory -> 2461870, usedMemory ->2763678
16/08/17 21:07:28 INFO reduce.LocalFetcher: localfetcher#98 about to shuffle output of map attempt_local1999544224_0005_m_000000_0 decomp: 308222 len: 308226 to MEMORY
16/08/17 21:07:28 INFO reduce.InMemoryMapOutput: Read 308222 bytes from map-output for attempt_local1999544224_0005_m_000000_0
16/08/17 21:07:28 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 308222, inMemoryMapOutputs.size() -> 10, commitMemory -> 2763678, usedMemory ->3071900
16/08/17 21:07:28 INFO reduce.LocalFetcher: localfetcher#98 about to shuffle output of map attempt_local1999544224_0005_m_000006_0 decomp: 313012 len: 313016 to MEMORY
16/08/17 21:07:28 INFO reduce.InMemoryMapOutput: Read 313012 bytes from map-output for attempt_local1999544224_0005_m_000006_0
16/08/17 21:07:28 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 313012, inMemoryMapOutputs.size() -> 11, commitMemory -> 3071900, usedMemory ->3384912
16/08/17 21:07:28 INFO reduce.LocalFetcher: localfetcher#98 about to shuffle output of map attempt_local1999544224_0005_m_000009_0 decomp: 299360 len: 299364 to MEMORY
16/08/17 21:07:28 INFO reduce.MergeManagerImpl: Merged 16 segments, 4702230 bytes to disk to satisfy reduce memory limit
16/08/17 21:07:28 INFO reduce.MergeManagerImpl: Merging 1 files, 4702204 bytes from disk
16/08/17 21:07:28 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:07:28 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:07:28 INFO reduce.InMemoryMapOutput: Read 299360 bytes from map-output for attempt_local1999544224_0005_m_000009_0
16/08/17 21:07:28 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 299360, inMemoryMapOutputs.size() -> 12, commitMemory -> 3384912, usedMemory ->3684272
16/08/17 21:07:28 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 4702178 bytes
16/08/17 21:07:28 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:28 INFO reduce.LocalFetcher: localfetcher#98 about to shuffle output of map attempt_local1999544224_0005_m_000015_0 decomp: 292678 len: 292682 to MEMORY
16/08/17 21:07:28 INFO reduce.InMemoryMapOutput: Read 292678 bytes from map-output for attempt_local1999544224_0005_m_000015_0
16/08/17 21:07:28 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 292678, inMemoryMapOutputs.size() -> 13, commitMemory -> 3684272, usedMemory ->3976950
16/08/17 21:07:28 INFO reduce.LocalFetcher: localfetcher#98 about to shuffle output of map attempt_local1999544224_0005_m_000002_0 decomp: 311066 len: 311070 to MEMORY
16/08/17 21:07:28 INFO reduce.InMemoryMapOutput: Read 311066 bytes from map-output for attempt_local1999544224_0005_m_000002_0
16/08/17 21:07:28 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 311066, inMemoryMapOutputs.size() -> 14, commitMemory -> 3976950, usedMemory ->4288016
16/08/17 21:07:28 INFO reduce.LocalFetcher: localfetcher#98 about to shuffle output of map attempt_local1999544224_0005_m_000008_0 decomp: 303554 len: 303558 to MEMORY
16/08/17 21:07:28 INFO reduce.InMemoryMapOutput: Read 303554 bytes from map-output for attempt_local1999544224_0005_m_000008_0
16/08/17 21:07:28 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 303554, inMemoryMapOutputs.size() -> 15, commitMemory -> 4288016, usedMemory ->4591570
16/08/17 21:07:28 INFO reduce.LocalFetcher: localfetcher#98 about to shuffle output of map attempt_local1999544224_0005_m_000014_0 decomp: 297854 len: 297858 to MEMORY
16/08/17 21:07:28 INFO reduce.InMemoryMapOutput: Read 297854 bytes from map-output for attempt_local1999544224_0005_m_000014_0
16/08/17 21:07:28 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 297854, inMemoryMapOutputs.size() -> 16, commitMemory -> 4591570, usedMemory ->4889424
16/08/17 21:07:28 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:07:28 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:28 INFO reduce.MergeManagerImpl: finalMerge called with 16 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:07:28 INFO mapred.Merger: Merging 16 sorted segments
16/08/17 21:07:28 INFO mapred.Merger: Down to the last merge-pass, with 16 segments left of total size: 4889020 bytes
16/08/17 21:07:28 INFO reduce.MergeManagerImpl: Merged 16 segments, 4889424 bytes to disk to satisfy reduce memory limit
16/08/17 21:07:28 INFO reduce.MergeManagerImpl: Merging 1 files, 4889398 bytes from disk
16/08/17 21:07:28 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:07:28 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:07:28 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 4889372 bytes
16/08/17 21:07:28 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:28 INFO mapred.Task: Task:attempt_local518136789_0007_r_000008_0 is done. And is in the process of committing
16/08/17 21:07:28 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:28 INFO mapred.Task: Task attempt_local518136789_0007_r_000008_0 is allowed to commit now
16/08/17 21:07:28 INFO output.FileOutputCommitter: Saved output of task 'attempt_local518136789_0007_r_000008_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/e_given_f_phrase/_temporary/0/task_local518136789_0007_r_000008
16/08/17 21:07:28 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:07:28 INFO mapred.Task: Task 'attempt_local518136789_0007_r_000008_0' done.
16/08/17 21:07:28 INFO mapred.LocalJobRunner: Finishing task: attempt_local518136789_0007_r_000008_0
16/08/17 21:07:28 INFO mapred.LocalJobRunner: Starting task: attempt_local518136789_0007_r_000009_0
16/08/17 21:07:28 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:28 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@11659dbc
16/08/17 21:07:28 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=365900576, maxSingleShuffleLimit=91475144, mergeThreshold=241494384, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:07:28 INFO reduce.EventFetcher: attempt_local518136789_0007_r_000009_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:07:28 INFO reduce.LocalFetcher: localfetcher#99 about to shuffle output of map attempt_local518136789_0007_m_000011_0 decomp: 295382 len: 295386 to MEMORY
16/08/17 21:07:28 INFO reduce.InMemoryMapOutput: Read 295382 bytes from map-output for attempt_local518136789_0007_m_000011_0
16/08/17 21:07:28 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 295382, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->295382
16/08/17 21:07:28 INFO reduce.LocalFetcher: localfetcher#99 about to shuffle output of map attempt_local518136789_0007_m_000004_0 decomp: 298092 len: 298096 to MEMORY
16/08/17 21:07:28 INFO reduce.InMemoryMapOutput: Read 298092 bytes from map-output for attempt_local518136789_0007_m_000004_0
16/08/17 21:07:28 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 298092, inMemoryMapOutputs.size() -> 2, commitMemory -> 295382, usedMemory ->593474
16/08/17 21:07:28 INFO reduce.LocalFetcher: localfetcher#99 about to shuffle output of map attempt_local518136789_0007_m_000010_0 decomp: 298230 len: 298234 to MEMORY
16/08/17 21:07:28 INFO reduce.InMemoryMapOutput: Read 298230 bytes from map-output for attempt_local518136789_0007_m_000010_0
16/08/17 21:07:28 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 298230, inMemoryMapOutputs.size() -> 3, commitMemory -> 593474, usedMemory ->891704
16/08/17 21:07:28 INFO reduce.LocalFetcher: localfetcher#99 about to shuffle output of map attempt_local518136789_0007_m_000000_0 decomp: 309600 len: 309604 to MEMORY
16/08/17 21:07:28 INFO reduce.InMemoryMapOutput: Read 309600 bytes from map-output for attempt_local518136789_0007_m_000000_0
16/08/17 21:07:28 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 309600, inMemoryMapOutputs.size() -> 4, commitMemory -> 891704, usedMemory ->1201304
16/08/17 21:07:28 INFO reduce.LocalFetcher: localfetcher#99 about to shuffle output of map attempt_local518136789_0007_m_000006_0 decomp: 295604 len: 295608 to MEMORY
16/08/17 21:07:28 INFO reduce.InMemoryMapOutput: Read 295604 bytes from map-output for attempt_local518136789_0007_m_000006_0
16/08/17 21:07:28 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 295604, inMemoryMapOutputs.size() -> 5, commitMemory -> 1201304, usedMemory ->1496908
16/08/17 21:07:28 INFO reduce.LocalFetcher: localfetcher#99 about to shuffle output of map attempt_local518136789_0007_m_000012_0 decomp: 289378 len: 289382 to MEMORY
16/08/17 21:07:28 INFO reduce.InMemoryMapOutput: Read 289378 bytes from map-output for attempt_local518136789_0007_m_000012_0
16/08/17 21:07:28 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 289378, inMemoryMapOutputs.size() -> 6, commitMemory -> 1496908, usedMemory ->1786286
16/08/17 21:07:28 INFO reduce.LocalFetcher: localfetcher#99 about to shuffle output of map attempt_local518136789_0007_m_000005_0 decomp: 309478 len: 309482 to MEMORY
16/08/17 21:07:28 INFO reduce.InMemoryMapOutput: Read 309478 bytes from map-output for attempt_local518136789_0007_m_000005_0
16/08/17 21:07:28 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 309478, inMemoryMapOutputs.size() -> 7, commitMemory -> 1786286, usedMemory ->2095764
16/08/17 21:07:28 INFO reduce.LocalFetcher: localfetcher#99 about to shuffle output of map attempt_local518136789_0007_m_000008_0 decomp: 291538 len: 291542 to MEMORY
16/08/17 21:07:28 INFO reduce.InMemoryMapOutput: Read 291538 bytes from map-output for attempt_local518136789_0007_m_000008_0
16/08/17 21:07:28 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 291538, inMemoryMapOutputs.size() -> 8, commitMemory -> 2095764, usedMemory ->2387302
16/08/17 21:07:28 INFO reduce.LocalFetcher: localfetcher#99 about to shuffle output of map attempt_local518136789_0007_m_000014_0 decomp: 280894 len: 280898 to MEMORY
16/08/17 21:07:28 INFO reduce.InMemoryMapOutput: Read 280894 bytes from map-output for attempt_local518136789_0007_m_000014_0
16/08/17 21:07:28 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 280894, inMemoryMapOutputs.size() -> 9, commitMemory -> 2387302, usedMemory ->2668196
16/08/17 21:07:28 INFO reduce.LocalFetcher: localfetcher#99 about to shuffle output of map attempt_local518136789_0007_m_000001_0 decomp: 294736 len: 294740 to MEMORY
16/08/17 21:07:28 INFO reduce.InMemoryMapOutput: Read 294736 bytes from map-output for attempt_local518136789_0007_m_000001_0
16/08/17 21:07:28 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 294736, inMemoryMapOutputs.size() -> 10, commitMemory -> 2668196, usedMemory ->2962932
16/08/17 21:07:28 INFO reduce.LocalFetcher: localfetcher#99 about to shuffle output of map attempt_local518136789_0007_m_000007_0 decomp: 306102 len: 306106 to MEMORY
16/08/17 21:07:28 INFO reduce.InMemoryMapOutput: Read 306102 bytes from map-output for attempt_local518136789_0007_m_000007_0
16/08/17 21:07:28 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 306102, inMemoryMapOutputs.size() -> 11, commitMemory -> 2962932, usedMemory ->3269034
16/08/17 21:07:28 INFO reduce.LocalFetcher: localfetcher#99 about to shuffle output of map attempt_local518136789_0007_m_000013_0 decomp: 295218 len: 295222 to MEMORY
16/08/17 21:07:28 INFO reduce.InMemoryMapOutput: Read 295218 bytes from map-output for attempt_local518136789_0007_m_000013_0
16/08/17 21:07:28 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 295218, inMemoryMapOutputs.size() -> 12, commitMemory -> 3269034, usedMemory ->3564252
16/08/17 21:07:28 INFO reduce.LocalFetcher: localfetcher#99 about to shuffle output of map attempt_local518136789_0007_m_000003_0 decomp: 295014 len: 295018 to MEMORY
16/08/17 21:07:28 INFO reduce.InMemoryMapOutput: Read 295014 bytes from map-output for attempt_local518136789_0007_m_000003_0
16/08/17 21:07:28 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 295014, inMemoryMapOutputs.size() -> 13, commitMemory -> 3564252, usedMemory ->3859266
16/08/17 21:07:28 INFO reduce.LocalFetcher: localfetcher#99 about to shuffle output of map attempt_local518136789_0007_m_000009_0 decomp: 289934 len: 289938 to MEMORY
16/08/17 21:07:28 INFO reduce.InMemoryMapOutput: Read 289934 bytes from map-output for attempt_local518136789_0007_m_000009_0
16/08/17 21:07:28 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 289934, inMemoryMapOutputs.size() -> 14, commitMemory -> 3859266, usedMemory ->4149200
16/08/17 21:07:29 INFO reduce.LocalFetcher: localfetcher#99 about to shuffle output of map attempt_local518136789_0007_m_000015_0 decomp: 284530 len: 284534 to MEMORY
16/08/17 21:07:29 INFO reduce.InMemoryMapOutput: Read 284530 bytes from map-output for attempt_local518136789_0007_m_000015_0
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 284530, inMemoryMapOutputs.size() -> 15, commitMemory -> 4149200, usedMemory ->4433730
16/08/17 21:07:29 INFO reduce.LocalFetcher: localfetcher#99 about to shuffle output of map attempt_local518136789_0007_m_000002_0 decomp: 298254 len: 298258 to MEMORY
16/08/17 21:07:29 INFO reduce.InMemoryMapOutput: Read 298254 bytes from map-output for attempt_local518136789_0007_m_000002_0
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 298254, inMemoryMapOutputs.size() -> 16, commitMemory -> 4433730, usedMemory ->4731984
16/08/17 21:07:29 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:07:29 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: finalMerge called with 16 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:07:29 INFO mapred.Merger: Merging 16 sorted segments
16/08/17 21:07:29 INFO mapred.Merger: Down to the last merge-pass, with 16 segments left of total size: 4731596 bytes
16/08/17 21:07:29 INFO mapred.Task: Task:attempt_local1999544224_0005_r_000008_0 is done. And is in the process of committing
16/08/17 21:07:29 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:29 INFO mapred.Task: Task attempt_local1999544224_0005_r_000008_0 is allowed to commit now
16/08/17 21:07:29 INFO output.FileOutputCommitter: Saved output of task 'attempt_local1999544224_0005_r_000008_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/f_given_e_phrase/_temporary/0/task_local1999544224_0005_r_000008
16/08/17 21:07:29 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:07:29 INFO mapred.Task: Task 'attempt_local1999544224_0005_r_000008_0' done.
16/08/17 21:07:29 INFO mapred.LocalJobRunner: Finishing task: attempt_local1999544224_0005_r_000008_0
16/08/17 21:07:29 INFO mapred.LocalJobRunner: Starting task: attempt_local1999544224_0005_r_000009_0
16/08/17 21:07:29 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:29 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@638c28c1
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=365900576, maxSingleShuffleLimit=91475144, mergeThreshold=241494384, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:07:29 INFO reduce.EventFetcher: attempt_local1999544224_0005_r_000009_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:07:29 INFO reduce.LocalFetcher: localfetcher#100 about to shuffle output of map attempt_local1999544224_0005_m_000004_0 decomp: 309040 len: 309044 to MEMORY
16/08/17 21:07:29 INFO reduce.InMemoryMapOutput: Read 309040 bytes from map-output for attempt_local1999544224_0005_m_000004_0
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 309040, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->309040
16/08/17 21:07:29 INFO reduce.LocalFetcher: localfetcher#100 about to shuffle output of map attempt_local1999544224_0005_m_000010_0 decomp: 300488 len: 300492 to MEMORY
16/08/17 21:07:29 INFO reduce.InMemoryMapOutput: Read 300488 bytes from map-output for attempt_local1999544224_0005_m_000010_0
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 300488, inMemoryMapOutputs.size() -> 2, commitMemory -> 309040, usedMemory ->609528
16/08/17 21:07:29 INFO reduce.LocalFetcher: localfetcher#100 about to shuffle output of map attempt_local1999544224_0005_m_000003_0 decomp: 315718 len: 315722 to MEMORY
16/08/17 21:07:29 INFO reduce.InMemoryMapOutput: Read 315718 bytes from map-output for attempt_local1999544224_0005_m_000003_0
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 315718, inMemoryMapOutputs.size() -> 3, commitMemory -> 609528, usedMemory ->925246
16/08/17 21:07:29 INFO reduce.LocalFetcher: localfetcher#100 about to shuffle output of map attempt_local1999544224_0005_m_000012_0 decomp: 311426 len: 311430 to MEMORY
16/08/17 21:07:29 INFO reduce.InMemoryMapOutput: Read 311426 bytes from map-output for attempt_local1999544224_0005_m_000012_0
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 311426, inMemoryMapOutputs.size() -> 4, commitMemory -> 925246, usedMemory ->1236672
16/08/17 21:07:29 INFO reduce.LocalFetcher: localfetcher#100 about to shuffle output of map attempt_local1999544224_0005_m_000005_0 decomp: 307350 len: 307354 to MEMORY
16/08/17 21:07:29 INFO reduce.InMemoryMapOutput: Read 307350 bytes from map-output for attempt_local1999544224_0005_m_000005_0
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 307350, inMemoryMapOutputs.size() -> 5, commitMemory -> 1236672, usedMemory ->1544022
16/08/17 21:07:29 INFO reduce.LocalFetcher: localfetcher#100 about to shuffle output of map attempt_local1999544224_0005_m_000011_0 decomp: 302744 len: 302748 to MEMORY
16/08/17 21:07:29 INFO reduce.InMemoryMapOutput: Read 302744 bytes from map-output for attempt_local1999544224_0005_m_000011_0
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 302744, inMemoryMapOutputs.size() -> 6, commitMemory -> 1544022, usedMemory ->1846766
16/08/17 21:07:29 INFO reduce.LocalFetcher: localfetcher#100 about to shuffle output of map attempt_local1999544224_0005_m_000001_0 decomp: 302148 len: 302152 to MEMORY
16/08/17 21:07:29 INFO reduce.InMemoryMapOutput: Read 302148 bytes from map-output for attempt_local1999544224_0005_m_000001_0
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 302148, inMemoryMapOutputs.size() -> 7, commitMemory -> 1846766, usedMemory ->2148914
16/08/17 21:07:29 INFO reduce.LocalFetcher: localfetcher#100 about to shuffle output of map attempt_local1999544224_0005_m_000007_0 decomp: 309114 len: 309118 to MEMORY
16/08/17 21:07:29 INFO reduce.InMemoryMapOutput: Read 309114 bytes from map-output for attempt_local1999544224_0005_m_000007_0
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 309114, inMemoryMapOutputs.size() -> 8, commitMemory -> 2148914, usedMemory ->2458028
16/08/17 21:07:29 INFO reduce.LocalFetcher: localfetcher#100 about to shuffle output of map attempt_local1999544224_0005_m_000013_0 decomp: 300840 len: 300844 to MEMORY
16/08/17 21:07:29 INFO reduce.InMemoryMapOutput: Read 300840 bytes from map-output for attempt_local1999544224_0005_m_000013_0
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 300840, inMemoryMapOutputs.size() -> 9, commitMemory -> 2458028, usedMemory ->2758868
16/08/17 21:07:29 INFO reduce.LocalFetcher: localfetcher#100 about to shuffle output of map attempt_local1999544224_0005_m_000000_0 decomp: 316322 len: 316326 to MEMORY
16/08/17 21:07:29 INFO reduce.InMemoryMapOutput: Read 316322 bytes from map-output for attempt_local1999544224_0005_m_000000_0
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 316322, inMemoryMapOutputs.size() -> 10, commitMemory -> 2758868, usedMemory ->3075190
16/08/17 21:07:29 INFO reduce.LocalFetcher: localfetcher#100 about to shuffle output of map attempt_local1999544224_0005_m_000006_0 decomp: 314788 len: 314792 to MEMORY
16/08/17 21:07:29 INFO reduce.InMemoryMapOutput: Read 314788 bytes from map-output for attempt_local1999544224_0005_m_000006_0
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 314788, inMemoryMapOutputs.size() -> 11, commitMemory -> 3075190, usedMemory ->3389978
16/08/17 21:07:29 INFO reduce.LocalFetcher: localfetcher#100 about to shuffle output of map attempt_local1999544224_0005_m_000009_0 decomp: 309178 len: 309182 to MEMORY
16/08/17 21:07:29 INFO reduce.InMemoryMapOutput: Read 309178 bytes from map-output for attempt_local1999544224_0005_m_000009_0
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 309178, inMemoryMapOutputs.size() -> 12, commitMemory -> 3389978, usedMemory ->3699156
16/08/17 21:07:29 INFO reduce.LocalFetcher: localfetcher#100 about to shuffle output of map attempt_local1999544224_0005_m_000015_0 decomp: 292874 len: 292878 to MEMORY
16/08/17 21:07:29 INFO reduce.InMemoryMapOutput: Read 292874 bytes from map-output for attempt_local1999544224_0005_m_000015_0
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 292874, inMemoryMapOutputs.size() -> 13, commitMemory -> 3699156, usedMemory ->3992030
16/08/17 21:07:29 INFO reduce.LocalFetcher: localfetcher#100 about to shuffle output of map attempt_local1999544224_0005_m_000002_0 decomp: 306770 len: 306774 to MEMORY
16/08/17 21:07:29 INFO reduce.InMemoryMapOutput: Read 306770 bytes from map-output for attempt_local1999544224_0005_m_000002_0
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 306770, inMemoryMapOutputs.size() -> 14, commitMemory -> 3992030, usedMemory ->4298800
16/08/17 21:07:29 INFO reduce.LocalFetcher: localfetcher#100 about to shuffle output of map attempt_local1999544224_0005_m_000008_0 decomp: 301750 len: 301754 to MEMORY
16/08/17 21:07:29 INFO reduce.InMemoryMapOutput: Read 301750 bytes from map-output for attempt_local1999544224_0005_m_000008_0
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 301750, inMemoryMapOutputs.size() -> 15, commitMemory -> 4298800, usedMemory ->4600550
16/08/17 21:07:29 INFO reduce.LocalFetcher: localfetcher#100 about to shuffle output of map attempt_local1999544224_0005_m_000014_0 decomp: 307762 len: 307766 to MEMORY
16/08/17 21:07:29 INFO reduce.InMemoryMapOutput: Read 307762 bytes from map-output for attempt_local1999544224_0005_m_000014_0
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 307762, inMemoryMapOutputs.size() -> 16, commitMemory -> 4600550, usedMemory ->4908312
16/08/17 21:07:29 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:07:29 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: finalMerge called with 16 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:07:29 INFO mapred.Merger: Merging 16 sorted segments
16/08/17 21:07:29 INFO mapred.Merger: Down to the last merge-pass, with 16 segments left of total size: 4907892 bytes
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: Merged 16 segments, 4731984 bytes to disk to satisfy reduce memory limit
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: Merging 1 files, 4731958 bytes from disk
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:07:29 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:07:29 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 4731916 bytes
16/08/17 21:07:29 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: Merged 16 segments, 4908312 bytes to disk to satisfy reduce memory limit
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: Merging 1 files, 4908286 bytes from disk
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:07:29 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:07:29 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 4908260 bytes
16/08/17 21:07:29 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:29 INFO mapred.Task: Task:attempt_local1999544224_0005_r_000009_0 is done. And is in the process of committing
16/08/17 21:07:29 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:29 INFO mapred.Task: Task attempt_local1999544224_0005_r_000009_0 is allowed to commit now
16/08/17 21:07:29 INFO output.FileOutputCommitter: Saved output of task 'attempt_local1999544224_0005_r_000009_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/f_given_e_phrase/_temporary/0/task_local1999544224_0005_r_000009
16/08/17 21:07:29 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:07:29 INFO mapred.Task: Task 'attempt_local1999544224_0005_r_000009_0' done.
16/08/17 21:07:29 INFO mapred.LocalJobRunner: Finishing task: attempt_local1999544224_0005_r_000009_0
16/08/17 21:07:29 INFO mapred.LocalJobRunner: Starting task: attempt_local1999544224_0005_r_000010_0
16/08/17 21:07:29 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:29 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@267b4701
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=365900576, maxSingleShuffleLimit=91475144, mergeThreshold=241494384, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:07:29 INFO reduce.EventFetcher: attempt_local1999544224_0005_r_000010_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:07:29 INFO reduce.LocalFetcher: localfetcher#101 about to shuffle output of map attempt_local1999544224_0005_m_000004_0 decomp: 313464 len: 313468 to MEMORY
16/08/17 21:07:29 INFO reduce.InMemoryMapOutput: Read 313464 bytes from map-output for attempt_local1999544224_0005_m_000004_0
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 313464, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->313464
16/08/17 21:07:29 INFO reduce.LocalFetcher: localfetcher#101 about to shuffle output of map attempt_local1999544224_0005_m_000010_0 decomp: 298018 len: 298022 to MEMORY
16/08/17 21:07:29 INFO reduce.InMemoryMapOutput: Read 298018 bytes from map-output for attempt_local1999544224_0005_m_000010_0
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 298018, inMemoryMapOutputs.size() -> 2, commitMemory -> 313464, usedMemory ->611482
16/08/17 21:07:29 INFO reduce.LocalFetcher: localfetcher#101 about to shuffle output of map attempt_local1999544224_0005_m_000003_0 decomp: 307846 len: 307850 to MEMORY
16/08/17 21:07:29 INFO reduce.InMemoryMapOutput: Read 307846 bytes from map-output for attempt_local1999544224_0005_m_000003_0
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 307846, inMemoryMapOutputs.size() -> 3, commitMemory -> 611482, usedMemory ->919328
16/08/17 21:07:29 INFO reduce.LocalFetcher: localfetcher#101 about to shuffle output of map attempt_local1999544224_0005_m_000012_0 decomp: 311800 len: 311804 to MEMORY
16/08/17 21:07:29 INFO reduce.InMemoryMapOutput: Read 311800 bytes from map-output for attempt_local1999544224_0005_m_000012_0
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 311800, inMemoryMapOutputs.size() -> 4, commitMemory -> 919328, usedMemory ->1231128
16/08/17 21:07:29 INFO reduce.LocalFetcher: localfetcher#101 about to shuffle output of map attempt_local1999544224_0005_m_000005_0 decomp: 306488 len: 306492 to MEMORY
16/08/17 21:07:29 INFO reduce.InMemoryMapOutput: Read 306488 bytes from map-output for attempt_local1999544224_0005_m_000005_0
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 306488, inMemoryMapOutputs.size() -> 5, commitMemory -> 1231128, usedMemory ->1537616
16/08/17 21:07:29 INFO reduce.LocalFetcher: localfetcher#101 about to shuffle output of map attempt_local1999544224_0005_m_000011_0 decomp: 293340 len: 293344 to MEMORY
16/08/17 21:07:29 INFO reduce.InMemoryMapOutput: Read 293340 bytes from map-output for attempt_local1999544224_0005_m_000011_0
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 293340, inMemoryMapOutputs.size() -> 6, commitMemory -> 1537616, usedMemory ->1830956
16/08/17 21:07:29 INFO reduce.LocalFetcher: localfetcher#101 about to shuffle output of map attempt_local1999544224_0005_m_000001_0 decomp: 314184 len: 314188 to MEMORY
16/08/17 21:07:29 INFO reduce.InMemoryMapOutput: Read 314184 bytes from map-output for attempt_local1999544224_0005_m_000001_0
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 314184, inMemoryMapOutputs.size() -> 7, commitMemory -> 1830956, usedMemory ->2145140
16/08/17 21:07:29 INFO reduce.LocalFetcher: localfetcher#101 about to shuffle output of map attempt_local1999544224_0005_m_000007_0 decomp: 308734 len: 308738 to MEMORY
16/08/17 21:07:29 INFO reduce.InMemoryMapOutput: Read 308734 bytes from map-output for attempt_local1999544224_0005_m_000007_0
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 308734, inMemoryMapOutputs.size() -> 8, commitMemory -> 2145140, usedMemory ->2453874
16/08/17 21:07:29 INFO reduce.LocalFetcher: localfetcher#101 about to shuffle output of map attempt_local1999544224_0005_m_000013_0 decomp: 299342 len: 299346 to MEMORY
16/08/17 21:07:29 INFO reduce.InMemoryMapOutput: Read 299342 bytes from map-output for attempt_local1999544224_0005_m_000013_0
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 299342, inMemoryMapOutputs.size() -> 9, commitMemory -> 2453874, usedMemory ->2753216
16/08/17 21:07:29 INFO reduce.LocalFetcher: localfetcher#101 about to shuffle output of map attempt_local1999544224_0005_m_000000_0 decomp: 300566 len: 300570 to MEMORY
16/08/17 21:07:29 INFO reduce.InMemoryMapOutput: Read 300566 bytes from map-output for attempt_local1999544224_0005_m_000000_0
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 300566, inMemoryMapOutputs.size() -> 10, commitMemory -> 2753216, usedMemory ->3053782
16/08/17 21:07:29 INFO reduce.LocalFetcher: localfetcher#101 about to shuffle output of map attempt_local1999544224_0005_m_000006_0 decomp: 303522 len: 303526 to MEMORY
16/08/17 21:07:29 INFO reduce.InMemoryMapOutput: Read 303522 bytes from map-output for attempt_local1999544224_0005_m_000006_0
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 303522, inMemoryMapOutputs.size() -> 11, commitMemory -> 3053782, usedMemory ->3357304
16/08/17 21:07:29 INFO reduce.LocalFetcher: localfetcher#101 about to shuffle output of map attempt_local1999544224_0005_m_000009_0 decomp: 293056 len: 293060 to MEMORY
16/08/17 21:07:29 INFO reduce.InMemoryMapOutput: Read 293056 bytes from map-output for attempt_local1999544224_0005_m_000009_0
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 293056, inMemoryMapOutputs.size() -> 12, commitMemory -> 3357304, usedMemory ->3650360
16/08/17 21:07:29 INFO reduce.LocalFetcher: localfetcher#101 about to shuffle output of map attempt_local1999544224_0005_m_000015_0 decomp: 306784 len: 306788 to MEMORY
16/08/17 21:07:29 INFO reduce.InMemoryMapOutput: Read 306784 bytes from map-output for attempt_local1999544224_0005_m_000015_0
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 306784, inMemoryMapOutputs.size() -> 13, commitMemory -> 3650360, usedMemory ->3957144
16/08/17 21:07:29 INFO reduce.LocalFetcher: localfetcher#101 about to shuffle output of map attempt_local1999544224_0005_m_000002_0 decomp: 299780 len: 299784 to MEMORY
16/08/17 21:07:29 INFO reduce.InMemoryMapOutput: Read 299780 bytes from map-output for attempt_local1999544224_0005_m_000002_0
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 299780, inMemoryMapOutputs.size() -> 14, commitMemory -> 3957144, usedMemory ->4256924
16/08/17 21:07:29 INFO reduce.LocalFetcher: localfetcher#101 about to shuffle output of map attempt_local1999544224_0005_m_000008_0 decomp: 306400 len: 306404 to MEMORY
16/08/17 21:07:29 INFO reduce.InMemoryMapOutput: Read 306400 bytes from map-output for attempt_local1999544224_0005_m_000008_0
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 306400, inMemoryMapOutputs.size() -> 15, commitMemory -> 4256924, usedMemory ->4563324
16/08/17 21:07:29 INFO reduce.LocalFetcher: localfetcher#101 about to shuffle output of map attempt_local1999544224_0005_m_000014_0 decomp: 288270 len: 288274 to MEMORY
16/08/17 21:07:29 INFO reduce.InMemoryMapOutput: Read 288270 bytes from map-output for attempt_local1999544224_0005_m_000014_0
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 288270, inMemoryMapOutputs.size() -> 16, commitMemory -> 4563324, usedMemory ->4851594
16/08/17 21:07:29 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:07:29 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: finalMerge called with 16 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:07:29 INFO mapred.Merger: Merging 16 sorted segments
16/08/17 21:07:29 INFO mapred.Merger: Down to the last merge-pass, with 16 segments left of total size: 4851226 bytes
16/08/17 21:07:29 INFO mapred.Task: Task:attempt_local518136789_0007_r_000009_0 is done. And is in the process of committing
16/08/17 21:07:29 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:29 INFO mapred.Task: Task attempt_local518136789_0007_r_000009_0 is allowed to commit now
16/08/17 21:07:29 INFO output.FileOutputCommitter: Saved output of task 'attempt_local518136789_0007_r_000009_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/e_given_f_phrase/_temporary/0/task_local518136789_0007_r_000009
16/08/17 21:07:29 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:07:29 INFO mapred.Task: Task 'attempt_local518136789_0007_r_000009_0' done.
16/08/17 21:07:29 INFO mapred.LocalJobRunner: Finishing task: attempt_local518136789_0007_r_000009_0
16/08/17 21:07:29 INFO mapred.LocalJobRunner: Starting task: attempt_local518136789_0007_r_000010_0
16/08/17 21:07:29 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:29 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@50623514
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=366267584, maxSingleShuffleLimit=91566896, mergeThreshold=241736608, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:07:29 INFO reduce.EventFetcher: attempt_local518136789_0007_r_000010_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:07:29 INFO reduce.LocalFetcher: localfetcher#102 about to shuffle output of map attempt_local518136789_0007_m_000011_0 decomp: 290274 len: 290278 to MEMORY
16/08/17 21:07:29 INFO reduce.InMemoryMapOutput: Read 290274 bytes from map-output for attempt_local518136789_0007_m_000011_0
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 290274, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->290274
16/08/17 21:07:29 INFO reduce.LocalFetcher: localfetcher#102 about to shuffle output of map attempt_local518136789_0007_m_000004_0 decomp: 292148 len: 292152 to MEMORY
16/08/17 21:07:29 INFO reduce.InMemoryMapOutput: Read 292148 bytes from map-output for attempt_local518136789_0007_m_000004_0
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 292148, inMemoryMapOutputs.size() -> 2, commitMemory -> 290274, usedMemory ->582422
16/08/17 21:07:29 INFO reduce.LocalFetcher: localfetcher#102 about to shuffle output of map attempt_local518136789_0007_m_000010_0 decomp: 284104 len: 284108 to MEMORY
16/08/17 21:07:29 INFO reduce.InMemoryMapOutput: Read 284104 bytes from map-output for attempt_local518136789_0007_m_000010_0
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 284104, inMemoryMapOutputs.size() -> 3, commitMemory -> 582422, usedMemory ->866526
16/08/17 21:07:29 INFO reduce.LocalFetcher: localfetcher#102 about to shuffle output of map attempt_local518136789_0007_m_000000_0 decomp: 298642 len: 298646 to MEMORY
16/08/17 21:07:29 INFO reduce.InMemoryMapOutput: Read 298642 bytes from map-output for attempt_local518136789_0007_m_000000_0
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 298642, inMemoryMapOutputs.size() -> 4, commitMemory -> 866526, usedMemory ->1165168
16/08/17 21:07:29 INFO reduce.LocalFetcher: localfetcher#102 about to shuffle output of map attempt_local518136789_0007_m_000006_0 decomp: 288276 len: 288280 to MEMORY
16/08/17 21:07:29 INFO reduce.InMemoryMapOutput: Read 288276 bytes from map-output for attempt_local518136789_0007_m_000006_0
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 288276, inMemoryMapOutputs.size() -> 5, commitMemory -> 1165168, usedMemory ->1453444
16/08/17 21:07:29 INFO reduce.LocalFetcher: localfetcher#102 about to shuffle output of map attempt_local518136789_0007_m_000012_0 decomp: 293886 len: 293890 to MEMORY
16/08/17 21:07:29 INFO reduce.InMemoryMapOutput: Read 293886 bytes from map-output for attempt_local518136789_0007_m_000012_0
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 293886, inMemoryMapOutputs.size() -> 6, commitMemory -> 1453444, usedMemory ->1747330
16/08/17 21:07:29 INFO reduce.LocalFetcher: localfetcher#102 about to shuffle output of map attempt_local518136789_0007_m_000005_0 decomp: 299498 len: 299502 to MEMORY
16/08/17 21:07:29 INFO reduce.InMemoryMapOutput: Read 299498 bytes from map-output for attempt_local518136789_0007_m_000005_0
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 299498, inMemoryMapOutputs.size() -> 7, commitMemory -> 1747330, usedMemory ->2046828
16/08/17 21:07:29 INFO reduce.LocalFetcher: localfetcher#102 about to shuffle output of map attempt_local518136789_0007_m_000008_0 decomp: 291984 len: 291988 to MEMORY
16/08/17 21:07:29 INFO reduce.InMemoryMapOutput: Read 291984 bytes from map-output for attempt_local518136789_0007_m_000008_0
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 291984, inMemoryMapOutputs.size() -> 8, commitMemory -> 2046828, usedMemory ->2338812
16/08/17 21:07:29 INFO reduce.LocalFetcher: localfetcher#102 about to shuffle output of map attempt_local518136789_0007_m_000014_0 decomp: 294434 len: 294438 to MEMORY
16/08/17 21:07:29 INFO reduce.InMemoryMapOutput: Read 294434 bytes from map-output for attempt_local518136789_0007_m_000014_0
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 294434, inMemoryMapOutputs.size() -> 9, commitMemory -> 2338812, usedMemory ->2633246
16/08/17 21:07:29 INFO reduce.LocalFetcher: localfetcher#102 about to shuffle output of map attempt_local518136789_0007_m_000001_0 decomp: 293416 len: 293420 to MEMORY
16/08/17 21:07:29 INFO reduce.InMemoryMapOutput: Read 293416 bytes from map-output for attempt_local518136789_0007_m_000001_0
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 293416, inMemoryMapOutputs.size() -> 10, commitMemory -> 2633246, usedMemory ->2926662
16/08/17 21:07:29 INFO reduce.LocalFetcher: localfetcher#102 about to shuffle output of map attempt_local518136789_0007_m_000007_0 decomp: 300936 len: 300940 to MEMORY
16/08/17 21:07:29 INFO reduce.InMemoryMapOutput: Read 300936 bytes from map-output for attempt_local518136789_0007_m_000007_0
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 300936, inMemoryMapOutputs.size() -> 11, commitMemory -> 2926662, usedMemory ->3227598
16/08/17 21:07:29 INFO reduce.LocalFetcher: localfetcher#102 about to shuffle output of map attempt_local518136789_0007_m_000013_0 decomp: 287690 len: 287694 to MEMORY
16/08/17 21:07:29 INFO reduce.InMemoryMapOutput: Read 287690 bytes from map-output for attempt_local518136789_0007_m_000013_0
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 287690, inMemoryMapOutputs.size() -> 12, commitMemory -> 3227598, usedMemory ->3515288
16/08/17 21:07:29 INFO reduce.LocalFetcher: localfetcher#102 about to shuffle output of map attempt_local518136789_0007_m_000003_0 decomp: 290966 len: 290970 to MEMORY
16/08/17 21:07:29 INFO reduce.InMemoryMapOutput: Read 290966 bytes from map-output for attempt_local518136789_0007_m_000003_0
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 290966, inMemoryMapOutputs.size() -> 13, commitMemory -> 3515288, usedMemory ->3806254
16/08/17 21:07:29 INFO reduce.LocalFetcher: localfetcher#102 about to shuffle output of map attempt_local518136789_0007_m_000009_0 decomp: 298120 len: 298124 to MEMORY
16/08/17 21:07:29 INFO reduce.InMemoryMapOutput: Read 298120 bytes from map-output for attempt_local518136789_0007_m_000009_0
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 298120, inMemoryMapOutputs.size() -> 14, commitMemory -> 3806254, usedMemory ->4104374
16/08/17 21:07:29 INFO reduce.LocalFetcher: localfetcher#102 about to shuffle output of map attempt_local518136789_0007_m_000015_0 decomp: 286608 len: 286612 to MEMORY
16/08/17 21:07:29 INFO reduce.InMemoryMapOutput: Read 286608 bytes from map-output for attempt_local518136789_0007_m_000015_0
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 286608, inMemoryMapOutputs.size() -> 15, commitMemory -> 4104374, usedMemory ->4390982
16/08/17 21:07:29 INFO reduce.LocalFetcher: localfetcher#102 about to shuffle output of map attempt_local518136789_0007_m_000002_0 decomp: 296024 len: 296028 to MEMORY
16/08/17 21:07:29 INFO reduce.InMemoryMapOutput: Read 296024 bytes from map-output for attempt_local518136789_0007_m_000002_0
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 296024, inMemoryMapOutputs.size() -> 16, commitMemory -> 4390982, usedMemory ->4687006
16/08/17 21:07:29 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:07:29 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: finalMerge called with 16 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:07:29 INFO mapred.Merger: Merging 16 sorted segments
16/08/17 21:07:29 INFO mapred.Merger: Down to the last merge-pass, with 16 segments left of total size: 4686618 bytes
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: Merged 16 segments, 4851594 bytes to disk to satisfy reduce memory limit
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: Merging 1 files, 4851568 bytes from disk
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:07:29 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:07:29 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 4851538 bytes
16/08/17 21:07:29 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: Merged 16 segments, 4687006 bytes to disk to satisfy reduce memory limit
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: Merging 1 files, 4686980 bytes from disk
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:07:29 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:07:29 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 4686958 bytes
16/08/17 21:07:29 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:29 INFO mapred.Task: Task:attempt_local518136789_0007_r_000010_0 is done. And is in the process of committing
16/08/17 21:07:29 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:29 INFO mapred.Task: Task attempt_local518136789_0007_r_000010_0 is allowed to commit now
16/08/17 21:07:29 INFO output.FileOutputCommitter: Saved output of task 'attempt_local518136789_0007_r_000010_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/e_given_f_phrase/_temporary/0/task_local518136789_0007_r_000010
16/08/17 21:07:29 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:07:29 INFO mapred.Task: Task 'attempt_local518136789_0007_r_000010_0' done.
16/08/17 21:07:29 INFO mapred.LocalJobRunner: Finishing task: attempt_local518136789_0007_r_000010_0
16/08/17 21:07:29 INFO mapred.LocalJobRunner: Starting task: attempt_local518136789_0007_r_000011_0
16/08/17 21:07:29 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:29 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@6581f036
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=366267584, maxSingleShuffleLimit=91566896, mergeThreshold=241736608, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:07:29 INFO reduce.EventFetcher: attempt_local518136789_0007_r_000011_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:07:29 INFO reduce.LocalFetcher: localfetcher#103 about to shuffle output of map attempt_local518136789_0007_m_000011_0 decomp: 292882 len: 292886 to MEMORY
16/08/17 21:07:29 INFO reduce.InMemoryMapOutput: Read 292882 bytes from map-output for attempt_local518136789_0007_m_000011_0
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 292882, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->292882
16/08/17 21:07:29 INFO reduce.LocalFetcher: localfetcher#103 about to shuffle output of map attempt_local518136789_0007_m_000004_0 decomp: 294698 len: 294702 to MEMORY
16/08/17 21:07:29 INFO reduce.InMemoryMapOutput: Read 294698 bytes from map-output for attempt_local518136789_0007_m_000004_0
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 294698, inMemoryMapOutputs.size() -> 2, commitMemory -> 292882, usedMemory ->587580
16/08/17 21:07:29 INFO reduce.LocalFetcher: localfetcher#103 about to shuffle output of map attempt_local518136789_0007_m_000010_0 decomp: 292172 len: 292176 to MEMORY
16/08/17 21:07:29 INFO reduce.InMemoryMapOutput: Read 292172 bytes from map-output for attempt_local518136789_0007_m_000010_0
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 292172, inMemoryMapOutputs.size() -> 3, commitMemory -> 587580, usedMemory ->879752
16/08/17 21:07:29 INFO reduce.LocalFetcher: localfetcher#103 about to shuffle output of map attempt_local518136789_0007_m_000000_0 decomp: 306452 len: 306456 to MEMORY
16/08/17 21:07:29 INFO reduce.InMemoryMapOutput: Read 306452 bytes from map-output for attempt_local518136789_0007_m_000000_0
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 306452, inMemoryMapOutputs.size() -> 4, commitMemory -> 879752, usedMemory ->1186204
16/08/17 21:07:29 INFO reduce.LocalFetcher: localfetcher#103 about to shuffle output of map attempt_local518136789_0007_m_000006_0 decomp: 295500 len: 295504 to MEMORY
16/08/17 21:07:29 INFO reduce.InMemoryMapOutput: Read 295500 bytes from map-output for attempt_local518136789_0007_m_000006_0
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 295500, inMemoryMapOutputs.size() -> 5, commitMemory -> 1186204, usedMemory ->1481704
16/08/17 21:07:29 INFO reduce.LocalFetcher: localfetcher#103 about to shuffle output of map attempt_local518136789_0007_m_000012_0 decomp: 291616 len: 291620 to MEMORY
16/08/17 21:07:29 INFO reduce.InMemoryMapOutput: Read 291616 bytes from map-output for attempt_local518136789_0007_m_000012_0
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 291616, inMemoryMapOutputs.size() -> 6, commitMemory -> 1481704, usedMemory ->1773320
16/08/17 21:07:29 INFO reduce.LocalFetcher: localfetcher#103 about to shuffle output of map attempt_local518136789_0007_m_000005_0 decomp: 299560 len: 299564 to MEMORY
16/08/17 21:07:29 INFO reduce.InMemoryMapOutput: Read 299560 bytes from map-output for attempt_local518136789_0007_m_000005_0
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 299560, inMemoryMapOutputs.size() -> 7, commitMemory -> 1773320, usedMemory ->2072880
16/08/17 21:07:29 INFO reduce.LocalFetcher: localfetcher#103 about to shuffle output of map attempt_local518136789_0007_m_000008_0 decomp: 297610 len: 297614 to MEMORY
16/08/17 21:07:29 INFO reduce.InMemoryMapOutput: Read 297610 bytes from map-output for attempt_local518136789_0007_m_000008_0
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 297610, inMemoryMapOutputs.size() -> 8, commitMemory -> 2072880, usedMemory ->2370490
16/08/17 21:07:29 INFO reduce.LocalFetcher: localfetcher#103 about to shuffle output of map attempt_local518136789_0007_m_000014_0 decomp: 285758 len: 285762 to MEMORY
16/08/17 21:07:29 INFO reduce.InMemoryMapOutput: Read 285758 bytes from map-output for attempt_local518136789_0007_m_000014_0
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 285758, inMemoryMapOutputs.size() -> 9, commitMemory -> 2370490, usedMemory ->2656248
16/08/17 21:07:29 INFO reduce.LocalFetcher: localfetcher#103 about to shuffle output of map attempt_local518136789_0007_m_000001_0 decomp: 299662 len: 299666 to MEMORY
16/08/17 21:07:29 INFO reduce.InMemoryMapOutput: Read 299662 bytes from map-output for attempt_local518136789_0007_m_000001_0
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 299662, inMemoryMapOutputs.size() -> 10, commitMemory -> 2656248, usedMemory ->2955910
16/08/17 21:07:29 INFO reduce.LocalFetcher: localfetcher#103 about to shuffle output of map attempt_local518136789_0007_m_000007_0 decomp: 297114 len: 297118 to MEMORY
16/08/17 21:07:29 INFO reduce.InMemoryMapOutput: Read 297114 bytes from map-output for attempt_local518136789_0007_m_000007_0
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 297114, inMemoryMapOutputs.size() -> 11, commitMemory -> 2955910, usedMemory ->3253024
16/08/17 21:07:29 INFO reduce.LocalFetcher: localfetcher#103 about to shuffle output of map attempt_local518136789_0007_m_000013_0 decomp: 299810 len: 299814 to MEMORY
16/08/17 21:07:29 INFO reduce.InMemoryMapOutput: Read 299810 bytes from map-output for attempt_local518136789_0007_m_000013_0
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 299810, inMemoryMapOutputs.size() -> 12, commitMemory -> 3253024, usedMemory ->3552834
16/08/17 21:07:29 INFO reduce.LocalFetcher: localfetcher#103 about to shuffle output of map attempt_local518136789_0007_m_000003_0 decomp: 292962 len: 292966 to MEMORY
16/08/17 21:07:29 INFO reduce.InMemoryMapOutput: Read 292962 bytes from map-output for attempt_local518136789_0007_m_000003_0
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 292962, inMemoryMapOutputs.size() -> 13, commitMemory -> 3552834, usedMemory ->3845796
16/08/17 21:07:29 INFO reduce.LocalFetcher: localfetcher#103 about to shuffle output of map attempt_local518136789_0007_m_000009_0 decomp: 299258 len: 299262 to MEMORY
16/08/17 21:07:29 INFO reduce.InMemoryMapOutput: Read 299258 bytes from map-output for attempt_local518136789_0007_m_000009_0
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 299258, inMemoryMapOutputs.size() -> 14, commitMemory -> 3845796, usedMemory ->4145054
16/08/17 21:07:29 INFO reduce.LocalFetcher: localfetcher#103 about to shuffle output of map attempt_local518136789_0007_m_000015_0 decomp: 291148 len: 291152 to MEMORY
16/08/17 21:07:29 INFO reduce.InMemoryMapOutput: Read 291148 bytes from map-output for attempt_local518136789_0007_m_000015_0
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 291148, inMemoryMapOutputs.size() -> 15, commitMemory -> 4145054, usedMemory ->4436202
16/08/17 21:07:29 INFO reduce.LocalFetcher: localfetcher#103 about to shuffle output of map attempt_local518136789_0007_m_000002_0 decomp: 313946 len: 313950 to MEMORY
16/08/17 21:07:29 INFO reduce.InMemoryMapOutput: Read 313946 bytes from map-output for attempt_local518136789_0007_m_000002_0
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 313946, inMemoryMapOutputs.size() -> 16, commitMemory -> 4436202, usedMemory ->4750148
16/08/17 21:07:29 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:07:29 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: finalMerge called with 16 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:07:29 INFO mapred.Merger: Merging 16 sorted segments
16/08/17 21:07:29 INFO mapred.Merger: Down to the last merge-pass, with 16 segments left of total size: 4749764 bytes
16/08/17 21:07:29 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)
	at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
16/08/17 21:07:29 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)
	at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
16/08/17 21:07:29 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)
	at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
16/08/17 21:07:29 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)
	at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
16/08/17 21:07:29 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)
	at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
16/08/17 21:07:29 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)
	at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
16/08/17 21:07:29 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)
	at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
16/08/17 21:07:29 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)
	at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
16/08/17 21:07:29 INFO mapred.Task: Task:attempt_local1999544224_0005_r_000010_0 is done. And is in the process of committing
16/08/17 21:07:29 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:29 INFO mapred.Task: Task attempt_local1999544224_0005_r_000010_0 is allowed to commit now
16/08/17 21:07:29 INFO output.FileOutputCommitter: Saved output of task 'attempt_local1999544224_0005_r_000010_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/f_given_e_phrase/_temporary/0/task_local1999544224_0005_r_000010
16/08/17 21:07:29 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:07:29 INFO mapred.Task: Task 'attempt_local1999544224_0005_r_000010_0' done.
16/08/17 21:07:29 INFO mapred.LocalJobRunner: Finishing task: attempt_local1999544224_0005_r_000010_0
16/08/17 21:07:29 INFO mapred.LocalJobRunner: Starting task: attempt_local1999544224_0005_r_000011_0
16/08/17 21:07:29 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:29 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@74cea8b9
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=366267584, maxSingleShuffleLimit=91566896, mergeThreshold=241736608, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:07:29 INFO reduce.EventFetcher: attempt_local1999544224_0005_r_000011_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:07:29 INFO reduce.LocalFetcher: localfetcher#104 about to shuffle output of map attempt_local1999544224_0005_m_000004_0 decomp: 311964 len: 311968 to MEMORY
16/08/17 21:07:29 INFO reduce.InMemoryMapOutput: Read 311964 bytes from map-output for attempt_local1999544224_0005_m_000004_0
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 311964, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->311964
16/08/17 21:07:29 INFO reduce.LocalFetcher: localfetcher#104 about to shuffle output of map attempt_local1999544224_0005_m_000010_0 decomp: 306654 len: 306658 to MEMORY
16/08/17 21:07:29 INFO reduce.InMemoryMapOutput: Read 306654 bytes from map-output for attempt_local1999544224_0005_m_000010_0
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 306654, inMemoryMapOutputs.size() -> 2, commitMemory -> 311964, usedMemory ->618618
16/08/17 21:07:29 INFO reduce.LocalFetcher: localfetcher#104 about to shuffle output of map attempt_local1999544224_0005_m_000003_0 decomp: 305062 len: 305066 to MEMORY
16/08/17 21:07:29 INFO reduce.InMemoryMapOutput: Read 305062 bytes from map-output for attempt_local1999544224_0005_m_000003_0
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 305062, inMemoryMapOutputs.size() -> 3, commitMemory -> 618618, usedMemory ->923680
16/08/17 21:07:29 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)
	at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
16/08/17 21:07:29 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)
	at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
16/08/17 21:07:29 INFO reduce.LocalFetcher: localfetcher#104 about to shuffle output of map attempt_local1999544224_0005_m_000012_0 decomp: 309328 len: 309332 to MEMORY
16/08/17 21:07:29 INFO reduce.InMemoryMapOutput: Read 309328 bytes from map-output for attempt_local1999544224_0005_m_000012_0
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 309328, inMemoryMapOutputs.size() -> 4, commitMemory -> 923680, usedMemory ->1233008
16/08/17 21:07:29 INFO reduce.LocalFetcher: localfetcher#104 about to shuffle output of map attempt_local1999544224_0005_m_000005_0 decomp: 330138 len: 330142 to MEMORY
16/08/17 21:07:29 INFO reduce.InMemoryMapOutput: Read 330138 bytes from map-output for attempt_local1999544224_0005_m_000005_0
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 330138, inMemoryMapOutputs.size() -> 5, commitMemory -> 1233008, usedMemory ->1563146
16/08/17 21:07:29 INFO reduce.LocalFetcher: localfetcher#104 about to shuffle output of map attempt_local1999544224_0005_m_000011_0 decomp: 309306 len: 309310 to MEMORY
16/08/17 21:07:29 INFO reduce.InMemoryMapOutput: Read 309306 bytes from map-output for attempt_local1999544224_0005_m_000011_0
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 309306, inMemoryMapOutputs.size() -> 6, commitMemory -> 1563146, usedMemory ->1872452
16/08/17 21:07:29 INFO reduce.LocalFetcher: localfetcher#104 about to shuffle output of map attempt_local1999544224_0005_m_000001_0 decomp: 313914 len: 313918 to MEMORY
16/08/17 21:07:29 INFO reduce.InMemoryMapOutput: Read 313914 bytes from map-output for attempt_local1999544224_0005_m_000001_0
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 313914, inMemoryMapOutputs.size() -> 7, commitMemory -> 1872452, usedMemory ->2186366
16/08/17 21:07:29 INFO reduce.LocalFetcher: localfetcher#104 about to shuffle output of map attempt_local1999544224_0005_m_000007_0 decomp: 311522 len: 311526 to MEMORY
16/08/17 21:07:29 INFO reduce.InMemoryMapOutput: Read 311522 bytes from map-output for attempt_local1999544224_0005_m_000007_0
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 311522, inMemoryMapOutputs.size() -> 8, commitMemory -> 2186366, usedMemory ->2497888
16/08/17 21:07:29 INFO reduce.LocalFetcher: localfetcher#104 about to shuffle output of map attempt_local1999544224_0005_m_000013_0 decomp: 305416 len: 305420 to MEMORY
16/08/17 21:07:29 INFO reduce.InMemoryMapOutput: Read 305416 bytes from map-output for attempt_local1999544224_0005_m_000013_0
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 305416, inMemoryMapOutputs.size() -> 9, commitMemory -> 2497888, usedMemory ->2803304
16/08/17 21:07:29 INFO reduce.LocalFetcher: localfetcher#104 about to shuffle output of map attempt_local1999544224_0005_m_000000_0 decomp: 318666 len: 318670 to MEMORY
16/08/17 21:07:29 INFO reduce.InMemoryMapOutput: Read 318666 bytes from map-output for attempt_local1999544224_0005_m_000000_0
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 318666, inMemoryMapOutputs.size() -> 10, commitMemory -> 2803304, usedMemory ->3121970
16/08/17 21:07:29 INFO reduce.LocalFetcher: localfetcher#104 about to shuffle output of map attempt_local1999544224_0005_m_000006_0 decomp: 301966 len: 301970 to MEMORY
16/08/17 21:07:29 INFO reduce.InMemoryMapOutput: Read 301966 bytes from map-output for attempt_local1999544224_0005_m_000006_0
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 301966, inMemoryMapOutputs.size() -> 11, commitMemory -> 3121970, usedMemory ->3423936
16/08/17 21:07:29 INFO reduce.LocalFetcher: localfetcher#104 about to shuffle output of map attempt_local1999544224_0005_m_000009_0 decomp: 305110 len: 305114 to MEMORY
16/08/17 21:07:29 INFO reduce.InMemoryMapOutput: Read 305110 bytes from map-output for attempt_local1999544224_0005_m_000009_0
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 305110, inMemoryMapOutputs.size() -> 12, commitMemory -> 3423936, usedMemory ->3729046
16/08/17 21:07:29 INFO reduce.LocalFetcher: localfetcher#104 about to shuffle output of map attempt_local1999544224_0005_m_000015_0 decomp: 299578 len: 299582 to MEMORY
16/08/17 21:07:29 INFO reduce.InMemoryMapOutput: Read 299578 bytes from map-output for attempt_local1999544224_0005_m_000015_0
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 299578, inMemoryMapOutputs.size() -> 13, commitMemory -> 3729046, usedMemory ->4028624
16/08/17 21:07:29 INFO reduce.LocalFetcher: localfetcher#104 about to shuffle output of map attempt_local1999544224_0005_m_000002_0 decomp: 321718 len: 321722 to MEMORY
16/08/17 21:07:29 INFO reduce.InMemoryMapOutput: Read 321718 bytes from map-output for attempt_local1999544224_0005_m_000002_0
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 321718, inMemoryMapOutputs.size() -> 14, commitMemory -> 4028624, usedMemory ->4350342
16/08/17 21:07:29 INFO reduce.LocalFetcher: localfetcher#104 about to shuffle output of map attempt_local1999544224_0005_m_000008_0 decomp: 305470 len: 305474 to MEMORY
16/08/17 21:07:29 INFO reduce.InMemoryMapOutput: Read 305470 bytes from map-output for attempt_local1999544224_0005_m_000008_0
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 305470, inMemoryMapOutputs.size() -> 15, commitMemory -> 4350342, usedMemory ->4655812
16/08/17 21:07:29 INFO reduce.LocalFetcher: localfetcher#104 about to shuffle output of map attempt_local1999544224_0005_m_000014_0 decomp: 302276 len: 302280 to MEMORY
16/08/17 21:07:29 INFO reduce.InMemoryMapOutput: Read 302276 bytes from map-output for attempt_local1999544224_0005_m_000014_0
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 302276, inMemoryMapOutputs.size() -> 16, commitMemory -> 4655812, usedMemory ->4958088
16/08/17 21:07:29 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:07:29 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: finalMerge called with 16 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:07:29 INFO mapred.Merger: Merging 16 sorted segments
16/08/17 21:07:29 INFO mapred.Merger: Down to the last merge-pass, with 16 segments left of total size: 4957696 bytes
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: Merged 16 segments, 4750148 bytes to disk to satisfy reduce memory limit
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: Merging 1 files, 4750122 bytes from disk
16/08/17 21:07:29 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:07:29 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:07:29 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 4750100 bytes
16/08/17 21:07:29 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:30 INFO reduce.MergeManagerImpl: Merged 16 segments, 4958088 bytes to disk to satisfy reduce memory limit
16/08/17 21:07:30 INFO reduce.MergeManagerImpl: Merging 1 files, 4958062 bytes from disk
16/08/17 21:07:30 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:07:30 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:07:30 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 4958016 bytes
16/08/17 21:07:30 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:30 INFO mapred.Task: Task:attempt_local518136789_0007_r_000011_0 is done. And is in the process of committing
16/08/17 21:07:30 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:30 INFO mapred.Task: Task attempt_local518136789_0007_r_000011_0 is allowed to commit now
16/08/17 21:07:30 INFO output.FileOutputCommitter: Saved output of task 'attempt_local518136789_0007_r_000011_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/e_given_f_phrase/_temporary/0/task_local518136789_0007_r_000011
16/08/17 21:07:30 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:07:30 INFO mapred.Task: Task 'attempt_local518136789_0007_r_000011_0' done.
16/08/17 21:07:30 INFO mapred.LocalJobRunner: Finishing task: attempt_local518136789_0007_r_000011_0
16/08/17 21:07:30 INFO mapred.LocalJobRunner: Starting task: attempt_local518136789_0007_r_000012_0
16/08/17 21:07:30 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:30 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@2cad512e
16/08/17 21:07:30 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=360762560, maxSingleShuffleLimit=90190640, mergeThreshold=238103296, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:07:30 INFO reduce.EventFetcher: attempt_local518136789_0007_r_000012_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:07:30 INFO reduce.LocalFetcher: localfetcher#105 about to shuffle output of map attempt_local518136789_0007_m_000011_0 decomp: 299432 len: 299436 to MEMORY
16/08/17 21:07:30 INFO reduce.InMemoryMapOutput: Read 299432 bytes from map-output for attempt_local518136789_0007_m_000011_0
16/08/17 21:07:30 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 299432, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->299432
16/08/17 21:07:30 INFO reduce.LocalFetcher: localfetcher#105 about to shuffle output of map attempt_local518136789_0007_m_000004_0 decomp: 312536 len: 312540 to MEMORY
16/08/17 21:07:30 INFO reduce.InMemoryMapOutput: Read 312536 bytes from map-output for attempt_local518136789_0007_m_000004_0
16/08/17 21:07:30 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 312536, inMemoryMapOutputs.size() -> 2, commitMemory -> 299432, usedMemory ->611968
16/08/17 21:07:30 INFO reduce.LocalFetcher: localfetcher#105 about to shuffle output of map attempt_local518136789_0007_m_000010_0 decomp: 297556 len: 297560 to MEMORY
16/08/17 21:07:30 INFO reduce.InMemoryMapOutput: Read 297556 bytes from map-output for attempt_local518136789_0007_m_000010_0
16/08/17 21:07:30 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 297556, inMemoryMapOutputs.size() -> 3, commitMemory -> 611968, usedMemory ->909524
16/08/17 21:07:30 INFO reduce.LocalFetcher: localfetcher#105 about to shuffle output of map attempt_local518136789_0007_m_000000_0 decomp: 292874 len: 292878 to MEMORY
16/08/17 21:07:30 INFO reduce.InMemoryMapOutput: Read 292874 bytes from map-output for attempt_local518136789_0007_m_000000_0
16/08/17 21:07:30 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 292874, inMemoryMapOutputs.size() -> 4, commitMemory -> 909524, usedMemory ->1202398
16/08/17 21:07:30 INFO reduce.LocalFetcher: localfetcher#105 about to shuffle output of map attempt_local518136789_0007_m_000006_0 decomp: 308982 len: 308986 to MEMORY
16/08/17 21:07:30 INFO reduce.InMemoryMapOutput: Read 308982 bytes from map-output for attempt_local518136789_0007_m_000006_0
16/08/17 21:07:30 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 308982, inMemoryMapOutputs.size() -> 5, commitMemory -> 1202398, usedMemory ->1511380
16/08/17 21:07:30 INFO reduce.LocalFetcher: localfetcher#105 about to shuffle output of map attempt_local518136789_0007_m_000012_0 decomp: 290174 len: 290178 to MEMORY
16/08/17 21:07:30 INFO reduce.InMemoryMapOutput: Read 290174 bytes from map-output for attempt_local518136789_0007_m_000012_0
16/08/17 21:07:30 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 290174, inMemoryMapOutputs.size() -> 6, commitMemory -> 1511380, usedMemory ->1801554
16/08/17 21:07:30 INFO reduce.LocalFetcher: localfetcher#105 about to shuffle output of map attempt_local518136789_0007_m_000005_0 decomp: 297460 len: 297464 to MEMORY
16/08/17 21:07:30 INFO reduce.InMemoryMapOutput: Read 297460 bytes from map-output for attempt_local518136789_0007_m_000005_0
16/08/17 21:07:30 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 297460, inMemoryMapOutputs.size() -> 7, commitMemory -> 1801554, usedMemory ->2099014
16/08/17 21:07:30 INFO reduce.LocalFetcher: localfetcher#105 about to shuffle output of map attempt_local518136789_0007_m_000008_0 decomp: 297918 len: 297922 to MEMORY
16/08/17 21:07:30 INFO reduce.InMemoryMapOutput: Read 297918 bytes from map-output for attempt_local518136789_0007_m_000008_0
16/08/17 21:07:30 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 297918, inMemoryMapOutputs.size() -> 8, commitMemory -> 2099014, usedMemory ->2396932
16/08/17 21:07:30 INFO reduce.LocalFetcher: localfetcher#105 about to shuffle output of map attempt_local518136789_0007_m_000014_0 decomp: 294828 len: 294832 to MEMORY
16/08/17 21:07:30 INFO reduce.InMemoryMapOutput: Read 294828 bytes from map-output for attempt_local518136789_0007_m_000014_0
16/08/17 21:07:30 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 294828, inMemoryMapOutputs.size() -> 9, commitMemory -> 2396932, usedMemory ->2691760
16/08/17 21:07:30 INFO reduce.LocalFetcher: localfetcher#105 about to shuffle output of map attempt_local518136789_0007_m_000001_0 decomp: 302732 len: 302736 to MEMORY
16/08/17 21:07:30 INFO reduce.InMemoryMapOutput: Read 302732 bytes from map-output for attempt_local518136789_0007_m_000001_0
16/08/17 21:07:30 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 302732, inMemoryMapOutputs.size() -> 10, commitMemory -> 2691760, usedMemory ->2994492
16/08/17 21:07:30 INFO reduce.LocalFetcher: localfetcher#105 about to shuffle output of map attempt_local518136789_0007_m_000007_0 decomp: 296866 len: 296870 to MEMORY
16/08/17 21:07:30 INFO reduce.InMemoryMapOutput: Read 296866 bytes from map-output for attempt_local518136789_0007_m_000007_0
16/08/17 21:07:30 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 296866, inMemoryMapOutputs.size() -> 11, commitMemory -> 2994492, usedMemory ->3291358
16/08/17 21:07:30 INFO reduce.LocalFetcher: localfetcher#105 about to shuffle output of map attempt_local518136789_0007_m_000013_0 decomp: 293224 len: 293228 to MEMORY
16/08/17 21:07:30 INFO reduce.InMemoryMapOutput: Read 293224 bytes from map-output for attempt_local518136789_0007_m_000013_0
16/08/17 21:07:30 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 293224, inMemoryMapOutputs.size() -> 12, commitMemory -> 3291358, usedMemory ->3584582
16/08/17 21:07:30 INFO reduce.LocalFetcher: localfetcher#105 about to shuffle output of map attempt_local518136789_0007_m_000003_0 decomp: 311166 len: 311170 to MEMORY
16/08/17 21:07:30 INFO reduce.InMemoryMapOutput: Read 311166 bytes from map-output for attempt_local518136789_0007_m_000003_0
16/08/17 21:07:30 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 311166, inMemoryMapOutputs.size() -> 13, commitMemory -> 3584582, usedMemory ->3895748
16/08/17 21:07:30 INFO reduce.LocalFetcher: localfetcher#105 about to shuffle output of map attempt_local518136789_0007_m_000009_0 decomp: 290400 len: 290404 to MEMORY
16/08/17 21:07:30 INFO reduce.InMemoryMapOutput: Read 290400 bytes from map-output for attempt_local518136789_0007_m_000009_0
16/08/17 21:07:30 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 290400, inMemoryMapOutputs.size() -> 14, commitMemory -> 3895748, usedMemory ->4186148
16/08/17 21:07:30 INFO reduce.LocalFetcher: localfetcher#105 about to shuffle output of map attempt_local518136789_0007_m_000015_0 decomp: 296010 len: 296014 to MEMORY
16/08/17 21:07:30 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)
	at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
16/08/17 21:07:30 INFO reduce.InMemoryMapOutput: Read 296010 bytes from map-output for attempt_local518136789_0007_m_000015_0
16/08/17 21:07:30 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 296010, inMemoryMapOutputs.size() -> 15, commitMemory -> 4186148, usedMemory ->4482158
16/08/17 21:07:30 INFO reduce.LocalFetcher: localfetcher#105 about to shuffle output of map attempt_local518136789_0007_m_000002_0 decomp: 311754 len: 311758 to MEMORY
16/08/17 21:07:30 INFO reduce.InMemoryMapOutput: Read 311754 bytes from map-output for attempt_local518136789_0007_m_000002_0
16/08/17 21:07:30 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 311754, inMemoryMapOutputs.size() -> 16, commitMemory -> 4482158, usedMemory ->4793912
16/08/17 21:07:30 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:07:30 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:30 INFO reduce.MergeManagerImpl: finalMerge called with 16 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:07:30 INFO mapred.Merger: Merging 16 sorted segments
16/08/17 21:07:30 INFO mapred.Merger: Down to the last merge-pass, with 16 segments left of total size: 4793540 bytes
16/08/17 21:07:30 INFO mapred.Task: Task:attempt_local1999544224_0005_r_000011_0 is done. And is in the process of committing
16/08/17 21:07:30 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:30 INFO mapred.Task: Task attempt_local1999544224_0005_r_000011_0 is allowed to commit now
16/08/17 21:07:30 INFO output.FileOutputCommitter: Saved output of task 'attempt_local1999544224_0005_r_000011_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/f_given_e_phrase/_temporary/0/task_local1999544224_0005_r_000011
16/08/17 21:07:30 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:07:30 INFO mapred.Task: Task 'attempt_local1999544224_0005_r_000011_0' done.
16/08/17 21:07:30 INFO mapred.LocalJobRunner: Finishing task: attempt_local1999544224_0005_r_000011_0
16/08/17 21:07:30 INFO mapred.LocalJobRunner: Starting task: attempt_local1999544224_0005_r_000012_0
16/08/17 21:07:30 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:30 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@2938f709
16/08/17 21:07:30 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=360762560, maxSingleShuffleLimit=90190640, mergeThreshold=238103296, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:07:30 INFO reduce.EventFetcher: attempt_local1999544224_0005_r_000012_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:07:30 INFO reduce.LocalFetcher: localfetcher#106 about to shuffle output of map attempt_local1999544224_0005_m_000004_0 decomp: 298962 len: 298966 to MEMORY
16/08/17 21:07:30 INFO reduce.InMemoryMapOutput: Read 298962 bytes from map-output for attempt_local1999544224_0005_m_000004_0
16/08/17 21:07:30 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 298962, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->298962
16/08/17 21:07:30 INFO reduce.LocalFetcher: localfetcher#106 about to shuffle output of map attempt_local1999544224_0005_m_000010_0 decomp: 303044 len: 303048 to MEMORY
16/08/17 21:07:30 INFO reduce.InMemoryMapOutput: Read 303044 bytes from map-output for attempt_local1999544224_0005_m_000010_0
16/08/17 21:07:30 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 303044, inMemoryMapOutputs.size() -> 2, commitMemory -> 298962, usedMemory ->602006
16/08/17 21:07:30 INFO reduce.LocalFetcher: localfetcher#106 about to shuffle output of map attempt_local1999544224_0005_m_000003_0 decomp: 306460 len: 306464 to MEMORY
16/08/17 21:07:30 INFO reduce.InMemoryMapOutput: Read 306460 bytes from map-output for attempt_local1999544224_0005_m_000003_0
16/08/17 21:07:30 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 306460, inMemoryMapOutputs.size() -> 3, commitMemory -> 602006, usedMemory ->908466
16/08/17 21:07:30 INFO reduce.LocalFetcher: localfetcher#106 about to shuffle output of map attempt_local1999544224_0005_m_000012_0 decomp: 297618 len: 297622 to MEMORY
16/08/17 21:07:30 INFO reduce.InMemoryMapOutput: Read 297618 bytes from map-output for attempt_local1999544224_0005_m_000012_0
16/08/17 21:07:30 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 297618, inMemoryMapOutputs.size() -> 4, commitMemory -> 908466, usedMemory ->1206084
16/08/17 21:07:30 INFO reduce.LocalFetcher: localfetcher#106 about to shuffle output of map attempt_local1999544224_0005_m_000005_0 decomp: 315462 len: 315466 to MEMORY
16/08/17 21:07:30 INFO reduce.InMemoryMapOutput: Read 315462 bytes from map-output for attempt_local1999544224_0005_m_000005_0
16/08/17 21:07:30 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 315462, inMemoryMapOutputs.size() -> 5, commitMemory -> 1206084, usedMemory ->1521546
16/08/17 21:07:30 INFO reduce.LocalFetcher: localfetcher#106 about to shuffle output of map attempt_local1999544224_0005_m_000011_0 decomp: 302554 len: 302558 to MEMORY
16/08/17 21:07:30 INFO reduce.InMemoryMapOutput: Read 302554 bytes from map-output for attempt_local1999544224_0005_m_000011_0
16/08/17 21:07:30 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 302554, inMemoryMapOutputs.size() -> 6, commitMemory -> 1521546, usedMemory ->1824100
16/08/17 21:07:30 INFO reduce.LocalFetcher: localfetcher#106 about to shuffle output of map attempt_local1999544224_0005_m_000001_0 decomp: 309472 len: 309476 to MEMORY
16/08/17 21:07:30 INFO reduce.InMemoryMapOutput: Read 309472 bytes from map-output for attempt_local1999544224_0005_m_000001_0
16/08/17 21:07:30 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 309472, inMemoryMapOutputs.size() -> 7, commitMemory -> 1824100, usedMemory ->2133572
16/08/17 21:07:30 INFO reduce.LocalFetcher: localfetcher#106 about to shuffle output of map attempt_local1999544224_0005_m_000007_0 decomp: 307788 len: 307792 to MEMORY
16/08/17 21:07:30 INFO reduce.InMemoryMapOutput: Read 307788 bytes from map-output for attempt_local1999544224_0005_m_000007_0
16/08/17 21:07:30 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 307788, inMemoryMapOutputs.size() -> 8, commitMemory -> 2133572, usedMemory ->2441360
16/08/17 21:07:30 INFO reduce.LocalFetcher: localfetcher#106 about to shuffle output of map attempt_local1999544224_0005_m_000013_0 decomp: 294260 len: 294264 to MEMORY
16/08/17 21:07:30 INFO reduce.InMemoryMapOutput: Read 294260 bytes from map-output for attempt_local1999544224_0005_m_000013_0
16/08/17 21:07:30 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 294260, inMemoryMapOutputs.size() -> 9, commitMemory -> 2441360, usedMemory ->2735620
16/08/17 21:07:30 INFO reduce.LocalFetcher: localfetcher#106 about to shuffle output of map attempt_local1999544224_0005_m_000000_0 decomp: 314204 len: 314208 to MEMORY
16/08/17 21:07:30 INFO reduce.InMemoryMapOutput: Read 314204 bytes from map-output for attempt_local1999544224_0005_m_000000_0
16/08/17 21:07:30 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 314204, inMemoryMapOutputs.size() -> 10, commitMemory -> 2735620, usedMemory ->3049824
16/08/17 21:07:30 INFO reduce.LocalFetcher: localfetcher#106 about to shuffle output of map attempt_local1999544224_0005_m_000006_0 decomp: 316490 len: 316494 to MEMORY
16/08/17 21:07:30 INFO reduce.InMemoryMapOutput: Read 316490 bytes from map-output for attempt_local1999544224_0005_m_000006_0
16/08/17 21:07:30 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 316490, inMemoryMapOutputs.size() -> 11, commitMemory -> 3049824, usedMemory ->3366314
16/08/17 21:07:30 INFO reduce.LocalFetcher: localfetcher#106 about to shuffle output of map attempt_local1999544224_0005_m_000009_0 decomp: 303742 len: 303746 to MEMORY
16/08/17 21:07:30 INFO reduce.InMemoryMapOutput: Read 303742 bytes from map-output for attempt_local1999544224_0005_m_000009_0
16/08/17 21:07:30 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 303742, inMemoryMapOutputs.size() -> 12, commitMemory -> 3366314, usedMemory ->3670056
16/08/17 21:07:30 INFO reduce.LocalFetcher: localfetcher#106 about to shuffle output of map attempt_local1999544224_0005_m_000015_0 decomp: 301436 len: 301440 to MEMORY
16/08/17 21:07:30 INFO reduce.InMemoryMapOutput: Read 301436 bytes from map-output for attempt_local1999544224_0005_m_000015_0
16/08/17 21:07:30 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 301436, inMemoryMapOutputs.size() -> 13, commitMemory -> 3670056, usedMemory ->3971492
16/08/17 21:07:30 INFO reduce.LocalFetcher: localfetcher#106 about to shuffle output of map attempt_local1999544224_0005_m_000002_0 decomp: 312190 len: 312194 to MEMORY
16/08/17 21:07:30 INFO reduce.InMemoryMapOutput: Read 312190 bytes from map-output for attempt_local1999544224_0005_m_000002_0
16/08/17 21:07:30 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 312190, inMemoryMapOutputs.size() -> 14, commitMemory -> 3971492, usedMemory ->4283682
16/08/17 21:07:30 INFO reduce.LocalFetcher: localfetcher#106 about to shuffle output of map attempt_local1999544224_0005_m_000008_0 decomp: 305060 len: 305064 to MEMORY
16/08/17 21:07:30 INFO reduce.InMemoryMapOutput: Read 305060 bytes from map-output for attempt_local1999544224_0005_m_000008_0
16/08/17 21:07:30 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 305060, inMemoryMapOutputs.size() -> 15, commitMemory -> 4283682, usedMemory ->4588742
16/08/17 21:07:30 INFO reduce.LocalFetcher: localfetcher#106 about to shuffle output of map attempt_local1999544224_0005_m_000014_0 decomp: 301172 len: 301176 to MEMORY
16/08/17 21:07:30 INFO reduce.InMemoryMapOutput: Read 301172 bytes from map-output for attempt_local1999544224_0005_m_000014_0
16/08/17 21:07:30 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 301172, inMemoryMapOutputs.size() -> 16, commitMemory -> 4588742, usedMemory ->4889914
16/08/17 21:07:30 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:07:30 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:30 INFO reduce.MergeManagerImpl: finalMerge called with 16 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:07:30 INFO mapred.Merger: Merging 16 sorted segments
16/08/17 21:07:30 INFO mapred.Merger: Down to the last merge-pass, with 16 segments left of total size: 4889586 bytes
16/08/17 21:07:30 INFO reduce.MergeManagerImpl: Merged 16 segments, 4793912 bytes to disk to satisfy reduce memory limit
16/08/17 21:07:30 INFO reduce.MergeManagerImpl: Merging 1 files, 4793886 bytes from disk
16/08/17 21:07:30 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:07:30 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:07:30 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 4793864 bytes
16/08/17 21:07:30 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:30 INFO reduce.MergeManagerImpl: Merged 16 segments, 4889914 bytes to disk to satisfy reduce memory limit
16/08/17 21:07:30 INFO reduce.MergeManagerImpl: Merging 1 files, 4889888 bytes from disk
16/08/17 21:07:30 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:07:30 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:07:30 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 4889870 bytes
16/08/17 21:07:30 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:30 INFO mapred.Task: Task:attempt_local518136789_0007_r_000012_0 is done. And is in the process of committing
16/08/17 21:07:30 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:30 INFO mapred.Task: Task attempt_local518136789_0007_r_000012_0 is allowed to commit now
16/08/17 21:07:30 INFO output.FileOutputCommitter: Saved output of task 'attempt_local518136789_0007_r_000012_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/e_given_f_phrase/_temporary/0/task_local518136789_0007_r_000012
16/08/17 21:07:30 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:07:30 INFO mapred.Task: Task 'attempt_local518136789_0007_r_000012_0' done.
16/08/17 21:07:30 INFO mapred.LocalJobRunner: Finishing task: attempt_local518136789_0007_r_000012_0
16/08/17 21:07:30 INFO mapred.LocalJobRunner: Starting task: attempt_local518136789_0007_r_000013_0
16/08/17 21:07:30 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:30 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@37e6cdc2
16/08/17 21:07:30 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=360762560, maxSingleShuffleLimit=90190640, mergeThreshold=238103296, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:07:30 INFO reduce.EventFetcher: attempt_local518136789_0007_r_000013_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:07:30 INFO reduce.LocalFetcher: localfetcher#107 about to shuffle output of map attempt_local518136789_0007_m_000011_0 decomp: 303350 len: 303354 to MEMORY
16/08/17 21:07:30 INFO reduce.InMemoryMapOutput: Read 303350 bytes from map-output for attempt_local518136789_0007_m_000011_0
16/08/17 21:07:30 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 303350, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->303350
16/08/17 21:07:30 INFO reduce.LocalFetcher: localfetcher#107 about to shuffle output of map attempt_local518136789_0007_m_000004_0 decomp: 303942 len: 303946 to MEMORY
16/08/17 21:07:30 INFO reduce.InMemoryMapOutput: Read 303942 bytes from map-output for attempt_local518136789_0007_m_000004_0
16/08/17 21:07:30 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 303942, inMemoryMapOutputs.size() -> 2, commitMemory -> 303350, usedMemory ->607292
16/08/17 21:07:30 INFO reduce.LocalFetcher: localfetcher#107 about to shuffle output of map attempt_local518136789_0007_m_000010_0 decomp: 297586 len: 297590 to MEMORY
16/08/17 21:07:30 INFO reduce.InMemoryMapOutput: Read 297586 bytes from map-output for attempt_local518136789_0007_m_000010_0
16/08/17 21:07:30 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 297586, inMemoryMapOutputs.size() -> 3, commitMemory -> 607292, usedMemory ->904878
16/08/17 21:07:30 INFO reduce.LocalFetcher: localfetcher#107 about to shuffle output of map attempt_local518136789_0007_m_000000_0 decomp: 296936 len: 296940 to MEMORY
16/08/17 21:07:30 INFO reduce.InMemoryMapOutput: Read 296936 bytes from map-output for attempt_local518136789_0007_m_000000_0
16/08/17 21:07:30 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 296936, inMemoryMapOutputs.size() -> 4, commitMemory -> 904878, usedMemory ->1201814
16/08/17 21:07:30 INFO reduce.LocalFetcher: localfetcher#107 about to shuffle output of map attempt_local518136789_0007_m_000006_0 decomp: 308692 len: 308696 to MEMORY
16/08/17 21:07:30 INFO reduce.InMemoryMapOutput: Read 308692 bytes from map-output for attempt_local518136789_0007_m_000006_0
16/08/17 21:07:30 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 308692, inMemoryMapOutputs.size() -> 5, commitMemory -> 1201814, usedMemory ->1510506
16/08/17 21:07:30 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)
	at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
16/08/17 21:07:30 INFO reduce.LocalFetcher: localfetcher#107 about to shuffle output of map attempt_local518136789_0007_m_000012_0 decomp: 299586 len: 299590 to MEMORY
16/08/17 21:07:30 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)
	at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
16/08/17 21:07:30 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)
	at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
16/08/17 21:07:30 INFO reduce.InMemoryMapOutput: Read 299586 bytes from map-output for attempt_local518136789_0007_m_000012_0
16/08/17 21:07:30 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 299586, inMemoryMapOutputs.size() -> 6, commitMemory -> 1510506, usedMemory ->1810092
16/08/17 21:07:30 INFO reduce.LocalFetcher: localfetcher#107 about to shuffle output of map attempt_local518136789_0007_m_000005_0 decomp: 300908 len: 300912 to MEMORY
16/08/17 21:07:30 INFO reduce.InMemoryMapOutput: Read 300908 bytes from map-output for attempt_local518136789_0007_m_000005_0
16/08/17 21:07:30 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 300908, inMemoryMapOutputs.size() -> 7, commitMemory -> 1810092, usedMemory ->2111000
16/08/17 21:07:30 INFO reduce.LocalFetcher: localfetcher#107 about to shuffle output of map attempt_local518136789_0007_m_000008_0 decomp: 294452 len: 294456 to MEMORY
16/08/17 21:07:30 INFO reduce.InMemoryMapOutput: Read 294452 bytes from map-output for attempt_local518136789_0007_m_000008_0
16/08/17 21:07:30 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 294452, inMemoryMapOutputs.size() -> 8, commitMemory -> 2111000, usedMemory ->2405452
16/08/17 21:07:30 INFO reduce.LocalFetcher: localfetcher#107 about to shuffle output of map attempt_local518136789_0007_m_000014_0 decomp: 283264 len: 283268 to MEMORY
16/08/17 21:07:30 INFO reduce.InMemoryMapOutput: Read 283264 bytes from map-output for attempt_local518136789_0007_m_000014_0
16/08/17 21:07:30 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 283264, inMemoryMapOutputs.size() -> 9, commitMemory -> 2405452, usedMemory ->2688716
16/08/17 21:07:30 INFO reduce.LocalFetcher: localfetcher#107 about to shuffle output of map attempt_local518136789_0007_m_000001_0 decomp: 296036 len: 296040 to MEMORY
16/08/17 21:07:30 INFO reduce.InMemoryMapOutput: Read 296036 bytes from map-output for attempt_local518136789_0007_m_000001_0
16/08/17 21:07:30 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 296036, inMemoryMapOutputs.size() -> 10, commitMemory -> 2688716, usedMemory ->2984752
16/08/17 21:07:30 INFO reduce.LocalFetcher: localfetcher#107 about to shuffle output of map attempt_local518136789_0007_m_000007_0 decomp: 304564 len: 304568 to MEMORY
16/08/17 21:07:30 INFO reduce.InMemoryMapOutput: Read 304564 bytes from map-output for attempt_local518136789_0007_m_000007_0
16/08/17 21:07:30 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 304564, inMemoryMapOutputs.size() -> 11, commitMemory -> 2984752, usedMemory ->3289316
16/08/17 21:07:30 INFO reduce.LocalFetcher: localfetcher#107 about to shuffle output of map attempt_local518136789_0007_m_000013_0 decomp: 297706 len: 297710 to MEMORY
16/08/17 21:07:30 INFO reduce.InMemoryMapOutput: Read 297706 bytes from map-output for attempt_local518136789_0007_m_000013_0
16/08/17 21:07:30 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 297706, inMemoryMapOutputs.size() -> 12, commitMemory -> 3289316, usedMemory ->3587022
16/08/17 21:07:30 INFO reduce.LocalFetcher: localfetcher#107 about to shuffle output of map attempt_local518136789_0007_m_000003_0 decomp: 306298 len: 306302 to MEMORY
16/08/17 21:07:30 INFO reduce.InMemoryMapOutput: Read 306298 bytes from map-output for attempt_local518136789_0007_m_000003_0
16/08/17 21:07:30 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 306298, inMemoryMapOutputs.size() -> 13, commitMemory -> 3587022, usedMemory ->3893320
16/08/17 21:07:30 INFO mapred.Task: Task:attempt_local1999544224_0005_r_000012_0 is done. And is in the process of committing
16/08/17 21:07:30 INFO reduce.LocalFetcher: localfetcher#107 about to shuffle output of map attempt_local518136789_0007_m_000009_0 decomp: 296774 len: 296778 to MEMORY
16/08/17 21:07:30 INFO reduce.InMemoryMapOutput: Read 296774 bytes from map-output for attempt_local518136789_0007_m_000009_0
16/08/17 21:07:30 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:30 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 296774, inMemoryMapOutputs.size() -> 14, commitMemory -> 3893320, usedMemory ->4190094
16/08/17 21:07:30 INFO mapred.Task: Task attempt_local1999544224_0005_r_000012_0 is allowed to commit now
16/08/17 21:07:30 INFO output.FileOutputCommitter: Saved output of task 'attempt_local1999544224_0005_r_000012_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/f_given_e_phrase/_temporary/0/task_local1999544224_0005_r_000012
16/08/17 21:07:30 INFO reduce.LocalFetcher: localfetcher#107 about to shuffle output of map attempt_local518136789_0007_m_000015_0 decomp: 298408 len: 298412 to MEMORY
16/08/17 21:07:30 INFO reduce.InMemoryMapOutput: Read 298408 bytes from map-output for attempt_local518136789_0007_m_000015_0
16/08/17 21:07:30 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 298408, inMemoryMapOutputs.size() -> 15, commitMemory -> 4190094, usedMemory ->4488502
16/08/17 21:07:30 INFO reduce.LocalFetcher: localfetcher#107 about to shuffle output of map attempt_local518136789_0007_m_000002_0 decomp: 308718 len: 308722 to MEMORY
16/08/17 21:07:30 INFO reduce.InMemoryMapOutput: Read 308718 bytes from map-output for attempt_local518136789_0007_m_000002_0
16/08/17 21:07:30 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 308718, inMemoryMapOutputs.size() -> 16, commitMemory -> 4488502, usedMemory ->4797220
16/08/17 21:07:30 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:07:30 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:07:30 INFO mapred.Task: Task 'attempt_local1999544224_0005_r_000012_0' done.
16/08/17 21:07:30 INFO mapred.LocalJobRunner: Finishing task: attempt_local1999544224_0005_r_000012_0
16/08/17 21:07:30 INFO mapred.LocalJobRunner: Starting task: attempt_local1999544224_0005_r_000013_0
16/08/17 21:07:30 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:30 INFO reduce.MergeManagerImpl: finalMerge called with 16 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:07:30 INFO mapred.Merger: Merging 16 sorted segments
16/08/17 21:07:30 INFO mapred.Merger: Down to the last merge-pass, with 16 segments left of total size: 4796848 bytes
16/08/17 21:07:30 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:30 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@5fa119f5
16/08/17 21:07:30 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=360762560, maxSingleShuffleLimit=90190640, mergeThreshold=238103296, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:07:30 INFO reduce.EventFetcher: attempt_local1999544224_0005_r_000013_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:07:30 INFO reduce.LocalFetcher: localfetcher#108 about to shuffle output of map attempt_local1999544224_0005_m_000004_0 decomp: 311586 len: 311590 to MEMORY
16/08/17 21:07:30 INFO reduce.InMemoryMapOutput: Read 311586 bytes from map-output for attempt_local1999544224_0005_m_000004_0
16/08/17 21:07:30 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 311586, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->311586
16/08/17 21:07:30 INFO reduce.LocalFetcher: localfetcher#108 about to shuffle output of map attempt_local1999544224_0005_m_000010_0 decomp: 310596 len: 310600 to MEMORY
16/08/17 21:07:30 INFO reduce.InMemoryMapOutput: Read 310596 bytes from map-output for attempt_local1999544224_0005_m_000010_0
16/08/17 21:07:30 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 310596, inMemoryMapOutputs.size() -> 2, commitMemory -> 311586, usedMemory ->622182
16/08/17 21:07:30 INFO reduce.LocalFetcher: localfetcher#108 about to shuffle output of map attempt_local1999544224_0005_m_000003_0 decomp: 299110 len: 299114 to MEMORY
16/08/17 21:07:30 INFO reduce.InMemoryMapOutput: Read 299110 bytes from map-output for attempt_local1999544224_0005_m_000003_0
16/08/17 21:07:30 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 299110, inMemoryMapOutputs.size() -> 3, commitMemory -> 622182, usedMemory ->921292
16/08/17 21:07:30 INFO reduce.LocalFetcher: localfetcher#108 about to shuffle output of map attempt_local1999544224_0005_m_000012_0 decomp: 300294 len: 300298 to MEMORY
16/08/17 21:07:30 INFO reduce.InMemoryMapOutput: Read 300294 bytes from map-output for attempt_local1999544224_0005_m_000012_0
16/08/17 21:07:30 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 300294, inMemoryMapOutputs.size() -> 4, commitMemory -> 921292, usedMemory ->1221586
16/08/17 21:07:30 INFO reduce.LocalFetcher: localfetcher#108 about to shuffle output of map attempt_local1999544224_0005_m_000005_0 decomp: 300228 len: 300232 to MEMORY
16/08/17 21:07:30 INFO reduce.InMemoryMapOutput: Read 300228 bytes from map-output for attempt_local1999544224_0005_m_000005_0
16/08/17 21:07:30 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 300228, inMemoryMapOutputs.size() -> 5, commitMemory -> 1221586, usedMemory ->1521814
16/08/17 21:07:30 INFO reduce.LocalFetcher: localfetcher#108 about to shuffle output of map attempt_local1999544224_0005_m_000011_0 decomp: 306306 len: 306310 to MEMORY
16/08/17 21:07:30 INFO reduce.InMemoryMapOutput: Read 306306 bytes from map-output for attempt_local1999544224_0005_m_000011_0
16/08/17 21:07:30 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 306306, inMemoryMapOutputs.size() -> 6, commitMemory -> 1521814, usedMemory ->1828120
16/08/17 21:07:30 INFO reduce.LocalFetcher: localfetcher#108 about to shuffle output of map attempt_local1999544224_0005_m_000001_0 decomp: 301304 len: 301308 to MEMORY
16/08/17 21:07:30 INFO reduce.InMemoryMapOutput: Read 301304 bytes from map-output for attempt_local1999544224_0005_m_000001_0
16/08/17 21:07:30 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 301304, inMemoryMapOutputs.size() -> 7, commitMemory -> 1828120, usedMemory ->2129424
16/08/17 21:07:30 INFO reduce.LocalFetcher: localfetcher#108 about to shuffle output of map attempt_local1999544224_0005_m_000007_0 decomp: 312490 len: 312494 to MEMORY
16/08/17 21:07:30 INFO reduce.InMemoryMapOutput: Read 312490 bytes from map-output for attempt_local1999544224_0005_m_000007_0
16/08/17 21:07:30 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 312490, inMemoryMapOutputs.size() -> 8, commitMemory -> 2129424, usedMemory ->2441914
16/08/17 21:07:30 INFO reduce.LocalFetcher: localfetcher#108 about to shuffle output of map attempt_local1999544224_0005_m_000013_0 decomp: 300886 len: 300890 to MEMORY
16/08/17 21:07:30 INFO reduce.InMemoryMapOutput: Read 300886 bytes from map-output for attempt_local1999544224_0005_m_000013_0
16/08/17 21:07:30 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 300886, inMemoryMapOutputs.size() -> 9, commitMemory -> 2441914, usedMemory ->2742800
16/08/17 21:07:30 INFO reduce.LocalFetcher: localfetcher#108 about to shuffle output of map attempt_local1999544224_0005_m_000000_0 decomp: 307880 len: 307884 to MEMORY
16/08/17 21:07:30 INFO reduce.InMemoryMapOutput: Read 307880 bytes from map-output for attempt_local1999544224_0005_m_000000_0
16/08/17 21:07:30 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 307880, inMemoryMapOutputs.size() -> 10, commitMemory -> 2742800, usedMemory ->3050680
16/08/17 21:07:30 INFO reduce.LocalFetcher: localfetcher#108 about to shuffle output of map attempt_local1999544224_0005_m_000006_0 decomp: 311754 len: 311758 to MEMORY
16/08/17 21:07:30 INFO reduce.InMemoryMapOutput: Read 311754 bytes from map-output for attempt_local1999544224_0005_m_000006_0
16/08/17 21:07:30 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 311754, inMemoryMapOutputs.size() -> 11, commitMemory -> 3050680, usedMemory ->3362434
16/08/17 21:07:30 INFO reduce.LocalFetcher: localfetcher#108 about to shuffle output of map attempt_local1999544224_0005_m_000009_0 decomp: 300038 len: 300042 to MEMORY
16/08/17 21:07:30 INFO reduce.InMemoryMapOutput: Read 300038 bytes from map-output for attempt_local1999544224_0005_m_000009_0
16/08/17 21:07:30 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 300038, inMemoryMapOutputs.size() -> 12, commitMemory -> 3362434, usedMemory ->3662472
16/08/17 21:07:30 INFO reduce.LocalFetcher: localfetcher#108 about to shuffle output of map attempt_local1999544224_0005_m_000015_0 decomp: 289714 len: 289718 to MEMORY
16/08/17 21:07:30 INFO reduce.InMemoryMapOutput: Read 289714 bytes from map-output for attempt_local1999544224_0005_m_000015_0
16/08/17 21:07:30 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 289714, inMemoryMapOutputs.size() -> 13, commitMemory -> 3662472, usedMemory ->3952186
16/08/17 21:07:30 INFO reduce.LocalFetcher: localfetcher#108 about to shuffle output of map attempt_local1999544224_0005_m_000002_0 decomp: 326682 len: 326686 to MEMORY
16/08/17 21:07:30 INFO reduce.InMemoryMapOutput: Read 326682 bytes from map-output for attempt_local1999544224_0005_m_000002_0
16/08/17 21:07:30 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 326682, inMemoryMapOutputs.size() -> 14, commitMemory -> 3952186, usedMemory ->4278868
16/08/17 21:07:30 INFO reduce.LocalFetcher: localfetcher#108 about to shuffle output of map attempt_local1999544224_0005_m_000008_0 decomp: 295362 len: 295366 to MEMORY
16/08/17 21:07:30 INFO reduce.InMemoryMapOutput: Read 295362 bytes from map-output for attempt_local1999544224_0005_m_000008_0
16/08/17 21:07:30 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 295362, inMemoryMapOutputs.size() -> 15, commitMemory -> 4278868, usedMemory ->4574230
16/08/17 21:07:30 INFO reduce.LocalFetcher: localfetcher#108 about to shuffle output of map attempt_local1999544224_0005_m_000014_0 decomp: 304038 len: 304042 to MEMORY
16/08/17 21:07:30 INFO reduce.InMemoryMapOutput: Read 304038 bytes from map-output for attempt_local1999544224_0005_m_000014_0
16/08/17 21:07:30 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 304038, inMemoryMapOutputs.size() -> 16, commitMemory -> 4574230, usedMemory ->4878268
16/08/17 21:07:30 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:07:30 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:30 INFO reduce.MergeManagerImpl: finalMerge called with 16 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:07:30 INFO mapred.Merger: Merging 16 sorted segments
16/08/17 21:07:30 INFO mapred.Merger: Down to the last merge-pass, with 16 segments left of total size: 4877908 bytes
16/08/17 21:07:30 INFO reduce.MergeManagerImpl: Merged 16 segments, 4797220 bytes to disk to satisfy reduce memory limit
16/08/17 21:07:30 INFO reduce.MergeManagerImpl: Merging 1 files, 4797194 bytes from disk
16/08/17 21:07:30 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:07:30 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:07:30 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 4797164 bytes
16/08/17 21:07:30 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:30 INFO reduce.MergeManagerImpl: Merged 16 segments, 4878268 bytes to disk to satisfy reduce memory limit
16/08/17 21:07:30 INFO reduce.MergeManagerImpl: Merging 1 files, 4878242 bytes from disk
16/08/17 21:07:30 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:07:30 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:07:30 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 4878220 bytes
16/08/17 21:07:30 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:31 INFO mapred.Task: Task:attempt_local518136789_0007_r_000013_0 is done. And is in the process of committing
16/08/17 21:07:31 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:31 INFO mapred.Task: Task attempt_local518136789_0007_r_000013_0 is allowed to commit now
16/08/17 21:07:31 INFO output.FileOutputCommitter: Saved output of task 'attempt_local518136789_0007_r_000013_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/e_given_f_phrase/_temporary/0/task_local518136789_0007_r_000013
16/08/17 21:07:31 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:07:31 INFO mapred.Task: Task 'attempt_local518136789_0007_r_000013_0' done.
16/08/17 21:07:31 INFO mapred.LocalJobRunner: Finishing task: attempt_local518136789_0007_r_000013_0
16/08/17 21:07:31 INFO mapred.LocalJobRunner: Starting task: attempt_local518136789_0007_r_000014_0
16/08/17 21:07:31 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:31 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@48f60c7c
16/08/17 21:07:31 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=366634592, maxSingleShuffleLimit=91658648, mergeThreshold=241978848, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:07:31 INFO reduce.EventFetcher: attempt_local518136789_0007_r_000014_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:07:31 INFO reduce.LocalFetcher: localfetcher#109 about to shuffle output of map attempt_local518136789_0007_m_000011_0 decomp: 290458 len: 290462 to MEMORY
16/08/17 21:07:31 INFO reduce.InMemoryMapOutput: Read 290458 bytes from map-output for attempt_local518136789_0007_m_000011_0
16/08/17 21:07:31 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 290458, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->290458
16/08/17 21:07:31 INFO reduce.LocalFetcher: localfetcher#109 about to shuffle output of map attempt_local518136789_0007_m_000004_0 decomp: 315002 len: 315006 to MEMORY
16/08/17 21:07:31 INFO reduce.InMemoryMapOutput: Read 315002 bytes from map-output for attempt_local518136789_0007_m_000004_0
16/08/17 21:07:31 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 315002, inMemoryMapOutputs.size() -> 2, commitMemory -> 290458, usedMemory ->605460
16/08/17 21:07:31 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)
	at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)
	at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
16/08/17 21:07:31 INFO reduce.LocalFetcher: localfetcher#109 about to shuffle output of map attempt_local518136789_0007_m_000010_0 decomp: 300714 len: 300718 to MEMORY
16/08/17 21:07:31 INFO reduce.InMemoryMapOutput: Read 300714 bytes from map-output for attempt_local518136789_0007_m_000010_0
16/08/17 21:07:31 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 300714, inMemoryMapOutputs.size() -> 3, commitMemory -> 605460, usedMemory ->906174
16/08/17 21:07:31 INFO reduce.LocalFetcher: localfetcher#109 about to shuffle output of map attempt_local518136789_0007_m_000000_0 decomp: 311150 len: 311154 to MEMORY
16/08/17 21:07:31 INFO reduce.InMemoryMapOutput: Read 311150 bytes from map-output for attempt_local518136789_0007_m_000000_0
16/08/17 21:07:31 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 311150, inMemoryMapOutputs.size() -> 4, commitMemory -> 906174, usedMemory ->1217324
16/08/17 21:07:31 INFO reduce.LocalFetcher: localfetcher#109 about to shuffle output of map attempt_local518136789_0007_m_000006_0 decomp: 295842 len: 295846 to MEMORY
16/08/17 21:07:31 INFO reduce.InMemoryMapOutput: Read 295842 bytes from map-output for attempt_local518136789_0007_m_000006_0
16/08/17 21:07:31 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 295842, inMemoryMapOutputs.size() -> 5, commitMemory -> 1217324, usedMemory ->1513166
16/08/17 21:07:31 INFO reduce.LocalFetcher: localfetcher#109 about to shuffle output of map attempt_local518136789_0007_m_000012_0 decomp: 304742 len: 304746 to MEMORY
16/08/17 21:07:31 INFO reduce.InMemoryMapOutput: Read 304742 bytes from map-output for attempt_local518136789_0007_m_000012_0
16/08/17 21:07:31 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 304742, inMemoryMapOutputs.size() -> 6, commitMemory -> 1513166, usedMemory ->1817908
16/08/17 21:07:31 INFO reduce.LocalFetcher: localfetcher#109 about to shuffle output of map attempt_local518136789_0007_m_000005_0 decomp: 300442 len: 300446 to MEMORY
16/08/17 21:07:31 INFO reduce.InMemoryMapOutput: Read 300442 bytes from map-output for attempt_local518136789_0007_m_000005_0
16/08/17 21:07:31 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 300442, inMemoryMapOutputs.size() -> 7, commitMemory -> 1817908, usedMemory ->2118350
16/08/17 21:07:31 INFO reduce.LocalFetcher: localfetcher#109 about to shuffle output of map attempt_local518136789_0007_m_000008_0 decomp: 289490 len: 289494 to MEMORY
16/08/17 21:07:31 INFO reduce.InMemoryMapOutput: Read 289490 bytes from map-output for attempt_local518136789_0007_m_000008_0
16/08/17 21:07:31 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 289490, inMemoryMapOutputs.size() -> 8, commitMemory -> 2118350, usedMemory ->2407840
16/08/17 21:07:31 INFO reduce.LocalFetcher: localfetcher#109 about to shuffle output of map attempt_local518136789_0007_m_000014_0 decomp: 304214 len: 304218 to MEMORY
16/08/17 21:07:31 INFO reduce.InMemoryMapOutput: Read 304214 bytes from map-output for attempt_local518136789_0007_m_000014_0
16/08/17 21:07:31 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 304214, inMemoryMapOutputs.size() -> 9, commitMemory -> 2407840, usedMemory ->2712054
16/08/17 21:07:31 INFO reduce.LocalFetcher: localfetcher#109 about to shuffle output of map attempt_local518136789_0007_m_000001_0 decomp: 313600 len: 313604 to MEMORY
16/08/17 21:07:31 INFO reduce.InMemoryMapOutput: Read 313600 bytes from map-output for attempt_local518136789_0007_m_000001_0
16/08/17 21:07:31 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 313600, inMemoryMapOutputs.size() -> 10, commitMemory -> 2712054, usedMemory ->3025654
16/08/17 21:07:31 INFO reduce.LocalFetcher: localfetcher#109 about to shuffle output of map attempt_local518136789_0007_m_000007_0 decomp: 292046 len: 292050 to MEMORY
16/08/17 21:07:31 INFO reduce.InMemoryMapOutput: Read 292046 bytes from map-output for attempt_local518136789_0007_m_000007_0
16/08/17 21:07:31 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 292046, inMemoryMapOutputs.size() -> 11, commitMemory -> 3025654, usedMemory ->3317700
16/08/17 21:07:31 INFO reduce.LocalFetcher: localfetcher#109 about to shuffle output of map attempt_local518136789_0007_m_000013_0 decomp: 298422 len: 298426 to MEMORY
16/08/17 21:07:31 INFO reduce.InMemoryMapOutput: Read 298422 bytes from map-output for attempt_local518136789_0007_m_000013_0
16/08/17 21:07:31 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 298422, inMemoryMapOutputs.size() -> 12, commitMemory -> 3317700, usedMemory ->3616122
16/08/17 21:07:31 INFO reduce.LocalFetcher: localfetcher#109 about to shuffle output of map attempt_local518136789_0007_m_000003_0 decomp: 332008 len: 332012 to MEMORY
16/08/17 21:07:31 INFO reduce.InMemoryMapOutput: Read 332008 bytes from map-output for attempt_local518136789_0007_m_000003_0
16/08/17 21:07:31 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 332008, inMemoryMapOutputs.size() -> 13, commitMemory -> 3616122, usedMemory ->3948130
16/08/17 21:07:31 INFO reduce.LocalFetcher: localfetcher#109 about to shuffle output of map attempt_local518136789_0007_m_000009_0 decomp: 300814 len: 300818 to MEMORY
16/08/17 21:07:31 INFO reduce.InMemoryMapOutput: Read 300814 bytes from map-output for attempt_local518136789_0007_m_000009_0
16/08/17 21:07:31 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 300814, inMemoryMapOutputs.size() -> 14, commitMemory -> 3948130, usedMemory ->4248944
16/08/17 21:07:31 INFO reduce.LocalFetcher: localfetcher#109 about to shuffle output of map attempt_local518136789_0007_m_000015_0 decomp: 297502 len: 297506 to MEMORY
16/08/17 21:07:31 INFO reduce.InMemoryMapOutput: Read 297502 bytes from map-output for attempt_local518136789_0007_m_000015_0
16/08/17 21:07:31 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 297502, inMemoryMapOutputs.size() -> 15, commitMemory -> 4248944, usedMemory ->4546446
16/08/17 21:07:31 INFO reduce.LocalFetcher: localfetcher#109 about to shuffle output of map attempt_local518136789_0007_m_000002_0 decomp: 304912 len: 304916 to MEMORY
16/08/17 21:07:31 INFO reduce.InMemoryMapOutput: Read 304912 bytes from map-output for attempt_local518136789_0007_m_000002_0
16/08/17 21:07:31 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 304912, inMemoryMapOutputs.size() -> 16, commitMemory -> 4546446, usedMemory ->4851358
16/08/17 21:07:31 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:07:31 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:31 INFO reduce.MergeManagerImpl: finalMerge called with 16 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:07:31 INFO mapred.Merger: Merging 16 sorted segments
16/08/17 21:07:31 INFO mapred.Merger: Down to the last merge-pass, with 16 segments left of total size: 4851022 bytes
16/08/17 21:07:31 INFO mapred.Task: Task:attempt_local1999544224_0005_r_000013_0 is done. And is in the process of committing
16/08/17 21:07:31 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:31 INFO mapred.Task: Task attempt_local1999544224_0005_r_000013_0 is allowed to commit now
16/08/17 21:07:31 INFO output.FileOutputCommitter: Saved output of task 'attempt_local1999544224_0005_r_000013_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/f_given_e_phrase/_temporary/0/task_local1999544224_0005_r_000013
16/08/17 21:07:31 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:07:31 INFO mapred.Task: Task 'attempt_local1999544224_0005_r_000013_0' done.
16/08/17 21:07:31 INFO mapred.LocalJobRunner: Finishing task: attempt_local1999544224_0005_r_000013_0
16/08/17 21:07:31 INFO mapred.LocalJobRunner: Starting task: attempt_local1999544224_0005_r_000014_0
16/08/17 21:07:31 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:31 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@12f8b5df
16/08/17 21:07:31 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=366634592, maxSingleShuffleLimit=91658648, mergeThreshold=241978848, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:07:31 INFO reduce.EventFetcher: attempt_local1999544224_0005_r_000014_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:07:31 INFO reduce.LocalFetcher: localfetcher#110 about to shuffle output of map attempt_local1999544224_0005_m_000004_0 decomp: 316912 len: 316916 to MEMORY
16/08/17 21:07:31 INFO reduce.InMemoryMapOutput: Read 316912 bytes from map-output for attempt_local1999544224_0005_m_000004_0
16/08/17 21:07:31 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 316912, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->316912
16/08/17 21:07:31 INFO reduce.LocalFetcher: localfetcher#110 about to shuffle output of map attempt_local1999544224_0005_m_000010_0 decomp: 309696 len: 309700 to MEMORY
16/08/17 21:07:31 INFO reduce.InMemoryMapOutput: Read 309696 bytes from map-output for attempt_local1999544224_0005_m_000010_0
16/08/17 21:07:31 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 309696, inMemoryMapOutputs.size() -> 2, commitMemory -> 316912, usedMemory ->626608
16/08/17 21:07:31 INFO reduce.LocalFetcher: localfetcher#110 about to shuffle output of map attempt_local1999544224_0005_m_000003_0 decomp: 341072 len: 341076 to MEMORY
16/08/17 21:07:31 INFO reduce.InMemoryMapOutput: Read 341072 bytes from map-output for attempt_local1999544224_0005_m_000003_0
16/08/17 21:07:31 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 341072, inMemoryMapOutputs.size() -> 3, commitMemory -> 626608, usedMemory ->967680
16/08/17 21:07:31 INFO reduce.LocalFetcher: localfetcher#110 about to shuffle output of map attempt_local1999544224_0005_m_000012_0 decomp: 307304 len: 307308 to MEMORY
16/08/17 21:07:31 INFO reduce.InMemoryMapOutput: Read 307304 bytes from map-output for attempt_local1999544224_0005_m_000012_0
16/08/17 21:07:31 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 307304, inMemoryMapOutputs.size() -> 4, commitMemory -> 967680, usedMemory ->1274984
16/08/17 21:07:31 INFO reduce.LocalFetcher: localfetcher#110 about to shuffle output of map attempt_local1999544224_0005_m_000005_0 decomp: 306354 len: 306358 to MEMORY
16/08/17 21:07:31 INFO reduce.InMemoryMapOutput: Read 306354 bytes from map-output for attempt_local1999544224_0005_m_000005_0
16/08/17 21:07:31 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 306354, inMemoryMapOutputs.size() -> 5, commitMemory -> 1274984, usedMemory ->1581338
16/08/17 21:07:31 INFO reduce.LocalFetcher: localfetcher#110 about to shuffle output of map attempt_local1999544224_0005_m_000011_0 decomp: 305506 len: 305510 to MEMORY
16/08/17 21:07:31 INFO reduce.InMemoryMapOutput: Read 305506 bytes from map-output for attempt_local1999544224_0005_m_000011_0
16/08/17 21:07:31 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 305506, inMemoryMapOutputs.size() -> 6, commitMemory -> 1581338, usedMemory ->1886844
16/08/17 21:07:31 INFO reduce.LocalFetcher: localfetcher#110 about to shuffle output of map attempt_local1999544224_0005_m_000001_0 decomp: 310114 len: 310118 to MEMORY
16/08/17 21:07:31 INFO reduce.InMemoryMapOutput: Read 310114 bytes from map-output for attempt_local1999544224_0005_m_000001_0
16/08/17 21:07:31 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 310114, inMemoryMapOutputs.size() -> 7, commitMemory -> 1886844, usedMemory ->2196958
16/08/17 21:07:31 INFO reduce.LocalFetcher: localfetcher#110 about to shuffle output of map attempt_local1999544224_0005_m_000007_0 decomp: 300050 len: 300054 to MEMORY
16/08/17 21:07:31 INFO reduce.InMemoryMapOutput: Read 300050 bytes from map-output for attempt_local1999544224_0005_m_000007_0
16/08/17 21:07:31 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 300050, inMemoryMapOutputs.size() -> 8, commitMemory -> 2196958, usedMemory ->2497008
16/08/17 21:07:31 INFO reduce.LocalFetcher: localfetcher#110 about to shuffle output of map attempt_local1999544224_0005_m_000013_0 decomp: 306616 len: 306620 to MEMORY
16/08/17 21:07:31 INFO reduce.InMemoryMapOutput: Read 306616 bytes from map-output for attempt_local1999544224_0005_m_000013_0
16/08/17 21:07:31 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 306616, inMemoryMapOutputs.size() -> 9, commitMemory -> 2497008, usedMemory ->2803624
16/08/17 21:07:31 INFO reduce.LocalFetcher: localfetcher#110 about to shuffle output of map attempt_local1999544224_0005_m_000000_0 decomp: 302712 len: 302716 to MEMORY
16/08/17 21:07:31 INFO reduce.InMemoryMapOutput: Read 302712 bytes from map-output for attempt_local1999544224_0005_m_000000_0
16/08/17 21:07:31 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 302712, inMemoryMapOutputs.size() -> 10, commitMemory -> 2803624, usedMemory ->3106336
16/08/17 21:07:31 INFO reduce.LocalFetcher: localfetcher#110 about to shuffle output of map attempt_local1999544224_0005_m_000006_0 decomp: 326418 len: 326422 to MEMORY
16/08/17 21:07:31 INFO reduce.InMemoryMapOutput: Read 326418 bytes from map-output for attempt_local1999544224_0005_m_000006_0
16/08/17 21:07:31 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 326418, inMemoryMapOutputs.size() -> 11, commitMemory -> 3106336, usedMemory ->3432754
16/08/17 21:07:31 INFO reduce.LocalFetcher: localfetcher#110 about to shuffle output of map attempt_local1999544224_0005_m_000009_0 decomp: 306330 len: 306334 to MEMORY
16/08/17 21:07:31 INFO reduce.InMemoryMapOutput: Read 306330 bytes from map-output for attempt_local1999544224_0005_m_000009_0
16/08/17 21:07:31 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 306330, inMemoryMapOutputs.size() -> 12, commitMemory -> 3432754, usedMemory ->3739084
16/08/17 21:07:31 INFO reduce.LocalFetcher: localfetcher#110 about to shuffle output of map attempt_local1999544224_0005_m_000015_0 decomp: 290988 len: 290992 to MEMORY
16/08/17 21:07:31 INFO reduce.InMemoryMapOutput: Read 290988 bytes from map-output for attempt_local1999544224_0005_m_000015_0
16/08/17 21:07:31 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 290988, inMemoryMapOutputs.size() -> 13, commitMemory -> 3739084, usedMemory ->4030072
16/08/17 21:07:31 INFO reduce.LocalFetcher: localfetcher#110 about to shuffle output of map attempt_local1999544224_0005_m_000002_0 decomp: 313864 len: 313868 to MEMORY
16/08/17 21:07:31 INFO reduce.InMemoryMapOutput: Read 313864 bytes from map-output for attempt_local1999544224_0005_m_000002_0
16/08/17 21:07:31 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 313864, inMemoryMapOutputs.size() -> 14, commitMemory -> 4030072, usedMemory ->4343936
16/08/17 21:07:31 INFO reduce.LocalFetcher: localfetcher#110 about to shuffle output of map attempt_local1999544224_0005_m_000008_0 decomp: 309542 len: 309546 to MEMORY
16/08/17 21:07:31 INFO reduce.InMemoryMapOutput: Read 309542 bytes from map-output for attempt_local1999544224_0005_m_000008_0
16/08/17 21:07:31 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 309542, inMemoryMapOutputs.size() -> 15, commitMemory -> 4343936, usedMemory ->4653478
16/08/17 21:07:31 INFO reduce.LocalFetcher: localfetcher#110 about to shuffle output of map attempt_local1999544224_0005_m_000014_0 decomp: 308690 len: 308694 to MEMORY
16/08/17 21:07:31 INFO reduce.InMemoryMapOutput: Read 308690 bytes from map-output for attempt_local1999544224_0005_m_000014_0
16/08/17 21:07:31 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 308690, inMemoryMapOutputs.size() -> 16, commitMemory -> 4653478, usedMemory ->4962168
16/08/17 21:07:31 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:07:31 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:31 INFO reduce.MergeManagerImpl: finalMerge called with 16 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:07:31 INFO mapred.Merger: Merging 16 sorted segments
16/08/17 21:07:31 INFO mapred.Merger: Down to the last merge-pass, with 16 segments left of total size: 4961788 bytes
16/08/17 21:07:31 INFO reduce.MergeManagerImpl: Merged 16 segments, 4851358 bytes to disk to satisfy reduce memory limit
16/08/17 21:07:31 INFO reduce.MergeManagerImpl: Merging 1 files, 4851332 bytes from disk
16/08/17 21:07:31 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:07:31 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:07:31 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 4851306 bytes
16/08/17 21:07:31 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:31 INFO reduce.MergeManagerImpl: Merged 16 segments, 4962168 bytes to disk to satisfy reduce memory limit
16/08/17 21:07:31 INFO reduce.MergeManagerImpl: Merging 1 files, 4962142 bytes from disk
16/08/17 21:07:31 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:07:31 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:07:31 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 4962100 bytes
16/08/17 21:07:31 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:31 INFO mapred.Task: Task:attempt_local518136789_0007_r_000014_0 is done. And is in the process of committing
16/08/17 21:07:31 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:31 INFO mapred.Task: Task attempt_local518136789_0007_r_000014_0 is allowed to commit now
16/08/17 21:07:31 INFO output.FileOutputCommitter: Saved output of task 'attempt_local518136789_0007_r_000014_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/e_given_f_phrase/_temporary/0/task_local518136789_0007_r_000014
16/08/17 21:07:31 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:07:31 INFO mapred.Task: Task 'attempt_local518136789_0007_r_000014_0' done.
16/08/17 21:07:31 INFO mapred.LocalJobRunner: Finishing task: attempt_local518136789_0007_r_000014_0
16/08/17 21:07:31 INFO mapred.LocalJobRunner: Starting task: attempt_local518136789_0007_r_000015_0
16/08/17 21:07:31 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:31 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@60116a04
16/08/17 21:07:31 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=366634592, maxSingleShuffleLimit=91658648, mergeThreshold=241978848, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:07:31 INFO reduce.EventFetcher: attempt_local518136789_0007_r_000015_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:07:31 INFO reduce.LocalFetcher: localfetcher#111 about to shuffle output of map attempt_local518136789_0007_m_000011_0 decomp: 298858 len: 298862 to MEMORY
16/08/17 21:07:31 INFO reduce.InMemoryMapOutput: Read 298858 bytes from map-output for attempt_local518136789_0007_m_000011_0
16/08/17 21:07:31 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 298858, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->298858
16/08/17 21:07:31 INFO reduce.LocalFetcher: localfetcher#111 about to shuffle output of map attempt_local518136789_0007_m_000004_0 decomp: 301800 len: 301804 to MEMORY
16/08/17 21:07:31 INFO reduce.InMemoryMapOutput: Read 301800 bytes from map-output for attempt_local518136789_0007_m_000004_0
16/08/17 21:07:31 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 301800, inMemoryMapOutputs.size() -> 2, commitMemory -> 298858, usedMemory ->600658
16/08/17 21:07:31 INFO reduce.LocalFetcher: localfetcher#111 about to shuffle output of map attempt_local518136789_0007_m_000010_0 decomp: 286414 len: 286418 to MEMORY
16/08/17 21:07:31 INFO reduce.InMemoryMapOutput: Read 286414 bytes from map-output for attempt_local518136789_0007_m_000010_0
16/08/17 21:07:31 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 286414, inMemoryMapOutputs.size() -> 3, commitMemory -> 600658, usedMemory ->887072
16/08/17 21:07:31 INFO reduce.LocalFetcher: localfetcher#111 about to shuffle output of map attempt_local518136789_0007_m_000000_0 decomp: 312050 len: 312054 to MEMORY
16/08/17 21:07:31 INFO reduce.InMemoryMapOutput: Read 312050 bytes from map-output for attempt_local518136789_0007_m_000000_0
16/08/17 21:07:31 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 312050, inMemoryMapOutputs.size() -> 4, commitMemory -> 887072, usedMemory ->1199122
16/08/17 21:07:31 INFO reduce.LocalFetcher: localfetcher#111 about to shuffle output of map attempt_local518136789_0007_m_000006_0 decomp: 304098 len: 304102 to MEMORY
16/08/17 21:07:31 INFO reduce.InMemoryMapOutput: Read 304098 bytes from map-output for attempt_local518136789_0007_m_000006_0
16/08/17 21:07:31 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 304098, inMemoryMapOutputs.size() -> 5, commitMemory -> 1199122, usedMemory ->1503220
16/08/17 21:07:31 INFO reduce.LocalFetcher: localfetcher#111 about to shuffle output of map attempt_local518136789_0007_m_000012_0 decomp: 288530 len: 288534 to MEMORY
16/08/17 21:07:31 INFO reduce.InMemoryMapOutput: Read 288530 bytes from map-output for attempt_local518136789_0007_m_000012_0
16/08/17 21:07:31 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 288530, inMemoryMapOutputs.size() -> 6, commitMemory -> 1503220, usedMemory ->1791750
16/08/17 21:07:31 INFO reduce.LocalFetcher: localfetcher#111 about to shuffle output of map attempt_local518136789_0007_m_000005_0 decomp: 309102 len: 309106 to MEMORY
16/08/17 21:07:31 INFO reduce.InMemoryMapOutput: Read 309102 bytes from map-output for attempt_local518136789_0007_m_000005_0
16/08/17 21:07:31 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 309102, inMemoryMapOutputs.size() -> 7, commitMemory -> 1791750, usedMemory ->2100852
16/08/17 21:07:31 INFO reduce.LocalFetcher: localfetcher#111 about to shuffle output of map attempt_local518136789_0007_m_000008_0 decomp: 296694 len: 296698 to MEMORY
16/08/17 21:07:31 INFO reduce.InMemoryMapOutput: Read 296694 bytes from map-output for attempt_local518136789_0007_m_000008_0
16/08/17 21:07:31 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 296694, inMemoryMapOutputs.size() -> 8, commitMemory -> 2100852, usedMemory ->2397546
16/08/17 21:07:31 INFO reduce.LocalFetcher: localfetcher#111 about to shuffle output of map attempt_local518136789_0007_m_000014_0 decomp: 294744 len: 294748 to MEMORY
16/08/17 21:07:31 INFO reduce.InMemoryMapOutput: Read 294744 bytes from map-output for attempt_local518136789_0007_m_000014_0
16/08/17 21:07:31 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 294744, inMemoryMapOutputs.size() -> 9, commitMemory -> 2397546, usedMemory ->2692290
16/08/17 21:07:31 INFO reduce.LocalFetcher: localfetcher#111 about to shuffle output of map attempt_local518136789_0007_m_000001_0 decomp: 315562 len: 315566 to MEMORY
16/08/17 21:07:31 INFO reduce.InMemoryMapOutput: Read 315562 bytes from map-output for attempt_local518136789_0007_m_000001_0
16/08/17 21:07:31 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 315562, inMemoryMapOutputs.size() -> 10, commitMemory -> 2692290, usedMemory ->3007852
16/08/17 21:07:31 INFO reduce.LocalFetcher: localfetcher#111 about to shuffle output of map attempt_local518136789_0007_m_000007_0 decomp: 306024 len: 306028 to MEMORY
16/08/17 21:07:31 INFO reduce.InMemoryMapOutput: Read 306024 bytes from map-output for attempt_local518136789_0007_m_000007_0
16/08/17 21:07:31 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 306024, inMemoryMapOutputs.size() -> 11, commitMemory -> 3007852, usedMemory ->3313876
16/08/17 21:07:31 INFO reduce.LocalFetcher: localfetcher#111 about to shuffle output of map attempt_local518136789_0007_m_000013_0 decomp: 291312 len: 291316 to MEMORY
16/08/17 21:07:31 INFO reduce.InMemoryMapOutput: Read 291312 bytes from map-output for attempt_local518136789_0007_m_000013_0
16/08/17 21:07:31 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 291312, inMemoryMapOutputs.size() -> 12, commitMemory -> 3313876, usedMemory ->3605188
16/08/17 21:07:31 INFO reduce.LocalFetcher: localfetcher#111 about to shuffle output of map attempt_local518136789_0007_m_000003_0 decomp: 307812 len: 307816 to MEMORY
16/08/17 21:07:31 INFO reduce.InMemoryMapOutput: Read 307812 bytes from map-output for attempt_local518136789_0007_m_000003_0
16/08/17 21:07:31 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 307812, inMemoryMapOutputs.size() -> 13, commitMemory -> 3605188, usedMemory ->3913000
16/08/17 21:07:31 INFO reduce.LocalFetcher: localfetcher#111 about to shuffle output of map attempt_local518136789_0007_m_000009_0 decomp: 289646 len: 289650 to MEMORY
16/08/17 21:07:31 INFO reduce.InMemoryMapOutput: Read 289646 bytes from map-output for attempt_local518136789_0007_m_000009_0
16/08/17 21:07:31 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 289646, inMemoryMapOutputs.size() -> 14, commitMemory -> 3913000, usedMemory ->4202646
16/08/17 21:07:31 INFO reduce.LocalFetcher: localfetcher#111 about to shuffle output of map attempt_local518136789_0007_m_000015_0 decomp: 292732 len: 292736 to MEMORY
16/08/17 21:07:31 INFO reduce.InMemoryMapOutput: Read 292732 bytes from map-output for attempt_local518136789_0007_m_000015_0
16/08/17 21:07:31 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 292732, inMemoryMapOutputs.size() -> 15, commitMemory -> 4202646, usedMemory ->4495378
16/08/17 21:07:31 INFO reduce.LocalFetcher: localfetcher#111 about to shuffle output of map attempt_local518136789_0007_m_000002_0 decomp: 299774 len: 299778 to MEMORY
16/08/17 21:07:31 INFO reduce.InMemoryMapOutput: Read 299774 bytes from map-output for attempt_local518136789_0007_m_000002_0
16/08/17 21:07:31 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 299774, inMemoryMapOutputs.size() -> 16, commitMemory -> 4495378, usedMemory ->4795152
16/08/17 21:07:31 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:07:31 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:31 INFO reduce.MergeManagerImpl: finalMerge called with 16 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:07:31 INFO mapred.Merger: Merging 16 sorted segments
16/08/17 21:07:31 INFO mapred.Merger: Down to the last merge-pass, with 16 segments left of total size: 4794772 bytes
16/08/17 21:07:31 INFO mapred.Task: Task:attempt_local1999544224_0005_r_000014_0 is done. And is in the process of committing
16/08/17 21:07:31 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:31 INFO mapred.Task: Task attempt_local1999544224_0005_r_000014_0 is allowed to commit now
16/08/17 21:07:31 INFO output.FileOutputCommitter: Saved output of task 'attempt_local1999544224_0005_r_000014_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/f_given_e_phrase/_temporary/0/task_local1999544224_0005_r_000014
16/08/17 21:07:31 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:07:31 INFO mapred.Task: Task 'attempt_local1999544224_0005_r_000014_0' done.
16/08/17 21:07:31 INFO mapred.LocalJobRunner: Finishing task: attempt_local1999544224_0005_r_000014_0
16/08/17 21:07:31 INFO mapred.LocalJobRunner: Starting task: attempt_local1999544224_0005_r_000015_0
16/08/17 21:07:31 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
16/08/17 21:07:31 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@1e82b2d6
16/08/17 21:07:31 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=366267584, maxSingleShuffleLimit=91566896, mergeThreshold=241736608, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/08/17 21:07:31 INFO reduce.EventFetcher: attempt_local1999544224_0005_r_000015_0 Thread started: EventFetcher for fetching Map Completion Events
16/08/17 21:07:31 INFO reduce.LocalFetcher: localfetcher#112 about to shuffle output of map attempt_local1999544224_0005_m_000004_0 decomp: 303132 len: 303136 to MEMORY
16/08/17 21:07:31 INFO reduce.InMemoryMapOutput: Read 303132 bytes from map-output for attempt_local1999544224_0005_m_000004_0
16/08/17 21:07:31 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 303132, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->303132
16/08/17 21:07:31 INFO reduce.LocalFetcher: localfetcher#112 about to shuffle output of map attempt_local1999544224_0005_m_000010_0 decomp: 291616 len: 291620 to MEMORY
16/08/17 21:07:31 INFO reduce.InMemoryMapOutput: Read 291616 bytes from map-output for attempt_local1999544224_0005_m_000010_0
16/08/17 21:07:31 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 291616, inMemoryMapOutputs.size() -> 2, commitMemory -> 303132, usedMemory ->594748
16/08/17 21:07:31 INFO reduce.LocalFetcher: localfetcher#112 about to shuffle output of map attempt_local1999544224_0005_m_000003_0 decomp: 294552 len: 294556 to MEMORY
16/08/17 21:07:31 INFO reduce.InMemoryMapOutput: Read 294552 bytes from map-output for attempt_local1999544224_0005_m_000003_0
16/08/17 21:07:31 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 294552, inMemoryMapOutputs.size() -> 3, commitMemory -> 594748, usedMemory ->889300
16/08/17 21:07:31 INFO reduce.LocalFetcher: localfetcher#112 about to shuffle output of map attempt_local1999544224_0005_m_000012_0 decomp: 300646 len: 300650 to MEMORY
16/08/17 21:07:31 INFO reduce.InMemoryMapOutput: Read 300646 bytes from map-output for attempt_local1999544224_0005_m_000012_0
16/08/17 21:07:31 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 300646, inMemoryMapOutputs.size() -> 4, commitMemory -> 889300, usedMemory ->1189946
16/08/17 21:07:31 INFO reduce.LocalFetcher: localfetcher#112 about to shuffle output of map attempt_local1999544224_0005_m_000005_0 decomp: 308226 len: 308230 to MEMORY
16/08/17 21:07:31 INFO reduce.InMemoryMapOutput: Read 308226 bytes from map-output for attempt_local1999544224_0005_m_000005_0
16/08/17 21:07:31 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 308226, inMemoryMapOutputs.size() -> 5, commitMemory -> 1189946, usedMemory ->1498172
16/08/17 21:07:31 INFO reduce.LocalFetcher: localfetcher#112 about to shuffle output of map attempt_local1999544224_0005_m_000011_0 decomp: 299996 len: 300000 to MEMORY
16/08/17 21:07:31 INFO reduce.InMemoryMapOutput: Read 299996 bytes from map-output for attempt_local1999544224_0005_m_000011_0
16/08/17 21:07:31 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 299996, inMemoryMapOutputs.size() -> 6, commitMemory -> 1498172, usedMemory ->1798168
16/08/17 21:07:31 INFO reduce.LocalFetcher: localfetcher#112 about to shuffle output of map attempt_local1999544224_0005_m_000001_0 decomp: 322876 len: 322880 to MEMORY
16/08/17 21:07:31 INFO reduce.InMemoryMapOutput: Read 322876 bytes from map-output for attempt_local1999544224_0005_m_000001_0
16/08/17 21:07:31 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 322876, inMemoryMapOutputs.size() -> 7, commitMemory -> 1798168, usedMemory ->2121044
16/08/17 21:07:31 INFO reduce.LocalFetcher: localfetcher#112 about to shuffle output of map attempt_local1999544224_0005_m_000007_0 decomp: 314540 len: 314544 to MEMORY
16/08/17 21:07:31 INFO reduce.InMemoryMapOutput: Read 314540 bytes from map-output for attempt_local1999544224_0005_m_000007_0
16/08/17 21:07:31 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 314540, inMemoryMapOutputs.size() -> 8, commitMemory -> 2121044, usedMemory ->2435584
16/08/17 21:07:31 INFO reduce.LocalFetcher: localfetcher#112 about to shuffle output of map attempt_local1999544224_0005_m_000013_0 decomp: 304690 len: 304694 to MEMORY
16/08/17 21:07:31 INFO reduce.InMemoryMapOutput: Read 304690 bytes from map-output for attempt_local1999544224_0005_m_000013_0
16/08/17 21:07:31 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 304690, inMemoryMapOutputs.size() -> 9, commitMemory -> 2435584, usedMemory ->2740274
16/08/17 21:07:31 INFO reduce.LocalFetcher: localfetcher#112 about to shuffle output of map attempt_local1999544224_0005_m_000000_0 decomp: 314272 len: 314276 to MEMORY
16/08/17 21:07:31 INFO reduce.InMemoryMapOutput: Read 314272 bytes from map-output for attempt_local1999544224_0005_m_000000_0
16/08/17 21:07:31 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 314272, inMemoryMapOutputs.size() -> 10, commitMemory -> 2740274, usedMemory ->3054546
16/08/17 21:07:31 INFO reduce.LocalFetcher: localfetcher#112 about to shuffle output of map attempt_local1999544224_0005_m_000006_0 decomp: 303136 len: 303140 to MEMORY
16/08/17 21:07:31 INFO reduce.InMemoryMapOutput: Read 303136 bytes from map-output for attempt_local1999544224_0005_m_000006_0
16/08/17 21:07:31 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 303136, inMemoryMapOutputs.size() -> 11, commitMemory -> 3054546, usedMemory ->3357682
16/08/17 21:07:31 INFO reduce.LocalFetcher: localfetcher#112 about to shuffle output of map attempt_local1999544224_0005_m_000009_0 decomp: 288928 len: 288932 to MEMORY
16/08/17 21:07:31 INFO reduce.InMemoryMapOutput: Read 288928 bytes from map-output for attempt_local1999544224_0005_m_000009_0
16/08/17 21:07:31 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 288928, inMemoryMapOutputs.size() -> 12, commitMemory -> 3357682, usedMemory ->3646610
16/08/17 21:07:31 INFO reduce.LocalFetcher: localfetcher#112 about to shuffle output of map attempt_local1999544224_0005_m_000015_0 decomp: 303652 len: 303656 to MEMORY
16/08/17 21:07:31 INFO reduce.InMemoryMapOutput: Read 303652 bytes from map-output for attempt_local1999544224_0005_m_000015_0
16/08/17 21:07:31 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 303652, inMemoryMapOutputs.size() -> 13, commitMemory -> 3646610, usedMemory ->3950262
16/08/17 21:07:31 INFO reduce.LocalFetcher: localfetcher#112 about to shuffle output of map attempt_local1999544224_0005_m_000002_0 decomp: 310448 len: 310452 to MEMORY
16/08/17 21:07:31 INFO reduce.InMemoryMapOutput: Read 310448 bytes from map-output for attempt_local1999544224_0005_m_000002_0
16/08/17 21:07:31 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 310448, inMemoryMapOutputs.size() -> 14, commitMemory -> 3950262, usedMemory ->4260710
16/08/17 21:07:31 INFO reduce.LocalFetcher: localfetcher#112 about to shuffle output of map attempt_local1999544224_0005_m_000008_0 decomp: 301460 len: 301464 to MEMORY
16/08/17 21:07:31 INFO reduce.InMemoryMapOutput: Read 301460 bytes from map-output for attempt_local1999544224_0005_m_000008_0
16/08/17 21:07:31 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 301460, inMemoryMapOutputs.size() -> 15, commitMemory -> 4260710, usedMemory ->4562170
16/08/17 21:07:31 INFO reduce.LocalFetcher: localfetcher#112 about to shuffle output of map attempt_local1999544224_0005_m_000014_0 decomp: 295920 len: 295924 to MEMORY
16/08/17 21:07:31 INFO reduce.MergeManagerImpl: Merged 16 segments, 4795152 bytes to disk to satisfy reduce memory limit
16/08/17 21:07:31 INFO reduce.InMemoryMapOutput: Read 295920 bytes from map-output for attempt_local1999544224_0005_m_000014_0
16/08/17 21:07:31 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 295920, inMemoryMapOutputs.size() -> 16, commitMemory -> 4562170, usedMemory ->4858090
16/08/17 21:07:31 INFO reduce.MergeManagerImpl: Merging 1 files, 4795126 bytes from disk
16/08/17 21:07:31 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:07:31 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:07:31 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
16/08/17 21:07:31 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 4795108 bytes
16/08/17 21:07:31 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:31 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:31 INFO reduce.MergeManagerImpl: finalMerge called with 16 in-memory map-outputs and 0 on-disk map-outputs
16/08/17 21:07:31 INFO mapred.Merger: Merging 16 sorted segments
16/08/17 21:07:31 INFO mapred.Merger: Down to the last merge-pass, with 16 segments left of total size: 4857726 bytes
16/08/17 21:07:31 INFO reduce.MergeManagerImpl: Merged 16 segments, 4858090 bytes to disk to satisfy reduce memory limit
16/08/17 21:07:31 INFO reduce.MergeManagerImpl: Merging 1 files, 4858064 bytes from disk
16/08/17 21:07:31 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
16/08/17 21:07:31 INFO mapred.Merger: Merging 1 sorted segments
16/08/17 21:07:31 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 4858042 bytes
16/08/17 21:07:31 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:32 INFO mapred.Task: Task:attempt_local518136789_0007_r_000015_0 is done. And is in the process of committing
16/08/17 21:07:32 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:32 INFO mapred.Task: Task attempt_local518136789_0007_r_000015_0 is allowed to commit now
16/08/17 21:07:32 INFO output.FileOutputCommitter: Saved output of task 'attempt_local518136789_0007_r_000015_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/e_given_f_phrase/_temporary/0/task_local518136789_0007_r_000015
16/08/17 21:07:32 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:07:32 INFO mapred.Task: Task 'attempt_local518136789_0007_r_000015_0' done.
16/08/17 21:07:32 INFO mapred.LocalJobRunner: Finishing task: attempt_local518136789_0007_r_000015_0
16/08/17 21:07:32 INFO mapred.LocalJobRunner: reduce task executor complete.
16/08/17 21:07:32 INFO mapred.Task: Task:attempt_local1999544224_0005_r_000015_0 is done. And is in the process of committing
16/08/17 21:07:32 INFO mapred.LocalJobRunner: 16 / 16 copied.
16/08/17 21:07:32 INFO mapred.Task: Task attempt_local1999544224_0005_r_000015_0 is allowed to commit now
16/08/17 21:07:32 INFO output.FileOutputCommitter: Saved output of task 'attempt_local1999544224_0005_r_000015_0' to file:/home/bpaulin/joshua-tutorial/runs/1/pipeline-es-en-hiero-_home_bpaulin_joshua-tutorial_runs_1/f_given_e_phrase/_temporary/0/task_local1999544224_0005_r_000015
16/08/17 21:07:32 INFO mapred.LocalJobRunner: reduce > reduce
16/08/17 21:07:32 INFO mapred.Task: Task 'attempt_local1999544224_0005_r_000015_0' done.
16/08/17 21:07:32 INFO mapred.LocalJobRunner: Finishing task: attempt_local1999544224_0005_r_000015_0
16/08/17 21:07:32 INFO mapred.LocalJobRunner: reduce task executor complete.
[SCHED] class edu.jhu.thrax.hadoop.features.mapred.TargetPhraseGivenSourceFeature in state SUCCESS
[SCHED] class edu.jhu.thrax.hadoop.features.mapred.SourcePhraseGivenTargetFeature in state SUCCESS
class edu.jhu.thrax.hadoop.features.mapred.SourcePhraseGivenTargetFeature	SUCCESS
class edu.jhu.thrax.hadoop.features.annotation.AnnotationFeatureJob	FAILED
class edu.jhu.thrax.hadoop.jobs.OutputJob	PREREQ_FAILED
class edu.jhu.thrax.hadoop.jobs.TargetWordGivenSourceWordProbabilityJob	SUCCESS
class edu.jhu.thrax.hadoop.features.mapred.TargetPhraseGivenSourceFeature	SUCCESS
class edu.jhu.thrax.hadoop.jobs.VocabularyJob	SUCCESS
class edu.jhu.thrax.hadoop.jobs.SourceWordGivenTargetWordProbabilityJob	SUCCESS
class edu.jhu.thrax.hadoop.jobs.ExtractionJob	SUCCESS