[MAHOUT-475] Replace Mapper with MultithreadedMapper to run job pairwiseSimilarity

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Not A Problem
• Affects Version/s: 0.4
• Fix Version/s: 0.4
• Assignee: Sean R. Owen
• Reporter: Han Hui Wen
• Created: 13/Aug/10 06:08
• Updated: 31/Jan/24 22:11
• Resolved: 13/Aug/10 11:10

Description

Because CooccurrencesMapper has huge computing,
Maybe we can replace Mapper with MultithreadedMapper.

And call the mapper
original:

    if (shouldRunNextPhase(parsedArgs, currentPhase)) {
      Job pairwiseSimilarity = prepareJob(weightsPath,
                               pairwiseSimilarityPath,
                               SequenceFileInputFormat.class,
                               CooccurrencesMapper.class,
                               WeightedRowPair.class,
                               Cooccurrence.class,
                               SimilarityReducer.class,
                               SimilarityMatrixEntryKey.class,
                               MatrixEntryWritable.class,
                               SequenceFileOutputFormat.class);

      Configuration pairwiseConf = pairwiseSimilarity.getConfiguration();
      pairwiseConf.set(DISTRIBUTED_SIMILARITY_CLASSNAME, distributedSimilarityClassname);
      pairwiseConf.setInt(NUMBER_OF_COLUMNS, numberOfColumns);
      pairwiseSimilarity.waitForCompletion(true);
    }

new:

    if (shouldRunNextPhase(parsedArgs, currentPhase)) {
      Job pairwiseSimilarity = prepareJob(weightsPath,
                               pairwiseSimilarityPath,
                               SequenceFileInputFormat.class,
                               CooccurrencesMapper.class,
                               WeightedRowPair.class,
                               Cooccurrence.class,
                               SimilarityReducer.class,
                               SimilarityMatrixEntryKey.class,
                               MatrixEntryWritable.class,
                               SequenceFileOutputFormat.class);

      
      Configuration pairwiseConf = pairwiseSimilarity.getConfiguration();
      pairwiseConf.set(DISTRIBUTED_SIMILARITY_CLASSNAME, distributedSimilarityClassname);
      pairwiseConf.setInt(NUMBER_OF_COLUMNS, numberOfColumns);
      MultithreadedMapper.setMapperClass(pairwiseSimilarity, CooccurrencesMapper.class);
      MultithreadedMapper.setNumberOfThreads(pairwiseSimilarity, numMapThreads);
      SequenceFileOutputFormat.setCompressOutput(pairwiseSimilarity, true);
      SequenceFileOutputFormat.setOutputCompressorClass(pairwiseSimilarity, GzipCodec.class);
      SequenceFileOutputFormat.setOutputCompressionType(pairwiseSimilarity, CompressionType.BLOCK);

      pairwiseSimilarity.waitForCompletion(true);
    }