Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-22096

use aggregateByKeyLocally to save one stage in calculating ItemFrequency in NaiveBayes

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Minor
    • Resolution: Incomplete
    • 2.2.0
    • None
    • ML

    Description

      NaiveBayes currently takes aggreateByKey followed by a collect to calculate frequency for each feature/label. We can implement a new function 'aggregateByKeyLocally' in RDD that merges locally on each mapper before sending results to a reducer to save one stage.
      We tested on NaiveBayes and see ~16% performance gain with these changes.
      performance data for NB.png

      Attachments

        Issue Links

          depends upon

          Sub-task - The sub-task of the issue SPARK-22098 Add aggregateByKeyLocally in RDD

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          links to

          Pull request #19318 [Github] Pull Request #19318 (VinceShieh)

          Web Link GitHub Pull Request #19318

          Activity

            People

              Unassigned Unassigned
              VinceXie Vincent
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: