Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-12026

ChiSqTest gets slower and slower over time when number of features is large

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.5.2
    • Fix Version/s: 1.6.1, 2.0.0
    • Component/s: MLlib
    • Labels:

      Description

      I've been running a ChiSqTest to pick features for feature reduction. My understanding is that internally it creates jobs to run on batches of 1000 features at a time.

      I was under the impression that the features are treated as independant, but this does not appear to be the case. When the number of features is large (160k in my case), each batch gets slower and slower. As an example, running on 25 m3.2xlarges on Amazon EMR, it started at just over 1 minute per batch. By the end, batches were taking over 30 minutes per batch.

        Attachments

        1. First Stages.png
          462 kB
          Hunter Kelly
        2. Latest Stages.png
          388 kB
          Hunter Kelly

          Activity

            People

            • Assignee:
              yuhaoyan yuhao yang
              Reporter:
              retnuH Hunter Kelly
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: