Uploaded image for project: 'Mahout'
  1. Mahout
  2. MAHOUT-1693

FunctionalMatrixView materializes row vectors in scala shell

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.10.0
    • Fix Version/s: 0.10.1
    • Component/s: Mahout spark shell, Math
    • Labels:
      None

      Description

      FunctionalMatrixView materializes row vectors in scala shell.

      Problem first reported by a user Michael Alton, Intel:

      "When I first tried to make a large matrix, I got an out of Java heap space error. I increased the memory incrementally until I got it to work. “export MAHOUT_HEAPSIZE=8000” didn’t work, but “export MAHOUT_HEAPSIZE=64000” did. The question is why do we need so much memory? A 5000x5000 matrix of doubles should only take up ~200MB of space?"

      Problem has been narrowed down to not override toString() method in FunctionalMatrixView which causes it to materialize all of the row vectors when run in Mahout Spark Shell.

        Attachments

          Activity

            People

            • Assignee:
              Andrew_Palumbo Andrew Palumbo
              Reporter:
              smarthi Suneel Marthi
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: