Uploaded image for project: 'Mahout'
  1. Mahout
  2. MAHOUT-154

Reduce memory usage with smarter data structures

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.2
    • Fix Version/s: 0.2
    • Labels:
      None

      Description

      Memory usage remains an issue. This issue tracks two changes with API implications that could reduce memory requirements:

      • use float, not double, for preference values. It is terribly unlikely that a float (4 bytes) is not enough precision to accurately represent user preferences, which are typically like "3.0" or "4.5". Using float instead of an 8-byte double saves 4 bytes per preference value, which is significant when loading tens of millions of prefs into memory
      • Preference[] is an inefficient way to store prefs, since it entails a great deal of Preference object overhead (48 bytes per pref is needed, of which 36 is overhead ) Using an abstraction like PreferenceArray which can use parallel arrays internally can cut at least 12 of the 36 bytes of overhead out – more if crazier data structures are used.

      So far these changes have reduced memory requirements by about 20% in my particular test case, which is significant.

      I am tracking this as an issue since like MAHOUT-151 it will entail API changes.

        Attachments

        1. MAHOUT-151-154.patch
          461 kB
          Sean R. Owen

          Activity

            People

            • Assignee:
              srowen Sean R. Owen
              Reporter:
              srowen Sean R. Owen
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Due:
                Created:
                Updated:
                Resolved: