Uploaded image for project: 'Mahout'
  1. Mahout
  2. MAHOUT-299

Collocations: improve performance by making Gram BinaryComparable

    XMLWordPrintableJSON

    Details

      Description

      Robin's profiling indicated that a large portion of a run was spent in readFields() in Gram due to the deserialization occuring as a part of Gram comparions for sorting. He pointed me to BinaryComparable and the implementation in Text.

      Like Text, in this new implementation, Gram stores its string in binary form. When encoding the string at construction time we allocate an extra character's worth of data to hold the Gram type information. When sorting Grams, the binary arrays are compared instead of deserializing and comparing fields.

        Attachments

        1. MAHOUT-299.patch
          36 kB
          Drew Farris

          Activity

            People

            • Assignee:
              drew.farris Drew Farris
              Reporter:
              drew.farris Drew Farris
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: