Uploaded image for project: 'Apache Nemo'
  1. Apache Nemo
  2. NEMO-325

Add (de)serialization for non-string key types in data metric collection

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Labels:

      Description

      I figured out that our data metric collection mechanism assumes that the key type of the data is String always.

      PartitionSizeEntry of our ControlMessage protobuf has string as the type of key, and DynOptDataOutputCollector just conducts String.valueOf(key) for the keys. (These strings are not deserialized after.)

      Because of this, if the type of the key of the data to analyze is not }}{{String, the skewness will not be detected properly.

      We need to add some (de)serialization step for the key instead of this assumption.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              sanha Sanha Lee
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: