Uploaded image for project: 'Samza'
  1. Samza
  2. SAMZA-2297

InMemorySystemAdmin offsets are off-by-one in some cases

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.3
    • Component/s: None
    • Labels:
      None

      Description

      1. When a stream is empty for an "InMemory" system, the admin returns that the oldest/newest offset is "0" and the upcoming offset is "1".
      2. The "newest" offset returned by the admin is one higher than the offset of the last message in the list. Therefore, consuming from this offset would give no messages.
      3. The "upcoming" offset returned by the admin is two higher than the offset of the last message in the list. However, the next message actually gets an offset which is (offset of the last message + 1).

      For case (1), when using UPCOMING offsets, this results in missing the first message in a stream when the job starts while the stream is empty but then messages get produced to the stream. The current usages use the InMemorySystemDescriptor which uses OLDEST offsets, so this issue is not observed in the current tests. However, internal streams don't use the descriptor, so when trying to move some other tests to an in-memory system, those tests run into this problem.

      For case (2)/(3), the current usage of the in-memory system doesn't seem to be impacted. It seems to be because the newest/upcoming offsets aren't used to consume in the current tests.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                cameronlee Cameron Lee
                Reporter:
                cameronlee Cameron Lee
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1h 10m
                  1h 10m