SequenceManager generated IDs are unreliable whilst clustering
Description
Environment
Attachments
Activity
Guus der Kinderen December 4, 2020 at 9:49 AM
Making SequenceManager cluster-aware, to ensure that it does not only hand out unique values, but also hands them out in the correct order in a cluster, is important, but does not completely resolve the issue.
The sequence value should be assigned to the to-be-written data in the same order as the data is to be ordered in the database (likely: the chronological order in which data was generated). If data is written to the database in a batched/staggered fashion by individual cluster nodes (for example when using the Archiver implementation) then the database insertion order might no longer reflect the desired order of elements!
To prevent issues, it is important to assign the SequenceManager-obtained value (or any other value used to order the data) as soon as the data is made available, and explicitly not only when the data is being written to the database / persistent storage.
Dan Caseley December 3, 2020 at 1:26 PM
Current thinking: Make SequenceManager cluster-aware to ensure all new unique values are sequential across the cluster.
Modifying old IDs could impact clients disproportionately, so probably best left alone.
Dan Caseley December 3, 2020 at 11:03 AM
Attached the XML Debugger log of this scenario occurring.
Guus der Kinderen December 2, 2020 at 3:39 PM
An additional consideration for this issue: apart from preventing inconsistent IDs to be used while storing new content to the database, old content that already is in the database needs to be fixed.
Details
Details
Assignee
Guus der KinderenReporter
Dan CaseleyFix versions
Priority
Major
The Conversations client was used to participate in a MUC on a cluster.
After force quitting and restarting the app, Conversations requests all messages since a previous time stamp via MAM2, then requests all messages after the last message (presumably in case of activity whilst fetching and processing the first batch).
In this scenario, the MUC had 3 messages that were returned in the first request. It should have returned 0 results for the second request, there having been no activity, but instead returned the middle message, duplicating it in the client.
This snapshot of the ofMucConversationLog shows the messages in conversation order, but with inconsistent messageIds (the second field). In this test user1 was on the Senior Node whilst users 2 & 3 were on the Junior.
IDs need to be consistent if they're to be used.
(This issue could also be considered a Monitoring Plugin bug, where it could use different cluster-safe fields to respond to the query)