Uploaded image for project: 'Ignite'
  1. Ignite
  2. IGNITE-8888

Possible data loss during starting of the nodes with empty pds

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.4
    • Fix Version/s: 2.7
    • Component/s: persistence
    • Labels:
      None

      Description

      Case: 

      1)Start 3 data nodes and activate the cluster with cache with 1 backup and PartitionLossPolicy.READ_ONLY_SAFE.
      2)Start client and add the data to your cache. Stop the client
      3)Stop DN2 and clear it pds and val
      4)Start DN2. Rebalance will start.
      5)During rebalance stop DN3.
      6)Start DN3.

      At this moment some partitions from DN2 marked as LOST and cache size will be less than expected.

      7) Run resetLostPartitions(caches).

      Now all partitions on DN2 marked as OWNING but cache size is still less than expected.

      Workaround:

      after step 6 do:

      7)force rebalance using deactivate/activate methods.
      8)wait for completion of rebalance

      Now cache size is expected but some partitions from DN2 marked as LOST

      9)Run resetLostPartitions(caches).

      Now cache size is OK and all partitions from DN2 marked as OWNING.

      However, looks like without force rebalance we have data loss here.

       

        Attachments

        1. reproducer.java
          8 kB
          Andrey Aleksandrov

          Issue Links

            Activity

              People

              • Assignee:
                astelmak Alexey Stelmak
                Reporter:
                aealeksandrov Andrey Aleksandrov
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: