[FINERACT-1909] Retry feature for commands - deadlock

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: Savings
• Labels: BeanSalad
• Environment: Local. ; ./gradlew clean ; ./gradlew :fineract-provider:jibDockerBuild -x test ; docker compose up
• Flags: Important
• Assignee: Unassigned
• Reporter: Peter Santa
• Created: 24/Mar/23 11:22
• Updated: 20/Nov/23 18:49

Description

Background

In our use case, Savings Account transactions have to be inserted, sometimes multiple transactions for the same account triggered by independent threads.

Tested on this commit hash: d21110f7b

application.properties modified to:

resilience4j.retry.instances.executeCommand.max-attempts=${FINERACT_COMMAND_PROCESSING_RETRY_MAX_ATTEMPTS:20}...

resilience4j.retry.instances.executeCommand.retryExceptions=${FINERACT_COMMAND_PROCESSING_RETRY_EXCEPTIONS:org.springframework.dao.CannotAcquireLockException,org.springframework.orm.ObjectOptimisticLockingFailureException,org.eclipse.persistence.exceptions.OptimisticLockException}

Build and run

./gradlew clean
./gradlew :fineract-provider:jibDockerBuild -x test 
docker compose up

Phenomenon

Performing the following command 100 times on 2 threads (sum: 200 deposit), by Postman runner:

POST url/savingsaccounts/savingsAccountId/transactions?command=deposit
{
"transactionDate": {
{currentDate}},
"transactionAmount": 1000,
"paymentTypeId": {
{paymentTypeId}},
"note": "",
"dateFormat": "yyyyMMdd",
"locale": "en"
{color:#000000}}

Expected result

Fineract retries the command if any of the configured exception is thrown, and all the transactions are performed successfully.

Actual result

Deadlock. See the logs attached.

2 commands were unsuccessful - internal server error, code 500 response on API.

Issue Links

• links to: GitHub Pull Request #3471
• links to: GitHub Pull Request #3560
• links to: GitHub Pull Request #3589