Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 3.4.0
    • None
    • None

    Description

      While working on the deletedMessageVault, we targeted in MAILBOX-381 a first implementation on top of MailRepositories

      That decision was unfortunate as then one can browse the content of the deletedMessageVault vault through webAdmin.

      Thus we decided to implement DeletedMessageVault as a separate component.

      That component has the following considerations:

      • Should have a low storage cost
      • Cold storage and delays upon reads are acceptable as one does not expect a restore/export to be fast
      • Find an efficient way to implement retention (deleting too old emails) without a full scan

      We decided to provide an implementation above the blobStore and a metaData store (cassandra)

      Retention will be implemented with object storage buckets (one month = 1 bucket). We could then simply delete a bucket when all the mails it contains are older than the retention period.

      Querying the deleteMessageVault will first be done with a full scan. Later enhancement can be done by providing a per-bucket (immutable) Lucene index - once the bucket is no longer the latest one.

      Attachments

        Activity

          btellier Benoit Tellier created issue -
          btellier Benoit Tellier made changes -
          Field Original Value New Value
          Description While working on the deletedMessageVault, we targeted in MAILBOX-381 a first implementation on top of MailRepositories

          That decision was unfortunate as then one can browse the content of the deletedMessageVault vault through webAdmin.

          Thus we decided to implement DeletedMessageVault as a separate component.

          That component has the following considerations:
           - Should have a low storage cost
           - Cold storage and delays upon reads are acceptable as one does not expect a restore/export to be fast
           - Find an efficient way to implement retention (deleting too old emails) without a full scan

          We decided to provide an implementation above the blobStore and a metaData store (cassandra)

          Retention will be implemented with object storage buckets (one month = 1 bucket). We could then simply delete a bucket when all the mails it contains are older than the retention period.

          Querying the deleteMessageVault will first be done with a full scan. Later enhancement can be done by providing a per-bucket (immutable) Lucene index - once the bucket is no longer the latest one.
          btellier Benoit Tellier added a comment - - edited

          I created the tasks for the tasks we plan to tackle in the very near future.

          Note that to complete this work we will also need:

          • Implement DeletedMessageVault::delete. There are some interactions with deduplication that can happen in the blobStore. We thus decided to postpone this task to gain some better insights. A temporary acceptable step might be to just delete the meta-data.Another alternative might be to delete all the deletedMessages sharing the same blob. Anyway this topic needs further discussions.
          • Webadmin Task for moving old data in mail repository to new vault (for those who already tried thius feature, we should allow a migration)
          • CassandraBlobStore should deal with containers - the proposed table structure needs review and further discussions.
          • We also plan to write a "how to" + documentation
          btellier Benoit Tellier added a comment - - edited I created the tasks for the tasks we plan to tackle in the very near future. Note that to complete this work we will also need: Implement DeletedMessageVault::delete. There are some interactions with deduplication that can happen in the blobStore. We thus decided to postpone this task to gain some better insights. A temporary acceptable step might be to just delete the meta-data.Another alternative might be to delete all the deletedMessages sharing the same blob. Anyway this topic needs further discussions. Webadmin Task for moving old data in mail repository to new vault (for those who already tried thius feature, we should allow a migration) CassandraBlobStore should deal with containers - the proposed table structure needs review and further discussions. We also plan to write a "how to" + documentation
          btellier Benoit Tellier added a comment -

          Story:

          As an admin, using webAdmin, I want to definitly delete a deletedMessage
          For instance, if it contains some critical information hence I want to delete it sooner than the retention period
          

          Regarding the 'single message delete' of the deletedMessageVault, we had two choices:

          • Deleting the metadata, which unreferences the blob, and let blob background deletion task do it's job

          This is bad as the content the admin wishes to deletes is still in the DeletedMessageVault.

          • Deleting the blob in the objectStore

          Then it means deleting other deletedMessages sharing this blob. This is acceptable as it is an admin triggered operation, and the side effect actually ensure all messages sharing the same MIME message would also no longer exist.

          We decided to implement the latest, of course documenting this behaviour will be needed.

          btellier Benoit Tellier added a comment - Story: As an admin, using webAdmin, I want to definitly delete a deletedMessage For instance, if it contains some critical information hence I want to delete it sooner than the retention period Regarding the 'single message delete' of the deletedMessageVault, we had two choices: Deleting the metadata, which unreferences the blob, and let blob background deletion task do it's job This is bad as the content the admin wishes to deletes is still in the DeletedMessageVault. Deleting the blob in the objectStore Then it means deleting other deletedMessages sharing this blob. This is acceptable as it is an admin triggered operation, and the side effect actually ensure all messages sharing the same MIME message would also no longer exist. We decided to implement the latest, of course documenting this behaviour will be needed.
          btellier Benoit Tellier added a comment -

          We are now removing previous implementations cf JAMES-2847

          btellier Benoit Tellier added a comment - We are now removing previous implementations cf JAMES-2847
          btellier Benoit Tellier made changes -
          Fix Version/s 3.4.0 [ 12344981 ]
          Resolution Fixed [ 1 ]
          Status Open [ 1 ] Resolved [ 5 ]
          btellier Benoit Tellier made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

          People

            Unassigned Unassigned
            btellier Benoit Tellier
            Votes:
            0 Vote for this issue
            Watchers:
            Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack