Uploaded image for project: 'Sentry'
  1. Sentry
  2. SENTRY-2302 Optimize HDFS ACL Synchronization.
  3. SENTRY-2305

Optimize time taken for persistence HMS snapshot by persisting in parallel

    Details

    • Type: Sub-task
    • Status: Patch Available
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 2.1.0
    • Fix Version/s: None
    • Component/s: Sentry
    • Labels:
      None

      Description

      There are couple of options

      1. Break the total snapshot into to batches and persist all of them in parallel in different transactions. As sentry uses repeatable_read isolation level we should be able to have parallel writes on the same table. This bring an issue if there is a failure in persisting any of the batches. This approach needs additional logic of cleaning the partially persisted snapshot. I’m evaluating this option.
        • Result: Initial results are promising. Time to persist the snapshot came down by 60%.
      2. Try disabling L1 Cache for persisting the snapshot.
      3. Try persisting the snapshot entries sequentially in separate transactions. As transactions which commit huge data might take longer as they take a lot of CPU cycles to keep the rollback log up to date.

        Attachments

        1. SENTRY-2305.001.patch
          11 kB
          kalyan kumar kalvagadda
        2. SENTRY-2305.002.patch
          14 kB
          kalyan kumar kalvagadda
        3. SENTRY-2305.003.patch
          17 kB
          kalyan kumar kalvagadda
        4. SENTRY-2305.004.patch
          18 kB
          kalyan kumar kalvagadda
        5. SENTRY-2305.005.patch
          21 kB
          kalyan kumar kalvagadda

          Issue Links

            Activity

              People

              • Assignee:
                kkalyan kalyan kumar kalvagadda
                Reporter:
                kkalyan kalyan kumar kalvagadda
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated: