Uploaded image for project: 'Jackrabbit Oak'
  1. Jackrabbit Oak
  2. OAK-9112

High CPU IOWAIT on Mongo due to Oak Version Garbage Collection

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Won't Fix
    • Affects Version/s: 1.8.22
    • Fix Version/s: None
    • Component/s: mongomk
    • Labels:
      None
    • Environment:

      Mongo: 3.4.9
      Hosteding: Scalegrid
      Instance Size: Small

      Description

      Over the previous weekend, our existing application went from having 100,000 documents in the "nodes" collection to 600,000 documents in the "nodes" collection. Since that time there appears to be an Oak background task that commences running near 100% CPU IOWAIT and heavy disk activity (read only). Please refer to charts below

      According to currentOp Mongo command, the queries being running look as follows:

      
      { desc: "conn4944", threadId: "140061124310784", connectionId: 4944, client: "18.207.114.70:47008", clientMetadata: \{ driver: { name: "mongo-java-driver", version: "unknown" }, os: \{ type: "Linux", name: "Linux", architecture: "amd64", version: "4.4.115-k8s" }, platform: "Java/IcedTea/1.8.0_212-b04" }, active: true, opid: 2215767, secs_running: 2, microsecs_running: \{ "$numberLong": "2280333" }, op: "query", ns: "prod-content-service.nodes", query: \{ find: "nodes", filter: { _id: "2:/prod/18d6b4d0-9ee8-4a06-804e-b3669f67dda8" }, limit: 1, singleBatch: true }, planSummary: "IDHACK", numYields: 0, locks: \{ Global: "r", Database: "r", Collection: "r" }, waitingForLock: false, lockStats: \{ Global: { acquireCount: { r: { "$numberLong": "2" } } }, Database: \{ acquireCount: { r: { "$numberLong": "1" } } }, Collection: \{ acquireCount: { r: { "$numberLong": "1" } } } } }
      

      and appears to be iterating over the entire collection.

      Can anyone provide any guidance on what is happening?

        Attachments

        1. image-2020-06-15-19-41-04-852.png
          106 kB
          David Snowsill
        2. image-2020-06-15-19-41-40-385.png
          62 kB
          David Snowsill
        3. screenshot-1.png
          1.36 MB
          David Snowsill

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              djsnowsill David Snowsill
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: