Accumulo
  1. Accumulo
  2. ACCUMULO-1940

Data file in !METADATA differs from in memory data

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 1.4.0, 1.4.1, 1.4.2, 1.4.3, 1.4.4, 1.5.0
    • Fix Version/s: 1.4.5, 1.5.1, 1.6.0
    • Component/s: test
    • Labels:

      Description

      Found during CI run with agitation.

      Got the first two error messages 5 times (assuming in a retry on failure block):

      Failed to do close consistency check for tablet c;79d0ab;7870a
      	java.lang.RuntimeException: Data file in !METADATA differ from in memory data c;79d0ab;7870a  {/t-0005h1j/A0005n8k.rf=797350457 19198312, /t-0005h1j/C0005skm.rf=798078368 19322025, /t-0005h1j/C0005tet.rf=89783168 2196349, /t-0005h1j/C0005u20.rf=90979448 2227972, /t-0005h1j/F0005u0v.rf=23410023 582233, /t-0005h1j/F0005u2p.rf=21958551 547159, /t-0005h1j/F0005u3g.rf=14395121 358893}  {/t-0005h1j/A0005n8k.rf=797350457 19198312, /t-0005h1j/C0005skm.rf=798078368 19322025, /t-0005h1j/C0005tet.rf=89783168 2196349, /t-0005h1j/C0005u20.rf=90979448 2227972, /t-0005h1j/F0005u2p.rf=21958551 547159, /t-0005h1j/F0005u3g.rf=14395121 358893}
      		at org.apache.accumulo.server.tabletserver.Tablet.closeConsistencyCheck(Tablet.java:2847)
      		at org.apache.accumulo.server.tabletserver.Tablet.completeClose(Tablet.java:2780)
      		at org.apache.accumulo.server.tabletserver.Tablet.close(Tablet.java:2658)
      		at org.apache.accumulo.server.tabletserver.TabletServer$UnloadTabletHandler.run(TabletServer.java:2357)
      		at org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
      		at org.apache.accumulo.trace.instrument.TraceRunnable.run(TraceRunnable.java:47)
      		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      		at org.apache.accumulo.trace.instrument.TraceRunnable.run(TraceRunnable.java:47)
      		at org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
      		at java.lang.Thread.run(Thread.java:744)
      

      Then, we logged that we failed the consistency check

      Consistency check fails, retrying java.lang.RuntimeException: Failed to do close consistency check for tablet c;79d0ab;7870a
      

      In the end, we gave up and closed it anyways.

      Tablet closed consistency check has failed for c;79d0ab;7870a giving up and closing
      

      Before all of this happened, we tried to bring this tablet online after a failure on a new tserver. During the minc as part of the recovery process, we failed to get the lease on the .rf_tmp file we tried to create. We failed this a couple of times, but eventually got the tmp file we needed and the recovery process completed and we could bring the tablet online. The difference between the in-memory version and the !METADATA version was this one flushed rfile that we created during this recovery process.

      The problem eventually fixed itself because the tablet was migrated to a different server and we just took what was (correctly) in the !METADATA table.

      There still is an unknown issue of how we missed the flush RFile in the DatafileManager's copy.

        Issue Links

          Activity

          Hide
          Josh Elser added a comment -

          I haven't re-run another multi-day test yet with this fix, but the fix does look good. I'll close for now and can re-open if it happens again.

          Show
          Josh Elser added a comment - I haven't re-run another multi-day test yet with this fix, but the fix does look good. I'll close for now and can re-open if it happens again.
          Hide
          Mike Drob added a comment -

          Eric Newton/Josh Elser, is this resolved?

          Show
          Mike Drob added a comment - Eric Newton / Josh Elser , is this resolved?
          Hide
          Keith Turner added a comment -

          I opened ACCUMULO-1949. Thinking about this some more, the changes in this ticket did not create that issue. It existed before, because the notification to the memory manager was not made until after all data was loaded into memory. Ideally as the tablet is recovered it would block (unless metadata table) if memory is full until other tablets can be flushed.

          Show
          Keith Turner added a comment - I opened ACCUMULO-1949 . Thinking about this some more, the changes in this ticket did not create that issue. It existed before, because the notification to the memory manager was not made until after all data was loaded into memory. Ideally as the tablet is recovered it would block (unless metadata table) if memory is full until other tablets can be flushed.
          Hide
          Keith Turner added a comment -

          It's probably a bad idea to be doing recovery in the ctor, or anything else that leaks "this" to other code that assumes that the tablet is fully constructed, like the MM.

          Agreed. There is another place where this is leaked, but it seems innocuous. I'll open a ticket for that.

          The concurrent MinC check worked, but the tablet load failed and retried. Then there were two Tablet objects in the server at the same time, and each was allowed to update the !METADATA table.

          ok, I see where the concurrency check is done now, I did not look deep enough earlier. The fix looks good, I will open a separate ticket to track the memory mgmt issue.

          Show
          Keith Turner added a comment - It's probably a bad idea to be doing recovery in the ctor, or anything else that leaks "this" to other code that assumes that the tablet is fully constructed, like the MM. Agreed. There is another place where this is leaked, but it seems innocuous. I'll open a ticket for that. The concurrent MinC check worked, but the tablet load failed and retried. Then there were two Tablet objects in the server at the same time, and each was allowed to update the !METADATA table. ok, I see where the concurrency check is done now, I did not look deep enough earlier. The fix looks good, I will open a separate ticket to track the memory mgmt issue.
          Hide
          Eric Newton added a comment -

          It's probably a bad idea to be doing recovery in the ctor, or anything else that leaks "this" to other code that assumes that the tablet is fully constructed, like the MM.

          The concurrent MinC check worked, but the tablet load failed and retried. Then there were two Tablet objects in the server at the same time, and each was allowed to update the !METADATA table.

          Show
          Eric Newton added a comment - It's probably a bad idea to be doing recovery in the ctor, or anything else that leaks "this" to other code that assumes that the tablet is fully constructed, like the MM. The concurrent MinC check worked, but the tablet load failed and retried. Then there were two Tablet objects in the server at the same time, and each was allowed to update the !METADATA table.
          Hide
          Keith Turner added a comment -

          The change that removes the line that informs the tablet server of the tablets memory usage may lead to the tablet server making bad decisions about memory usage. Instead of doing this maybe the minor compaction after recovery should probably go through the normal checks that prevent concurrent minor compactions.

          Show
          Keith Turner added a comment - The change that removes the line that informs the tablet server of the tablets memory usage may lead to the tablet server making bad decisions about memory usage. Instead of doing this maybe the minor compaction after recovery should probably go through the normal checks that prevent concurrent minor compactions.
          Hide
          Eric Newton added a comment -

          The memory manager can start a minor compaction after it learns of the tablet's existence. It learns this when the tablet server reports how much memory it is using. Unfortunately, it does this right after a recovery in the Tablet's constructor. In this case, the memory manager started a minor compaction before the tablet was online. This caused the AssignmentManager's minor compaction to fail. The AssignmentManager reloaded the tablet at a later time, and no data was lost, but the updates to the METADATA table made by the MemoryManager's MinC were not seen until the consistency check.

          Show
          Eric Newton added a comment - The memory manager can start a minor compaction after it learns of the tablet's existence. It learns this when the tablet server reports how much memory it is using. Unfortunately, it does this right after a recovery in the Tablet's constructor. In this case, the memory manager started a minor compaction before the tablet was online. This caused the AssignmentManager's minor compaction to fail. The AssignmentManager reloaded the tablet at a later time, and no data was lost, but the updates to the METADATA table made by the MemoryManager's MinC were not seen until the consistency check.
          Hide
          ASF subversion and git services added a comment -

          Commit 9b6b9cf104ff332cffdd4900d8057557e64e0ec8 in branch refs/heads/master from Eric Newton
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=9b6b9cf ]

          ACCUMULO-1940 do not expose a new tablet to the memory manager until after it is online

          Show
          ASF subversion and git services added a comment - Commit 9b6b9cf104ff332cffdd4900d8057557e64e0ec8 in branch refs/heads/master from Eric Newton [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=9b6b9cf ] ACCUMULO-1940 do not expose a new tablet to the memory manager until after it is online
          Hide
          ASF subversion and git services added a comment -

          Commit 82477f08aa64e2a8a1cf7f6af0db5ce954801ac8 in branch refs/heads/master from Eric Newton
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=82477f0 ]

          ACCUMULO-1940 do not expose a new tablet to the memory manager until after it is online

          Show
          ASF subversion and git services added a comment - Commit 82477f08aa64e2a8a1cf7f6af0db5ce954801ac8 in branch refs/heads/master from Eric Newton [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=82477f0 ] ACCUMULO-1940 do not expose a new tablet to the memory manager until after it is online
          Hide
          ASF subversion and git services added a comment -

          Commit 9b6b9cf104ff332cffdd4900d8057557e64e0ec8 in branch refs/heads/1.6.0-SNAPSHOT from Eric Newton
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=9b6b9cf ]

          ACCUMULO-1940 do not expose a new tablet to the memory manager until after it is online

          Show
          ASF subversion and git services added a comment - Commit 9b6b9cf104ff332cffdd4900d8057557e64e0ec8 in branch refs/heads/1.6.0-SNAPSHOT from Eric Newton [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=9b6b9cf ] ACCUMULO-1940 do not expose a new tablet to the memory manager until after it is online
          Hide
          ASF subversion and git services added a comment -

          Commit 82477f08aa64e2a8a1cf7f6af0db5ce954801ac8 in branch refs/heads/1.6.0-SNAPSHOT from Eric Newton
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=82477f0 ]

          ACCUMULO-1940 do not expose a new tablet to the memory manager until after it is online

          Show
          ASF subversion and git services added a comment - Commit 82477f08aa64e2a8a1cf7f6af0db5ce954801ac8 in branch refs/heads/1.6.0-SNAPSHOT from Eric Newton [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=82477f0 ] ACCUMULO-1940 do not expose a new tablet to the memory manager until after it is online
          Hide
          ASF subversion and git services added a comment -

          Commit 82477f08aa64e2a8a1cf7f6af0db5ce954801ac8 in branch refs/heads/1.5.1-SNAPSHOT from Eric Newton
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=82477f0 ]

          ACCUMULO-1940 do not expose a new tablet to the memory manager until after it is online

          Show
          ASF subversion and git services added a comment - Commit 82477f08aa64e2a8a1cf7f6af0db5ce954801ac8 in branch refs/heads/1.5.1-SNAPSHOT from Eric Newton [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=82477f0 ] ACCUMULO-1940 do not expose a new tablet to the memory manager until after it is online
          Hide
          ASF subversion and git services added a comment -

          Commit 82477f08aa64e2a8a1cf7f6af0db5ce954801ac8 in branch refs/heads/1.4.5-SNAPSHOT from Eric Newton
          [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=82477f0 ]

          ACCUMULO-1940 do not expose a new tablet to the memory manager until after it is online

          Show
          ASF subversion and git services added a comment - Commit 82477f08aa64e2a8a1cf7f6af0db5ce954801ac8 in branch refs/heads/1.4.5-SNAPSHOT from Eric Newton [ https://git-wip-us.apache.org/repos/asf?p=accumulo.git;h=82477f0 ] ACCUMULO-1940 do not expose a new tablet to the memory manager until after it is online
          Hide
          Eric Newton added a comment -

          One possibility for the lost value in DatafileManager is if the tablet is loaded twice.

          Show
          Eric Newton added a comment - One possibility for the lost value in DatafileManager is if the tablet is loaded twice.
          Hide
          Josh Elser added a comment - - edited

          I haven't been able to figure out how the DatafileManager got into an inconsistent state. bringMinorCompactionOnline(...) appears to have completed successfully which means that we should have correctly placed an entry for this flush file into the Map. Unless I missed it, there were no other compactions with this tablet either that could have altered the collection of files.

          Show
          Josh Elser added a comment - - edited I haven't been able to figure out how the DatafileManager got into an inconsistent state. bringMinorCompactionOnline(...) appears to have completed successfully which means that we should have correctly placed an entry for this flush file into the Map. Unless I missed it, there were no other compactions with this tablet either that could have altered the collection of files.

            People

            • Assignee:
              Eric Newton
              Reporter:
              Josh Elser
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development