[KUDU-2842] Data race in CatalogManager::GetTableLocations

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.10.0
• Fix Version/s: 1.11.0
• Component/s: master
• Code Review: https://gerrit.cloudera.org/c/14263/
• Assignee: Andrew Wong
• Reporter: Adar Dembo
• Created: 09/Jun/19 21:51
• Updated: 20/Sep/19 21:30
• Resolved: 20/Sep/19 21:30

Description

Saw this on a test run.

I think the problem is that TSInfosDict is reused for all calls to BuildLocationsForTablet for a particular table, and the map inside it points to peer UUIDs by reference (i.e. StringPiece) instead of by copy. Thus, when a given BuildLocationsForTablet call completes, the tablet lock is released and the catalog manager is free to destroy that tablet's TabletInfo (i.e. via committing a mutation in ProcessTabletReport). But the very next call to BuildLocationsForTablet may cause TSInfosDict map keys to be read and compared, even though the memory backing those keys no longer exists.

Assigning to Will because he committed 586e957f7 but cc'ing tlipcon as he was the original author of the code.

Issue Links

• is caused by: KUDU-2711 Improve performance of GetTableLocations for large tables
• is duplicated by: KUDU-2945 ksck_remote-test: locking a mutex that has been destroyed in the Master