[IMPALA-8266] Event filtering logic may not filter all the events

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: Impala 3.2.0
• Epic Color: ghx-label-1
• Assignee: Vihang Karajgaonkar
• Reporter: Vihang Karajgaonkar
• Created: 01/Mar/19 00:53
• Updated: 05/Mar/19 23:07
• Resolved: 05/Mar/19 23:07

Description

Here is a the logic to filter out create events from a given batch of events.

Iterator<MetastoreEvent> it = metastoreEvents.iterator();
      // filter out the create events which has a corresponding drop event later
      int fromIndex = 0;
      int numFilteredEvents = 0;
      int inputSize = metastoreEvents.size();
      while (it.hasNext()) {
        MetastoreEvent current = it.next();
        if (fromIndex < metastoreEvents.size() && current.isRemovedAfter(
            metastoreEvents.subList(fromIndex + 1, metastoreEvents.size()))) {
          LOG.info(current.debugString("Filtering out this event since the object is "
              + "either removed or renamed later in the event stream"));
          it.remove();
          numFilteredEvents++;
        }
        fromIndex++;
      }

If the event list contains CREATE_DATABASE, CREATE_TABLE, DROP_TABLE, DROP_DATABASE events, it is possible that we only filter out CREATE_DATABASE event and not CREATE_TABLE event. This is because the fromIndex above gets incremented event iteration of the while loop and hence when it is evaluating if there is a inverse event later on in the stream, it starts to look from DROP_DATABASE event onwards.

The fix is simple, the fromIndex needs to be incremented only when the item from the list is not removed.

Issue Links

• links to: gerrit_review