Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: Impala 3.2.0
    • Component/s: None
    • Labels:
      None
    • Epic Color:
      ghx-label-1

      Description

      Here is a the logic to filter out create events from a given batch of events.

      Iterator<MetastoreEvent> it = metastoreEvents.iterator();
            // filter out the create events which has a corresponding drop event later
            int fromIndex = 0;
            int numFilteredEvents = 0;
            int inputSize = metastoreEvents.size();
            while (it.hasNext()) {
              MetastoreEvent current = it.next();
              if (fromIndex < metastoreEvents.size() && current.isRemovedAfter(
                  metastoreEvents.subList(fromIndex + 1, metastoreEvents.size()))) {
                LOG.info(current.debugString("Filtering out this event since the object is "
                    + "either removed or renamed later in the event stream"));
                it.remove();
                numFilteredEvents++;
              }
              fromIndex++;
            }
      

      If the event list contains CREATE_DATABASE, CREATE_TABLE, DROP_TABLE, DROP_DATABASE events, it is possible that we only filter out CREATE_DATABASE event and not CREATE_TABLE event. This is because the fromIndex above gets incremented event iteration of the while loop and hence when it is evaluating if there is a inverse event later on in the stream, it starts to look from DROP_DATABASE event onwards.

      The fix is simple, the fromIndex needs to be incremented only when the item from the list is not removed.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                vihangk1 Vihang Karajgaonkar
                Reporter:
                vihangk1 Vihang Karajgaonkar
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: