Currently we're invoking task.onPartitionsRevoked and commitOffsets on any consumer onPartitionsRevoked callback. Since this happens even on the first rebalance (which is somewhat unexpected given that partitions cannot actually be revoked at that point), we will not be fully setup in the SinkTask and this can throw exceptions.
It turns out this is easy to miss for two reasons. First, the code is likely to throw an exception, but the consumer catches this and handles it. We end up not missing any important steps as a result. Second, the logging was a bit screwed up and we weren't getting the full exception stacktrace, just the toString(), so it was easy to miss that there was a problem.