[ACCUMULO-3751] Iterator Redesign

• Type: Wish
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: tserver
• Assignee: Unassigned
• Reporter: Shana Hutchison
• Created: 26/Apr/15 02:30
• Updated: 23/Apr/19 20:26

Description

Many Accumulo users have pointed out issues and places for improvement in the iterator stack formed atop the SortedKeyValueIterator interface. This issue aims to gather thoughts and requirements on what would make a new iterator stack, ideally reverse compatible with the current stack.

close method for iterators

See ACCUMULO-1280. Iterators do not have full lifecycle control, since a tablet server may "tear down" an iterator after it returns from a seek or next call. Iterators that start other threads, access external resources or perform some other action requiring cleanup must either initialize and tear down those actions within a call to seek or next, which is usually prohibitively expensive, or they keep state anyway and "hope for the best," possibly by putting cleanup code in the finalize method, which is not guaranteed to be called by the JVM but is a better option than nothing.

Current advice to iterator writers is to "not do" these kinds of operations that require stateful cleanup. Adding a close-like method that the tablet server guarantees will be called (via try-finally) before an iterator is torn down for any reason would make these iterators much more stable and easier to write.

billie.rinaldi has suggested using the Closeable or AutoCloseable interface. The tablet server could call close() on any SKVI that also implements AutoCloseable.

It would also be nice for the iterator to know why it is being closed, e.g., because

• the scan/compact range on the current tablet finished
• the source is switching
• the scan batch finished, and we're waiting for the client to request more batches
• some interrupt occurred
• to give CPU time or memory to other iterators for fairness

Such a reason could be passed to the iterator in the same way that the tablet server has a MajorCompactionReason.

Iterator Performance

Some have noticed that Accumulo is CPU-bound because it bottlenecks on the large numbers of serialize/deserialize operations, object creations and data copying present in the iterator stack. afuchs made a bunch of changes to system iterators that increased performance significantly in ACCUMULO-3079.

We may want to consider more fundamental changes, like putting the row, column family, qualifier, visibility, delete marker and timestamp in a single byte[] with a field delimitter byteacter, rather than keeping them in separate byte[]s. This gives an added bonus of easy key comparisons. Why not also store the Value with the Key rather than split the two into separate objects? Imagine a getTopEntry operation that returns a byte[] that holds all the components of the Key and Value.

We should adopt a philosophy of "reuse/alias byte[] buffers as often as possible," copying only when we need to save a copy. Imagine one extreme where we pass a single byte[] down the iterator stack rather than a Key or Value wrapping scattered buffers. If we were to consider changes along this route, we ought to create features that make it easy to grab and manipulate data in the byte[] as easily as the Key and Value objects, perhaps through well-documented static methods. This is critical for usability since users are used to object-oriented style manipulations of Key and Value.
For reverse compatibility, create an interface that extends SKVI and has methods for passing byte[] references directly, converting the byte[] to old Key/Value objects for iterators that do not implement the interface extension.

State-save/restore for iterators

The current information an iterator has to build up the state it needs are (1) the Map<String,String> options passed to init, (2) the IteratorEnvironment passed to init, and (3) the range passed to seek. When an iterator is torn down, the seek call after it is next reconstructed has a start key equal to the last key returned (non-inclusive).

There are a few ways we could imagine letting iterators save their state.

1. We could allow an iterator to add to or even modify the Map<String,String> options passed to init. This should be sufficient for most iterators to save their state and reload.
2. ACCUMULO-625 proposes a kind of "state cookie" that is emitted from an iterator (maybe as the return value of a close method) and is sent back to the client, so that a client could re-start a scan with this cookie and re-create the iterator in that state at the tablet server. This seems more powerful but perhaps harder engineering than #1.

Iterator safety

jstoneham put forward the idea of encapsulating user iterators in a security manager in ACCUMULO-1188.

kturner had an idea for running iterators in separate processes, and then suggested using tablet server rolling restarts to handle failing iterators.

elserj thought about giving long-running iterators the ability to stop their processing when their scan thread is interrupted but before the iterator returns in ACCUMULO-3348. Similar to the AutoCloseable suggestion above, we may realize this by checking whether iterators implement InterruptibleIterator and calling their setInterruptFlag method when they need interruption.

Issue Links

• relates to: ACCUMULO-1188 Sandbox iterators
• relates to: ACCUMULO-625 consider augmenting session state with "breadcrumbs"
• relates to: ACCUMULO-3079 improve system iterator performance by collapsing call stack
• relates to: ACCUMULO-1280 Add close method to iterators
• relates to: ACCUMULO-3348 Interrupt the ScanTask's thread when the Session is cancelled.