Accumulo
  1. Accumulo
  2. ACCUMULO-697

Break Scanner parameterization from Key,Value to Key,{Something}

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Won't Fix
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: client
    • Labels:
      None

      Description

      When writing a custom iterator, many times the iterator has some semantic knowledge of what each Key/Value being returned actually means (e.g. A word count could be returning Key/Value but really is returning an Integer/Long count in the Value). This forces the client to know what is going to be returned and handle the cast/transformation.

      I believe it should be fairly straightforward to encapsulate this transformation inside the Accumulo client code. I plan on investigating the possibility of changing the ScannerBase impl, or perhaps making a TypedScannerBase, in which the iterator at the "top" of the stack for a scan can return something other than a Value to the client.

        Activity

        Hide
        William Slacum added a comment -

        Does this work with client side iterators only? The top level iterator will usually be running server side, so you'll need to have an interface to set a serializer for some object at the top of the stack (and possibly another deserializer if you have custom iterators beneath it) and set a deserializer at the client level. At some point on the server side, the iterator will have to transform the user type into a byte[], which is basically what a Value is. On the client side, the user will still have to supply code to deserialize the byte[], which can already be accomplished with no code changes by using a library such as Google Collections/Guava or Apache Commons.

        Show
        William Slacum added a comment - Does this work with client side iterators only? The top level iterator will usually be running server side, so you'll need to have an interface to set a serializer for some object at the top of the stack (and possibly another deserializer if you have custom iterators beneath it) and set a deserializer at the client level. At some point on the server side, the iterator will have to transform the user type into a byte[] , which is basically what a Value is. On the client side, the user will still have to supply code to deserialize the byte[] , which can already be accomplished with no code changes by using a library such as Google Collections/Guava or Apache Commons.
        Hide
        Adam Fuchs added a comment -

        I like the concept, but does this go far enough? If Values aren't special, then are Keys special, and if so then why? Should we make our SortedKeyValueIterator implement Iterable<? extends Object> ? Then the bottom level iterator (RFile reader) would include KeyValue or Entry<Key,Value> objects, the top level iterator for scans would have to have objects that are serializable, and the top level iterator for compactions would have to implement Iterable<Entry<Key,Value>>.

        One of the problems we have with iterators now is that the Key and Value are accessed with separate methods, even though they're always read off of disk together. Splitting up the Key and Value on the server side is sort of arbitrary and could reduce our ability to parallelize iterators (if we ever decide that's something we want to do).

        Another problem is that SortedKeyValueIterator falls somewhere in between Java's Iterator and Iterable interfaces. SortedKeyValueIterator holds onto filters, aggregation parameters, etc. that make it act like a collection, and it keeps a pointer to somewhere in that collection like an Iterator. I think we should change SortedKeyValueIterator into more like an immutable collection, or a consistent, isolated, unchanging view of the data, and have it implement Iterable. That might open up opportunities for automating optimization of queries on the server side, or better support for built-in iterator tree definition languages.

        Show
        Adam Fuchs added a comment - I like the concept, but does this go far enough? If Values aren't special, then are Keys special, and if so then why? Should we make our SortedKeyValueIterator implement Iterable<? extends Object> ? Then the bottom level iterator (RFile reader) would include KeyValue or Entry<Key,Value> objects, the top level iterator for scans would have to have objects that are serializable, and the top level iterator for compactions would have to implement Iterable<Entry<Key,Value>>. One of the problems we have with iterators now is that the Key and Value are accessed with separate methods, even though they're always read off of disk together. Splitting up the Key and Value on the server side is sort of arbitrary and could reduce our ability to parallelize iterators (if we ever decide that's something we want to do). Another problem is that SortedKeyValueIterator falls somewhere in between Java's Iterator and Iterable interfaces. SortedKeyValueIterator holds onto filters, aggregation parameters, etc. that make it act like a collection, and it keeps a pointer to somewhere in that collection like an Iterator. I think we should change SortedKeyValueIterator into more like an immutable collection, or a consistent, isolated, unchanging view of the data, and have it implement Iterable. That might open up opportunities for automating optimization of queries on the server side, or better support for built-in iterator tree definition languages.
        Hide
        jv added a comment -

        I don't know why this is something we want to integrate into our API. A utility can easily be written to do whatever transformations you want on top of the scanner's iterator, so why push this down into our api? I think a change like this could only serve to make the api harder to use for little to no real gain.

        Show
        jv added a comment - I don't know why this is something we want to integrate into our API. A utility can easily be written to do whatever transformations you want on top of the scanner's iterator, so why push this down into our api? I think a change like this could only serve to make the api harder to use for little to no real gain.
        Hide
        Josh Elser added a comment -

        John, I should have mentioned that the "inspiration" for something like this came from the TypedValueCombiner. If someone is already instrumenting code to run in Accumulo (writing an SKVI), I see no reason why to not let them actually deal in the types which they want, rather than forcing them to deal with everything in terms of Key/Value.

        Adam and Bill definitely hit on a larger issue of how far you can go with such an idea, but the scope of what I'll attempt here is much smaller than that.

        Show
        Josh Elser added a comment - John, I should have mentioned that the "inspiration" for something like this came from the TypedValueCombiner. If someone is already instrumenting code to run in Accumulo (writing an SKVI), I see no reason why to not let them actually deal in the types which they want, rather than forcing them to deal with everything in terms of Key/Value. Adam and Bill definitely hit on a larger issue of how far you can go with such an idea, but the scope of what I'll attempt here is much smaller than that.
        Hide
        Billie Rinaldi added a comment -

        A related feature in our API is InputFormatBase<K,V> which you can extend to provide any type of K,V. This was to support the ChunkInputFormat which wraps a set of Values in an InputStream.

        I could see making ScannerBase extend Iterable<E> instead of Iterable<Entry<Key,Value>>. Then individual scanner implementations could provide whatever types they wanted, e.g. Iterable<KeyValue>, Iterable<Entry<Key,Value>>, Iterable<Entry<Key,Whatever>>. We could model the use of these after the IsolatedScanner and ClientSideIteratorScanner, e.g. Scanner scanner = new ClientSideIteratorScanner(connector.createScanner(tableName, authorizations)). The default types could stay Entry<Key,Value>, and you would use a client side scanner wrapper to translate them.

        Show
        Billie Rinaldi added a comment - A related feature in our API is InputFormatBase<K,V> which you can extend to provide any type of K,V. This was to support the ChunkInputFormat which wraps a set of Values in an InputStream. I could see making ScannerBase extend Iterable<E> instead of Iterable<Entry<Key,Value>>. Then individual scanner implementations could provide whatever types they wanted, e.g. Iterable<KeyValue>, Iterable<Entry<Key,Value>>, Iterable<Entry<Key,Whatever>>. We could model the use of these after the IsolatedScanner and ClientSideIteratorScanner, e.g. Scanner scanner = new ClientSideIteratorScanner(connector.createScanner(tableName, authorizations)). The default types could stay Entry<Key,Value>, and you would use a client side scanner wrapper to translate them.
        Hide
        David Medinets added a comment -

        What is K and V? Where are they defined? I know they mean key and value but I'd like to grok the fundamentals. Sorry for this basic question, and pardon me for asking it here, but I don't know what kind of web search would give me an answer.

        Show
        David Medinets added a comment - What is K and V? Where are they defined? I know they mean key and value but I'd like to grok the fundamentals. Sorry for this basic question, and pardon me for asking it here, but I don't know what kind of web search would give me an answer.
        Hide
        Josh Elser added a comment -

        What is K and V? Where are they defined? I know they mean key and value but I'd like to grok the fundamentals. Sorry for this basic question, and pardon me for asking it here, but I don't know what kind of web search would give me an answer.

        http://en.wikipedia.org/wiki/Generics_in_Java

        Show
        Josh Elser added a comment - What is K and V? Where are they defined? I know they mean key and value but I'd like to grok the fundamentals. Sorry for this basic question, and pardon me for asking it here, but I don't know what kind of web search would give me an answer. http://en.wikipedia.org/wiki/Generics_in_Java
        Hide
        David Medinets added a comment -

        Thank you. I've used generics before but hadn't explicitly thought about K and V simply being placeholders.

        Show
        David Medinets added a comment - Thank you. I've used generics before but hadn't explicitly thought about K and V simply being placeholders.
        Hide
        Josh Elser added a comment -

        I believe I'm going to throw in the towel on this one. I got some simple client-side implementations working that are functional, but I never fully went down the thrift/tserver side of things.

        I think the code I wrote would fit better into Keith's Typo.

        Show
        Josh Elser added a comment - I believe I'm going to throw in the towel on this one. I got some simple client-side implementations working that are functional, but I never fully went down the thrift/tserver side of things. I think the code I wrote would fit better into Keith's Typo.

          People

          • Assignee:
            Josh Elser
            Reporter:
            Josh Elser
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development