Details

    • Type: New Feature
    • Status: Closed
    • Priority: Minor
    • Resolution: Implemented
    • Affects Version/s: 1.1.0
    • Fix Version/s: 1.3.0
    • Labels:
      None

      Description

      Flink provides mutable Value type implementations of Java primitives along with efficient serializers and comparators. It would be useful to have corresponding ValueArray implementations backed by primitive rather than object arrays, along with an ArrayableValue interface tying a Value to its ValueArray.

        Issue Links

          Activity

          Hide
          StephanEwen Stephan Ewen added a comment -

          I would vote to try and have a generic type system extension mechanism and then realize this issue via that extension mechanism.

          FLINK-3042 proposes such a mechanism.

          Show
          StephanEwen Stephan Ewen added a comment - I would vote to try and have a generic type system extension mechanism and then realize this issue via that extension mechanism. FLINK-3042 proposes such a mechanism.
          Hide
          greghogan Greg Hogan added a comment -

          Stephan Ewen I was looking to use primitive arrays rather than object arrays so that the data stays in contiguous memory. Something like

          public interface ValueArray<T> extends Iterable<T>, IOReadableWritable, Serializable, ResettableValue<ValueArray<T>>, CopyableValue<ValueArray<T>> {
          
          	void clear();
          
          	void add(T value);
          }
          
          public class LongValueArray implements ValueArray<LongValue> {
          
          	private long[] data;
          
          	private int position;
          
          	@Override
          	public void clear() {
          		position = 0;
          	}
          
          	@Override
          	public void add(LongValue value) {
          		data[position++] = value.getValue();
          	}
          
          	...
          }
          
          Show
          greghogan Greg Hogan added a comment - Stephan Ewen I was looking to use primitive arrays rather than object arrays so that the data stays in contiguous memory. Something like public interface ValueArray<T> extends Iterable<T>, IOReadableWritable, Serializable, ResettableValue<ValueArray<T>>, CopyableValue<ValueArray<T>> { void clear(); void add(T value); } public class LongValueArray implements ValueArray<LongValue> { private long [] data; private int position; @Override public void clear() { position = 0; } @Override public void add(LongValue value) { data[position++] = value.getValue(); } ... }
          Hide
          StephanEwen Stephan Ewen added a comment -

          Makes sense.

          You can probably still get started with these ValueArrayTypes, defining the type, the serializer, and the type information. That is the same, no matter where the types go.

          To make the type info available: Do you think it helps to first have the generic extension mechanism? Then we could simply allow people to add these types in libraries, and gradually move core types to the core.

          You can probably still go ahead using our custom type info by using the .returns(...) method, or the ResultTypeQueryable interface, even without the type being added to core, or the extension mechanism being in place.

          Show
          StephanEwen Stephan Ewen added a comment - Makes sense. You can probably still get started with these ValueArrayTypes, defining the type, the serializer, and the type information. That is the same, no matter where the types go. To make the type info available: Do you think it helps to first have the generic extension mechanism? Then we could simply allow people to add these types in libraries, and gradually move core types to the core. You can probably still go ahead using our custom type info by using the .returns(...) method, or the ResultTypeQueryable interface, even without the type being added to core, or the extension mechanism being in place.
          Hide
          greghogan Greg Hogan added a comment - - edited

          I was thinking this could be moved to the Gelly library since we no longer require integration into the TypeExtractor; however, we still need the Value types to map to the proper ValueArray type.

          Show
          greghogan Greg Hogan added a comment - - edited I was thinking this could be moved to the Gelly library since we no longer require integration into the TypeExtractor ; however, we still need the Value types to map to the proper ValueArray type.
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user greghogan opened a pull request:

          https://github.com/apache/flink/pull/3382

          FLINK-3695 [gelly] ValueArray types

          Provide compact and efficiently serializable and comparable array implementations for Flink mutable Value types and Java primitives.

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/greghogan/flink 3695_valuearray_types

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/flink/pull/3382.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #3382


          commit aac1f8f2d2ca0e7eba4ab52fba94db62e9c3ef41
          Author: Greg Hogan <code@greghogan.com>
          Date: 2017-02-21T16:40:22Z

          FLINK-3695 [gelly] ValueArray types

          Provide compact and efficiently serializable and comparable array
          implementations for Flink mutable Value types and Java primitives.


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user greghogan opened a pull request: https://github.com/apache/flink/pull/3382 FLINK-3695 [gelly] ValueArray types Provide compact and efficiently serializable and comparable array implementations for Flink mutable Value types and Java primitives. You can merge this pull request into a Git repository by running: $ git pull https://github.com/greghogan/flink 3695_valuearray_types Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/3382.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3382 commit aac1f8f2d2ca0e7eba4ab52fba94db62e9c3ef41 Author: Greg Hogan <code@greghogan.com> Date: 2017-02-21T16:40:22Z FLINK-3695 [gelly] ValueArray types Provide compact and efficiently serializable and comparable array implementations for Flink mutable Value types and Java primitives.
          Hide
          greghogan Greg Hogan added a comment -

          Stephan Ewen Here is the ValueArray interface I have been developing against. Next would be StringValueArray and also for tuples.

          Show
          greghogan Greg Hogan added a comment - Stephan Ewen Here is the ValueArray interface I have been developing against. Next would be StringValueArray and also for tuples.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user greghogan commented on the issue:

          https://github.com/apache/flink/pull/3382

          @vasia, if you are available to review, I think the easiest way to do so is to first review the classes for `LongValue` then compare the diff for `IntValue` and `StringValue`. I think we are forced to duplicate code as the value types to keep the method calls monomorphic.

          Show
          githubbot ASF GitHub Bot added a comment - Github user greghogan commented on the issue: https://github.com/apache/flink/pull/3382 @vasia, if you are available to review, I think the easiest way to do so is to first review the classes for `LongValue` then compare the diff for `IntValue` and `StringValue`. I think we are forced to duplicate code as the value types to keep the method calls monomorphic.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user greghogan commented on the issue:

          https://github.com/apache/flink/pull/3382

          Failing test is unrelated. Merging this after offline discussion with @vasia ...

          Show
          githubbot ASF GitHub Bot added a comment - Github user greghogan commented on the issue: https://github.com/apache/flink/pull/3382 Failing test is unrelated. Merging this after offline discussion with @vasia ...
          Hide
          greghogan Greg Hogan added a comment -

          Implemented in 963f46e7179db034fd1d444469f4af58eac87409

          Show
          greghogan Greg Hogan added a comment - Implemented in 963f46e7179db034fd1d444469f4af58eac87409
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user greghogan closed the pull request at:

          https://github.com/apache/flink/pull/3382

          Show
          githubbot ASF GitHub Bot added a comment - Github user greghogan closed the pull request at: https://github.com/apache/flink/pull/3382

            People

            • Assignee:
              greghogan Greg Hogan
              Reporter:
              greghogan Greg Hogan
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development