Uploaded image for project: 'Giraph (Retired)'
  1. Giraph (Retired)
  2. GIRAPH-907

refactor giraph code to support multiple implementations of vertexId data

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 1.1.0
    • None
    • None

    Description

      currently MessageStores & EdgeStores expect ByteArrayVertexIdData objects. but this is too restrictive,
      refactor giraph code to support multiple VertexId structs (for instance ByteBuf, OneMessageToMultipleIds, etc.)

      Attachments

        1. GIRAPH-907.patch
          176 kB
          Pavan Kumar

        Activity

          pavanka Pavan Kumar added a comment -

          https://reviews.apache.org/r/22157/
          also posted on review board

          pavanka Pavan Kumar added a comment - https://reviews.apache.org/r/22157/ also posted on review board
          hudson Hudson added a comment -

          ABORTED: Integrated in Giraph-trunk-Commit #1444 (See https://builds.apache.org/job/Giraph-trunk-Commit/1444/)
          GIRAPH-907: refactor giraph code to support multiple implementations of vertexId data (pavanka) (pavanka: http://git-wip-us.apache.org/repos/asf?p=giraph.git&a=commit&h=535a333b7776f7a229b76c7656e928d8a21d21bc)

          • giraph-core/src/main/java/org/apache/giraph/comm/requests/WritableRequest.java
          • giraph-core/src/main/java/org/apache/giraph/utils/ByteUtils.java
          • CHANGELOG
          • giraph-core/src/main/java/org/apache/giraph/utils/ByteStructVertexIdMessageBytesIterator.java
          • giraph-core/src/main/java/org/apache/giraph/utils/UnsafeArrayReads.java
          • giraph-core/src/main/java/org/apache/giraph/utils/ByteStructVertexIdIterator.java
          • giraph-core/src/main/java/org/apache/giraph/comm/SendMessageCache.java
          • giraph-core/src/main/java/org/apache/giraph/comm/netty/NettyWorkerClientRequestProcessor.java
          • giraph-core/src/main/java/org/apache/giraph/comm/messages/ByteArrayMessagesPerVertexStore.java
          • giraph-core/src/main/java/org/apache/giraph/utils/ByteStructVertexIdDataIterator.java
          • giraph-core/src/main/java/org/apache/giraph/utils/ExtendedDataOutput.java
          • giraph-core/src/main/java/org/apache/giraph/comm/messages/OneMessagePerVertexStore.java
          • giraph-core/src/main/java/org/apache/giraph/comm/messages/primitives/IntByteArrayMessageStore.java
          • giraph-core/src/main/java/org/apache/giraph/utils/VertexIdEdgeIterator.java
          • giraph-core/src/main/java/org/apache/giraph/edge/primitives/IntEdgeStore.java
          • giraph-core/src/main/java/org/apache/giraph/utils/ByteStructIterable.java
          • giraph-core/src/main/java/org/apache/giraph/utils/UnsafeByteArrayOutputStream.java
          • giraph-core/src/main/java/org/apache/giraph/utils/ByteArrayIterator.java
          • giraph-core/src/main/java/org/apache/giraph/comm/netty/handler/RequestServerHandler.java
          • giraph-core/src/main/java/org/apache/giraph/comm/requests/SendWorkerEdgesRequest.java
          • giraph-core/src/main/java/org/apache/giraph/utils/ByteArrayVertexIdData.java
          • giraph-core/src/main/java/org/apache/giraph/edge/SimpleEdgeStore.java
          • giraph-core/src/main/java/org/apache/giraph/utils/ByteArrayIterable.java
          • giraph-core/src/main/java/org/apache/giraph/conf/GiraphConstants.java
          • giraph-core/src/main/java/org/apache/giraph/comm/messages/out_of_core/DiskBackedMessageStore.java
          • giraph-core/src/main/java/org/apache/giraph/comm/SendVertexIdDataCache.java
          • giraph-core/src/main/java/org/apache/giraph/utils/ExtendedByteArrayDataOutput.java
          • giraph-core/src/main/java/org/apache/giraph/comm/netty/handler/RequestDecoder.java
          • giraph-core/src/main/java/org/apache/giraph/edge/EdgeStore.java
          • giraph-core/src/main/java/org/apache/giraph/utils/ExtendedByteArrayDataInput.java
          • giraph-core/src/main/java/org/apache/giraph/utils/VerboseByteArrayMessageWrite.java
          • giraph-core/src/main/java/org/apache/giraph/comm/netty/handler/ResponseEncoder.java
          • giraph-core/src/main/java/org/apache/giraph/comm/messages/primitives/LongByteArrayMessageStore.java
          • giraph-core/src/main/java/org/apache/giraph/utils/ByteArrayVertexIdEdges.java
          • giraph-core/src/main/java/org/apache/giraph/utils/RepresentativeByteArrayIterable.java
          • giraph-core/src/main/java/org/apache/giraph/utils/RepresentativeByteStructIterator.java
          • giraph-core/src/main/java/org/apache/giraph/utils/VertexIdMessageIterator.java
          • giraph-core/src/main/java/org/apache/giraph/comm/messages/primitives/IntFloatMessageStore.java
          • giraph-core/src/main/java/org/apache/giraph/utils/AbstractVertexIdData.java
          • giraph-core/src/main/java/org/apache/giraph/conf/GiraphConfiguration.java
          • giraph-core/src/main/java/org/apache/giraph/conf/ImmutableClassesGiraphConfiguration.java
          • giraph-core/src/main/java/org/apache/giraph/utils/RepresentativeByteArrayIterator.java
          • giraph-core/src/main/java/org/apache/giraph/comm/SendMessageToAllCache.java
          • giraph-core/src/main/java/org/apache/giraph/utils/UnsafeByteArrayInputStream.java
          • giraph-core/src/main/java/org/apache/giraph/utils/RepresentativeByteStructIterable.java
          • giraph-core/src/test/java/org/apache/giraph/utils/MockUtils.java
          • giraph-core/src/main/java/org/apache/giraph/utils/ByteStructVertexIdEdgeIterator.java
          • giraph-core/src/main/java/org/apache/giraph/comm/requests/SendWorkerDataRequest.java
          • giraph-core/src/main/java/org/apache/giraph/comm/messages/MessagesIterable.java
          • giraph-core/src/main/java/org/apache/giraph/comm/messages/primitives/LongDoubleMessageStore.java
          • giraph-core/src/main/java/org/apache/giraph/utils/VertexIdDataIterator.java
          • giraph-core/src/main/java/org/apache/giraph/utils/VertexIdEdges.java
          • giraph-core/src/test/java/org/apache/giraph/comm/RequestFailureTest.java
          • giraph-core/src/main/java/org/apache/giraph/utils/VertexIdMessages.java
          • giraph-core/src/main/java/org/apache/giraph/edge/primitives/LongEdgeStore.java
          • giraph-core/src/main/java/org/apache/giraph/comm/messages/MessageStore.java
          • giraph-core/src/main/java/org/apache/giraph/utils/VerboseByteStructMessageWrite.java
          • giraph-core/src/main/java/org/apache/giraph/utils/RequestUtils.java
          • giraph-core/src/test/java/org/apache/giraph/comm/RequestTest.java
          • giraph-core/src/main/java/org/apache/giraph/utils/VertexIdIterator.java
          • giraph-core/src/main/java/org/apache/giraph/utils/UnsafeReads.java
          • giraph-core/src/main/java/org/apache/giraph/utils/VertexIdData.java
          • giraph-core/src/main/java/org/apache/giraph/comm/netty/InboundByteCounter.java
          • giraph-core/src/main/java/org/apache/giraph/edge/AbstractEdgeStore.java
          • giraph-core/src/main/java/org/apache/giraph/utils/ByteStructVertexIdMessageIterator.java
          • giraph-core/src/main/java/org/apache/giraph/utils/VertexIdMessageBytesIterator.java
          • giraph-core/src/main/java/org/apache/giraph/comm/requests/SendWorkerMessagesRequest.java
          • giraph-core/src/main/java/org/apache/giraph/comm/netty/handler/RequestEncoder.java
          • giraph-core/src/main/java/org/apache/giraph/comm/SendEdgeCache.java
          • giraph-core/src/main/java/org/apache/giraph/utils/ByteStructIterator.java
          • giraph-core/src/main/java/org/apache/giraph/utils/ByteArrayVertexIdMessages.java
          hudson Hudson added a comment - ABORTED: Integrated in Giraph-trunk-Commit #1444 (See https://builds.apache.org/job/Giraph-trunk-Commit/1444/ ) GIRAPH-907 : refactor giraph code to support multiple implementations of vertexId data (pavanka) (pavanka: http://git-wip-us.apache.org/repos/asf?p=giraph.git&a=commit&h=535a333b7776f7a229b76c7656e928d8a21d21bc ) giraph-core/src/main/java/org/apache/giraph/comm/requests/WritableRequest.java giraph-core/src/main/java/org/apache/giraph/utils/ByteUtils.java CHANGELOG giraph-core/src/main/java/org/apache/giraph/utils/ByteStructVertexIdMessageBytesIterator.java giraph-core/src/main/java/org/apache/giraph/utils/UnsafeArrayReads.java giraph-core/src/main/java/org/apache/giraph/utils/ByteStructVertexIdIterator.java giraph-core/src/main/java/org/apache/giraph/comm/SendMessageCache.java giraph-core/src/main/java/org/apache/giraph/comm/netty/NettyWorkerClientRequestProcessor.java giraph-core/src/main/java/org/apache/giraph/comm/messages/ByteArrayMessagesPerVertexStore.java giraph-core/src/main/java/org/apache/giraph/utils/ByteStructVertexIdDataIterator.java giraph-core/src/main/java/org/apache/giraph/utils/ExtendedDataOutput.java giraph-core/src/main/java/org/apache/giraph/comm/messages/OneMessagePerVertexStore.java giraph-core/src/main/java/org/apache/giraph/comm/messages/primitives/IntByteArrayMessageStore.java giraph-core/src/main/java/org/apache/giraph/utils/VertexIdEdgeIterator.java giraph-core/src/main/java/org/apache/giraph/edge/primitives/IntEdgeStore.java giraph-core/src/main/java/org/apache/giraph/utils/ByteStructIterable.java giraph-core/src/main/java/org/apache/giraph/utils/UnsafeByteArrayOutputStream.java giraph-core/src/main/java/org/apache/giraph/utils/ByteArrayIterator.java giraph-core/src/main/java/org/apache/giraph/comm/netty/handler/RequestServerHandler.java giraph-core/src/main/java/org/apache/giraph/comm/requests/SendWorkerEdgesRequest.java giraph-core/src/main/java/org/apache/giraph/utils/ByteArrayVertexIdData.java giraph-core/src/main/java/org/apache/giraph/edge/SimpleEdgeStore.java giraph-core/src/main/java/org/apache/giraph/utils/ByteArrayIterable.java giraph-core/src/main/java/org/apache/giraph/conf/GiraphConstants.java giraph-core/src/main/java/org/apache/giraph/comm/messages/out_of_core/DiskBackedMessageStore.java giraph-core/src/main/java/org/apache/giraph/comm/SendVertexIdDataCache.java giraph-core/src/main/java/org/apache/giraph/utils/ExtendedByteArrayDataOutput.java giraph-core/src/main/java/org/apache/giraph/comm/netty/handler/RequestDecoder.java giraph-core/src/main/java/org/apache/giraph/edge/EdgeStore.java giraph-core/src/main/java/org/apache/giraph/utils/ExtendedByteArrayDataInput.java giraph-core/src/main/java/org/apache/giraph/utils/VerboseByteArrayMessageWrite.java giraph-core/src/main/java/org/apache/giraph/comm/netty/handler/ResponseEncoder.java giraph-core/src/main/java/org/apache/giraph/comm/messages/primitives/LongByteArrayMessageStore.java giraph-core/src/main/java/org/apache/giraph/utils/ByteArrayVertexIdEdges.java giraph-core/src/main/java/org/apache/giraph/utils/RepresentativeByteArrayIterable.java giraph-core/src/main/java/org/apache/giraph/utils/RepresentativeByteStructIterator.java giraph-core/src/main/java/org/apache/giraph/utils/VertexIdMessageIterator.java giraph-core/src/main/java/org/apache/giraph/comm/messages/primitives/IntFloatMessageStore.java giraph-core/src/main/java/org/apache/giraph/utils/AbstractVertexIdData.java giraph-core/src/main/java/org/apache/giraph/conf/GiraphConfiguration.java giraph-core/src/main/java/org/apache/giraph/conf/ImmutableClassesGiraphConfiguration.java giraph-core/src/main/java/org/apache/giraph/utils/RepresentativeByteArrayIterator.java giraph-core/src/main/java/org/apache/giraph/comm/SendMessageToAllCache.java giraph-core/src/main/java/org/apache/giraph/utils/UnsafeByteArrayInputStream.java giraph-core/src/main/java/org/apache/giraph/utils/RepresentativeByteStructIterable.java giraph-core/src/test/java/org/apache/giraph/utils/MockUtils.java giraph-core/src/main/java/org/apache/giraph/utils/ByteStructVertexIdEdgeIterator.java giraph-core/src/main/java/org/apache/giraph/comm/requests/SendWorkerDataRequest.java giraph-core/src/main/java/org/apache/giraph/comm/messages/MessagesIterable.java giraph-core/src/main/java/org/apache/giraph/comm/messages/primitives/LongDoubleMessageStore.java giraph-core/src/main/java/org/apache/giraph/utils/VertexIdDataIterator.java giraph-core/src/main/java/org/apache/giraph/utils/VertexIdEdges.java giraph-core/src/test/java/org/apache/giraph/comm/RequestFailureTest.java giraph-core/src/main/java/org/apache/giraph/utils/VertexIdMessages.java giraph-core/src/main/java/org/apache/giraph/edge/primitives/LongEdgeStore.java giraph-core/src/main/java/org/apache/giraph/comm/messages/MessageStore.java giraph-core/src/main/java/org/apache/giraph/utils/VerboseByteStructMessageWrite.java giraph-core/src/main/java/org/apache/giraph/utils/RequestUtils.java giraph-core/src/test/java/org/apache/giraph/comm/RequestTest.java giraph-core/src/main/java/org/apache/giraph/utils/VertexIdIterator.java giraph-core/src/main/java/org/apache/giraph/utils/UnsafeReads.java giraph-core/src/main/java/org/apache/giraph/utils/VertexIdData.java giraph-core/src/main/java/org/apache/giraph/comm/netty/InboundByteCounter.java giraph-core/src/main/java/org/apache/giraph/edge/AbstractEdgeStore.java giraph-core/src/main/java/org/apache/giraph/utils/ByteStructVertexIdMessageIterator.java giraph-core/src/main/java/org/apache/giraph/utils/VertexIdMessageBytesIterator.java giraph-core/src/main/java/org/apache/giraph/comm/requests/SendWorkerMessagesRequest.java giraph-core/src/main/java/org/apache/giraph/comm/netty/handler/RequestEncoder.java giraph-core/src/main/java/org/apache/giraph/comm/SendEdgeCache.java giraph-core/src/main/java/org/apache/giraph/utils/ByteStructIterator.java giraph-core/src/main/java/org/apache/giraph/utils/ByteArrayVertexIdMessages.java

          People

            pavanka Pavan Kumar
            pavanka Pavan Kumar
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: