Hama
  1. Hama
  2. HAMA-704

Optimization of memory usage during message processing

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.6.1
    • Component/s: graph
    • Labels:
      None

      Description

      <vertex, message> map seems consume a lot of memory. We should figure out an efficient way to reduce memory.

      1. mytest.patch
        1.0 kB
        Edward J. Yoon
      2. patch.txt
        3 kB
        Edward J. Yoon
      3. removeMsgMap.patch
        17 kB
        Edward J. Yoon
      4. patch.txt
        22 kB
        Edward J. Yoon
      5. hama-704_v05.patch
        26 kB
        Edward J. Yoon
      6. localdisk.patch
        24 kB
        Edward J. Yoon
      7. HAMA-704.patch-v1
        42 kB
        Suraj Menon
      8. HAMA-704-v2.patch
        44 kB
        Suraj Menon
      9. HAMA-704_1337.patch
        98 kB
        Thomas Jungblut
      10. HAMA-704_1338.patch
        98 kB
        Thomas Jungblut
      11. HAMA-704_1339.patch
        103 kB
        Thomas Jungblut

        Issue Links

          Activity

          Edward J. Yoon created issue -
          Suraj Menon made changes -
          Field Original Value New Value
          Link This issue depends on HAMA-559 [ HAMA-559 ]
          Suraj Menon made changes -
          Link This issue relates to HAMA-706 [ HAMA-706 ]
          Hide
          Edward J. Yoon added a comment -

          The parseMessages() method parses 'GraphJobMessage' object to extract vertexID and the messages sent from other vertices in the previous iteration, and finally returns the HashMap <VertexID, Messages>. Then,

          for(Map.entry<VertexID, MessagesList> e : msgMap) {
            vertices.get(e.getKey()).compute(e.getValue());
          }
          

          Thus, the Previous messages and new messages to be sent to next iteration should be all in memory, during computing every vertices.

          My simple idea is the use of local disk. Then, memory usage will be reduced by between 40% to 50%.

          Show
          Edward J. Yoon added a comment - The parseMessages() method parses 'GraphJobMessage' object to extract vertexID and the messages sent from other vertices in the previous iteration, and finally returns the HashMap <VertexID, Messages>. Then, for (Map.entry<VertexID, MessagesList> e : msgMap) { vertices.get(e.getKey()).compute(e.getValue()); } Thus, the Previous messages and new messages to be sent to next iteration should be all in memory, during computing every vertices. My simple idea is the use of local disk. Then, memory usage will be reduced by between 40% to 50%.
          Hide
          Edward J. Yoon added a comment -

          Fortunately, write once read once.

          Show
          Edward J. Yoon added a comment - Fortunately, write once read once.
          Hide
          Edward J. Yoon added a comment -

          Hmm, seems not easy.

          Show
          Edward J. Yoon added a comment - Hmm, seems not easy.
          Hide
          Edward J. Yoon added a comment -
          Index: src/main/java/org/apache/hama/graph/GraphJobRunner.java
          ===================================================================
          --- src/main/java/org/apache/hama/graph/GraphJobRunner.java	(revision 1427035)
          +++ src/main/java/org/apache/hama/graph/GraphJobRunner.java	(working copy)
          @@ -179,6 +179,7 @@
                 BSPPeer<Writable, Writable, Writable, Writable, GraphJobMessage> peer)
                 throws IOException {
               int activeVertices = 0;
          +    int i = 0;
               for (Vertex<V, E, M> vertex : vertices) {
                 List<M> msgs = messages.get(vertex.getVertexID());
                 // If there are newly received messages, restart.
          @@ -202,6 +203,12 @@
                     activeVertices++;
                   }
                 }
          +      
          +      messages.remove(vertex.getVertexID());
          +      if((i % (vertices.size() / 10)) == 0) {
          +        System.gc();
          +      }
          +      i++;
               }
           
               aggregationRunner.sendAggregatorValues(peer, activeVertices);
          
          Show
          Edward J. Yoon added a comment - Index: src/main/java/org/apache/hama/graph/GraphJobRunner.java =================================================================== --- src/main/java/org/apache/hama/graph/GraphJobRunner.java (revision 1427035) +++ src/main/java/org/apache/hama/graph/GraphJobRunner.java (working copy) @@ -179,6 +179,7 @@ BSPPeer<Writable, Writable, Writable, Writable, GraphJobMessage> peer) throws IOException { int activeVertices = 0; + int i = 0; for (Vertex<V, E, M> vertex : vertices) { List<M> msgs = messages.get(vertex.getVertexID()); // If there are newly received messages, restart. @@ -202,6 +203,12 @@ activeVertices++; } } + + messages.remove(vertex.getVertexID()); + if ((i % (vertices.size() / 10)) == 0) { + System .gc(); + } + i++; } aggregationRunner.sendAggregatorValues(peer, activeVertices);
          Hide
          Edward J. Yoon added a comment -

          This looks helpful. But I'm not sure whether this is good idea. Please someone can review this?

          Show
          Edward J. Yoon added a comment - This looks helpful. But I'm not sure whether this is good idea. Please someone can review this?
          Edward J. Yoon made changes -
          Attachment mytest.patch [ 12563934 ]
          Hide
          Suraj Menon added a comment -

          First of all, our goal should be to reduce GC overhead and limit memory usage.
          Secondly, System.gc is not reliable, it just improves the odds. [Item 7 of "Effective Java" Second Edition].

          Show
          Suraj Menon added a comment - First of all, our goal should be to reduce GC overhead and limit memory usage. Secondly, System.gc is not reliable, it just improves the odds. [Item 7 of "Effective Java" Second Edition] .
          Hide
          Edward J. Yoon added a comment -

          You're right. System.gc seems not a fundamental solution.

          Show
          Edward J. Yoon added a comment - You're right. System.gc seems not a fundamental solution.
          Edward J. Yoon made changes -
          Fix Version/s 0.6.1 [ 12322959 ]
          Hide
          Edward J. Yoon added a comment -

          Tested local disk version and System.gc on 32 nodes cluster.

          It seems no way to avoid creation of memory garbage for grouping messages by vertexID. So, writing parsed message to disk is not helpful.

          On the other hand, System.gc calling was helpful. Should we close this issue as a 'won't fix'?

          Show
          Edward J. Yoon added a comment - Tested local disk version and System.gc on 32 nodes cluster. It seems no way to avoid creation of memory garbage for grouping messages by vertexID. So, writing parsed message to disk is not helpful. On the other hand, System.gc calling was helpful. Should we close this issue as a 'won't fix'?
          Edward J. Yoon made changes -
          Priority Minor [ 4 ] Critical [ 2 ]
          Hide
          Edward J. Yoon added a comment -

          I'm thinking about external merge sort http://en.wikipedia.org/wiki/External_sorting

          Show
          Edward J. Yoon added a comment - I'm thinking about external merge sort http://en.wikipedia.org/wiki/External_sorting
          Hide
          Edward J. Yoon added a comment -

          Once HAMA-723 is done, we can sort messages by vertex ID. Then, we can read messages of vertex and perform vertex.compute() sequentially.

          Show
          Edward J. Yoon added a comment - Once HAMA-723 is done, we can sort messages by vertex ID. Then, we can read messages of vertex and perform vertex.compute() sequentially.
          Edward J. Yoon made changes -
          Attachment patch.txt [ 12567805 ]
          Hide
          Edward J. Yoon added a comment -

          NOTE: add MessageIterator<M> to avoid creation of iterator objects.

          Show
          Edward J. Yoon added a comment - NOTE: add MessageIterator<M> to avoid creation of iterator objects.
          Hide
          Suraj Menon added a comment -

          You are right, for the graph job, sorting messages would make it easier to join the messages per vertex reading them using SpilledInputBuffer. So if you are sure that the messages could be held in memory, you can use a spilling buffer inside the VerticeInfo class for storing the vertex messages. Please use the spilling buffer implementation from HAMA-721 patch for performance.

          Show
          Suraj Menon added a comment - You are right, for the graph job, sorting messages would make it easier to join the messages per vertex reading them using SpilledInputBuffer. So if you are sure that the messages could be held in memory, you can use a spilling buffer inside the VerticeInfo class for storing the vertex messages. Please use the spilling buffer implementation from HAMA-721 patch for performance.
          Hide
          Edward J. Yoon added a comment -

          This patch will reduce memory usage dramatically.

          Show
          Edward J. Yoon added a comment - This patch will reduce memory usage dramatically.
          Edward J. Yoon made changes -
          Attachment removeMsgMap.patch [ 12567937 ]
          Edward J. Yoon made changes -
          Fix Version/s 0.6.1 [ 12322959 ]
          Hide
          Edward J. Yoon added a comment -

          I was changed vertices map to list for reducing memory usage but would like to change again. It looks like insignificant.

          Show
          Edward J. Yoon added a comment - I was changed vertices map to list for reducing memory usage but would like to change again. It looks like insignificant.
          Hide
          Edward J. Yoon added a comment -

          Very slow :/

          Show
          Edward J. Yoon added a comment - Very slow :/
          Hide
          Edward J. Yoon added a comment -

          It looks like need to use multithreading.

          Show
          Edward J. Yoon added a comment - It looks like need to use multithreading.
          Hide
          Edward J. Yoon added a comment -

          NOTE: if (!aggregationRunner.receiveAggregatedValues(peer, iteration)) and initVertices()

          Show
          Edward J. Yoon added a comment - NOTE: if (!aggregationRunner.receiveAggregatedValues(peer, iteration)) and initVertices()
          Hide
          Edward J. Yoon added a comment - - edited

          Comparing was the reason of slow performance. Below will improve the performance.

            @Override
            public int compareTo(GraphJobMessage that) {
              if (this.flag != that.flag) {
                return (this.flag < that.flag) ? -1 : 1;
              } else {
                if (this.isVertexMessage()) {
                  return this.getVertexId().toString()
                      .compareTo(that.getVertexId().toString());
                } else {
                  return 0;
                }
              }
            }
          
          Show
          Edward J. Yoon added a comment - - edited Comparing was the reason of slow performance. Below will improve the performance. @Override public int compareTo(GraphJobMessage that) { if ( this .flag != that.flag) { return ( this .flag < that.flag) ? -1 : 1; } else { if ( this .isVertexMessage()) { return this .getVertexId().toString() .compareTo(that.getVertexId().toString()); } else { return 0; } } }
          Hide
          Suraj Menon added a comment -

          Hi Edward, yes that was one thing which I wanted to suggest, but again, please don't compare on toString(). WritableComparable extends Writable. This means we can change V implements WritableComparable. I am afraid, toString().compare is not a good idea.

          ((new IntWritable(11)).compare(new IntWritable(2))) != ((new IntWritable(11)).toString()).compare(new IntWritable(2)).toString())
          

          Also, on performance, let's decide based on data. We need to check how much GC time did Hama-0.6 release need. And with serialized storage, have we reduced the same.
          Remember right now we are writing and reading message twice on both sender and receiver side. Once we make it single write on sender side and single read on receiver side, we should be able to improve numbers there.

          Show
          Suraj Menon added a comment - Hi Edward, yes that was one thing which I wanted to suggest, but again, please don't compare on toString(). WritableComparable extends Writable. This means we can change V implements WritableComparable. I am afraid, toString().compare is not a good idea. ((new IntWritable(11)).compare(new IntWritable(2))) != ((new IntWritable(11)).toString()).compare(new IntWritable(2)).toString()) Also, on performance, let's decide based on data. We need to check how much GC time did Hama-0.6 release need. And with serialized storage, have we reduced the same. Remember right now we are writing and reading message twice on both sender and receiver side. Once we make it single write on sender side and single read on receiver side, we should be able to improve numbers there.
          Hide
          Edward J. Yoon added a comment -

          I see, I'll change V as a WritableComparable.

          Show
          Edward J. Yoon added a comment - I see, I'll change V as a WritableComparable.
          Hide
          Edward J. Yoon added a comment -

          Attach my current status.

          I'll be vacation 6th ~ 12th (Korean Holiday).

          Show
          Edward J. Yoon added a comment - Attach my current status. I'll be vacation 6th ~ 12th (Korean Holiday).
          Edward J. Yoon made changes -
          Attachment patch.txt [ 12568003 ]
          Hide
          Edward J. Yoon added a comment - - edited

          I'm reading old code closely. We're synchronize additionally to check whether updated vertices are exist. So, the total superstep was 2 times of algorithm iteration. With sorted message queue, we can reduce synchronization times by half.

          I think, my patch will improve both the performance and memory usage.

          Show
          Edward J. Yoon added a comment - - edited I'm reading old code closely. We're synchronize additionally to check whether updated vertices are exist. So, the total superstep was 2 times of algorithm iteration. With sorted message queue, we can reduce synchronization times by half. I think, my patch will improve both the performance and memory usage.
          Hide
          Edward J. Yoon added a comment -

          attach my changes.

          Show
          Edward J. Yoon added a comment - attach my changes.
          Edward J. Yoon made changes -
          Attachment hama-704_v05.patch [ 12569169 ]
          Hide
          Edward J. Yoon added a comment -

          I'll evaluate/commit this tomorrow.

          Show
          Edward J. Yoon added a comment - I'll evaluate/commit this tomorrow.
          Hide
          Edward J. Yoon added a comment -

          Reading master aggregation code :/ I need more time.

          Show
          Edward J. Yoon added a comment - Reading master aggregation code :/ I need more time.
          Hide
          Edward J. Yoon added a comment -

          My another test patch.

          Show
          Edward J. Yoon added a comment - My another test patch.
          Edward J. Yoon made changes -
          Attachment localdisk.patch [ 12569340 ]
          Hide
          Suraj Menon added a comment -

          I suggest we leave this at having a TreeSet<Vertex> and SortedMessageQueue for messages and have an Iterator instance that goes over these two sorted queues in order grouping the vertices and messages. We can avoid reading the messages twice with HAMA-734. Memory usage then can be reduced with sorted spilling queues.

          Show
          Suraj Menon added a comment - I suggest we leave this at having a TreeSet<Vertex> and SortedMessageQueue for messages and have an Iterator instance that goes over these two sorted queues in order grouping the vertices and messages. We can avoid reading the messages twice with HAMA-734 . Memory usage then can be reduced with sorted spilling queues.
          Hide
          Edward J. Yoon added a comment -

          Yes, that's what I doing now. BTW, TreeSet<Vertex> for what?

          Show
          Edward J. Yoon added a comment - Yes, that's what I doing now. BTW, TreeSet<Vertex> for what?
          Hide
          Edward J. Yoon added a comment - - edited

          My question is whether this is really helpful. According to my simple Java test, it looks like helpful. 1) program throws OOM exception but 2) is ok.

          // 1. current TRUNK
          
          Map<Integer, List<Integer>> map;
          for(int i = 0; i < 1000000000; i++) {
            msg = new ArrayList<Integer>();
            for(int j = 0; j < 100000; j++) {
              msg.add(j);
            }
          
            map.put(i, msg);
          }
          

          vs.

          // 1. call compute() within loop
          
          for(int i = 0; i < 1000000000; i++) {
            msg = new ArrayList<Integer>();
            for(int j = 0; j < 100000; j++) {
              msg.add(j);
            }
          
            // i.compute(msg);
          }
          
          

          And, I couldn't test with huge graph yet but, performance will be decreased by cost of sort.

          With my single machine, Pagerank on 1,600,000 edges graph:

          TRUNK 12 secs
          TRUNK + SortedMessageQueue 39 secs

          My patch and + SortedMessageQueue 39 secs.

          Show
          Edward J. Yoon added a comment - - edited My question is whether this is really helpful. According to my simple Java test, it looks like helpful. 1) program throws OOM exception but 2) is ok. // 1. current TRUNK Map< Integer , List< Integer >> map; for ( int i = 0; i < 1000000000; i++) { msg = new ArrayList< Integer >(); for ( int j = 0; j < 100000; j++) { msg.add(j); } map.put(i, msg); } vs. // 1. call compute() within loop for ( int i = 0; i < 1000000000; i++) { msg = new ArrayList< Integer >(); for ( int j = 0; j < 100000; j++) { msg.add(j); } // i.compute(msg); } And, I couldn't test with huge graph yet but, performance will be decreased by cost of sort. With my single machine, Pagerank on 1,600,000 edges graph: TRUNK 12 secs TRUNK + SortedMessageQueue 39 secs – My patch and + SortedMessageQueue 39 secs.
          Hide
          Suraj Menon added a comment -

          Yes! Sorting would take more time, especially till messaging is synchronous.
          I am not an expert, but can we call compute before aggregation by master?

          As we decided, you can check this on the new branch.

          On a side note, have you profiled and checked the number of objects created in the process? Every compare in SortedQueue involves creating new String object. Can you check if removing and doing VertexId.compare(otherVertexId) makes any difference.

          I would suggest that the requirements get defined here. The design should get things to work on 16GB 400 node cluster as well as a 4GB 15 node cluster. In any scenario, if we reach the main memory limit, we would have to spill to disk or find some other modes of persistence.

          Show
          Suraj Menon added a comment - Yes! Sorting would take more time, especially till messaging is synchronous. I am not an expert, but can we call compute before aggregation by master? As we decided, you can check this on the new branch. On a side note, have you profiled and checked the number of objects created in the process? Every compare in SortedQueue involves creating new String object. Can you check if removing and doing VertexId.compare(otherVertexId) makes any difference. I would suggest that the requirements get defined here. The design should get things to work on 16GB 400 node cluster as well as a 4GB 15 node cluster. In any scenario, if we reach the main memory limit, we would have to spill to disk or find some other modes of persistence.
          Hide
          Edward J. Yoon added a comment -

          Yes, if aggregationRunner is enabled, it looks like there's no way to avoid additional synchronization.

          Show
          Edward J. Yoon added a comment - Yes, if aggregationRunner is enabled, it looks like there's no way to avoid additional synchronization.
          Hide
          Edward J. Yoon added a comment -

          P.S., Current graph job uses too much memory space. http://wiki.apache.org/hama/Benchmarks was maximum on 2 Oracle BDA full rack :/ I think if this is right way, we should accept slower performance for reducing memory usage.

          Show
          Edward J. Yoon added a comment - P.S., Current graph job uses too much memory space. http://wiki.apache.org/hama/Benchmarks was maximum on 2 Oracle BDA full rack :/ I think if this is right way, we should accept slower performance for reducing memory usage.
          Hide
          Edward J. Yoon added a comment -

          >>call compute before aggregation by master

          Sequence doesn't matter. The problem is additional sync between every vertex computing supersteps.

          Show
          Edward J. Yoon added a comment - >>call compute before aggregation by master Sequence doesn't matter. The problem is additional sync between every vertex computing supersteps.
          Hide
          Edward J. Yoon added a comment -

          The design should get things to work on 16GB 400 node cluster as well as a 4GB 15 node cluster.

          Suraj, I don't want to consider such extreme scenarios at the moment.

          Suraj: so I wanted to have a brief discussion about the whole matter
          there is no escape .. we have to think about persisting the vertices outiside main memory
          4:51 PM
          Suraj: that is why Thomas was thinking of making things Disk based.

          As I told you before, each Task loads 256MB vertices and uses 30GB memory. I don't know why you think "disk-based vertices" is helpful? Memory usage is continuously increased by vertex.setValue()?

          Show
          Edward J. Yoon added a comment - The design should get things to work on 16GB 400 node cluster as well as a 4GB 15 node cluster. Suraj, I don't want to consider such extreme scenarios at the moment. Suraj: so I wanted to have a brief discussion about the whole matter there is no escape .. we have to think about persisting the vertices outiside main memory 4:51 PM Suraj: that is why Thomas was thinking of making things Disk based. As I told you before, each Task loads 256MB vertices and uses 30GB memory. I don't know why you think "disk-based vertices" is helpful? Memory usage is continuously increased by vertex.setValue()?
          Hide
          Edward J. Yoon added a comment -

          Another question, can vertexID in GraphJobMessage become a WritableComparable without interface change?

          Show
          Edward J. Yoon added a comment - Another question, can vertexID in GraphJobMessage become a WritableComparable without interface change?
          Hide
          Thomas Jungblut added a comment - - edited

          Edward,

          a single vertex is composed out of multiple objects:

          • Vertex ID
          • Vertex Value
          • A List for the outgoing edges (containing default array of 10 objects)
          • Plus 3 objects per Edge (Edge itself, ID and Value).

          Let's sum the memory together that is only the Java object overhead (8 bytes per object[1]).

          5 * 8 (for the private references, including ref to self) + n * 8 * 3 (for the edges) = 24 * n + 40 bytes.

          So you have 40 bytes for each vertex and 24 bytes for each edge ONLY object overhead. If you want to assume that everything is an IntWritable (4 byte core size), you can calculate for yourself how much memory is occupied by all vertices with a sparse graph of 4 outlinks of each vertex. (Tip, it is nearly a KB per vertex).

          I don't know why you think "disk-based vertices" is helpful?

          It is helpful, because you can just hold a single vertex in memory and refill it constantly with data from disk. The problem Pregel has, is that vertex/edge values can (should) change during computation. So you would either have to write all the file completely in every iteration (that is mapreduce without sorting), or take multiple files and and constantly rewrite a "value file" that contains only the vertex values in a smaller file. On SSD systems (rarely used, but your BDA has them?) you can use a random access file to directly change the value (that is what graphchi does), this file can be mmap'ed by the OS to main memory everytime it is needed and if enough memory is available.

          The question is also, if memory should only be allocated for fast messaging and the computation can completely be done on disk with a cache for frequently active vertices or if everything should be in memory first and spilled if needed.

          Good luck with that.

          [1] http://stackoverflow.com/questions/258120/what-is-the-memory-consumption-of-an-object-in-java

          Show
          Thomas Jungblut added a comment - - edited Edward, a single vertex is composed out of multiple objects: Vertex ID Vertex Value A List for the outgoing edges (containing default array of 10 objects) Plus 3 objects per Edge (Edge itself, ID and Value). Let's sum the memory together that is only the Java object overhead (8 bytes per object [1] ). 5 * 8 (for the private references, including ref to self) + n * 8 * 3 (for the edges) = 24 * n + 40 bytes. So you have 40 bytes for each vertex and 24 bytes for each edge ONLY object overhead. If you want to assume that everything is an IntWritable (4 byte core size), you can calculate for yourself how much memory is occupied by all vertices with a sparse graph of 4 outlinks of each vertex. (Tip, it is nearly a KB per vertex ). I don't know why you think "disk-based vertices" is helpful? It is helpful, because you can just hold a single vertex in memory and refill it constantly with data from disk. The problem Pregel has, is that vertex/edge values can (should) change during computation. So you would either have to write all the file completely in every iteration (that is mapreduce without sorting), or take multiple files and and constantly rewrite a "value file" that contains only the vertex values in a smaller file. On SSD systems (rarely used, but your BDA has them?) you can use a random access file to directly change the value (that is what graphchi does), this file can be mmap'ed by the OS to main memory everytime it is needed and if enough memory is available. The question is also, if memory should only be allocated for fast messaging and the computation can completely be done on disk with a cache for frequently active vertices or if everything should be in memory first and spilled if needed. Good luck with that. [1] http://stackoverflow.com/questions/258120/what-is-the-memory-consumption-of-an-object-in-java
          Hide
          Edward J. Yoon added a comment -

          Thanks for your kind explanation. However, those objects are fixed and won't continually eating more memory right?

          Show
          Edward J. Yoon added a comment - Thanks for your kind explanation. However, those objects are fixed and won't continually eating more memory right?
          Hide
          Thomas Jungblut added a comment -

          What you describe sounds like a memory leak. However it is difficult to tell if that is merely a coincidence and the messages suck up all the memory or if there is a memory leak in the vertex.

          So the safest bet would be to turn on a dummy message manager that discards all messages and constantly sends a small amount of messages to random vertices. If you then see a memory increase (with a profiler, not with top) while computation there is a leak.

          Show
          Thomas Jungblut added a comment - What you describe sounds like a memory leak. However it is difficult to tell if that is merely a coincidence and the messages suck up all the memory or if there is a memory leak in the vertex. So the safest bet would be to turn on a dummy message manager that discards all messages and constantly sends a small amount of messages to random vertices. If you then see a memory increase (with a profiler, not with top) while computation there is a leak.
          Hide
          Suraj Menon added a comment -

          The attachment has a solution for HAMA-734. The advantages/disadvantages of sorting is also shown. The tests pass except for in PageRankVertex. I see messages coming for vertexes not present. I can look into it. Please let me know about using BSPMessageCoupledStorage. If we accept this, the problem strictly becomes a storage issue between vertices and messages.

          Show
          Suraj Menon added a comment - The attachment has a solution for HAMA-734 . The advantages/disadvantages of sorting is also shown. The tests pass except for in PageRankVertex. I see messages coming for vertexes not present. I can look into it. Please let me know about using BSPMessageCoupledStorage. If we accept this, the problem strictly becomes a storage issue between vertices and messages.
          Suraj Menon made changes -
          Attachment HAMA-704.patch-v1 [ 12569843 ]
          Hide
          Edward J. Yoon added a comment -

          Vertices in-memory and objects overhead are acceptable to me. Let's don't think about it at the moment.

          Show
          Edward J. Yoon added a comment - Vertices in-memory and objects overhead are acceptable to me. Let's don't think about it at the moment.
          Hide
          Suraj Menon added a comment -

          Reading over the conversation between you and Thomas again and from my experience today, do you think the Map<Vertex, List<M>> is growing because non-existent(the vertices that belong to some other peers) vertices are getting added as program executes. This could be easily found by keeping a tab on the key count in the HashMap.

          Show
          Suraj Menon added a comment - Reading over the conversation between you and Thomas again and from my experience today, do you think the Map<Vertex, List<M>> is growing because non-existent(the vertices that belong to some other peers) vertices are getting added as program executes. This could be easily found by keeping a tab on the key count in the HashMap.
          Hide
          Edward J. Yoon added a comment -

          The problem is that we don't know the Max of Map<Vertex, List<M>>.

          Show
          Edward J. Yoon added a comment - The problem is that we don't know the Max of Map<Vertex, List<M>>.
          Hide
          Suraj Menon added a comment -

          I am sorry, I take back my comments; looking at why binary search is failing eventhough the element is in the list.

          Show
          Suraj Menon added a comment - I am sorry, I take back my comments; looking at why binary search is failing eventhough the element is in the list.
          Hide
          Edward J. Yoon added a comment -

          If vertices and its objects overhead are calculable, that's acceptable to me. Maybe we can think about it later.

          "each Task loads 256MB vertices and uses 30GB memory" -> means that Max 30GB to finish the whole job. Mem usage graph was notched.

          Show
          Edward J. Yoon added a comment - If vertices and its objects overhead are calculable, that's acceptable to me. Maybe we can think about it later. "each Task loads 256MB vertices and uses 30GB memory" -> means that Max 30GB to finish the whole job. Mem usage graph was notched.
          Hide
          Thomas Jungblut added a comment -

          The problem is that we don't know the Max of Map<Vertex, List<M>>.

          You can certainly run every vertex id (in the vertexmap) through the partitioner and check if the peer index is the same. Once known you know that this is the max, so you can track throughout the supersteps. But vertices usually don't add to the map after partitioning, you can make the map/list/set unmodifiable.
          If you step over some vertices that doesn't belong there, kudos to the partitioning voodoo.

          In case of the messaging map, it is discarded every superstep so I don't think that creates memory leaks- at least lots of garbage (only if the vertex implementation itself starts to buffer messages on it's own so the references can't escape the GC).

          "each Task loads 256MB vertices and uses 30GB memory"

          How is the outlink distribution? I don't think that this is a scenario of typical powerlaw distributions here. Usually the factor was x10, so 256mb sequence file create 2gigs of memory. Which is huge, bit still acceptable per task.

          Show
          Thomas Jungblut added a comment - The problem is that we don't know the Max of Map<Vertex, List<M>>. You can certainly run every vertex id (in the vertexmap) through the partitioner and check if the peer index is the same. Once known you know that this is the max, so you can track throughout the supersteps. But vertices usually don't add to the map after partitioning, you can make the map/list/set unmodifiable. If you step over some vertices that doesn't belong there, kudos to the partitioning voodoo. In case of the messaging map, it is discarded every superstep so I don't think that creates memory leaks- at least lots of garbage (only if the vertex implementation itself starts to buffer messages on it's own so the references can't escape the GC). "each Task loads 256MB vertices and uses 30GB memory" How is the outlink distribution? I don't think that this is a scenario of typical powerlaw distributions here. Usually the factor was x10, so 256mb sequence file create 2gigs of memory. Which is huge, bit still acceptable per task.
          Hide
          Edward J. Yoon added a comment -

          Helpful comments. I was mean the max space of Map<Vertex, List<M>>. The max number of vertex and its messages can be calculable but, max space is not if message is flexible type such as Text or Array.

          Show
          Edward J. Yoon added a comment - Helpful comments. I was mean the max space of Map<Vertex, List<M>>. The max number of vertex and its messages can be calculable but, max space is not if message is flexible type such as Text or Array.
          Hide
          Edward J. Yoon added a comment -

          So, I expect that we can reduce memory consumption by removing Map<Vertex, List<M>> and using of spilling queue.

          Show
          Edward J. Yoon added a comment - So, I expect that we can reduce memory consumption by removing Map<Vertex, List<M>> and using of spilling queue.
          Hide
          Thomas Jungblut added a comment - - edited

          If the incoming messages are already sorted by the vertex id from every peer, it is really easy to merge them together in linear runtime and on disk. Hadoop has quite good merging code in that regard, so I don't know if there is some effort in reusing the code.

          However, this won't fix the complete problem as the second half of the memory is occupied by the graph itself. Maybe HAMA-732 solves it efficiently.
          Oh and BTW, for fault tolerance the changing part of the graph must be persisted after every superstep anyways. So it is just a short-term decision not to save the graph onto a secondary storage.

          Show
          Thomas Jungblut added a comment - - edited If the incoming messages are already sorted by the vertex id from every peer, it is really easy to merge them together in linear runtime and on disk. Hadoop has quite good merging code in that regard, so I don't know if there is some effort in reusing the code. However, this won't fix the complete problem as the second half of the memory is occupied by the graph itself. Maybe HAMA-732 solves it efficiently. Oh and BTW, for fault tolerance the changing part of the graph must be persisted after every superstep anyways. So it is just a short-term decision not to save the graph onto a secondary storage.
          Hide
          Suraj Menon added a comment -

          I have to handle synchronization in the patch and hence the issue. I was not expecting this given receiver queue is synchronized. I hope everyone is fine with the idea of coupling message storage? Please let me know in HAMA-734. I shall get back to this after working on async messaging and sorting.

          Show
          Suraj Menon added a comment - I have to handle synchronization in the patch and hence the issue. I was not expecting this given receiver queue is synchronized. I hope everyone is fine with the idea of coupling message storage? Please let me know in HAMA-734 . I shall get back to this after working on async messaging and sorting.
          Hide
          Thomas Jungblut added a comment -

          So, I expect that we can reduce memory consumption by removing Map<Vertex, List<M>> and using of spilling queue.

          Another consumer of memory is the MapWritable in the GraphJobMessage. It would be nice if this can be replaced with a stream iterator once there is the means for sorted/merged queues. Then the messages can be iterated in ascending order of the vertexid and easily be joined with the vertices in whatever storage we have without the need for random access.

          Show
          Thomas Jungblut added a comment - So, I expect that we can reduce memory consumption by removing Map<Vertex, List<M>> and using of spilling queue. Another consumer of memory is the MapWritable in the GraphJobMessage. It would be nice if this can be replaced with a stream iterator once there is the means for sorted/merged queues. Then the messages can be iterated in ascending order of the vertexid and easily be joined with the vertices in whatever storage we have without the need for random access.
          Hide
          Suraj Menon added a comment -

          passes test. A decision has to be made whether to make GraphJobMessage a comparable. Only vertice graph job messages are comparable in my opinion.

          Show
          Suraj Menon added a comment - passes test. A decision has to be made whether to make GraphJobMessage a comparable. Only vertice graph job messages are comparable in my opinion.
          Suraj Menon made changes -
          Attachment HAMA-704-v2.patch [ 12570110 ]
          Hide
          Thomas Jungblut added a comment - - edited

          VertexID in the vertex must be comparable. That is actually enough for everything.

          I was just profiling the memory leak by using 1 mio. pagerank vertices and 10 edges each (50mb).
          Here is much more detailed memory analysis:

          After Reading Vertices to RAM in setup (Superstep2)

          600mb raw heap usage.
          418199256 bytes occupied by the vertices.
          287999928 bytes occupied by Text objects (used as Vertex Key 48000000 bytes, rest is edge bytes)
          237999192 bytes occupied by Edges (Text Objects and Null references)

          In the first superstep

          1,5gb heap
          Vertex memory keeps constant. Messages are as follows:
          5 mio. GraphJobMessages (only half of the out edges) 225mb. So with all messages, this sums up to a bit less than 500 mb (10 times the graph size!).
          Each vertex message contains ~40 bytes, 20 Text, 20 DoubleWritable.

          In the fourth superstep (of 6 in total)

          GC'd to 1,1GB again
          BSPMessageBundle contains 4,1 mio messages and is only one time in memory. However the linked list in that hashmap of the bundle contains 100 MB of data.
          Maybe we can switch to an arraylist again, they are much sparser in memory because they aren't doubly linked and we should release the reference of it once it is send via RPC.

          However, everything is collected properly, so there is no memory leak in my opinion.

          BTW: is it intended in the VerticesInfo to do a linear search for every vertex? That is slow like hell.

          Show
          Thomas Jungblut added a comment - - edited VertexID in the vertex must be comparable. That is actually enough for everything. I was just profiling the memory leak by using 1 mio. pagerank vertices and 10 edges each (50mb). Here is much more detailed memory analysis: After Reading Vertices to RAM in setup (Superstep2) 600mb raw heap usage. 418199256 bytes occupied by the vertices. 287999928 bytes occupied by Text objects (used as Vertex Key 48000000 bytes, rest is edge bytes) 237999192 bytes occupied by Edges (Text Objects and Null references) In the first superstep 1,5gb heap Vertex memory keeps constant. Messages are as follows: 5 mio. GraphJobMessages (only half of the out edges) 225mb. So with all messages, this sums up to a bit less than 500 mb (10 times the graph size!). Each vertex message contains ~40 bytes, 20 Text, 20 DoubleWritable. In the fourth superstep (of 6 in total) GC'd to 1,1GB again BSPMessageBundle contains 4,1 mio messages and is only one time in memory. However the linked list in that hashmap of the bundle contains 100 MB of data. Maybe we can switch to an arraylist again, they are much sparser in memory because they aren't doubly linked and we should release the reference of it once it is send via RPC. However, everything is collected properly, so there is no memory leak in my opinion. BTW: is it intended in the VerticesInfo to do a linear search for every vertex? That is slow like hell.
          Hide
          Suraj Menon added a comment -

          Are these numbers for trunk or with patch?

          > is it intended in the VerticesInfo to do a linear search for every vertex? That is slow like hell.
          Hence the Collections.sort and binarySearch. We need the check only for addition of vertices right?

          Show
          Suraj Menon added a comment - Are these numbers for trunk or with patch? > is it intended in the VerticesInfo to do a linear search for every vertex? That is slow like hell. Hence the Collections.sort and binarySearch. We need the check only for addition of vertices right?
          Hide
          Edward J. Yoon added a comment -

          The vertex.compute() should be executed even if there's no messages until it's halted by user.

          Show
          Edward J. Yoon added a comment - The vertex.compute() should be executed even if there's no messages until it's halted by user.
          Hide
          Thomas Jungblut added a comment -

          Are these numbers for trunk or with patch?

          With HAMA-735 on trunk. And I should note that this was also created with Java7. Anyway, it should yield us as baseline to detect improvements.

          Hence the Collections.sort and binarySearch. We need the check only for addition of vertices right?

          That would help in speed, I wonder if the partitioner/user should deduplicate the graph by IDs.

          The vertex.compute() should be executed even if there's no messages until it's halted by user.

          Exactly.

          Show
          Thomas Jungblut added a comment - Are these numbers for trunk or with patch? With HAMA-735 on trunk. And I should note that this was also created with Java7. Anyway, it should yield us as baseline to detect improvements. Hence the Collections.sort and binarySearch. We need the check only for addition of vertices right? That would help in speed, I wonder if the partitioner/user should deduplicate the graph by IDs. The vertex.compute() should be executed even if there's no messages until it's halted by user. Exactly.
          Hide
          Thomas Jungblut added a comment -

          So I want to share the overall plan now.

          Our requirements are:

          • Vertex and Edge Values (+possible user added state) change through every superstep
          • Vertex can be active or not, but if it is inactive it is just activated by a message until it is voteToHalt()
          • Be as memory savvy as possible, for future FT needs it is necessary to spill changing parts to disk to be restartable.

          So how can we tackle this with an architectecture?

          The graph part

          We divide the graph into two parts:
          1. a static graph that only contains the ID's they are sorted ascending by the vertex id, followed by it's outlink Ids. (VertexID1,EdgeID1,EdgeID2,VertexID2,EdgeID5 etc...).
          2. a changing graph (soft/liquid graph) that contains the soft attributes that change very often. It's format would be in a parallel fashion to the static graph so both can be read in paralle. (VertexValue1,EdgeValue1,EdgeValue2,VertexValue2,EdgeValue5 etc...)

          Both files are on disk, the static graph must be written while receiving the vertices from the partitioning + beeing sorted. The soft part is written everytime to disk (or directmemory or whatever sink defined by the VerticesInfo) during a superstep after the vertex computed its new state.

          Therefore we need some additional bookkeeping. For the activeness of vertex we can use a BitSet that has an index mapping (we can index-map everything because it is sorted), e.G. bit 1 is the vertex1 beeing active when set. To seek to the next active vertex without deserializing the whole graph, we need another index mapping by a long[] that contains the byte-offset where the values of a given vertex at the given index starts. This is majorly for seeking purposes, so we can seek to the next active vertex in the set without deserializing the between part of the graph. Anyways, this has really really low overhead in memory (pretty much none) and it is scalable like hell so we can make GraphChi look like a real loser.

          Suraj and me talked about it, and he is going to make the messaging more scalable- I focus on the graph storages according to the plan above. Our interface between the two is a sorted messaging (ascending by vertex ID) queue. As long as Suraj is working on it, I will use the SortedMessagingQueue in memory to simulate the behaviour.

          Show
          Thomas Jungblut added a comment - So I want to share the overall plan now. Our requirements are: Vertex and Edge Values (+possible user added state) change through every superstep Vertex can be active or not, but if it is inactive it is just activated by a message until it is voteToHalt() Be as memory savvy as possible, for future FT needs it is necessary to spill changing parts to disk to be restartable. So how can we tackle this with an architectecture? The graph part We divide the graph into two parts: 1. a static graph that only contains the ID's they are sorted ascending by the vertex id, followed by it's outlink Ids. (VertexID1,EdgeID1,EdgeID2,VertexID2,EdgeID5 etc...). 2. a changing graph (soft/liquid graph) that contains the soft attributes that change very often. It's format would be in a parallel fashion to the static graph so both can be read in paralle. (VertexValue1,EdgeValue1,EdgeValue2,VertexValue2,EdgeValue5 etc...) Both files are on disk, the static graph must be written while receiving the vertices from the partitioning + beeing sorted. The soft part is written everytime to disk (or directmemory or whatever sink defined by the VerticesInfo) during a superstep after the vertex computed its new state. Therefore we need some additional bookkeeping. For the activeness of vertex we can use a BitSet that has an index mapping (we can index-map everything because it is sorted), e.G. bit 1 is the vertex1 beeing active when set. To seek to the next active vertex without deserializing the whole graph, we need another index mapping by a long[] that contains the byte-offset where the values of a given vertex at the given index starts. This is majorly for seeking purposes, so we can seek to the next active vertex in the set without deserializing the between part of the graph. Anyways, this has really really low overhead in memory (pretty much none) and it is scalable like hell so we can make GraphChi look like a real loser. Suraj and me talked about it, and he is going to make the messaging more scalable- I focus on the graph storages according to the plan above. Our interface between the two is a sorted messaging (ascending by vertex ID) queue. As long as Suraj is working on it, I will use the SortedMessagingQueue in memory to simulate the behaviour.
          Hide
          Thomas Jungblut added a comment -

          First scratch in HAMA-704_1337.patch, everything within maven and locally works so far (even pagerank!).
          I will profile this tomorrow or a bit later today to see if it has given any major improvement on the RAM usage.

          Still I'm not completely sure about many things, do you think we should open a review board thread for it? Please let me know about suggestions and honest opinions you have.

          Also a small drawback: we can't skip reading inactive vertices every iteration because the values file must be rewritten using all vertex values (otherwise there are obviously strange results). However, I added the possibility to do so by a strategy pattern if needed or we have a better idea how to not rewrite the whole values every iteration.
          Also for aggregators to work correctly, I had to replay messages between the supersteps (between the master and the slave synchronization). Maybe we can get a better synchronization strategy that doesn't clear the queues so we don't have to loop-back them again for the next step.

          Show
          Thomas Jungblut added a comment - First scratch in HAMA-704 _1337.patch, everything within maven and locally works so far (even pagerank!). I will profile this tomorrow or a bit later today to see if it has given any major improvement on the RAM usage. Still I'm not completely sure about many things, do you think we should open a review board thread for it? Please let me know about suggestions and honest opinions you have. Also a small drawback: we can't skip reading inactive vertices every iteration because the values file must be rewritten using all vertex values (otherwise there are obviously strange results). However, I added the possibility to do so by a strategy pattern if needed or we have a better idea how to not rewrite the whole values every iteration. Also for aggregators to work correctly, I had to replay messages between the supersteps (between the master and the slave synchronization). Maybe we can get a better synchronization strategy that doesn't clear the queues so we don't have to loop-back them again for the next step.
          Thomas Jungblut made changes -
          Attachment HAMA-704_1337.patch [ 12570458 ]
          Thomas Jungblut made changes -
          Attachment HAMA-704_1337.patch [ 12570458 ]
          Hide
          Thomas Jungblut added a comment -

          Oh please wait a bit, found a bug in bipartite matching :/

          Show
          Thomas Jungblut added a comment - Oh please wait a bit, found a bug in bipartite matching :/
          Hide
          Edward J. Yoon added a comment -

          Since my all machines are upgraded to hdfs 2.0, I can't test huge job right now. But everything looks fine to me.

          Show
          Edward J. Yoon added a comment - Since my all machines are upgraded to hdfs 2.0, I can't test huge job right now. But everything looks fine to me.
          Hide
          Thomas Jungblut added a comment -

          How can you see? I have deleted the attachment

          I reuploaded it into the 704_1337.patch-
          Bipartite Matching works now again.

          Now you can review.

          Show
          Thomas Jungblut added a comment - How can you see? I have deleted the attachment I reuploaded it into the 704_1337.patch- Bipartite Matching works now again. Now you can review.
          Thomas Jungblut made changes -
          Attachment HAMA-704_1337.patch [ 12570463 ]
          Hide
          Thomas Jungblut added a comment -

          First profiling

          > First Superstep after messages are completely send and in their outgoing queue.

          Strong reachable Objects consume 637 mb. (strong refers that they can't be collected because they are reachable)
          Messages consume the most of the memory, if we subtract them out of the memory we have:
          24mb for the DiskVerticesInfo, most of it consumed by the long[] that stores the offset.

          Messages consume a bit over 600mb, the rest of it is data structure overhead.

          Great stuff. Now we have to get the messaging going

          Show
          Thomas Jungblut added a comment - First profiling > First Superstep after messages are completely send and in their outgoing queue. Strong reachable Objects consume 637 mb. (strong refers that they can't be collected because they are reachable) Messages consume the most of the memory, if we subtract them out of the memory we have: 24mb for the DiskVerticesInfo, most of it consumed by the long[] that stores the offset. Messages consume a bit over 600mb, the rest of it is data structure overhead. Great stuff. Now we have to get the messaging going
          Hide
          Suraj Menon added a comment -

          I was pursuing this idea long time back. Let me know what you think. Create an array of say 8 or 16 spilling queues, basically partitioning the messages into vertice corresponding buckets. So spilling queue[0] would be responsible for messages for first numVertices/16 vertices, queues[1] for the next numVertices/16. You may prefetch and sort them in memory (still risky but a lot reduced risk) as you linearly iterate through the vertices. This way you will keep only 1/8th of the messages in memory. I am pursuing the changes for messaging. For sorted spilling queue I would have to sort throughout the whole messages. This reduces that. Let me know of what you think. I do understand the messages don't have uniform distribution across vertices and we don't need a complete B-tree implementation. Just one thread sorting the next partition as the other thread read is reading the current partition sorted messages.

          Show
          Suraj Menon added a comment - I was pursuing this idea long time back. Let me know what you think. Create an array of say 8 or 16 spilling queues, basically partitioning the messages into vertice corresponding buckets. So spilling queue [0] would be responsible for messages for first numVertices/16 vertices, queues [1] for the next numVertices/16. You may prefetch and sort them in memory (still risky but a lot reduced risk) as you linearly iterate through the vertices. This way you will keep only 1/8th of the messages in memory. I am pursuing the changes for messaging. For sorted spilling queue I would have to sort throughout the whole messages. This reduces that. Let me know of what you think. I do understand the messages don't have uniform distribution across vertices and we don't need a complete B-tree implementation. Just one thread sorting the next partition as the other thread read is reading the current partition sorted messages.
          Hide
          Thomas Jungblut added a comment -

          fixing a small bug where the zeroth file always remains in the temporary path.

          Newest patch is now HAMA-704_1338.patch.

          Thanks.

          Show
          Thomas Jungblut added a comment - fixing a small bug where the zeroth file always remains in the temporary path. Newest patch is now HAMA-704 _1338.patch. Thanks.
          Thomas Jungblut made changes -
          Attachment HAMA-704_1338.patch [ 12570465 ]
          Hide
          Thomas Jungblut added a comment -

          Create an array of say 8 or 16 spilling queues, basically partitioning the messages into vertice corresponding buckets. So spilling queue[0] would be responsible for messages for first numVertices/16 vertices, queues[1] for the next numVertices/16.

          That is cool, you just have to make sure that they are sorted and getCurrentMessage returns them in this order. Nothing much else is needed
          Your approach has a bit of a problem, as the currentMessage must be the lowest in all the received messages. So a heap structure wouldn't be wrong in this case although very difficult to maintain the heap property on disk. Your bucketing must ensure that the order is correct, otherwise sorting the buckets in itself won't yield a correct working algorithm.

          For sorted spilling queue I would have to sort throughout the whole messages.

          I guess we can't get over that easily. But I told you in chat already that sorting on sender side and merging on receiver side is in this case better and easier to implement.

          I do understand the messages don't have uniform distribution across vertices and we don't need a complete B-tree implementation.

          The good thing is that a b-tree is maintaining order over all keys, so you can iterate messages with minimal number of disk seeks.

          The message distribution is related to the number of active vertices- in case of pagerank every vertex is active for all iterations (that is why I benchmark it usually).
          We track how many vertices are active, so we can give a hint to the queue how much it should cache.

          I guess we shouldn't get that fancy for the first shot, improvement can always be done later on.

          Show
          Thomas Jungblut added a comment - Create an array of say 8 or 16 spilling queues, basically partitioning the messages into vertice corresponding buckets. So spilling queue [0] would be responsible for messages for first numVertices/16 vertices, queues [1] for the next numVertices/16. That is cool, you just have to make sure that they are sorted and getCurrentMessage returns them in this order. Nothing much else is needed Your approach has a bit of a problem, as the currentMessage must be the lowest in all the received messages. So a heap structure wouldn't be wrong in this case although very difficult to maintain the heap property on disk. Your bucketing must ensure that the order is correct, otherwise sorting the buckets in itself won't yield a correct working algorithm. For sorted spilling queue I would have to sort throughout the whole messages. I guess we can't get over that easily. But I told you in chat already that sorting on sender side and merging on receiver side is in this case better and easier to implement. I do understand the messages don't have uniform distribution across vertices and we don't need a complete B-tree implementation. The good thing is that a b-tree is maintaining order over all keys, so you can iterate messages with minimal number of disk seeks. The message distribution is related to the number of active vertices- in case of pagerank every vertex is active for all iterations (that is why I benchmark it usually). We track how many vertices are active, so we can give a hint to the queue how much it should cache. I guess we shouldn't get that fancy for the first shot, improvement can always be done later on.
          Hide
          Thomas Jungblut added a comment -

          fixed the review board suggestions, added a testcase for the default priority queue behaviour.

          Any objections? Otherwise I would like to commit and close this issue.

          Show
          Thomas Jungblut added a comment - fixed the review board suggestions, added a testcase for the default priority queue behaviour. Any objections? Otherwise I would like to commit and close this issue.
          Thomas Jungblut made changes -
          Attachment HAMA-704_1339.patch [ 12570745 ]
          Thomas Jungblut made changes -
          Link This issue is related to HAMA-738 [ HAMA-738 ]
          Hide
          Edward J. Yoon added a comment -

          +1

          Show
          Edward J. Yoon added a comment - +1
          Hide
          Thomas Jungblut added a comment -

          Thank you all for review

          Show
          Thomas Jungblut added a comment - Thank you all for review
          Thomas Jungblut made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Assignee Edward J. Yoon [ udanax ] Thomas Jungblut [ thomas.jungblut ]
          Fix Version/s 0.7.0 [ 12320349 ]
          Resolution Fixed [ 1 ]
          Hide
          Hudson added a comment -

          Integrated in Hama trunk #104 (See https://builds.apache.org/job/Hama%20trunk/104/)
          HAMA-704: Optimization of memory usage during message processing (Revision 1449666)

          Result = FAILURE
          tjungblut :
          Files :

          • /hama/trunk/CHANGES.txt
          • /hama/trunk/conf/hama-default.xml
          • /hama/trunk/core/src/main/java/org/apache/hama/bsp/SimpleTaskScheduler.java
          • /hama/trunk/core/src/main/java/org/apache/hama/bsp/ft/AsyncRcvdMsgCheckpointImpl.java
          • /hama/trunk/core/src/main/java/org/apache/hama/bsp/ft/BSPFaultTolerantService.java
          • /hama/trunk/core/src/main/java/org/apache/hama/util/ReflectionUtils.java
          • /hama/trunk/core/src/test/java/org/apache/hama/bsp/TestCheckpoint.java
          • /hama/trunk/core/src/test/java/org/apache/hama/monitor/TestFederator.java
          • /hama/trunk/examples/src/main/java/org/apache/hama/examples/BipartiteMatching.java
          • /hama/trunk/examples/src/main/java/org/apache/hama/examples/InlinkCount.java
          • /hama/trunk/examples/src/main/java/org/apache/hama/examples/MindistSearch.java
          • /hama/trunk/examples/src/main/java/org/apache/hama/examples/PageRank.java
          • /hama/trunk/examples/src/main/java/org/apache/hama/examples/SSSP.java
          • /hama/trunk/examples/src/main/java/org/apache/hama/examples/util/TextPair.java
          • /hama/trunk/examples/src/test/java/org/apache/hama/examples/BipartiteMatchingTest.java
          • /hama/trunk/examples/src/test/java/org/apache/hama/examples/PageRankTest.java
          • /hama/trunk/examples/src/test/java/org/apache/hama/examples/SSSPTest.java
          • /hama/trunk/graph/src/main/java/org/apache/hama/graph/AggregationRunner.java
          • /hama/trunk/graph/src/main/java/org/apache/hama/graph/DiskVerticesInfo.java
          • /hama/trunk/graph/src/main/java/org/apache/hama/graph/Edge.java
          • /hama/trunk/graph/src/main/java/org/apache/hama/graph/GraphJob.java
          • /hama/trunk/graph/src/main/java/org/apache/hama/graph/GraphJobMessage.java
          • /hama/trunk/graph/src/main/java/org/apache/hama/graph/GraphJobRunner.java
          • /hama/trunk/graph/src/main/java/org/apache/hama/graph/IDSkippingIterator.java
          • /hama/trunk/graph/src/main/java/org/apache/hama/graph/IVerticesInfo.java
          • /hama/trunk/graph/src/main/java/org/apache/hama/graph/ListVerticesInfo.java
          • /hama/trunk/graph/src/main/java/org/apache/hama/graph/Vertex.java
          • /hama/trunk/graph/src/main/java/org/apache/hama/graph/VertexInterface.java
          • /hama/trunk/graph/src/main/java/org/apache/hama/graph/VertexMessageIterable.java
          • /hama/trunk/graph/src/main/java/org/apache/hama/graph/VerticesInfo.java
          • /hama/trunk/graph/src/test/java/org/apache/hama/graph/TestAbsDiffAggregator.java
          • /hama/trunk/graph/src/test/java/org/apache/hama/graph/TestAverageAggregator.java
          • /hama/trunk/graph/src/test/java/org/apache/hama/graph/TestDiskVerticesInfo.java
          • /hama/trunk/graph/src/test/java/org/apache/hama/graph/TestGraphJobMessage.java
          • /hama/trunk/graph/src/test/java/org/apache/hama/graph/TestMinMaxAggregator.java
          • /hama/trunk/graph/src/test/java/org/apache/hama/graph/TestSubmitGraphJob.java
          • /hama/trunk/graph/src/test/java/org/apache/hama/graph/TestSumAggregator.java
          • /hama/trunk/graph/src/test/java/org/apache/hama/graph/example/PageRank.java
          Show
          Hudson added a comment - Integrated in Hama trunk #104 (See https://builds.apache.org/job/Hama%20trunk/104/ ) HAMA-704 : Optimization of memory usage during message processing (Revision 1449666) Result = FAILURE tjungblut : Files : /hama/trunk/CHANGES.txt /hama/trunk/conf/hama-default.xml /hama/trunk/core/src/main/java/org/apache/hama/bsp/SimpleTaskScheduler.java /hama/trunk/core/src/main/java/org/apache/hama/bsp/ft/AsyncRcvdMsgCheckpointImpl.java /hama/trunk/core/src/main/java/org/apache/hama/bsp/ft/BSPFaultTolerantService.java /hama/trunk/core/src/main/java/org/apache/hama/util/ReflectionUtils.java /hama/trunk/core/src/test/java/org/apache/hama/bsp/TestCheckpoint.java /hama/trunk/core/src/test/java/org/apache/hama/monitor/TestFederator.java /hama/trunk/examples/src/main/java/org/apache/hama/examples/BipartiteMatching.java /hama/trunk/examples/src/main/java/org/apache/hama/examples/InlinkCount.java /hama/trunk/examples/src/main/java/org/apache/hama/examples/MindistSearch.java /hama/trunk/examples/src/main/java/org/apache/hama/examples/PageRank.java /hama/trunk/examples/src/main/java/org/apache/hama/examples/SSSP.java /hama/trunk/examples/src/main/java/org/apache/hama/examples/util/TextPair.java /hama/trunk/examples/src/test/java/org/apache/hama/examples/BipartiteMatchingTest.java /hama/trunk/examples/src/test/java/org/apache/hama/examples/PageRankTest.java /hama/trunk/examples/src/test/java/org/apache/hama/examples/SSSPTest.java /hama/trunk/graph/src/main/java/org/apache/hama/graph/AggregationRunner.java /hama/trunk/graph/src/main/java/org/apache/hama/graph/DiskVerticesInfo.java /hama/trunk/graph/src/main/java/org/apache/hama/graph/Edge.java /hama/trunk/graph/src/main/java/org/apache/hama/graph/GraphJob.java /hama/trunk/graph/src/main/java/org/apache/hama/graph/GraphJobMessage.java /hama/trunk/graph/src/main/java/org/apache/hama/graph/GraphJobRunner.java /hama/trunk/graph/src/main/java/org/apache/hama/graph/IDSkippingIterator.java /hama/trunk/graph/src/main/java/org/apache/hama/graph/IVerticesInfo.java /hama/trunk/graph/src/main/java/org/apache/hama/graph/ListVerticesInfo.java /hama/trunk/graph/src/main/java/org/apache/hama/graph/Vertex.java /hama/trunk/graph/src/main/java/org/apache/hama/graph/VertexInterface.java /hama/trunk/graph/src/main/java/org/apache/hama/graph/VertexMessageIterable.java /hama/trunk/graph/src/main/java/org/apache/hama/graph/VerticesInfo.java /hama/trunk/graph/src/test/java/org/apache/hama/graph/TestAbsDiffAggregator.java /hama/trunk/graph/src/test/java/org/apache/hama/graph/TestAverageAggregator.java /hama/trunk/graph/src/test/java/org/apache/hama/graph/TestDiskVerticesInfo.java /hama/trunk/graph/src/test/java/org/apache/hama/graph/TestGraphJobMessage.java /hama/trunk/graph/src/test/java/org/apache/hama/graph/TestMinMaxAggregator.java /hama/trunk/graph/src/test/java/org/apache/hama/graph/TestSubmitGraphJob.java /hama/trunk/graph/src/test/java/org/apache/hama/graph/TestSumAggregator.java /hama/trunk/graph/src/test/java/org/apache/hama/graph/example/PageRank.java
          Hide
          Hudson added a comment -

          Integrated in Hama trunk #105 (See https://builds.apache.org/job/Hama%20trunk/105/)
          HAMA-704: adding licence (Revision 1449676)

          Result = SUCCESS
          tjungblut :
          Files :

          • /hama/trunk/graph/src/test/java/org/apache/hama/graph/TestGraphJobMessage.java
          Show
          Hudson added a comment - Integrated in Hama trunk #105 (See https://builds.apache.org/job/Hama%20trunk/105/ ) HAMA-704 : adding licence (Revision 1449676) Result = SUCCESS tjungblut : Files : /hama/trunk/graph/src/test/java/org/apache/hama/graph/TestGraphJobMessage.java
          Hide
          Hudson added a comment -

          Integrated in Hama-Nightly #840 (See https://builds.apache.org/job/Hama-Nightly/840/)
          HAMA-704: adding licence (Revision 1449676)
          HAMA-704: Optimization of memory usage during message processing (Revision 1449666)

          Result = SUCCESS
          tjungblut :
          Files :

          • /hama/trunk/graph/src/test/java/org/apache/hama/graph/TestGraphJobMessage.java

          tjungblut :
          Files :

          • /hama/trunk/CHANGES.txt
          • /hama/trunk/conf/hama-default.xml
          • /hama/trunk/core/src/main/java/org/apache/hama/bsp/SimpleTaskScheduler.java
          • /hama/trunk/core/src/main/java/org/apache/hama/bsp/ft/AsyncRcvdMsgCheckpointImpl.java
          • /hama/trunk/core/src/main/java/org/apache/hama/bsp/ft/BSPFaultTolerantService.java
          • /hama/trunk/core/src/main/java/org/apache/hama/util/ReflectionUtils.java
          • /hama/trunk/core/src/test/java/org/apache/hama/bsp/TestCheckpoint.java
          • /hama/trunk/core/src/test/java/org/apache/hama/monitor/TestFederator.java
          • /hama/trunk/examples/src/main/java/org/apache/hama/examples/BipartiteMatching.java
          • /hama/trunk/examples/src/main/java/org/apache/hama/examples/InlinkCount.java
          • /hama/trunk/examples/src/main/java/org/apache/hama/examples/MindistSearch.java
          • /hama/trunk/examples/src/main/java/org/apache/hama/examples/PageRank.java
          • /hama/trunk/examples/src/main/java/org/apache/hama/examples/SSSP.java
          • /hama/trunk/examples/src/main/java/org/apache/hama/examples/util/TextPair.java
          • /hama/trunk/examples/src/test/java/org/apache/hama/examples/BipartiteMatchingTest.java
          • /hama/trunk/examples/src/test/java/org/apache/hama/examples/PageRankTest.java
          • /hama/trunk/examples/src/test/java/org/apache/hama/examples/SSSPTest.java
          • /hama/trunk/graph/src/main/java/org/apache/hama/graph/AggregationRunner.java
          • /hama/trunk/graph/src/main/java/org/apache/hama/graph/DiskVerticesInfo.java
          • /hama/trunk/graph/src/main/java/org/apache/hama/graph/Edge.java
          • /hama/trunk/graph/src/main/java/org/apache/hama/graph/GraphJob.java
          • /hama/trunk/graph/src/main/java/org/apache/hama/graph/GraphJobMessage.java
          • /hama/trunk/graph/src/main/java/org/apache/hama/graph/GraphJobRunner.java
          • /hama/trunk/graph/src/main/java/org/apache/hama/graph/IDSkippingIterator.java
          • /hama/trunk/graph/src/main/java/org/apache/hama/graph/IVerticesInfo.java
          • /hama/trunk/graph/src/main/java/org/apache/hama/graph/ListVerticesInfo.java
          • /hama/trunk/graph/src/main/java/org/apache/hama/graph/Vertex.java
          • /hama/trunk/graph/src/main/java/org/apache/hama/graph/VertexInterface.java
          • /hama/trunk/graph/src/main/java/org/apache/hama/graph/VertexMessageIterable.java
          • /hama/trunk/graph/src/main/java/org/apache/hama/graph/VerticesInfo.java
          • /hama/trunk/graph/src/test/java/org/apache/hama/graph/TestAbsDiffAggregator.java
          • /hama/trunk/graph/src/test/java/org/apache/hama/graph/TestAverageAggregator.java
          • /hama/trunk/graph/src/test/java/org/apache/hama/graph/TestDiskVerticesInfo.java
          • /hama/trunk/graph/src/test/java/org/apache/hama/graph/TestGraphJobMessage.java
          • /hama/trunk/graph/src/test/java/org/apache/hama/graph/TestMinMaxAggregator.java
          • /hama/trunk/graph/src/test/java/org/apache/hama/graph/TestSubmitGraphJob.java
          • /hama/trunk/graph/src/test/java/org/apache/hama/graph/TestSumAggregator.java
          • /hama/trunk/graph/src/test/java/org/apache/hama/graph/example/PageRank.java
          Show
          Hudson added a comment - Integrated in Hama-Nightly #840 (See https://builds.apache.org/job/Hama-Nightly/840/ ) HAMA-704 : adding licence (Revision 1449676) HAMA-704 : Optimization of memory usage during message processing (Revision 1449666) Result = SUCCESS tjungblut : Files : /hama/trunk/graph/src/test/java/org/apache/hama/graph/TestGraphJobMessage.java tjungblut : Files : /hama/trunk/CHANGES.txt /hama/trunk/conf/hama-default.xml /hama/trunk/core/src/main/java/org/apache/hama/bsp/SimpleTaskScheduler.java /hama/trunk/core/src/main/java/org/apache/hama/bsp/ft/AsyncRcvdMsgCheckpointImpl.java /hama/trunk/core/src/main/java/org/apache/hama/bsp/ft/BSPFaultTolerantService.java /hama/trunk/core/src/main/java/org/apache/hama/util/ReflectionUtils.java /hama/trunk/core/src/test/java/org/apache/hama/bsp/TestCheckpoint.java /hama/trunk/core/src/test/java/org/apache/hama/monitor/TestFederator.java /hama/trunk/examples/src/main/java/org/apache/hama/examples/BipartiteMatching.java /hama/trunk/examples/src/main/java/org/apache/hama/examples/InlinkCount.java /hama/trunk/examples/src/main/java/org/apache/hama/examples/MindistSearch.java /hama/trunk/examples/src/main/java/org/apache/hama/examples/PageRank.java /hama/trunk/examples/src/main/java/org/apache/hama/examples/SSSP.java /hama/trunk/examples/src/main/java/org/apache/hama/examples/util/TextPair.java /hama/trunk/examples/src/test/java/org/apache/hama/examples/BipartiteMatchingTest.java /hama/trunk/examples/src/test/java/org/apache/hama/examples/PageRankTest.java /hama/trunk/examples/src/test/java/org/apache/hama/examples/SSSPTest.java /hama/trunk/graph/src/main/java/org/apache/hama/graph/AggregationRunner.java /hama/trunk/graph/src/main/java/org/apache/hama/graph/DiskVerticesInfo.java /hama/trunk/graph/src/main/java/org/apache/hama/graph/Edge.java /hama/trunk/graph/src/main/java/org/apache/hama/graph/GraphJob.java /hama/trunk/graph/src/main/java/org/apache/hama/graph/GraphJobMessage.java /hama/trunk/graph/src/main/java/org/apache/hama/graph/GraphJobRunner.java /hama/trunk/graph/src/main/java/org/apache/hama/graph/IDSkippingIterator.java /hama/trunk/graph/src/main/java/org/apache/hama/graph/IVerticesInfo.java /hama/trunk/graph/src/main/java/org/apache/hama/graph/ListVerticesInfo.java /hama/trunk/graph/src/main/java/org/apache/hama/graph/Vertex.java /hama/trunk/graph/src/main/java/org/apache/hama/graph/VertexInterface.java /hama/trunk/graph/src/main/java/org/apache/hama/graph/VertexMessageIterable.java /hama/trunk/graph/src/main/java/org/apache/hama/graph/VerticesInfo.java /hama/trunk/graph/src/test/java/org/apache/hama/graph/TestAbsDiffAggregator.java /hama/trunk/graph/src/test/java/org/apache/hama/graph/TestAverageAggregator.java /hama/trunk/graph/src/test/java/org/apache/hama/graph/TestDiskVerticesInfo.java /hama/trunk/graph/src/test/java/org/apache/hama/graph/TestGraphJobMessage.java /hama/trunk/graph/src/test/java/org/apache/hama/graph/TestMinMaxAggregator.java /hama/trunk/graph/src/test/java/org/apache/hama/graph/TestSubmitGraphJob.java /hama/trunk/graph/src/test/java/org/apache/hama/graph/TestSumAggregator.java /hama/trunk/graph/src/test/java/org/apache/hama/graph/example/PageRank.java
          Thomas Jungblut made changes -
          Link This issue blocks HAMA-723 [ HAMA-723 ]
          Gavin made changes -
          Link This issue depends on HAMA-559 [ HAMA-559 ]
          Gavin made changes -
          Link This issue depends upon HAMA-559 [ HAMA-559 ]
          Gavin made changes -
          Link This issue blocks HAMA-723 [ HAMA-723 ]
          Gavin made changes -
          Link This issue is depended upon by HAMA-723 [ HAMA-723 ]
          Edward J. Yoon made changes -
          Fix Version/s 0.7.0 [ 12320349 ]

            People

            • Assignee:
              Thomas Jungblut
              Reporter:
              Edward J. Yoon
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development