Details

    • Type: Sub-task Sub-task
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: graph
    • Labels:
      None

      Description

      Currently, the out-of-core partitions are assigned to memory or to disk statically. Using an LRU cache should help keeping in-memory only the partitions that are actively accessed, given a job that does not access all the graph at each superstep (traversals) and a good data partitioning (non random).

      1. GIRAPH-461.patch
        45 kB
        Claudio Martella
      2. GIRAPH-461.patch
        47 kB
        Claudio Martella
      3. GIRAPH-461.patch
        47 kB
        Claudio Martella

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          Integrated in Giraph-trunk-Commit #668 (See https://builds.apache.org/job/Giraph-trunk-Commit/668/)
          GIRAPH-461: Convert static assignment of in-memory partitions with LRU cache (Revision 62c12fa0b899ee76962d6e3fa05298e2a23d5e68)

          Result = SUCCESS
          claudio : http://git-wip-us.apache.org/repos/asf?p=giraph.git&a=commit&h=62c12fa0b899ee76962d6e3fa05298e2a23d5e68
          Files :

          • giraph-core/src/main/java/org/apache/giraph/partition/DiskBackedPartitionStore.java
          • giraph-core/src/test/java/org/apache/giraph/partition/TestPartitionStores.java
          • giraph-core/src/main/java/org/apache/giraph/bsp/CentralizedServiceWorker.java
          • giraph-core/src/main/java/org/apache/giraph/partition/SimplePartitionStore.java
          • giraph-core/src/main/java/org/apache/giraph/partition/PartitionStore.java
          • giraph-core/src/main/java/org/apache/giraph/worker/BspServiceWorker.java
          • giraph-core/src/main/java/org/apache/giraph/comm/netty/NettyWorkerServer.java
          • giraph-core/src/test/java/org/apache/giraph/comm/RequestTest.java
          • CHANGELOG
          • giraph-core/src/main/java/org/apache/giraph/graph/ComputeCallable.java
          Show
          Hudson added a comment - Integrated in Giraph-trunk-Commit #668 (See https://builds.apache.org/job/Giraph-trunk-Commit/668/ ) GIRAPH-461 : Convert static assignment of in-memory partitions with LRU cache (Revision 62c12fa0b899ee76962d6e3fa05298e2a23d5e68) Result = SUCCESS claudio : http://git-wip-us.apache.org/repos/asf?p=giraph.git&a=commit&h=62c12fa0b899ee76962d6e3fa05298e2a23d5e68 Files : giraph-core/src/main/java/org/apache/giraph/partition/DiskBackedPartitionStore.java giraph-core/src/test/java/org/apache/giraph/partition/TestPartitionStores.java giraph-core/src/main/java/org/apache/giraph/bsp/CentralizedServiceWorker.java giraph-core/src/main/java/org/apache/giraph/partition/SimplePartitionStore.java giraph-core/src/main/java/org/apache/giraph/partition/PartitionStore.java giraph-core/src/main/java/org/apache/giraph/worker/BspServiceWorker.java giraph-core/src/main/java/org/apache/giraph/comm/netty/NettyWorkerServer.java giraph-core/src/test/java/org/apache/giraph/comm/RequestTest.java CHANGELOG giraph-core/src/main/java/org/apache/giraph/graph/ComputeCallable.java
          Hide
          Claudio Martella added a comment -

          Ran this multiple times. Looks like performance of trunk when OOC is a bit slower. I would not consider this very reliable, let's say the behave the same, as expected on PR. I'm committing this one.

          13/02/05 12:37:49 INFO mapred.JobClient: Giraph Timers
          13/02/05 12:37:49 INFO mapred.JobClient: Total (milliseconds)=585851
          13/02/05 12:37:49 INFO mapred.JobClient: Superstep 3 (milliseconds)=56282
          13/02/05 12:37:49 INFO mapred.JobClient: Superstep 4 (milliseconds)=38479
          13/02/05 12:37:49 INFO mapred.JobClient: Superstep 10 (milliseconds)=6282
          13/02/05 12:37:49 INFO mapred.JobClient: Setup (milliseconds)=20225
          13/02/05 12:37:49 INFO mapred.JobClient: Shutdown (milliseconds)=139
          13/02/05 12:37:49 INFO mapred.JobClient: Superstep 7 (milliseconds)=48529
          13/02/05 12:37:49 INFO mapred.JobClient: Superstep 9 (milliseconds)=47064
          13/02/05 12:37:49 INFO mapred.JobClient: Superstep 0 (milliseconds)=55454
          13/02/05 12:37:49 INFO mapred.JobClient: Superstep 8 (milliseconds)=52432
          13/02/05 12:37:49 INFO mapred.JobClient: Input superstep (milliseconds)=64258
          13/02/05 12:37:49 INFO mapred.JobClient: Superstep 6 (milliseconds)=52334
          13/02/05 12:37:49 INFO mapred.JobClient: Superstep 5 (milliseconds)=48943
          13/02/05 12:37:49 INFO mapred.JobClient: Superstep 2 (milliseconds)=42660
          13/02/05 12:37:49 INFO mapred.JobClient: Superstep 1 (milliseconds)=52766
          13/02/05 12:37:49 INFO mapred.JobClient: Giraph Stats
          13/02/05 12:37:49 INFO mapred.JobClient: Aggregate edges=1000000000
          13/02/05 12:37:49 INFO mapred.JobClient: Superstep=11
          13/02/05 12:37:49 INFO mapred.JobClient: Last checkpointed superstep=0
          13/02/05 12:37:49 INFO mapred.JobClient: Current workers=60
          13/02/05 12:37:49 INFO mapred.JobClient: Current master task partition=0
          13/02/05 12:37:49 INFO mapred.JobClient: Sent messages=0
          13/02/05 12:37:49 INFO mapred.JobClient: Aggregate finished vertices=10000000
          13/02/05 12:37:49 INFO mapred.JobClient: Aggregate vertices=10000000
          13/02/05 12:37:49 INFO mapred.JobClient: File Output Format Counters
          13/02/05 12:37:49 INFO mapred.JobClient: Bytes Written=0
          13/02/05 12:37:49 INFO mapred.JobClient: FileSystemCounters
          13/02/05 12:37:49 INFO mapred.JobClient: HDFS_BYTES_READ=2684
          13/02/05 12:37:49 INFO mapred.JobClient: FILE_BYTES_WRITTEN=1397805
          13/02/05 12:37:49 INFO mapred.JobClient: File Input Format Counters
          13/02/05 12:37:49 INFO mapred.JobClient: Bytes Read=0
          13/02/05 12:37:49 INFO mapred.JobClient: Map-Reduce Framework
          13/02/05 12:37:49 INFO mapred.JobClient: Map input records=61
          13/02/05 12:37:49 INFO mapred.JobClient: Physical memory (bytes) snapshot=68512210944
          13/02/05 12:37:49 INFO mapred.JobClient: Spilled Records=0
          13/02/05 12:37:49 INFO mapred.JobClient: CPU time spent (ms)=17221490
          13/02/05 12:37:49 INFO mapred.JobClient: Total committed heap usage (bytes)=58657275904
          13/02/05 12:37:49 INFO mapred.JobClient: Virtual memory (bytes) snapshot=371997724672
          13/02/05 12:37:49 INFO mapred.JobClient: Map output records=0
          13/02/05 12:37:49 INFO mapred.JobClient: SPLIT_RAW_BYTES=2684

          Show
          Claudio Martella added a comment - Ran this multiple times. Looks like performance of trunk when OOC is a bit slower. I would not consider this very reliable, let's say the behave the same, as expected on PR. I'm committing this one. 13/02/05 12:37:49 INFO mapred.JobClient: Giraph Timers 13/02/05 12:37:49 INFO mapred.JobClient: Total (milliseconds)=585851 13/02/05 12:37:49 INFO mapred.JobClient: Superstep 3 (milliseconds)=56282 13/02/05 12:37:49 INFO mapred.JobClient: Superstep 4 (milliseconds)=38479 13/02/05 12:37:49 INFO mapred.JobClient: Superstep 10 (milliseconds)=6282 13/02/05 12:37:49 INFO mapred.JobClient: Setup (milliseconds)=20225 13/02/05 12:37:49 INFO mapred.JobClient: Shutdown (milliseconds)=139 13/02/05 12:37:49 INFO mapred.JobClient: Superstep 7 (milliseconds)=48529 13/02/05 12:37:49 INFO mapred.JobClient: Superstep 9 (milliseconds)=47064 13/02/05 12:37:49 INFO mapred.JobClient: Superstep 0 (milliseconds)=55454 13/02/05 12:37:49 INFO mapred.JobClient: Superstep 8 (milliseconds)=52432 13/02/05 12:37:49 INFO mapred.JobClient: Input superstep (milliseconds)=64258 13/02/05 12:37:49 INFO mapred.JobClient: Superstep 6 (milliseconds)=52334 13/02/05 12:37:49 INFO mapred.JobClient: Superstep 5 (milliseconds)=48943 13/02/05 12:37:49 INFO mapred.JobClient: Superstep 2 (milliseconds)=42660 13/02/05 12:37:49 INFO mapred.JobClient: Superstep 1 (milliseconds)=52766 13/02/05 12:37:49 INFO mapred.JobClient: Giraph Stats 13/02/05 12:37:49 INFO mapred.JobClient: Aggregate edges=1000000000 13/02/05 12:37:49 INFO mapred.JobClient: Superstep=11 13/02/05 12:37:49 INFO mapred.JobClient: Last checkpointed superstep=0 13/02/05 12:37:49 INFO mapred.JobClient: Current workers=60 13/02/05 12:37:49 INFO mapred.JobClient: Current master task partition=0 13/02/05 12:37:49 INFO mapred.JobClient: Sent messages=0 13/02/05 12:37:49 INFO mapred.JobClient: Aggregate finished vertices=10000000 13/02/05 12:37:49 INFO mapred.JobClient: Aggregate vertices=10000000 13/02/05 12:37:49 INFO mapred.JobClient: File Output Format Counters 13/02/05 12:37:49 INFO mapred.JobClient: Bytes Written=0 13/02/05 12:37:49 INFO mapred.JobClient: FileSystemCounters 13/02/05 12:37:49 INFO mapred.JobClient: HDFS_BYTES_READ=2684 13/02/05 12:37:49 INFO mapred.JobClient: FILE_BYTES_WRITTEN=1397805 13/02/05 12:37:49 INFO mapred.JobClient: File Input Format Counters 13/02/05 12:37:49 INFO mapred.JobClient: Bytes Read=0 13/02/05 12:37:49 INFO mapred.JobClient: Map-Reduce Framework 13/02/05 12:37:49 INFO mapred.JobClient: Map input records=61 13/02/05 12:37:49 INFO mapred.JobClient: Physical memory (bytes) snapshot=68512210944 13/02/05 12:37:49 INFO mapred.JobClient: Spilled Records=0 13/02/05 12:37:49 INFO mapred.JobClient: CPU time spent (ms)=17221490 13/02/05 12:37:49 INFO mapred.JobClient: Total committed heap usage (bytes)=58657275904 13/02/05 12:37:49 INFO mapred.JobClient: Virtual memory (bytes) snapshot=371997724672 13/02/05 12:37:49 INFO mapred.JobClient: Map output records=0 13/02/05 12:37:49 INFO mapred.JobClient: SPLIT_RAW_BYTES=2684
          Hide
          Claudio Martella added a comment -

          Ok, I will show that performance does not degrade explicitly (although I'd claim that these already do). To show that performance is actually improved for certain algorithms and data partitioning, is out of scope currently, as it would require to build a particular graph with a particular partitioning, and that would impact on many different things (e.g. load balancing). I agree that it would be nice to see it with numbers, and I plan to test this further in the near future as part of my current research. Lets say we should be happy if this does not make things worse, as we get the fix to multithreading, and we get a strategy that can strongly improve performance within certain conditions.

          One thing has to be considered though. The other condition for which we could gain performance wrt to the static assignment is in the other point where we load the partitions, i.e. when resolving mutations. Again, by considering vertices distributed uniformly, we should not see a big impact. But we have acted on a best-effort with LRU.

          Show
          Claudio Martella added a comment - Ok, I will show that performance does not degrade explicitly (although I'd claim that these already do). To show that performance is actually improved for certain algorithms and data partitioning, is out of scope currently, as it would require to build a particular graph with a particular partitioning, and that would impact on many different things (e.g. load balancing). I agree that it would be nice to see it with numbers, and I plan to test this further in the near future as part of my current research. Lets say we should be happy if this does not make things worse, as we get the fix to multithreading, and we get a strategy that can strongly improve performance within certain conditions. One thing has to be considered though. The other condition for which we could gain performance wrt to the static assignment is in the other point where we load the partitions, i.e. when resolving mutations. Again, by considering vertices distributed uniformly, we should not see a big impact. But we have acted on a best-effort with LRU.
          Hide
          Alessandro Presta added a comment -

          True, OOC with multithreading has to be fixed anyway (thanks for doing that!), but the LRU cache is an orthogonal improvement, so it would be good to see some examples where it proves beneficial (like you said, it depends on the algorithm and graph structure) and also make sure that performance doesn't degrade in other cases. We can disregard multithreading in this sort of comparison.

          Show
          Alessandro Presta added a comment - True, OOC with multithreading has to be fixed anyway (thanks for doing that!), but the LRU cache is an orthogonal improvement, so it would be good to see some examples where it proves beneficial (like you said, it depends on the algorithm and graph structure) and also make sure that performance doesn't degrade in other cases. We can disregard multithreading in this sort of comparison.
          Hide
          Claudio Martella added a comment -

          Alessandro, I could, but the current out-of-core graph is broken for multiple threads, so it has to be substituted anyway. Nonetheless, IIRC, the previous OOC performance was comparable to in-memory, as it is this one, hence I thought it did not make much sense. If you want, I can give it a run.

          Show
          Claudio Martella added a comment - Alessandro, I could, but the current out-of-core graph is broken for multiple threads, so it has to be substituted anyway. Nonetheless, IIRC, the previous OOC performance was comparable to in-memory, as it is this one, hence I thought it did not make much sense. If you want, I can give it a run.
          Hide
          Alessandro Presta added a comment -

          Shouldn't you compare this patch with trunk running out-of-core?

          Show
          Alessandro Presta added a comment - Shouldn't you compare this patch with trunk running out-of-core?
          Hide
          Claudio Martella added a comment -

          Fixed a missing line.

          Show
          Claudio Martella added a comment - Fixed a missing line.
          Hide
          Claudio Martella added a comment -

          hadoop jar giraph-0.2-SNAPSHOT-for-hadoop-1.0.2-jar-with-dependencies.jar org.apache.giraph.benchmark.PageRankBenchmark -w 60 -c 2 -e 100 -V 10000000 -v -s 10

          trunk:
          13/01/29 20:40:53 INFO mapred.JobClient: Giraph Timers
          13/01/29 20:40:53 INFO mapred.JobClient: Total (milliseconds)=492403
          13/01/29 20:40:53 INFO mapred.JobClient: Superstep 3 (milliseconds)=40243
          13/01/29 20:40:53 INFO mapred.JobClient: Superstep 4 (milliseconds)=45430
          13/01/29 20:40:53 INFO mapred.JobClient: Superstep 10 (milliseconds)=713
          13/01/29 20:40:53 INFO mapred.JobClient: Setup (milliseconds)=20832
          13/01/29 20:40:53 INFO mapred.JobClient: Shutdown (milliseconds)=56
          13/01/29 20:40:53 INFO mapred.JobClient: Superstep 7 (milliseconds)=36753
          13/01/29 20:40:53 INFO mapred.JobClient: Superstep 9 (milliseconds)=36363
          13/01/29 20:40:53 INFO mapred.JobClient: Superstep 0 (milliseconds)=39558
          13/01/29 20:40:53 INFO mapred.JobClient: Superstep 8 (milliseconds)=44548
          13/01/29 20:40:53 INFO mapred.JobClient: Input superstep (milliseconds)=59184
          13/01/29 20:40:53 INFO mapred.JobClient: Superstep 6 (milliseconds)=40777
          13/01/29 20:40:53 INFO mapred.JobClient: Superstep 5 (milliseconds)=43962
          13/01/29 20:40:53 INFO mapred.JobClient: Superstep 2 (milliseconds)=37325
          13/01/29 20:40:53 INFO mapred.JobClient: Superstep 1 (milliseconds)=46655
          13/01/29 20:40:53 INFO mapred.JobClient: Giraph Stats
          13/01/29 20:40:53 INFO mapred.JobClient: Aggregate edges=1000000000
          13/01/29 20:40:53 INFO mapred.JobClient: Superstep=11
          13/01/29 20:40:53 INFO mapred.JobClient: Last checkpointed superstep=0
          13/01/29 20:40:53 INFO mapred.JobClient: Current workers=60
          13/01/29 20:40:53 INFO mapred.JobClient: Current master task partition=0
          13/01/29 20:40:53 INFO mapred.JobClient: Sent messages=0
          13/01/29 20:40:53 INFO mapred.JobClient: Aggregate finished vertices=10000000
          13/01/29 20:40:53 INFO mapred.JobClient: Aggregate vertices=10000000
          13/01/29 20:40:53 INFO mapred.JobClient: File Output Format Counters
          13/01/29 20:40:53 INFO mapred.JobClient: Bytes Written=0
          13/01/29 20:40:53 INFO mapred.JobClient: FileSystemCounters
          13/01/29 20:40:53 INFO mapred.JobClient: HDFS_BYTES_READ=2684
          13/01/29 20:40:53 INFO mapred.JobClient: FILE_BYTES_WRITTEN=1388228
          13/01/29 20:40:53 INFO mapred.JobClient: File Input Format Counters
          13/01/29 20:40:53 INFO mapred.JobClient: Bytes Read=0
          13/01/29 20:40:53 INFO mapred.JobClient: Map-Reduce Framework
          13/01/29 20:40:53 INFO mapred.JobClient: Map input records=61
          13/01/29 20:40:53 INFO mapred.JobClient: Physical memory (bytes) snapshot=71703965696
          13/01/29 20:40:53 INFO mapred.JobClient: Spilled Records=0
          13/01/29 20:40:53 INFO mapred.JobClient: CPU time spent (ms)=15141630
          13/01/29 20:40:53 INFO mapred.JobClient: Total committed heap usage (bytes)=58151337984
          13/01/29 20:40:53 INFO mapred.JobClient: Virtual memory (bytes) snapshot=371313995776
          13/01/29 20:40:53 INFO mapred.JobClient: Map output records=0
          13/01/29 20:40:53 INFO mapred.JobClient: SPLIT_RAW_BYTES=2684

          GIRAPH-461:
          in memory:
          13/01/29 19:35:53 INFO mapred.JobClient: Giraph Timers
          13/01/29 19:35:53 INFO mapred.JobClient: Total (milliseconds)=427511
          13/01/29 19:35:53 INFO mapred.JobClient: Superstep 3 (milliseconds)=37341
          13/01/29 19:35:53 INFO mapred.JobClient: Superstep 4 (milliseconds)=35458
          13/01/29 19:35:53 INFO mapred.JobClient: Superstep 10 (milliseconds)=852
          13/01/29 19:35:53 INFO mapred.JobClient: Setup (milliseconds)=24825
          13/01/29 19:35:53 INFO mapred.JobClient: Shutdown (milliseconds)=50
          13/01/29 19:35:53 INFO mapred.JobClient: Superstep 7 (milliseconds)=37557
          13/01/29 19:35:53 INFO mapred.JobClient: Superstep 9 (milliseconds)=33961
          13/01/29 19:35:53 INFO mapred.JobClient: Superstep 0 (milliseconds)=33048
          13/01/29 19:35:53 INFO mapred.JobClient: Superstep 8 (milliseconds)=36345
          13/01/29 19:35:53 INFO mapred.JobClient: Input superstep (milliseconds)=44420
          13/01/29 19:35:53 INFO mapred.JobClient: Superstep 6 (milliseconds)=33635
          13/01/29 19:35:53 INFO mapred.JobClient: Superstep 5 (milliseconds)=41885
          13/01/29 19:35:53 INFO mapred.JobClient: Superstep 2 (milliseconds)=35046
          13/01/29 19:35:53 INFO mapred.JobClient: Superstep 1 (milliseconds)=33083
          13/01/29 19:35:53 INFO mapred.JobClient: Giraph Stats
          13/01/29 19:35:53 INFO mapred.JobClient: Aggregate edges=1000000000
          13/01/29 19:35:53 INFO mapred.JobClient: Superstep=11
          13/01/29 19:35:53 INFO mapred.JobClient: Last checkpointed superstep=0
          13/01/29 19:35:53 INFO mapred.JobClient: Current workers=60
          13/01/29 19:35:53 INFO mapred.JobClient: Current master task partition=0
          13/01/29 19:35:53 INFO mapred.JobClient: Sent messages=0
          13/01/29 19:35:53 INFO mapred.JobClient: Aggregate finished vertices=10000000
          13/01/29 19:35:53 INFO mapred.JobClient: Aggregate vertices=10000000
          13/01/29 19:35:53 INFO mapred.JobClient: File Output Format Counters
          13/01/29 19:35:53 INFO mapred.JobClient: Bytes Written=0
          13/01/29 19:35:53 INFO mapred.JobClient: FileSystemCounters
          13/01/29 19:35:53 INFO mapred.JobClient: HDFS_BYTES_READ=2684
          13/01/29 19:35:53 INFO mapred.JobClient: FILE_BYTES_WRITTEN=1388228
          13/01/29 19:35:53 INFO mapred.JobClient: File Input Format Counters
          13/01/29 19:35:53 INFO mapred.JobClient: Bytes Read=0
          13/01/29 19:35:53 INFO mapred.JobClient: Map-Reduce Framework
          13/01/29 19:35:53 INFO mapred.JobClient: Map input records=61
          13/01/29 19:35:53 INFO mapred.JobClient: Physical memory (bytes) snapshot=71627419648
          13/01/29 19:35:53 INFO mapred.JobClient: Spilled Records=0
          13/01/29 19:35:53 INFO mapred.JobClient: CPU time spent (ms)=15020990
          13/01/29 19:35:53 INFO mapred.JobClient: Total committed heap usage (bytes)=57611911168
          13/01/29 19:35:53 INFO mapred.JobClient: Virtual memory (bytes) snapshot=371123154944
          13/01/29 19:35:53 INFO mapred.JobClient: Map output records=0
          13/01/29 19:35:53 INFO mapred.JobClient: SPLIT_RAW_BYTES=2684

          ooh graph (2 partitions in memory out of 49):
          13/01/29 19:54:57 INFO mapred.JobClient: Giraph Timers
          13/01/29 19:54:57 INFO mapred.JobClient: Total (milliseconds)=508004
          13/01/29 19:54:57 INFO mapred.JobClient: Superstep 3 (milliseconds)=38085
          13/01/29 19:54:57 INFO mapred.JobClient: Superstep 4 (milliseconds)=40789
          13/01/29 19:54:57 INFO mapred.JobClient: Superstep 10 (milliseconds)=811
          13/01/29 19:54:57 INFO mapred.JobClient: Setup (milliseconds)=25612
          13/01/29 19:54:57 INFO mapred.JobClient: Shutdown (milliseconds)=699
          13/01/29 19:54:57 INFO mapred.JobClient: Superstep 7 (milliseconds)=44806
          13/01/29 19:54:57 INFO mapred.JobClient: Superstep 9 (milliseconds)=41873
          13/01/29 19:54:57 INFO mapred.JobClient: Superstep 0 (milliseconds)=46329
          13/01/29 19:54:57 INFO mapred.JobClient: Superstep 8 (milliseconds)=46272
          13/01/29 19:54:57 INFO mapred.JobClient: Input superstep (milliseconds)=52395
          13/01/29 19:54:57 INFO mapred.JobClient: Superstep 6 (milliseconds)=44337
          13/01/29 19:54:57 INFO mapred.JobClient: Superstep 5 (milliseconds)=39379
          13/01/29 19:54:57 INFO mapred.JobClient: Superstep 2 (milliseconds)=40452
          13/01/29 19:54:57 INFO mapred.JobClient: Superstep 1 (milliseconds)=46155
          13/01/29 19:54:57 INFO mapred.JobClient: Giraph Stats
          13/01/29 19:54:57 INFO mapred.JobClient: Aggregate edges=1000000000
          13/01/29 19:54:57 INFO mapred.JobClient: Superstep=11
          13/01/29 19:54:57 INFO mapred.JobClient: Last checkpointed superstep=0
          13/01/29 19:54:57 INFO mapred.JobClient: Current workers=60
          13/01/29 19:54:57 INFO mapred.JobClient: Current master task partition=0
          13/01/29 19:54:57 INFO mapred.JobClient: Sent messages=0
          13/01/29 19:54:57 INFO mapred.JobClient: Aggregate finished vertices=10000000
          13/01/29 19:54:57 INFO mapred.JobClient: Aggregate vertices=10000000
          13/01/29 19:54:57 INFO mapred.JobClient: File Output Format Counters
          13/01/29 19:54:57 INFO mapred.JobClient: Bytes Written=0
          13/01/29 19:54:57 INFO mapred.JobClient: FileSystemCounters
          13/01/29 19:54:57 INFO mapred.JobClient: HDFS_BYTES_READ=2684
          13/01/29 19:54:57 INFO mapred.JobClient: FILE_BYTES_WRITTEN=1388228
          13/01/29 19:54:57 INFO mapred.JobClient: File Input Format Counters
          13/01/29 19:54:57 INFO mapred.JobClient: Bytes Read=0
          13/01/29 19:54:57 INFO mapred.JobClient: Map-Reduce Framework
          13/01/29 19:54:57 INFO mapred.JobClient: Map input records=61
          13/01/29 19:54:57 INFO mapred.JobClient: Physical memory (bytes) snapshot=71368736768
          13/01/29 19:54:57 INFO mapred.JobClient: Spilled Records=0
          13/01/29 19:54:57 INFO mapred.JobClient: CPU time spent (ms)=15289390
          13/01/29 19:54:57 INFO mapred.JobClient: Total committed heap usage (bytes)=57278595072
          13/01/29 19:54:57 INFO mapred.JobClient: Virtual memory (bytes) snapshot=370911342592
          13/01/29 19:54:57 INFO mapred.JobClient: Map output records=0
          13/01/29 19:54:57 INFO mapred.JobClient: SPLIT_RAW_BYTES=2684

          in memory (2 compute threads per worker):
          13/01/29 20:30:49 INFO mapred.JobClient: Giraph Timers
          13/01/29 20:30:49 INFO mapred.JobClient: Total (milliseconds)=487379
          13/01/29 20:30:49 INFO mapred.JobClient: Superstep 3 (milliseconds)=46092
          13/01/29 20:30:49 INFO mapred.JobClient: Superstep 4 (milliseconds)=44840
          13/01/29 20:30:49 INFO mapred.JobClient: Superstep 10 (milliseconds)=745
          13/01/29 20:30:49 INFO mapred.JobClient: Setup (milliseconds)=23013
          13/01/29 20:30:49 INFO mapred.JobClient: Shutdown (milliseconds)=126
          13/01/29 20:30:49 INFO mapred.JobClient: Superstep 7 (milliseconds)=40620
          13/01/29 20:30:49 INFO mapred.JobClient: Superstep 9 (milliseconds)=39630
          13/01/29 20:30:49 INFO mapred.JobClient: Superstep 0 (milliseconds)=38221
          13/01/29 20:30:49 INFO mapred.JobClient: Superstep 8 (milliseconds)=40406
          13/01/29 20:30:49 INFO mapred.JobClient: Input superstep (milliseconds)=49762
          13/01/29 20:30:49 INFO mapred.JobClient: Superstep 6 (milliseconds)=45054
          13/01/29 20:30:49 INFO mapred.JobClient: Superstep 5 (milliseconds)=40220
          13/01/29 20:30:49 INFO mapred.JobClient: Superstep 2 (milliseconds)=40817
          13/01/29 20:30:49 INFO mapred.JobClient: Superstep 1 (milliseconds)=37830
          13/01/29 20:30:49 INFO mapred.JobClient: Giraph Stats
          13/01/29 20:30:49 INFO mapred.JobClient: Aggregate edges=1000000000
          13/01/29 20:30:49 INFO mapred.JobClient: Superstep=11
          13/01/29 20:30:49 INFO mapred.JobClient: Last checkpointed superstep=0
          13/01/29 20:30:49 INFO mapred.JobClient: Current workers=60
          13/01/29 20:30:49 INFO mapred.JobClient: Current master task partition=0
          13/01/29 20:30:49 INFO mapred.JobClient: Sent messages=0
          13/01/29 20:30:49 INFO mapred.JobClient: Aggregate finished vertices=10000000
          13/01/29 20:30:49 INFO mapred.JobClient: Aggregate vertices=10000000
          13/01/29 20:30:49 INFO mapred.JobClient: File Output Format Counters
          13/01/29 20:30:49 INFO mapred.JobClient: Bytes Written=0
          13/01/29 20:30:49 INFO mapred.JobClient: FileSystemCounters
          13/01/29 20:30:49 INFO mapred.JobClient: HDFS_BYTES_READ=2684
          13/01/29 20:30:49 INFO mapred.JobClient: FILE_BYTES_WRITTEN=1388228
          13/01/29 20:30:49 INFO mapred.JobClient: File Input Format Counters
          13/01/29 20:30:49 INFO mapred.JobClient: Bytes Read=0
          13/01/29 20:30:49 INFO mapred.JobClient: Map-Reduce Framework
          13/01/29 20:30:49 INFO mapred.JobClient: Map input records=61
          13/01/29 20:30:49 INFO mapred.JobClient: Physical memory (bytes) snapshot=71895678976
          13/01/29 20:30:49 INFO mapred.JobClient: Spilled Records=0
          13/01/29 20:30:49 INFO mapred.JobClient: CPU time spent (ms)=15134650
          13/01/29 20:30:49 INFO mapred.JobClient: Total committed heap usage (bytes)=57982255104
          13/01/29 20:30:49 INFO mapred.JobClient: Virtual memory (bytes) snapshot=371448213504
          13/01/29 20:30:49 INFO mapred.JobClient: Map output records=0
          13/01/29 20:30:49 INFO mapred.JobClient: SPLIT_RAW_BYTES=2684

          ooh graph (2 partitions in memory out of 49, 2 compute threads per worker):
          13/01/29 20:11:28 INFO mapred.JobClient: Giraph Timers
          13/01/29 20:11:28 INFO mapred.JobClient: Total (milliseconds)=506380
          13/01/29 20:11:28 INFO mapred.JobClient: Superstep 3 (milliseconds)=41677
          13/01/29 20:11:28 INFO mapred.JobClient: Superstep 4 (milliseconds)=41285
          13/01/29 20:11:28 INFO mapred.JobClient: Superstep 10 (milliseconds)=764
          13/01/29 20:11:28 INFO mapred.JobClient: Setup (milliseconds)=24574
          13/01/29 20:11:28 INFO mapred.JobClient: Shutdown (milliseconds)=82
          13/01/29 20:11:28 INFO mapred.JobClient: Superstep 7 (milliseconds)=43183
          13/01/29 20:11:28 INFO mapred.JobClient: Superstep 9 (milliseconds)=46654
          13/01/29 20:11:28 INFO mapred.JobClient: Superstep 0 (milliseconds)=50955
          13/01/29 20:11:28 INFO mapred.JobClient: Superstep 8 (milliseconds)=40413
          13/01/29 20:11:28 INFO mapred.JobClient: Input superstep (milliseconds)=43584
          13/01/29 20:11:28 INFO mapred.JobClient: Superstep 6 (milliseconds)=46638
          13/01/29 20:11:28 INFO mapred.JobClient: Superstep 5 (milliseconds)=46107
          13/01/29 20:11:28 INFO mapred.JobClient: Superstep 2 (milliseconds)=39321
          13/01/29 20:11:28 INFO mapred.JobClient: Superstep 1 (milliseconds)=41139
          13/01/29 20:11:28 INFO mapred.JobClient: Giraph Stats
          13/01/29 20:11:28 INFO mapred.JobClient: Aggregate edges=1000000000
          13/01/29 20:11:28 INFO mapred.JobClient: Superstep=11
          13/01/29 20:11:28 INFO mapred.JobClient: Last checkpointed superstep=0
          13/01/29 20:11:28 INFO mapred.JobClient: Current workers=60
          13/01/29 20:11:28 INFO mapred.JobClient: Current master task partition=0
          13/01/29 20:11:28 INFO mapred.JobClient: Sent messages=0
          13/01/29 20:11:28 INFO mapred.JobClient: Aggregate finished vertices=10000000
          13/01/29 20:11:28 INFO mapred.JobClient: Aggregate vertices=10000000
          13/01/29 20:11:28 INFO mapred.JobClient: File Output Format Counters
          13/01/29 20:11:28 INFO mapred.JobClient: Bytes Written=0
          13/01/29 20:11:28 INFO mapred.JobClient: FileSystemCounters
          13/01/29 20:11:28 INFO mapred.JobClient: HDFS_BYTES_READ=2684
          13/01/29 20:11:28 INFO mapred.JobClient: FILE_BYTES_WRITTEN=1388228
          13/01/29 20:11:28 INFO mapred.JobClient: File Input Format Counters
          13/01/29 20:11:28 INFO mapred.JobClient: Bytes Read=0
          13/01/29 20:11:28 INFO mapred.JobClient: Map-Reduce Framework
          13/01/29 20:11:28 INFO mapred.JobClient: Map input records=61
          13/01/29 20:11:28 INFO mapred.JobClient: Physical memory (bytes) snapshot=71620620288
          13/01/29 20:11:28 INFO mapred.JobClient: Spilled Records=0
          13/01/29 20:11:28 INFO mapred.JobClient: CPU time spent (ms)=15279810
          13/01/29 20:11:28 INFO mapred.JobClient: Total committed heap usage (bytes)=57294782464
          13/01/29 20:11:28 INFO mapred.JobClient: Virtual memory (bytes) snapshot=370988941312
          13/01/29 20:11:28 INFO mapred.JobClient: Map output records=0
          13/01/29 20:11:28 INFO mapred.JobClient: SPLIT_RAW_BYTES=2684

          Show
          Claudio Martella added a comment - hadoop jar giraph-0.2-SNAPSHOT-for-hadoop-1.0.2-jar-with-dependencies.jar org.apache.giraph.benchmark.PageRankBenchmark -w 60 -c 2 -e 100 -V 10000000 -v -s 10 trunk: 13/01/29 20:40:53 INFO mapred.JobClient: Giraph Timers 13/01/29 20:40:53 INFO mapred.JobClient: Total (milliseconds)=492403 13/01/29 20:40:53 INFO mapred.JobClient: Superstep 3 (milliseconds)=40243 13/01/29 20:40:53 INFO mapred.JobClient: Superstep 4 (milliseconds)=45430 13/01/29 20:40:53 INFO mapred.JobClient: Superstep 10 (milliseconds)=713 13/01/29 20:40:53 INFO mapred.JobClient: Setup (milliseconds)=20832 13/01/29 20:40:53 INFO mapred.JobClient: Shutdown (milliseconds)=56 13/01/29 20:40:53 INFO mapred.JobClient: Superstep 7 (milliseconds)=36753 13/01/29 20:40:53 INFO mapred.JobClient: Superstep 9 (milliseconds)=36363 13/01/29 20:40:53 INFO mapred.JobClient: Superstep 0 (milliseconds)=39558 13/01/29 20:40:53 INFO mapred.JobClient: Superstep 8 (milliseconds)=44548 13/01/29 20:40:53 INFO mapred.JobClient: Input superstep (milliseconds)=59184 13/01/29 20:40:53 INFO mapred.JobClient: Superstep 6 (milliseconds)=40777 13/01/29 20:40:53 INFO mapred.JobClient: Superstep 5 (milliseconds)=43962 13/01/29 20:40:53 INFO mapred.JobClient: Superstep 2 (milliseconds)=37325 13/01/29 20:40:53 INFO mapred.JobClient: Superstep 1 (milliseconds)=46655 13/01/29 20:40:53 INFO mapred.JobClient: Giraph Stats 13/01/29 20:40:53 INFO mapred.JobClient: Aggregate edges=1000000000 13/01/29 20:40:53 INFO mapred.JobClient: Superstep=11 13/01/29 20:40:53 INFO mapred.JobClient: Last checkpointed superstep=0 13/01/29 20:40:53 INFO mapred.JobClient: Current workers=60 13/01/29 20:40:53 INFO mapred.JobClient: Current master task partition=0 13/01/29 20:40:53 INFO mapred.JobClient: Sent messages=0 13/01/29 20:40:53 INFO mapred.JobClient: Aggregate finished vertices=10000000 13/01/29 20:40:53 INFO mapred.JobClient: Aggregate vertices=10000000 13/01/29 20:40:53 INFO mapred.JobClient: File Output Format Counters 13/01/29 20:40:53 INFO mapred.JobClient: Bytes Written=0 13/01/29 20:40:53 INFO mapred.JobClient: FileSystemCounters 13/01/29 20:40:53 INFO mapred.JobClient: HDFS_BYTES_READ=2684 13/01/29 20:40:53 INFO mapred.JobClient: FILE_BYTES_WRITTEN=1388228 13/01/29 20:40:53 INFO mapred.JobClient: File Input Format Counters 13/01/29 20:40:53 INFO mapred.JobClient: Bytes Read=0 13/01/29 20:40:53 INFO mapred.JobClient: Map-Reduce Framework 13/01/29 20:40:53 INFO mapred.JobClient: Map input records=61 13/01/29 20:40:53 INFO mapred.JobClient: Physical memory (bytes) snapshot=71703965696 13/01/29 20:40:53 INFO mapred.JobClient: Spilled Records=0 13/01/29 20:40:53 INFO mapred.JobClient: CPU time spent (ms)=15141630 13/01/29 20:40:53 INFO mapred.JobClient: Total committed heap usage (bytes)=58151337984 13/01/29 20:40:53 INFO mapred.JobClient: Virtual memory (bytes) snapshot=371313995776 13/01/29 20:40:53 INFO mapred.JobClient: Map output records=0 13/01/29 20:40:53 INFO mapred.JobClient: SPLIT_RAW_BYTES=2684 GIRAPH-461 : in memory: 13/01/29 19:35:53 INFO mapred.JobClient: Giraph Timers 13/01/29 19:35:53 INFO mapred.JobClient: Total (milliseconds)=427511 13/01/29 19:35:53 INFO mapred.JobClient: Superstep 3 (milliseconds)=37341 13/01/29 19:35:53 INFO mapred.JobClient: Superstep 4 (milliseconds)=35458 13/01/29 19:35:53 INFO mapred.JobClient: Superstep 10 (milliseconds)=852 13/01/29 19:35:53 INFO mapred.JobClient: Setup (milliseconds)=24825 13/01/29 19:35:53 INFO mapred.JobClient: Shutdown (milliseconds)=50 13/01/29 19:35:53 INFO mapred.JobClient: Superstep 7 (milliseconds)=37557 13/01/29 19:35:53 INFO mapred.JobClient: Superstep 9 (milliseconds)=33961 13/01/29 19:35:53 INFO mapred.JobClient: Superstep 0 (milliseconds)=33048 13/01/29 19:35:53 INFO mapred.JobClient: Superstep 8 (milliseconds)=36345 13/01/29 19:35:53 INFO mapred.JobClient: Input superstep (milliseconds)=44420 13/01/29 19:35:53 INFO mapred.JobClient: Superstep 6 (milliseconds)=33635 13/01/29 19:35:53 INFO mapred.JobClient: Superstep 5 (milliseconds)=41885 13/01/29 19:35:53 INFO mapred.JobClient: Superstep 2 (milliseconds)=35046 13/01/29 19:35:53 INFO mapred.JobClient: Superstep 1 (milliseconds)=33083 13/01/29 19:35:53 INFO mapred.JobClient: Giraph Stats 13/01/29 19:35:53 INFO mapred.JobClient: Aggregate edges=1000000000 13/01/29 19:35:53 INFO mapred.JobClient: Superstep=11 13/01/29 19:35:53 INFO mapred.JobClient: Last checkpointed superstep=0 13/01/29 19:35:53 INFO mapred.JobClient: Current workers=60 13/01/29 19:35:53 INFO mapred.JobClient: Current master task partition=0 13/01/29 19:35:53 INFO mapred.JobClient: Sent messages=0 13/01/29 19:35:53 INFO mapred.JobClient: Aggregate finished vertices=10000000 13/01/29 19:35:53 INFO mapred.JobClient: Aggregate vertices=10000000 13/01/29 19:35:53 INFO mapred.JobClient: File Output Format Counters 13/01/29 19:35:53 INFO mapred.JobClient: Bytes Written=0 13/01/29 19:35:53 INFO mapred.JobClient: FileSystemCounters 13/01/29 19:35:53 INFO mapred.JobClient: HDFS_BYTES_READ=2684 13/01/29 19:35:53 INFO mapred.JobClient: FILE_BYTES_WRITTEN=1388228 13/01/29 19:35:53 INFO mapred.JobClient: File Input Format Counters 13/01/29 19:35:53 INFO mapred.JobClient: Bytes Read=0 13/01/29 19:35:53 INFO mapred.JobClient: Map-Reduce Framework 13/01/29 19:35:53 INFO mapred.JobClient: Map input records=61 13/01/29 19:35:53 INFO mapred.JobClient: Physical memory (bytes) snapshot=71627419648 13/01/29 19:35:53 INFO mapred.JobClient: Spilled Records=0 13/01/29 19:35:53 INFO mapred.JobClient: CPU time spent (ms)=15020990 13/01/29 19:35:53 INFO mapred.JobClient: Total committed heap usage (bytes)=57611911168 13/01/29 19:35:53 INFO mapred.JobClient: Virtual memory (bytes) snapshot=371123154944 13/01/29 19:35:53 INFO mapred.JobClient: Map output records=0 13/01/29 19:35:53 INFO mapred.JobClient: SPLIT_RAW_BYTES=2684 ooh graph (2 partitions in memory out of 49): 13/01/29 19:54:57 INFO mapred.JobClient: Giraph Timers 13/01/29 19:54:57 INFO mapred.JobClient: Total (milliseconds)=508004 13/01/29 19:54:57 INFO mapred.JobClient: Superstep 3 (milliseconds)=38085 13/01/29 19:54:57 INFO mapred.JobClient: Superstep 4 (milliseconds)=40789 13/01/29 19:54:57 INFO mapred.JobClient: Superstep 10 (milliseconds)=811 13/01/29 19:54:57 INFO mapred.JobClient: Setup (milliseconds)=25612 13/01/29 19:54:57 INFO mapred.JobClient: Shutdown (milliseconds)=699 13/01/29 19:54:57 INFO mapred.JobClient: Superstep 7 (milliseconds)=44806 13/01/29 19:54:57 INFO mapred.JobClient: Superstep 9 (milliseconds)=41873 13/01/29 19:54:57 INFO mapred.JobClient: Superstep 0 (milliseconds)=46329 13/01/29 19:54:57 INFO mapred.JobClient: Superstep 8 (milliseconds)=46272 13/01/29 19:54:57 INFO mapred.JobClient: Input superstep (milliseconds)=52395 13/01/29 19:54:57 INFO mapred.JobClient: Superstep 6 (milliseconds)=44337 13/01/29 19:54:57 INFO mapred.JobClient: Superstep 5 (milliseconds)=39379 13/01/29 19:54:57 INFO mapred.JobClient: Superstep 2 (milliseconds)=40452 13/01/29 19:54:57 INFO mapred.JobClient: Superstep 1 (milliseconds)=46155 13/01/29 19:54:57 INFO mapred.JobClient: Giraph Stats 13/01/29 19:54:57 INFO mapred.JobClient: Aggregate edges=1000000000 13/01/29 19:54:57 INFO mapred.JobClient: Superstep=11 13/01/29 19:54:57 INFO mapred.JobClient: Last checkpointed superstep=0 13/01/29 19:54:57 INFO mapred.JobClient: Current workers=60 13/01/29 19:54:57 INFO mapred.JobClient: Current master task partition=0 13/01/29 19:54:57 INFO mapred.JobClient: Sent messages=0 13/01/29 19:54:57 INFO mapred.JobClient: Aggregate finished vertices=10000000 13/01/29 19:54:57 INFO mapred.JobClient: Aggregate vertices=10000000 13/01/29 19:54:57 INFO mapred.JobClient: File Output Format Counters 13/01/29 19:54:57 INFO mapred.JobClient: Bytes Written=0 13/01/29 19:54:57 INFO mapred.JobClient: FileSystemCounters 13/01/29 19:54:57 INFO mapred.JobClient: HDFS_BYTES_READ=2684 13/01/29 19:54:57 INFO mapred.JobClient: FILE_BYTES_WRITTEN=1388228 13/01/29 19:54:57 INFO mapred.JobClient: File Input Format Counters 13/01/29 19:54:57 INFO mapred.JobClient: Bytes Read=0 13/01/29 19:54:57 INFO mapred.JobClient: Map-Reduce Framework 13/01/29 19:54:57 INFO mapred.JobClient: Map input records=61 13/01/29 19:54:57 INFO mapred.JobClient: Physical memory (bytes) snapshot=71368736768 13/01/29 19:54:57 INFO mapred.JobClient: Spilled Records=0 13/01/29 19:54:57 INFO mapred.JobClient: CPU time spent (ms)=15289390 13/01/29 19:54:57 INFO mapred.JobClient: Total committed heap usage (bytes)=57278595072 13/01/29 19:54:57 INFO mapred.JobClient: Virtual memory (bytes) snapshot=370911342592 13/01/29 19:54:57 INFO mapred.JobClient: Map output records=0 13/01/29 19:54:57 INFO mapred.JobClient: SPLIT_RAW_BYTES=2684 in memory (2 compute threads per worker): 13/01/29 20:30:49 INFO mapred.JobClient: Giraph Timers 13/01/29 20:30:49 INFO mapred.JobClient: Total (milliseconds)=487379 13/01/29 20:30:49 INFO mapred.JobClient: Superstep 3 (milliseconds)=46092 13/01/29 20:30:49 INFO mapred.JobClient: Superstep 4 (milliseconds)=44840 13/01/29 20:30:49 INFO mapred.JobClient: Superstep 10 (milliseconds)=745 13/01/29 20:30:49 INFO mapred.JobClient: Setup (milliseconds)=23013 13/01/29 20:30:49 INFO mapred.JobClient: Shutdown (milliseconds)=126 13/01/29 20:30:49 INFO mapred.JobClient: Superstep 7 (milliseconds)=40620 13/01/29 20:30:49 INFO mapred.JobClient: Superstep 9 (milliseconds)=39630 13/01/29 20:30:49 INFO mapred.JobClient: Superstep 0 (milliseconds)=38221 13/01/29 20:30:49 INFO mapred.JobClient: Superstep 8 (milliseconds)=40406 13/01/29 20:30:49 INFO mapred.JobClient: Input superstep (milliseconds)=49762 13/01/29 20:30:49 INFO mapred.JobClient: Superstep 6 (milliseconds)=45054 13/01/29 20:30:49 INFO mapred.JobClient: Superstep 5 (milliseconds)=40220 13/01/29 20:30:49 INFO mapred.JobClient: Superstep 2 (milliseconds)=40817 13/01/29 20:30:49 INFO mapred.JobClient: Superstep 1 (milliseconds)=37830 13/01/29 20:30:49 INFO mapred.JobClient: Giraph Stats 13/01/29 20:30:49 INFO mapred.JobClient: Aggregate edges=1000000000 13/01/29 20:30:49 INFO mapred.JobClient: Superstep=11 13/01/29 20:30:49 INFO mapred.JobClient: Last checkpointed superstep=0 13/01/29 20:30:49 INFO mapred.JobClient: Current workers=60 13/01/29 20:30:49 INFO mapred.JobClient: Current master task partition=0 13/01/29 20:30:49 INFO mapred.JobClient: Sent messages=0 13/01/29 20:30:49 INFO mapred.JobClient: Aggregate finished vertices=10000000 13/01/29 20:30:49 INFO mapred.JobClient: Aggregate vertices=10000000 13/01/29 20:30:49 INFO mapred.JobClient: File Output Format Counters 13/01/29 20:30:49 INFO mapred.JobClient: Bytes Written=0 13/01/29 20:30:49 INFO mapred.JobClient: FileSystemCounters 13/01/29 20:30:49 INFO mapred.JobClient: HDFS_BYTES_READ=2684 13/01/29 20:30:49 INFO mapred.JobClient: FILE_BYTES_WRITTEN=1388228 13/01/29 20:30:49 INFO mapred.JobClient: File Input Format Counters 13/01/29 20:30:49 INFO mapred.JobClient: Bytes Read=0 13/01/29 20:30:49 INFO mapred.JobClient: Map-Reduce Framework 13/01/29 20:30:49 INFO mapred.JobClient: Map input records=61 13/01/29 20:30:49 INFO mapred.JobClient: Physical memory (bytes) snapshot=71895678976 13/01/29 20:30:49 INFO mapred.JobClient: Spilled Records=0 13/01/29 20:30:49 INFO mapred.JobClient: CPU time spent (ms)=15134650 13/01/29 20:30:49 INFO mapred.JobClient: Total committed heap usage (bytes)=57982255104 13/01/29 20:30:49 INFO mapred.JobClient: Virtual memory (bytes) snapshot=371448213504 13/01/29 20:30:49 INFO mapred.JobClient: Map output records=0 13/01/29 20:30:49 INFO mapred.JobClient: SPLIT_RAW_BYTES=2684 ooh graph (2 partitions in memory out of 49, 2 compute threads per worker): 13/01/29 20:11:28 INFO mapred.JobClient: Giraph Timers 13/01/29 20:11:28 INFO mapred.JobClient: Total (milliseconds)=506380 13/01/29 20:11:28 INFO mapred.JobClient: Superstep 3 (milliseconds)=41677 13/01/29 20:11:28 INFO mapred.JobClient: Superstep 4 (milliseconds)=41285 13/01/29 20:11:28 INFO mapred.JobClient: Superstep 10 (milliseconds)=764 13/01/29 20:11:28 INFO mapred.JobClient: Setup (milliseconds)=24574 13/01/29 20:11:28 INFO mapred.JobClient: Shutdown (milliseconds)=82 13/01/29 20:11:28 INFO mapred.JobClient: Superstep 7 (milliseconds)=43183 13/01/29 20:11:28 INFO mapred.JobClient: Superstep 9 (milliseconds)=46654 13/01/29 20:11:28 INFO mapred.JobClient: Superstep 0 (milliseconds)=50955 13/01/29 20:11:28 INFO mapred.JobClient: Superstep 8 (milliseconds)=40413 13/01/29 20:11:28 INFO mapred.JobClient: Input superstep (milliseconds)=43584 13/01/29 20:11:28 INFO mapred.JobClient: Superstep 6 (milliseconds)=46638 13/01/29 20:11:28 INFO mapred.JobClient: Superstep 5 (milliseconds)=46107 13/01/29 20:11:28 INFO mapred.JobClient: Superstep 2 (milliseconds)=39321 13/01/29 20:11:28 INFO mapred.JobClient: Superstep 1 (milliseconds)=41139 13/01/29 20:11:28 INFO mapred.JobClient: Giraph Stats 13/01/29 20:11:28 INFO mapred.JobClient: Aggregate edges=1000000000 13/01/29 20:11:28 INFO mapred.JobClient: Superstep=11 13/01/29 20:11:28 INFO mapred.JobClient: Last checkpointed superstep=0 13/01/29 20:11:28 INFO mapred.JobClient: Current workers=60 13/01/29 20:11:28 INFO mapred.JobClient: Current master task partition=0 13/01/29 20:11:28 INFO mapred.JobClient: Sent messages=0 13/01/29 20:11:28 INFO mapred.JobClient: Aggregate finished vertices=10000000 13/01/29 20:11:28 INFO mapred.JobClient: Aggregate vertices=10000000 13/01/29 20:11:28 INFO mapred.JobClient: File Output Format Counters 13/01/29 20:11:28 INFO mapred.JobClient: Bytes Written=0 13/01/29 20:11:28 INFO mapred.JobClient: FileSystemCounters 13/01/29 20:11:28 INFO mapred.JobClient: HDFS_BYTES_READ=2684 13/01/29 20:11:28 INFO mapred.JobClient: FILE_BYTES_WRITTEN=1388228 13/01/29 20:11:28 INFO mapred.JobClient: File Input Format Counters 13/01/29 20:11:28 INFO mapred.JobClient: Bytes Read=0 13/01/29 20:11:28 INFO mapred.JobClient: Map-Reduce Framework 13/01/29 20:11:28 INFO mapred.JobClient: Map input records=61 13/01/29 20:11:28 INFO mapred.JobClient: Physical memory (bytes) snapshot=71620620288 13/01/29 20:11:28 INFO mapred.JobClient: Spilled Records=0 13/01/29 20:11:28 INFO mapred.JobClient: CPU time spent (ms)=15279810 13/01/29 20:11:28 INFO mapred.JobClient: Total committed heap usage (bytes)=57294782464 13/01/29 20:11:28 INFO mapred.JobClient: Virtual memory (bytes) snapshot=370988941312 13/01/29 20:11:28 INFO mapred.JobClient: Map output records=0 13/01/29 20:11:28 INFO mapred.JobClient: SPLIT_RAW_BYTES=2684
          Show
          Claudio Martella added a comment - https://reviews.apache.org/r/9278/
          Hide
          Eli Reisman added a comment -

          I don't have cluster access yet, might be a while on my end. Sorry. Will check back when I get it.

          Show
          Eli Reisman added a comment - I don't have cluster access yet, might be a while on my end. Sorry. Will check back when I get it.
          Hide
          Claudio Martella added a comment -

          If anybody who can successfully run trunk on his cluster is willing to give this a go (I can rebase if necessary), it would be helpful.

          Show
          Claudio Martella added a comment - If anybody who can successfully run trunk on his cluster is willing to give this a go (I can rebase if necessary), it would be helpful.
          Hide
          Claudio Martella added a comment -

          This one passes local tests and my ad-hoc tests. Cannot run pseudo-distributed tests or on real cluster (also on trunk).

          Show
          Claudio Martella added a comment - This one passes local tests and my ad-hoc tests. Cannot run pseudo-distributed tests or on real cluster (also on trunk).

            People

            • Assignee:
              Claudio Martella
              Reporter:
              Claudio Martella
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development