Description
As we already know, kryo serializer shows quite awesome performance. I suggest that we use kryo serializer for everything.
Attached patch uses kryo for message serialization and below is its results:
15/03/10 18:08:08 INFO bsp.BSPJobClient: TASK_INPUT_RECORDS=30000 15/03/10 18:08:08 INFO bsp.FileInputFormat: Total input paths to process : 3 15/03/10 18:08:09 INFO bsp.BSPJobClient: Running job: job_201503101804_0007 15/03/10 18:08:12 INFO bsp.BSPJobClient: Current supersteps number: 0 15/03/10 18:09:34 INFO bsp.BSPJobClient: Current supersteps number: 1 15/03/10 18:11:24 INFO bsp.BSPJobClient: Current supersteps number: 2 15/03/10 18:11:27 INFO bsp.BSPJobClient: Current supersteps number: 3 15/03/10 18:14:33 INFO bsp.BSPJobClient: Current supersteps number: 4 15/03/10 18:14:36 INFO bsp.BSPJobClient: Current supersteps number: 5 15/03/10 18:17:20 INFO bsp.BSPJobClient: Current supersteps number: 7 15/03/10 18:18:12 INFO bsp.BSPJobClient: Current supersteps number: 9 15/03/10 18:18:15 INFO bsp.BSPJobClient: Current supersteps number: 11 15/03/10 18:18:30 INFO bsp.BSPJobClient: The total number of supersteps: 11 15/03/10 18:18:30 INFO bsp.BSPJobClient: Counters: 11 15/03/10 18:18:30 INFO bsp.BSPJobClient: org.apache.hama.graph.GraphJobRunner$GraphJobCounter 15/03/10 18:18:30 INFO bsp.BSPJobClient: ITERATIONS=4 15/03/10 18:18:30 INFO bsp.BSPJobClient: INPUT_VERTICES=30000 15/03/10 18:18:30 INFO bsp.BSPJobClient: org.apache.hama.bsp.JobInProgress$JobCounter 15/03/10 18:18:30 INFO bsp.BSPJobClient: SUPERSTEPS=11 15/03/10 18:18:30 INFO bsp.BSPJobClient: LAUNCHED_TASKS=3 15/03/10 18:18:30 INFO bsp.BSPJobClient: org.apache.hama.bsp.BSPPeerImpl$PeerCounter 15/03/10 18:18:30 INFO bsp.BSPJobClient: SUPERSTEP_SUM=33 15/03/10 18:18:30 INFO bsp.BSPJobClient: TIME_IN_SYNC_MS=596707 15/03/10 18:18:30 INFO bsp.BSPJobClient: IO_BYTES_READ=84870046 15/03/10 18:18:30 INFO bsp.BSPJobClient: TOTAL_MESSAGES_SENT=44717766 15/03/10 18:18:30 INFO bsp.BSPJobClient: TASK_INPUT_RECORDS=30000 15/03/10 18:18:30 INFO bsp.BSPJobClient: TOTAL_MESSAGE_BYTES_TRANSFERED=361694719 15/03/10 18:18:30 INFO bsp.BSPJobClient: TOTAL_MESSAGES_RECEIVED=270035 Job Finished in 639.248 seconds .... Kryo version: 15/03/10 18:23:17 INFO bsp.BSPJobClient: TASK_INPUT_RECORDS=30000 15/03/10 18:23:17 INFO bsp.FileInputFormat: Total input paths to process : 3 15/03/10 18:23:17 INFO bsp.BSPJobClient: Running job: job_201503101822_0001 15/03/10 18:23:20 INFO bsp.BSPJobClient: Current supersteps number: 0 15/03/10 18:23:35 INFO bsp.BSPJobClient: Current supersteps number: 1 15/03/10 18:24:16 INFO bsp.BSPJobClient: Current supersteps number: 2 15/03/10 18:24:19 INFO bsp.BSPJobClient: Current supersteps number: 3 15/03/10 18:25:23 INFO bsp.BSPJobClient: Current supersteps number: 4 15/03/10 18:25:26 INFO bsp.BSPJobClient: Current supersteps number: 5 15/03/10 18:26:47 INFO bsp.BSPJobClient: Current supersteps number: 7 15/03/10 18:26:53 INFO bsp.BSPJobClient: Current supersteps number: 9 15/03/10 18:26:56 INFO bsp.BSPJobClient: Current supersteps number: 11 15/03/10 18:27:10 INFO bsp.BSPJobClient: The total number of supersteps: 11 15/03/10 18:27:10 INFO bsp.BSPJobClient: Counters: 11 15/03/10 18:27:10 INFO bsp.BSPJobClient: org.apache.hama.graph.GraphJobRunner$GraphJobCounter 15/03/10 18:27:10 INFO bsp.BSPJobClient: ITERATIONS=4 15/03/10 18:27:10 INFO bsp.BSPJobClient: INPUT_VERTICES=30000 15/03/10 18:27:10 INFO bsp.BSPJobClient: org.apache.hama.bsp.JobInProgress$JobCounter 15/03/10 18:27:10 INFO bsp.BSPJobClient: SUPERSTEPS=11 15/03/10 18:27:10 INFO bsp.BSPJobClient: LAUNCHED_TASKS=3 15/03/10 18:27:10 INFO bsp.BSPJobClient: org.apache.hama.bsp.BSPPeerImpl$PeerCounter 15/03/10 18:27:10 INFO bsp.BSPJobClient: SUPERSTEP_SUM=33 15/03/10 18:27:10 INFO bsp.BSPJobClient: TIME_IN_SYNC_MS=185464 15/03/10 18:27:10 INFO bsp.BSPJobClient: IO_BYTES_READ=84870046 15/03/10 18:27:10 INFO bsp.BSPJobClient: TOTAL_MESSAGES_SENT=44717766 15/03/10 18:27:10 INFO bsp.BSPJobClient: TASK_INPUT_RECORDS=30000 15/03/10 18:27:10 INFO bsp.BSPJobClient: TOTAL_MESSAGE_BYTES_TRANSFERED=573865062 15/03/10 18:27:10 INFO bsp.BSPJobClient: TOTAL_MESSAGES_RECEIVED=270039 Job Finished in 250.217 seconds
Attachments
Attachments
1.
|
Add Kryo Server/Client based Messenger | Resolved | Unassigned |