Uploaded image for project: 'Hama'
  1. Hama
  2. HAMA-932

Use Kryo Serializer for message and bundle serializations.

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.6.4
    • Fix Version/s: 0.7.0
    • Component/s: bsp core
    • Labels:
      None

      Description

      As we already know, kryo serializer shows quite awesome performance. I suggest that we use kryo serializer for everything.

      Attached patch uses kryo for message serialization and below is its results:

      15/03/10 18:08:08 INFO bsp.BSPJobClient:     TASK_INPUT_RECORDS=30000
      15/03/10 18:08:08 INFO bsp.FileInputFormat: Total input paths to process : 3
      15/03/10 18:08:09 INFO bsp.BSPJobClient: Running job: job_201503101804_0007
      15/03/10 18:08:12 INFO bsp.BSPJobClient: Current supersteps number: 0
      15/03/10 18:09:34 INFO bsp.BSPJobClient: Current supersteps number: 1
      15/03/10 18:11:24 INFO bsp.BSPJobClient: Current supersteps number: 2
      15/03/10 18:11:27 INFO bsp.BSPJobClient: Current supersteps number: 3
      15/03/10 18:14:33 INFO bsp.BSPJobClient: Current supersteps number: 4
      15/03/10 18:14:36 INFO bsp.BSPJobClient: Current supersteps number: 5
      15/03/10 18:17:20 INFO bsp.BSPJobClient: Current supersteps number: 7
      15/03/10 18:18:12 INFO bsp.BSPJobClient: Current supersteps number: 9
      15/03/10 18:18:15 INFO bsp.BSPJobClient: Current supersteps number: 11
      15/03/10 18:18:30 INFO bsp.BSPJobClient: The total number of supersteps: 11
      15/03/10 18:18:30 INFO bsp.BSPJobClient: Counters: 11
      15/03/10 18:18:30 INFO bsp.BSPJobClient:   org.apache.hama.graph.GraphJobRunner$GraphJobCounter
      15/03/10 18:18:30 INFO bsp.BSPJobClient:     ITERATIONS=4
      15/03/10 18:18:30 INFO bsp.BSPJobClient:     INPUT_VERTICES=30000
      15/03/10 18:18:30 INFO bsp.BSPJobClient:   org.apache.hama.bsp.JobInProgress$JobCounter
      15/03/10 18:18:30 INFO bsp.BSPJobClient:     SUPERSTEPS=11
      15/03/10 18:18:30 INFO bsp.BSPJobClient:     LAUNCHED_TASKS=3
      15/03/10 18:18:30 INFO bsp.BSPJobClient:   org.apache.hama.bsp.BSPPeerImpl$PeerCounter
      15/03/10 18:18:30 INFO bsp.BSPJobClient:     SUPERSTEP_SUM=33
      15/03/10 18:18:30 INFO bsp.BSPJobClient:     TIME_IN_SYNC_MS=596707
      15/03/10 18:18:30 INFO bsp.BSPJobClient:     IO_BYTES_READ=84870046
      15/03/10 18:18:30 INFO bsp.BSPJobClient:     TOTAL_MESSAGES_SENT=44717766
      15/03/10 18:18:30 INFO bsp.BSPJobClient:     TASK_INPUT_RECORDS=30000
      15/03/10 18:18:30 INFO bsp.BSPJobClient:     TOTAL_MESSAGE_BYTES_TRANSFERED=361694719
      15/03/10 18:18:30 INFO bsp.BSPJobClient:     TOTAL_MESSAGES_RECEIVED=270035
      Job Finished in 639.248 seconds
      
      
      .... 
      Kryo version:
      
      15/03/10 18:23:17 INFO bsp.BSPJobClient:     TASK_INPUT_RECORDS=30000
      15/03/10 18:23:17 INFO bsp.FileInputFormat: Total input paths to process : 3
      15/03/10 18:23:17 INFO bsp.BSPJobClient: Running job: job_201503101822_0001
      15/03/10 18:23:20 INFO bsp.BSPJobClient: Current supersteps number: 0
      15/03/10 18:23:35 INFO bsp.BSPJobClient: Current supersteps number: 1
      15/03/10 18:24:16 INFO bsp.BSPJobClient: Current supersteps number: 2
      15/03/10 18:24:19 INFO bsp.BSPJobClient: Current supersteps number: 3
      15/03/10 18:25:23 INFO bsp.BSPJobClient: Current supersteps number: 4
      15/03/10 18:25:26 INFO bsp.BSPJobClient: Current supersteps number: 5
      15/03/10 18:26:47 INFO bsp.BSPJobClient: Current supersteps number: 7
      15/03/10 18:26:53 INFO bsp.BSPJobClient: Current supersteps number: 9
      15/03/10 18:26:56 INFO bsp.BSPJobClient: Current supersteps number: 11
      15/03/10 18:27:10 INFO bsp.BSPJobClient: The total number of supersteps: 11
      15/03/10 18:27:10 INFO bsp.BSPJobClient: Counters: 11
      15/03/10 18:27:10 INFO bsp.BSPJobClient:   org.apache.hama.graph.GraphJobRunner$GraphJobCounter
      15/03/10 18:27:10 INFO bsp.BSPJobClient:     ITERATIONS=4
      15/03/10 18:27:10 INFO bsp.BSPJobClient:     INPUT_VERTICES=30000
      15/03/10 18:27:10 INFO bsp.BSPJobClient:   org.apache.hama.bsp.JobInProgress$JobCounter
      15/03/10 18:27:10 INFO bsp.BSPJobClient:     SUPERSTEPS=11
      15/03/10 18:27:10 INFO bsp.BSPJobClient:     LAUNCHED_TASKS=3
      15/03/10 18:27:10 INFO bsp.BSPJobClient:   org.apache.hama.bsp.BSPPeerImpl$PeerCounter
      15/03/10 18:27:10 INFO bsp.BSPJobClient:     SUPERSTEP_SUM=33
      15/03/10 18:27:10 INFO bsp.BSPJobClient:     TIME_IN_SYNC_MS=185464
      15/03/10 18:27:10 INFO bsp.BSPJobClient:     IO_BYTES_READ=84870046
      15/03/10 18:27:10 INFO bsp.BSPJobClient:     TOTAL_MESSAGES_SENT=44717766
      15/03/10 18:27:10 INFO bsp.BSPJobClient:     TASK_INPUT_RECORDS=30000
      15/03/10 18:27:10 INFO bsp.BSPJobClient:     TOTAL_MESSAGE_BYTES_TRANSFERED=573865062
      15/03/10 18:27:10 INFO bsp.BSPJobClient:     TOTAL_MESSAGES_RECEIVED=270039
      Job Finished in 250.217 seconds
      

        Attachments

        1. kryo.patch
          13 kB
          Edward J. Yoon
        There are no Sub-Tasks for this issue.

          Activity

            People

            • Assignee:
              udanax Edward J. Yoon
              Reporter:
              udanax Edward J. Yoon
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: