[IGNITE-5996] Load test with >100 servers: slow topology setup and hanging during preloading

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.1
• Assignee: Unassigned
• Reporter: Ksenia Rybakova
• Created: 08/Aug/17 15:32
• Updated: 08/Aug/17 15:44

Description

Load test config:

• CacheRandomOperationBenchmark
• 30 caches:
  2 atomic partitioned
  2 atomic partitioned with offheap eviction
  atomic replicated
  2 atomic replicated with offheap eviction
  3 atomic partitioned onheap
  3 atomic replicated onheap
  4 atomic partitioned indexed
  2 transactional partitioned
  2 transactional partitioned with offheap eviction
  transactional replicated
  2 transactional replicated with offheap eviction
  3 transactional partitioned onheap
  3 transactional replicated onheap
• Preload amount: 100K-500K
• Key range: 200K
• 7 client nodes, 119 server nodes
• Persistent store is disabled
  Complete configs are attached.

Topology is set up only after ~15 minutes. "Failed to wait for partition map exchange" periodically appears in logs.

Then test hung during preloading. 28 of 30 caches are loaded, but 2 remain empty.