Uploaded image for project: 'Ignite'
  1. Ignite
  2. IGNITE-9281 [ML] Starter ML tasks
  3. IGNITE-9432

Investigate ability to make ScanQuery faster

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 2.7
    • Fix Version/s: 2.9
    • Component/s: ml
    • Labels:
    • Ignite Flags:
      Docs Required

      Description

      When we make ScanQuery via Binary Client Protocol it works very slowly in case we need to retrieve a big amount of objects (all objects in our case). The most slower part is a waiting for response from Apache Ignite. This also mentioned on devlist http://apache-ignite-developers.2346864.n4.nabble.com/How-to-reduce-Scan-Query-execution-time-td34212.html.

      To reproduce the problem we've prepared a small example: slow-scan-query-reproducer. In this example we creates a cache with 500 objects 1Mb each (on localhost), we make ScanQuery with different page sizes and calculate time between the moment when the request has been sent and the moment when the response is ready to be receive. The measurements are here:

      Page size 5 Mb, waiting time 119.85 ± 6.72 ms
      Page size 10 Mb, waiting time 157.70 ± 15.35 ms
      Page size 20 Mb, waiting time 204.50 ± 19.18 ms
      Page size 50 Mb, waiting time 264.70 ± 22.30 ms
      Page size 100 Mb, waiting time 463.35 ± 17.12 ms
      Page size 150 Mb, waiting time 672.50 ± 21.98 ms

      As result we spend ~4ms per every megabyte on something. It means that we will be able to achieve 250Mb/s throughput in best case. It's too slow, isn't it?

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              dmitrievanthony Anton Dmitriev
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: