Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-8582

/update/json/docs is 4x slower than /update for indexing a list of json docs

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 5.3.2, 5.4.1
    • Fix Version/s: 5.5, 6.0
    • Component/s: update
    • Labels:
      None

      Description

      Indexing a ~650 MB json file containing a list of 2.2 million json documents, I found that bin/post had become 4x slower after SOLR-7042. Memory consumption has also gone up and I can no longer index this file with a 512mb heap.

      The difference is because we now default to /update/json/docs instead of /update. This can be verified on trunk:

      time curl 'http://localhost:8983/solr/gettingstarted/update' --data-binary @/hdd/solr-data/imdb.json 
      {"responseHeader":{"status":0,"QTime":161869}}
      ​
      real	2m42.044s
      user	0m0.292s
      sys	0m0.493s
      ​
      time curl 'http://localhost:8983/solr/gettingstarted/update/json/docs' --data-binary @/hdd/solr-data/imdb.json 
      {"responseHeader":{"status":0,"QTime":686264}}
      ​
      real	11m26.478s
      user	0m0.324s
      sys	0m0.552s
      

        Attachments

        1. SOLR-8582.patch
          2 kB
          Noble Paul
        2. SOLR-8582.patch
          4 kB
          Noble Paul

          Activity

            People

            • Assignee:
              noble.paul Noble Paul
              Reporter:
              shalinmangar Shalin Shekhar Mangar
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: