Details

    • Type: Improvement
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 6.1, 7.0
    • Component/s: None
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      Like postings, points make heavy use of sorting at indexing time, so we should try to leverage radix sort too?

      1. LUCENE-7903.patch
        7 kB
        Adrien Grand
      2. LUCENE-7306.patch
        28 kB
        Adrien Grand

        Activity

        Hide
        jpountz Adrien Grand added a comment -

        Here is a simple patch that uses radix sorting on the last dimension (which is convenient since the bytes for the dimension and for the doc id are contiguous).

        I used IndexAndSearchOpenStreetMaps to benchmark. The indexing time went from 344s to 327s (-5%). Here are the first 30 logs for merging points in both cases:

        Master

        SM 0 [2016-05-26T16:28:35.224Z; Thread-0]: 2414 msec to merge points [1096300 docs]
        SM 0 [2016-05-26T16:28:39.390Z; Thread-0]: 1899 msec to merge points [1096300 docs]
        SM 0 [2016-05-26T16:28:43.443Z; Thread-0]: 1869 msec to merge points [1096300 docs]
        SM 0 [2016-05-26T16:28:47.426Z; Thread-0]: 1812 msec to merge points [1096300 docs]
        SM 0 [2016-05-26T16:28:51.444Z; Thread-0]: 1850 msec to merge points [1096300 docs]
        SM 0 [2016-05-26T16:28:55.422Z; Thread-0]: 1819 msec to merge points [1096300 docs]
        SM 0 [2016-05-26T16:28:59.409Z; Thread-0]: 1823 msec to merge points [1096300 docs]
        SM 0 [2016-05-26T16:29:03.368Z; Thread-0]: 1817 msec to merge points [1096300 docs]
        SM 0 [2016-05-26T16:29:07.296Z; Thread-0]: 1802 msec to merge points [1096300 docs]
        SM 0 [2016-05-26T16:29:11.205Z; Thread-0]: 1793 msec to merge points [1096300 docs]
        SM 0 [2016-05-26T16:29:34.980Z; Thread-0]: 23722 msec to merge points [10963000 docs]
        SM 0 [2016-05-26T16:29:38.934Z; Thread-0]: 1798 msec to merge points [1096300 docs]
        SM 0 [2016-05-26T16:29:42.844Z; Thread-0]: 1779 msec to merge points [1096300 docs]
        SM 0 [2016-05-26T16:29:46.849Z; Thread-0]: 1797 msec to merge points [1096300 docs]
        SM 0 [2016-05-26T16:29:50.866Z; Thread-0]: 1802 msec to merge points [1096300 docs]
        SM 0 [2016-05-26T16:29:54.917Z; Thread-0]: 1820 msec to merge points [1096300 docs]
        SM 0 [2016-05-26T16:29:58.965Z; Thread-0]: 1823 msec to merge points [1096300 docs]
        SM 0 [2016-05-26T16:30:02.889Z; Thread-0]: 1783 msec to merge points [1096300 docs]
        SM 0 [2016-05-26T16:30:06.815Z; Thread-0]: 1785 msec to merge points [1096300 docs]
        SM 0 [2016-05-26T16:30:10.835Z; Thread-0]: 1876 msec to merge points [1096300 docs]
        SM 0 [2016-05-26T16:30:14.759Z; Thread-0]: 1790 msec to merge points [1096300 docs]
        SM 0 [2016-05-26T16:30:37.886Z; Thread-0]: 23085 msec to merge points [10963000 docs]
        SM 0 [2016-05-26T16:30:41.777Z; Thread-0]: 1783 msec to merge points [1096300 docs]
        SM 0 [2016-05-26T16:30:45.837Z; Thread-0]: 1783 msec to merge points [1096300 docs]
        SM 0 [2016-05-26T16:30:49.731Z; Thread-0]: 1785 msec to merge points [1096300 docs]
        SM 0 [2016-05-26T16:30:53.624Z; Thread-0]: 1776 msec to merge points [1096300 docs]
        SM 0 [2016-05-26T16:30:57.536Z; Thread-0]: 1782 msec to merge points [1096300 docs]
        SM 0 [2016-05-26T16:31:01.512Z; Thread-0]: 1787 msec to merge points [1096300 docs]
        SM 0 [2016-05-26T16:31:05.477Z; Thread-0]: 1786 msec to merge points [1096300 docs]
        SM 0 [2016-05-26T16:31:09.889Z; Thread-0]: 1770 msec to merge points [1096300 docs]
        

        Patch

        SM 0 [2016-05-26T16:20:21.241Z; Thread-0]: 2405 msec to merge points [1096300 docs]
        SM 0 [2016-05-26T16:20:25.072Z; Thread-0]: 1583 msec to merge points [1096300 docs]
        SM 0 [2016-05-26T16:20:28.834Z; Thread-0]: 1537 msec to merge points [1096300 docs]
        SM 0 [2016-05-26T16:20:32.546Z; Thread-0]: 1489 msec to merge points [1096300 docs]
        SM 0 [2016-05-26T16:20:36.426Z; Thread-0]: 1524 msec to merge points [1096300 docs]
        SM 0 [2016-05-26T16:20:40.263Z; Thread-0]: 1519 msec to merge points [1096300 docs]
        SM 0 [2016-05-26T16:20:44.123Z; Thread-0]: 1511 msec to merge points [1096300 docs]
        SM 0 [2016-05-26T16:20:48.013Z; Thread-0]: 1506 msec to merge points [1096300 docs]
        SM 0 [2016-05-26T16:20:51.807Z; Thread-0]: 1486 msec to merge points [1096300 docs]
        SM 0 [2016-05-26T16:20:55.882Z; Thread-0]: 1479 msec to merge points [1096300 docs]
        SM 0 [2016-05-26T16:21:17.042Z; Thread-0]: 21106 msec to merge points [10963000 docs]
        SM 0 [2016-05-26T16:21:20.872Z; Thread-0]: 1517 msec to merge points [1096300 docs]
        SM 0 [2016-05-26T16:21:24.629Z; Thread-0]: 1467 msec to merge points [1096300 docs]
        SM 0 [2016-05-26T16:21:28.408Z; Thread-0]: 1479 msec to merge points [1096300 docs]
        SM 0 [2016-05-26T16:21:32.219Z; Thread-0]: 1485 msec to merge points [1096300 docs]
        SM 0 [2016-05-26T16:21:36.108Z; Thread-0]: 1501 msec to merge points [1096300 docs]
        SM 0 [2016-05-26T16:21:39.982Z; Thread-0]: 1504 msec to merge points [1096300 docs]
        SM 0 [2016-05-26T16:21:44.836Z; Thread-0]: 1502 msec to merge points [1096300 docs]
        SM 0 [2016-05-26T16:21:48.717Z; Thread-0]: 1499 msec to merge points [1096300 docs]
        SM 0 [2016-05-26T16:21:52.548Z; Thread-0]: 1503 msec to merge points [1096300 docs]
        SM 0 [2016-05-26T16:21:56.436Z; Thread-0]: 1514 msec to merge points [1096300 docs]
        SM 0 [2016-05-26T16:22:17.361Z; Thread-0]: 20883 msec to merge points [10963000 docs]
        SM 0 [2016-05-26T16:22:21.197Z; Thread-0]: 1515 msec to merge points [1096300 docs]
        SM 0 [2016-05-26T16:22:25.077Z; Thread-0]: 1513 msec to merge points [1096300 docs]
        SM 0 [2016-05-26T16:22:28.864Z; Thread-0]: 1504 msec to merge points [1096300 docs]
        SM 0 [2016-05-26T16:22:32.675Z; Thread-0]: 1494 msec to merge points [1096300 docs]
        SM 0 [2016-05-26T16:22:36.503Z; Thread-0]: 1500 msec to merge points [1096300 docs]
        SM 0 [2016-05-26T16:22:40.337Z; Thread-0]: 1516 msec to merge points [1096300 docs]
        SM 0 [2016-05-26T16:22:44.281Z; Thread-0]: 1496 msec to merge points [1096300 docs]
        SM 0 [2016-05-26T16:22:48.721Z; Thread-0]: 1503 msec to merge points [1096300 docs]
        
        Show
        jpountz Adrien Grand added a comment - Here is a simple patch that uses radix sorting on the last dimension (which is convenient since the bytes for the dimension and for the doc id are contiguous). I used IndexAndSearchOpenStreetMaps to benchmark. The indexing time went from 344s to 327s (-5%). Here are the first 30 logs for merging points in both cases: Master SM 0 [2016-05-26T16:28:35.224Z; Thread -0]: 2414 msec to merge points [1096300 docs] SM 0 [2016-05-26T16:28:39.390Z; Thread -0]: 1899 msec to merge points [1096300 docs] SM 0 [2016-05-26T16:28:43.443Z; Thread -0]: 1869 msec to merge points [1096300 docs] SM 0 [2016-05-26T16:28:47.426Z; Thread -0]: 1812 msec to merge points [1096300 docs] SM 0 [2016-05-26T16:28:51.444Z; Thread -0]: 1850 msec to merge points [1096300 docs] SM 0 [2016-05-26T16:28:55.422Z; Thread -0]: 1819 msec to merge points [1096300 docs] SM 0 [2016-05-26T16:28:59.409Z; Thread -0]: 1823 msec to merge points [1096300 docs] SM 0 [2016-05-26T16:29:03.368Z; Thread -0]: 1817 msec to merge points [1096300 docs] SM 0 [2016-05-26T16:29:07.296Z; Thread -0]: 1802 msec to merge points [1096300 docs] SM 0 [2016-05-26T16:29:11.205Z; Thread -0]: 1793 msec to merge points [1096300 docs] SM 0 [2016-05-26T16:29:34.980Z; Thread -0]: 23722 msec to merge points [10963000 docs] SM 0 [2016-05-26T16:29:38.934Z; Thread -0]: 1798 msec to merge points [1096300 docs] SM 0 [2016-05-26T16:29:42.844Z; Thread -0]: 1779 msec to merge points [1096300 docs] SM 0 [2016-05-26T16:29:46.849Z; Thread -0]: 1797 msec to merge points [1096300 docs] SM 0 [2016-05-26T16:29:50.866Z; Thread -0]: 1802 msec to merge points [1096300 docs] SM 0 [2016-05-26T16:29:54.917Z; Thread -0]: 1820 msec to merge points [1096300 docs] SM 0 [2016-05-26T16:29:58.965Z; Thread -0]: 1823 msec to merge points [1096300 docs] SM 0 [2016-05-26T16:30:02.889Z; Thread -0]: 1783 msec to merge points [1096300 docs] SM 0 [2016-05-26T16:30:06.815Z; Thread -0]: 1785 msec to merge points [1096300 docs] SM 0 [2016-05-26T16:30:10.835Z; Thread -0]: 1876 msec to merge points [1096300 docs] SM 0 [2016-05-26T16:30:14.759Z; Thread -0]: 1790 msec to merge points [1096300 docs] SM 0 [2016-05-26T16:30:37.886Z; Thread -0]: 23085 msec to merge points [10963000 docs] SM 0 [2016-05-26T16:30:41.777Z; Thread -0]: 1783 msec to merge points [1096300 docs] SM 0 [2016-05-26T16:30:45.837Z; Thread -0]: 1783 msec to merge points [1096300 docs] SM 0 [2016-05-26T16:30:49.731Z; Thread -0]: 1785 msec to merge points [1096300 docs] SM 0 [2016-05-26T16:30:53.624Z; Thread -0]: 1776 msec to merge points [1096300 docs] SM 0 [2016-05-26T16:30:57.536Z; Thread -0]: 1782 msec to merge points [1096300 docs] SM 0 [2016-05-26T16:31:01.512Z; Thread -0]: 1787 msec to merge points [1096300 docs] SM 0 [2016-05-26T16:31:05.477Z; Thread -0]: 1786 msec to merge points [1096300 docs] SM 0 [2016-05-26T16:31:09.889Z; Thread -0]: 1770 msec to merge points [1096300 docs] Patch SM 0 [2016-05-26T16:20:21.241Z; Thread -0]: 2405 msec to merge points [1096300 docs] SM 0 [2016-05-26T16:20:25.072Z; Thread -0]: 1583 msec to merge points [1096300 docs] SM 0 [2016-05-26T16:20:28.834Z; Thread -0]: 1537 msec to merge points [1096300 docs] SM 0 [2016-05-26T16:20:32.546Z; Thread -0]: 1489 msec to merge points [1096300 docs] SM 0 [2016-05-26T16:20:36.426Z; Thread -0]: 1524 msec to merge points [1096300 docs] SM 0 [2016-05-26T16:20:40.263Z; Thread -0]: 1519 msec to merge points [1096300 docs] SM 0 [2016-05-26T16:20:44.123Z; Thread -0]: 1511 msec to merge points [1096300 docs] SM 0 [2016-05-26T16:20:48.013Z; Thread -0]: 1506 msec to merge points [1096300 docs] SM 0 [2016-05-26T16:20:51.807Z; Thread -0]: 1486 msec to merge points [1096300 docs] SM 0 [2016-05-26T16:20:55.882Z; Thread -0]: 1479 msec to merge points [1096300 docs] SM 0 [2016-05-26T16:21:17.042Z; Thread -0]: 21106 msec to merge points [10963000 docs] SM 0 [2016-05-26T16:21:20.872Z; Thread -0]: 1517 msec to merge points [1096300 docs] SM 0 [2016-05-26T16:21:24.629Z; Thread -0]: 1467 msec to merge points [1096300 docs] SM 0 [2016-05-26T16:21:28.408Z; Thread -0]: 1479 msec to merge points [1096300 docs] SM 0 [2016-05-26T16:21:32.219Z; Thread -0]: 1485 msec to merge points [1096300 docs] SM 0 [2016-05-26T16:21:36.108Z; Thread -0]: 1501 msec to merge points [1096300 docs] SM 0 [2016-05-26T16:21:39.982Z; Thread -0]: 1504 msec to merge points [1096300 docs] SM 0 [2016-05-26T16:21:44.836Z; Thread -0]: 1502 msec to merge points [1096300 docs] SM 0 [2016-05-26T16:21:48.717Z; Thread -0]: 1499 msec to merge points [1096300 docs] SM 0 [2016-05-26T16:21:52.548Z; Thread -0]: 1503 msec to merge points [1096300 docs] SM 0 [2016-05-26T16:21:56.436Z; Thread -0]: 1514 msec to merge points [1096300 docs] SM 0 [2016-05-26T16:22:17.361Z; Thread -0]: 20883 msec to merge points [10963000 docs] SM 0 [2016-05-26T16:22:21.197Z; Thread -0]: 1515 msec to merge points [1096300 docs] SM 0 [2016-05-26T16:22:25.077Z; Thread -0]: 1513 msec to merge points [1096300 docs] SM 0 [2016-05-26T16:22:28.864Z; Thread -0]: 1504 msec to merge points [1096300 docs] SM 0 [2016-05-26T16:22:32.675Z; Thread -0]: 1494 msec to merge points [1096300 docs] SM 0 [2016-05-26T16:22:36.503Z; Thread -0]: 1500 msec to merge points [1096300 docs] SM 0 [2016-05-26T16:22:40.337Z; Thread -0]: 1516 msec to merge points [1096300 docs] SM 0 [2016-05-26T16:22:44.281Z; Thread -0]: 1496 msec to merge points [1096300 docs] SM 0 [2016-05-26T16:22:48.721Z; Thread -0]: 1503 msec to merge points [1096300 docs]
        Hide
        mikemccand Michael McCandless added a comment -

        +1, wonderful!

        Show
        mikemccand Michael McCandless added a comment - +1, wonderful!
        Hide
        jpountz Adrien Grand added a comment -

        I started playing with the last dimension since it was a low hanging fruit, but I'll explore if we can make things better for the other dimensions and the heap writer too.

        Show
        jpountz Adrien Grand added a comment - I started playing with the last dimension since it was a low hanging fruit, but I'll explore if we can make things better for the other dimensions and the heap writer too.
        Hide
        jpountz Adrien Grand added a comment -

        I switched to radix sort for other dimensions and HeapPointsWriter too and the speedup is quite spectacular. This time it only took 241s to index (-30%). Both flushing and merging are significantly faster:

        Master flush

        IW 0 [2016-05-26T16:28:30.643Z; Thread-0]: 261 msec to write points
        IW 0 [2016-05-26T16:28:30.987Z; Thread-0]: 176 msec to write points
        IW 0 [2016-05-26T16:28:31.238Z; Thread-0]: 172 msec to write points
        IW 0 [2016-05-26T16:28:31.457Z; Thread-0]: 146 msec to write points
        IW 0 [2016-05-26T16:28:31.676Z; Thread-0]: 147 msec to write points
        IW 0 [2016-05-26T16:28:31.894Z; Thread-0]: 145 msec to write points
        IW 0 [2016-05-26T16:28:32.114Z; Thread-0]: 146 msec to write points
        IW 0 [2016-05-26T16:28:32.330Z; Thread-0]: 143 msec to write points
        IW 0 [2016-05-26T16:28:32.549Z; Thread-0]: 146 msec to write points
        IW 0 [2016-05-26T16:28:32.774Z; Thread-0]: 147 msec to write points
        IW 0 [2016-05-26T16:28:35.450Z; Thread-0]: 157 msec to write points
        IW 0 [2016-05-26T16:28:35.704Z; Thread-0]: 150 msec to write points
        IW 0 [2016-05-26T16:28:35.925Z; Thread-0]: 152 msec to write points
        IW 0 [2016-05-26T16:28:36.141Z; Thread-0]: 148 msec to write points
        IW 0 [2016-05-26T16:28:36.357Z; Thread-0]: 149 msec to write points
        

        Patch flush

        IW 0 [2016-05-27T10:48:42.854Z; Thread-0]: 213 msec to write points
        IW 0 [2016-05-27T10:48:43.117Z; Thread-0]: 138 msec to write points
        IW 0 [2016-05-27T10:48:43.343Z; Thread-0]: 108 msec to write points
        IW 0 [2016-05-27T10:48:43.518Z; Thread-0]: 91 msec to write points
        IW 0 [2016-05-27T10:48:43.686Z; Thread-0]: 97 msec to write points
        IW 0 [2016-05-27T10:48:43.856Z; Thread-0]: 99 msec to write points
        IW 0 [2016-05-27T10:48:44.039Z; Thread-0]: 100 msec to write points
        IW 0 [2016-05-27T10:48:44.231Z; Thread-0]: 100 msec to write points
        IW 0 [2016-05-27T10:48:44.415Z; Thread-0]: 97 msec to write points
        IW 0 [2016-05-27T10:48:44.603Z; Thread-0]: 96 msec to write points
        IW 0 [2016-05-27T10:48:47.069Z; Thread-0]: 107 msec to write points
        IW 0 [2016-05-27T10:48:47.230Z; Thread-0]: 92 msec to write points
        IW 0 [2016-05-27T10:48:47.391Z; Thread-0]: 92 msec to write points
        IW 0 [2016-05-27T10:48:47.550Z; Thread-0]: 92 msec to write points
        IW 0 [2016-05-27T10:48:47.711Z; Thread-0]: 93 msec to write points
        

        Master merge

        SM 0 [2016-05-26T16:28:35.224Z; Thread-0]: 2414 msec to merge points [1096300 docs]
        SM 0 [2016-05-26T16:28:39.390Z; Thread-0]: 1899 msec to merge points [1096300 docs]
        SM 0 [2016-05-26T16:28:43.443Z; Thread-0]: 1869 msec to merge points [1096300 docs]
        SM 0 [2016-05-26T16:28:47.426Z; Thread-0]: 1812 msec to merge points [1096300 docs]
        SM 0 [2016-05-26T16:28:51.444Z; Thread-0]: 1850 msec to merge points [1096300 docs]
        SM 0 [2016-05-26T16:28:55.422Z; Thread-0]: 1819 msec to merge points [1096300 docs]
        SM 0 [2016-05-26T16:28:59.409Z; Thread-0]: 1823 msec to merge points [1096300 docs]
        SM 0 [2016-05-26T16:29:03.368Z; Thread-0]: 1817 msec to merge points [1096300 docs]
        SM 0 [2016-05-26T16:29:07.296Z; Thread-0]: 1802 msec to merge points [1096300 docs]
        SM 0 [2016-05-26T16:29:11.205Z; Thread-0]: 1793 msec to merge points [1096300 docs]
        SM 0 [2016-05-26T16:29:34.980Z; Thread-0]: 23722 msec to merge points [10963000 docs]
        SM 0 [2016-05-26T16:29:38.934Z; Thread-0]: 1798 msec to merge points [1096300 docs]
        SM 0 [2016-05-26T16:29:42.844Z; Thread-0]: 1779 msec to merge points [1096300 docs]
        SM 0 [2016-05-26T16:29:46.849Z; Thread-0]: 1797 msec to merge points [1096300 docs]
        SM 0 [2016-05-26T16:29:50.866Z; Thread-0]: 1802 msec to merge points [1096300 docs]
        

        Patch merge

        SM 0 [2016-05-27T10:48:46.890Z; Thread-0]: 2252 msec to merge points [1096300 docs]
        SM 0 [2016-05-27T10:48:49.943Z; Thread-0]: 1184 msec to merge points [1096300 docs]
        SM 0 [2016-05-27T10:48:52.722Z; Thread-0]: 1157 msec to merge points [1096300 docs]
        SM 0 [2016-05-27T10:48:55.440Z; Thread-0]: 1140 msec to merge points [1096300 docs]
        SM 0 [2016-05-27T10:48:58.146Z; Thread-0]: 1169 msec to merge points [1096300 docs]
        SM 0 [2016-05-27T10:49:00.915Z; Thread-0]: 1200 msec to merge points [1096300 docs]
        SM 0 [2016-05-27T10:49:03.684Z; Thread-0]: 1149 msec to merge points [1096300 docs]
        SM 0 [2016-05-27T10:49:06.436Z; Thread-0]: 1126 msec to merge points [1096300 docs]
        SM 0 [2016-05-27T10:49:09.169Z; Thread-0]: 1148 msec to merge points [1096300 docs]
        SM 0 [2016-05-27T10:49:11.903Z; Thread-0]: 1165 msec to merge points [1096300 docs]
        SM 0 [2016-05-27T10:49:28.140Z; Thread-0]: 16179 msec to merge points [10963000 docs]
        SM 0 [2016-05-27T10:49:30.913Z; Thread-0]: 1159 msec to merge points [1096300 docs]
        SM 0 [2016-05-27T10:49:33.609Z; Thread-0]: 1135 msec to merge points [1096300 docs]
        SM 0 [2016-05-27T10:49:36.383Z; Thread-0]: 1149 msec to merge points [1096300 docs]
        SM 0 [2016-05-27T10:49:39.118Z; Thread-0]: 1143 msec to merge points [1096300 docs]
        
        Show
        jpountz Adrien Grand added a comment - I switched to radix sort for other dimensions and HeapPointsWriter too and the speedup is quite spectacular. This time it only took 241s to index (-30%). Both flushing and merging are significantly faster: Master flush IW 0 [2016-05-26T16:28:30.643Z; Thread-0]: 261 msec to write points IW 0 [2016-05-26T16:28:30.987Z; Thread-0]: 176 msec to write points IW 0 [2016-05-26T16:28:31.238Z; Thread-0]: 172 msec to write points IW 0 [2016-05-26T16:28:31.457Z; Thread-0]: 146 msec to write points IW 0 [2016-05-26T16:28:31.676Z; Thread-0]: 147 msec to write points IW 0 [2016-05-26T16:28:31.894Z; Thread-0]: 145 msec to write points IW 0 [2016-05-26T16:28:32.114Z; Thread-0]: 146 msec to write points IW 0 [2016-05-26T16:28:32.330Z; Thread-0]: 143 msec to write points IW 0 [2016-05-26T16:28:32.549Z; Thread-0]: 146 msec to write points IW 0 [2016-05-26T16:28:32.774Z; Thread-0]: 147 msec to write points IW 0 [2016-05-26T16:28:35.450Z; Thread-0]: 157 msec to write points IW 0 [2016-05-26T16:28:35.704Z; Thread-0]: 150 msec to write points IW 0 [2016-05-26T16:28:35.925Z; Thread-0]: 152 msec to write points IW 0 [2016-05-26T16:28:36.141Z; Thread-0]: 148 msec to write points IW 0 [2016-05-26T16:28:36.357Z; Thread-0]: 149 msec to write points Patch flush IW 0 [2016-05-27T10:48:42.854Z; Thread-0]: 213 msec to write points IW 0 [2016-05-27T10:48:43.117Z; Thread-0]: 138 msec to write points IW 0 [2016-05-27T10:48:43.343Z; Thread-0]: 108 msec to write points IW 0 [2016-05-27T10:48:43.518Z; Thread-0]: 91 msec to write points IW 0 [2016-05-27T10:48:43.686Z; Thread-0]: 97 msec to write points IW 0 [2016-05-27T10:48:43.856Z; Thread-0]: 99 msec to write points IW 0 [2016-05-27T10:48:44.039Z; Thread-0]: 100 msec to write points IW 0 [2016-05-27T10:48:44.231Z; Thread-0]: 100 msec to write points IW 0 [2016-05-27T10:48:44.415Z; Thread-0]: 97 msec to write points IW 0 [2016-05-27T10:48:44.603Z; Thread-0]: 96 msec to write points IW 0 [2016-05-27T10:48:47.069Z; Thread-0]: 107 msec to write points IW 0 [2016-05-27T10:48:47.230Z; Thread-0]: 92 msec to write points IW 0 [2016-05-27T10:48:47.391Z; Thread-0]: 92 msec to write points IW 0 [2016-05-27T10:48:47.550Z; Thread-0]: 92 msec to write points IW 0 [2016-05-27T10:48:47.711Z; Thread-0]: 93 msec to write points Master merge SM 0 [2016-05-26T16:28:35.224Z; Thread-0]: 2414 msec to merge points [1096300 docs] SM 0 [2016-05-26T16:28:39.390Z; Thread-0]: 1899 msec to merge points [1096300 docs] SM 0 [2016-05-26T16:28:43.443Z; Thread-0]: 1869 msec to merge points [1096300 docs] SM 0 [2016-05-26T16:28:47.426Z; Thread-0]: 1812 msec to merge points [1096300 docs] SM 0 [2016-05-26T16:28:51.444Z; Thread-0]: 1850 msec to merge points [1096300 docs] SM 0 [2016-05-26T16:28:55.422Z; Thread-0]: 1819 msec to merge points [1096300 docs] SM 0 [2016-05-26T16:28:59.409Z; Thread-0]: 1823 msec to merge points [1096300 docs] SM 0 [2016-05-26T16:29:03.368Z; Thread-0]: 1817 msec to merge points [1096300 docs] SM 0 [2016-05-26T16:29:07.296Z; Thread-0]: 1802 msec to merge points [1096300 docs] SM 0 [2016-05-26T16:29:11.205Z; Thread-0]: 1793 msec to merge points [1096300 docs] SM 0 [2016-05-26T16:29:34.980Z; Thread-0]: 23722 msec to merge points [10963000 docs] SM 0 [2016-05-26T16:29:38.934Z; Thread-0]: 1798 msec to merge points [1096300 docs] SM 0 [2016-05-26T16:29:42.844Z; Thread-0]: 1779 msec to merge points [1096300 docs] SM 0 [2016-05-26T16:29:46.849Z; Thread-0]: 1797 msec to merge points [1096300 docs] SM 0 [2016-05-26T16:29:50.866Z; Thread-0]: 1802 msec to merge points [1096300 docs] Patch merge SM 0 [2016-05-27T10:48:46.890Z; Thread-0]: 2252 msec to merge points [1096300 docs] SM 0 [2016-05-27T10:48:49.943Z; Thread-0]: 1184 msec to merge points [1096300 docs] SM 0 [2016-05-27T10:48:52.722Z; Thread-0]: 1157 msec to merge points [1096300 docs] SM 0 [2016-05-27T10:48:55.440Z; Thread-0]: 1140 msec to merge points [1096300 docs] SM 0 [2016-05-27T10:48:58.146Z; Thread-0]: 1169 msec to merge points [1096300 docs] SM 0 [2016-05-27T10:49:00.915Z; Thread-0]: 1200 msec to merge points [1096300 docs] SM 0 [2016-05-27T10:49:03.684Z; Thread-0]: 1149 msec to merge points [1096300 docs] SM 0 [2016-05-27T10:49:06.436Z; Thread-0]: 1126 msec to merge points [1096300 docs] SM 0 [2016-05-27T10:49:09.169Z; Thread-0]: 1148 msec to merge points [1096300 docs] SM 0 [2016-05-27T10:49:11.903Z; Thread-0]: 1165 msec to merge points [1096300 docs] SM 0 [2016-05-27T10:49:28.140Z; Thread-0]: 16179 msec to merge points [10963000 docs] SM 0 [2016-05-27T10:49:30.913Z; Thread-0]: 1159 msec to merge points [1096300 docs] SM 0 [2016-05-27T10:49:33.609Z; Thread-0]: 1135 msec to merge points [1096300 docs] SM 0 [2016-05-27T10:49:36.383Z; Thread-0]: 1149 msec to merge points [1096300 docs] SM 0 [2016-05-27T10:49:39.118Z; Thread-0]: 1143 msec to merge points [1096300 docs]
        Hide
        dsmiley David Smiley added a comment -

        wow.

        I like how in the patch how you refactored out a MSBRadixSorter (which is well documented) and then made StringMSBRadixSorter simpler.

        Show
        dsmiley David Smiley added a comment - wow . I like how in the patch how you refactored out a MSBRadixSorter (which is well documented) and then made StringMSBRadixSorter simpler.
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 358d6f7e6be0b546c4c4a08e0277969d623fbbc0 in lucene-solr's branch refs/heads/branch_6x from Adrien Grand
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=358d6f7 ]

        LUCENE-7306: Speed up points indexing/merging with radix sort.

        Show
        jira-bot ASF subversion and git services added a comment - Commit 358d6f7e6be0b546c4c4a08e0277969d623fbbc0 in lucene-solr's branch refs/heads/branch_6x from Adrien Grand [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=358d6f7 ] LUCENE-7306 : Speed up points indexing/merging with radix sort.
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit ecec5244df39705b00272f3a5c783c0f915a4c31 in lucene-solr's branch refs/heads/master from Adrien Grand
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=ecec524 ]

        LUCENE-7306: Speed up points indexing/merging with radix sort.

        Show
        jira-bot ASF subversion and git services added a comment - Commit ecec5244df39705b00272f3a5c783c0f915a4c31 in lucene-solr's branch refs/heads/master from Adrien Grand [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=ecec524 ] LUCENE-7306 : Speed up points indexing/merging with radix sort.
        Hide
        jpountz Adrien Grand added a comment -

        Thanks for having a look David!

        Show
        jpountz Adrien Grand added a comment - Thanks for having a look David!
        Hide
        jpountz Adrien Grand added a comment -

        Mike's benchmarks confirm the speedup with a +36% jump in indexing speed: http://people.apache.org/~mikemccand/geobench.html

        Show
        jpountz Adrien Grand added a comment - Mike's benchmarks confirm the speedup with a +36% jump in indexing speed: http://people.apache.org/~mikemccand/geobench.html
        Hide
        mikemccand Michael McCandless added a comment -

        Mike's benchmarks confirm the speedup with a +36% jump in indexing speed

        Wow

        I will add an annotation! Thanks Adrien Grand.

        Show
        mikemccand Michael McCandless added a comment - Mike's benchmarks confirm the speedup with a +36% jump in indexing speed Wow I will add an annotation! Thanks Adrien Grand .
        Hide
        mikemccand Michael McCandless added a comment -

        Here's the direct URL to the indexing chart: http://people.apache.org/~mikemccand/geobench.html#index-times

        Looks like Adrien Grand already added the annotation (thanks!), I just pushed it.

        Show
        mikemccand Michael McCandless added a comment - Here's the direct URL to the indexing chart: http://people.apache.org/~mikemccand/geobench.html#index-times Looks like Adrien Grand already added the annotation (thanks!), I just pushed it.

          People

          • Assignee:
            jpountz Adrien Grand
            Reporter:
            jpountz Adrien Grand
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development