Solr
  1. Solr
  2. SOLR-5667

Performance problem when not using hdfs block cache.

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.7, 6.0
    • Component/s: None
    • Labels:
      None

      Description

      Reads are terribly slow due to what looks like a too small buffer in the buffered input. This has consequences for SOLR-1301, as we turn off the block cache when writing indexes with map-reduce.

        Issue Links

          Activity

          Hide
          ASF subversion and git services added a comment -

          Commit 1563766 from Mark Miller in branch 'dev/trunk'
          [ https://svn.apache.org/r1563766 ]

          SOLR-5667: Performance problem when not using hdfs block cache.

          Show
          ASF subversion and git services added a comment - Commit 1563766 from Mark Miller in branch 'dev/trunk' [ https://svn.apache.org/r1563766 ] SOLR-5667 : Performance problem when not using hdfs block cache.
          Hide
          ASF subversion and git services added a comment -

          Commit 1563770 from Mark Miller in branch 'dev/branches/branch_4x'
          [ https://svn.apache.org/r1563770 ]

          SOLR-5667: Performance problem when not using hdfs block cache.

          Show
          ASF subversion and git services added a comment - Commit 1563770 from Mark Miller in branch 'dev/branches/branch_4x' [ https://svn.apache.org/r1563770 ] SOLR-5667 : Performance problem when not using hdfs block cache.
          Hide
          Mark Miller added a comment -

          I put in a 32k buffer for now. Other parts in the code use a 32k buffer already (when the block cache is on), and initial performance testing showed it to performant. It fixes the performance bug.

          I do think we want to look at the optimal settings and sizes throughout the hdfs directory and block cache code, but that should be done more holistically rather than as part of this discreet performance bug fix.

          Show
          Mark Miller added a comment - I put in a 32k buffer for now. Other parts in the code use a 32k buffer already (when the block cache is on), and initial performance testing showed it to performant. It fixes the performance bug. I do think we want to look at the optimal settings and sizes throughout the hdfs directory and block cache code, but that should be done more holistically rather than as part of this discreet performance bug fix.

            People

            • Assignee:
              Mark Miller
              Reporter:
              Mark Miller
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development