Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-5667

Performance problem when not using hdfs block cache.

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.7, 6.0
    • Component/s: None
    • Labels:
      None

      Description

      Reads are terribly slow due to what looks like a too small buffer in the buffered input. This has consequences for SOLR-1301, as we turn off the block cache when writing indexes with map-reduce.

        Issue Links

          Activity

          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 1563766 from Mark Miller in branch 'dev/trunk'
          [ https://svn.apache.org/r1563766 ]

          SOLR-5667: Performance problem when not using hdfs block cache.

          Show
          jira-bot ASF subversion and git services added a comment - Commit 1563766 from Mark Miller in branch 'dev/trunk' [ https://svn.apache.org/r1563766 ] SOLR-5667 : Performance problem when not using hdfs block cache.
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 1563770 from Mark Miller in branch 'dev/branches/branch_4x'
          [ https://svn.apache.org/r1563770 ]

          SOLR-5667: Performance problem when not using hdfs block cache.

          Show
          jira-bot ASF subversion and git services added a comment - Commit 1563770 from Mark Miller in branch 'dev/branches/branch_4x' [ https://svn.apache.org/r1563770 ] SOLR-5667 : Performance problem when not using hdfs block cache.
          Hide
          markrmiller@gmail.com Mark Miller added a comment -

          I put in a 32k buffer for now. Other parts in the code use a 32k buffer already (when the block cache is on), and initial performance testing showed it to performant. It fixes the performance bug.

          I do think we want to look at the optimal settings and sizes throughout the hdfs directory and block cache code, but that should be done more holistically rather than as part of this discreet performance bug fix.

          Show
          markrmiller@gmail.com Mark Miller added a comment - I put in a 32k buffer for now. Other parts in the code use a 32k buffer already (when the block cache is on), and initial performance testing showed it to performant. It fixes the performance bug. I do think we want to look at the optimal settings and sizes throughout the hdfs directory and block cache code, but that should be done more holistically rather than as part of this discreet performance bug fix.

            People

            • Assignee:
              markrmiller@gmail.com Mark Miller
              Reporter:
              markrmiller@gmail.com Mark Miller
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development