Uploaded image for project: 'CarbonData'
  1. CarbonData
  2. CARBONDATA-3447

Index Server Performance Improvement

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.6.0
    • Component/s: core
    • Labels:
      None

      Description

      Problem:

      When number of splits are high, index server performance is slow as compared to old flow(Driver caching). This is because data is transferred over network is more and causing performance bottleneck.   

      Solution:

      1. If data transferred is less we can sent through network, but when it grows we can write to file and only send file name and in Main driver it will read the file and construct input split. 
      2. Use snappy to compress the data, so data transferred through network/written to file size will be less, so IO time wont impact performance 
      3. In main driver pruning is done in multiple thread, added same for index executor as now index executor will do the pruning
      4. In case of block cache no need to send blockletdetailinfo object as size is more and same can be constructed in executor from file footer

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                kumarvishal09 Kumar Vishal
                Reporter:
                kumarvishal09 Kumar Vishal
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 0.5h
                  0.5h