Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-15608

full repair getting failed for 1 large table version 3.11.2

    XMLWordPrintableJSON

    Details

    • Severity:
      Critical
    • Complexity:
      Challenging
    • Platform:
      All, Linux
    • Impacts:
      None, Clients
    • Test and Documentation Plan:
      Hide

      nil

      Show
      nil

      Description

      nodetool repair -pr test table1
      [2020-02-28 17:56:00,192] Starting repair command #1 (7a634f80-5a25-11ea-8b5a-3b5edc400697), repairing keyspace test with repair options (parallelism: parallel, primary range: true, incremental: true, job threads: 1, ColumnFamilies: [table1], dataCenters: [], hosts: [], # of ranges: 256, pull repair: false)
      [2020-02-28 17:56:17,173] Repair session 7ae020a1-5a25-11ea-8b5a-3b5edc400697 for range [(-1398494450650640175,-1398484603620354999]] finished (progress: 1%)
      [2020-02-28 17:56:22,633] Repair session 7ad3c491-5a25-11ea-8b5a-3b5edc400697 for range [(-2055080187802596770,-2055069591393124910]] finished (progress: 2%)
      [2020-02-28 17:56:23,888] Repair session 7ace4651-5a25-11ea-8b5a-3b5edc400697 for range [(-5916076129714268351,-5916054112636649354]] finished (progress: 2%)
      [2020-02-28 17:56:27,675] Repair session 7ad83161-5a25-11ea-8b5a-3b5edc400697 for range [(3262419392837600448,3262453314317655259]] finished (progress: 3%)
      [2020-02-28 17:56:31,674] Repair session 7ad32851-5a25-11ea-8b5a-3b5edc400697 for range [(2146167582342729236,2146207858477993492]] finished (progress: 3%)
      [2020-02-28 17:56:52,964] Repair session 7af15eb0-5a25-11ea-8b5a-3b5edc400697 for range [(-7088663015651548617,-7088601571353808807]] finished (progress: 3%)
      [2020-02-28 17:57:03,237] Repair session 7acf30b2-5a25-11ea-8b5a-3b5edc400697 for range [(-5392084277480362928,-5391992322619247178]] finished (progress: 4%)
      [2020-02-28 17:57:08,937] Repair session 7ada5443-5a25-11ea-8b5a-3b5edc400697 for range [(-2247846360993532740,-2247745485112718002]] finished (progress: 4%)
      [2020-02-28 17:57:14,102] Repair session 7ae1a740-5a25-11ea-8b5a-3b5edc400697 for range [(4804514170425290662,4804611019437533765]] finished (progress: 5%)
      [2020-02-28 17:57:18,190] Repair session 7ae6fe71-5a25-11ea-8b5a-3b5edc400697 for range [(4989724097640378549,4989803009764075856]] finished (progress: 5%)
      [2020-02-28 17:57:27,549] Repair session 7acfccf1-5a25-11ea-8b5a-3b5edc400697 for range [(4512226173347132723,4512312572318114003]] finished (progress: 5%)
      [2020-02-28 17:57:37,534] Repair session 7ae1ce50-5a25-11ea-8b5a-3b5edc400697 for range [(7146381406763943813,7146516394221309657]] finished (progress: 6%)
      Feb 28, 2020 5:59:00 PM ClientCommunicatorAdmin Checker-run
      WARNING: Failed to check the connection: java.net.SocketTimeoutException: Read timed out
      [2020-02-28 17:59:36,419] Repair session 7aeacf01-5a25-11ea-8b5a-3b5edc400697 for range [(4717535938324360043,4717683452072792471]] finished (progress: 6%)

      Current setup Details:

      Allocated Memory: 128gb

      No of nodes:12 DC1 and 12 DC2

      Sr No Parameter Name Old value New Value File Name
      1 Min Heap Size -Xms -Xms32G -Xms64G jvm.options
      2 Max Heap Size -Xmx -Xms32G -Xms64G jvm.options
      3 Young generation size -Xmn #-Xmn800M -Xmn3072M jvm.options
      4 GC SETTINGS CMS G1GC jvm.options
      5 Disk Read_ahead_kb 4096 64 /sys/class/block/sdd/queue/read_ahead_kb
      6 Disk Rotational 1 1 /sys/class/block/sdd/queue/rotational
      7 block devices attributes 8192 128 blockdev --setra 128 /dev/sdd (To set it)
      8 commitlog_segment_size_in_mb 32 64 cassandra.yaml
      9 memtable_heap_space_in_mb 1/4 the size of the heap 8192 cassandra.yaml
      10 memtable_offheap_space_in_mb 1/4 the size of the heap 8192 cassandra.yaml
      11 phi_convict_threshold #8 12 cassandra.yaml

       

        Attachments

        1. system.log
          313 kB
          rsasupport

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              rsasupport rsasupport
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: