Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-10291

Bootstrap hangs on adding new node

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Normal
    • Resolution: Unresolved
    • None
    • None
    • Debian 7 64 bit
      HotSpot JDK 1.7.0_79
      Cassandra-2.2.1 via apt-get
      1x Intel Quad-Core Xeon E3-1230 / 16GB / 4x1TB SATA / 3x1TB RAID0 data drive

    • Normal

    Description

      Adding new node in heavy loaded environment freeze bootstrap. No errors are reported in log files. Some of other other nodes throws "String didn't validate" error, but I;m not sure that this is related.
      After restarting node it start bootstrap again and hangs after some time .

      nodetool netstats shows :
      /data/XXX/XXXX/tmp-la-1184-big-Data.db 5126078789/18345924701 bytes(27%) received from idx:0/192.168.220.16
      /data/XXX/XXXX/tmp-la-1233-big-Data.db 7213706459/18600941671 bytes(38%) received from idx:0/192.168.220.22
      /data/XXX/XXXX/tmp-la-1599-big-Data.db 8492408759/17572043398 bytes(48%) received from idx:0/192.168.220.12
      /data/XXX/XXXX/tmp-la-2066-big-Data.db 15773981555/18508127610 bytes(85%) received from idx:0/192.168.220.18
      /data/XXX/XXXX/tmp-la-211-big-Data.db 8274231066/17172754085 bytes(48%) received from idx:0/192.168.220.20

      but listing files on local FS shows "No such file or directory"

      This happens only if there is significant amount of data. I have 1.5 TB per node on 13 node cluster, we use STCS compaction strategy and flat network topology .

      Attachments

        1. system.log
          860 kB
          Ara Sadoyan
        2. nodetool.txt
          4 kB
          Ara Sadoyan
        3. logs_netstats.tar.gz
          161 kB
          Ara Sadoyan

        Activity

          People

            Unassigned Unassigned
            sadoyan Ara Sadoyan
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated: