Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-11850

cannot use cql since upgrading python to 2.7.11+

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Fix Version/s: 2.1.16, 2.2.8, 3.0.9, 3.8
    • Component/s: Tools
    • Labels:
    • Environment:

      Development

      Description

      OS: Debian GNU/Linux stretch/sid
      Kernel: 4.5.0-2-amd64 #1 SMP Debian 4.5.4-1 (2016-05-16) x86_64 GNU/Linux
      Python version: 2.7.11+ (default, May 9 2016, 15:54:33)
      [GCC 5.3.1 20160429]

      cqlsh --version: cqlsh 5.0.1
      cassandra -v: 3.5 (also occurs with 3.0.6)

      Issue:
      when running cqlsh, it returns the following error:

      cqlsh -u dbarpt_usr01
      Password: *****

      Connection error: ('Unable to connect to any servers',

      {'odbasandbox1': TypeError('ref() does not take keyword arguments',)}

      )

      I cleared PYTHONPATH:

      python -c "import json; print dir(json); print json._version_"
      ['JSONDecoder', 'JSONEncoder', '__all__', '__author__', '__builtins__', '__doc__', '__file__', '__name__', '__package__', '__path__', '__version__', '_default_decoder', '_default_encoder', 'decoder', 'dump', 'dumps', 'encoder', 'load', 'loads', 'scanner']
      2.0.9

      Java based clients can connect to Cassandra with no issue. Just CQLSH and Python clients cannot.

      nodetool status also works.

      Thank you for your help.

        Issue Links

          Activity

          Hide
          mshuler Michael Shuler added a comment -

          Thanks - this is just an example, which does not get automatically updated. I just committed an update to the site for the latest debian repo 39x for the example.

          Show
          mshuler Michael Shuler added a comment - Thanks - this is just an example, which does not get automatically updated. I just committed an update to the site for the latest debian repo 39x for the example.
          Hide
          Stefania Stefania added a comment -

          Sure, have fun with Cassandra.

          I think the 36x is meant as an example on the documentation download page because it says for example for version 3.6, just before giving the sample command. I don't think we update the documentation for each release, but just in case I am cc-ing Michael Shuler.

          Show
          Stefania Stefania added a comment - Sure, have fun with Cassandra. I think the 36x is meant as an example on the documentation download page because it says for example for version 3.6 , just before giving the sample command. I don't think we update the documentation for each release, but just in case I am cc-ing Michael Shuler .
          Hide
          0nST 0nST added a comment -

          Thanks for the info, indeed an upgrade to 3.9 fixed it. I had assumed that http://cassandra.apache.org/download/ was the most updated version:

          echo "deb http://www.apache.org/dist/cassandra/debian 36x main" | sudo tee -a /etc/apt/sources.list.d/cassandra.sources.list

          Should be replaced with

          echo "deb http://www.apache.org/dist/cassandra/debian 39x main" | sudo tee -a /etc/apt/sources.list.d/cassandra.sources.list

          I'm looking forward to having a good play with Cassandra.

          Show
          0nST 0nST added a comment - Thanks for the info, indeed an upgrade to 3.9 fixed it. I had assumed that http://cassandra.apache.org/download/ was the most updated version: echo "deb http://www.apache.org/dist/cassandra/debian 36x main" | sudo tee -a /etc/apt/sources.list.d/cassandra.sources.list Should be replaced with echo "deb http://www.apache.org/dist/cassandra/debian 39x main" | sudo tee -a /etc/apt/sources.list.d/cassandra.sources.list I'm looking forward to having a good play with Cassandra.
          Hide
          Stefania Stefania added a comment -

          The earliest release of the 3.X series with this fix is 3.8. The current release of the 3.X family is 3.9, which is available for download.

          Show
          Stefania Stefania added a comment - The earliest release of the 3.X series with this fix is 3.8. The current release of the 3.X family is 3.9, which is available for download.
          Hide
          0nST 0nST added a comment -

          As of today 24th Nov 2016 the problem still exists with:

          Python 2.7.12
          Cassandra 3.6
          Ubuntu 16.04.1 LTS

          Temporary fix is:

          apt-get install python-pip
          pip install cassandra-driver
          export CQLSH_NO_BUNDLED=TRUE

          Show
          0nST 0nST added a comment - As of today 24th Nov 2016 the problem still exists with: Python 2.7.12 Cassandra 3.6 Ubuntu 16.04.1 LTS Temporary fix is: apt-get install python-pip pip install cassandra-driver export CQLSH_NO_BUNDLED=TRUE
          Hide
          Stefania Stefania added a comment - - edited

          You are correct, none of the versions with this fix have been released yet. The easiest workaround is, as you said, to downgrade Python. I'm pretty sure 2.7.11 should also be fine, since I was only able to reproduce this problem with 2.7.12.

          Installing version 3.5.0 or later of the Cassandra python driver is also an option, but there were some other minor issues that this patch fixed. Therefore, it is recommended to downgrade Python, if at all possible. If this is not possible, the driver can be installed by following these instructions. Further, cqlsh will only override its internal driver with a version installed elsewhere by setting the environment variable CQLSH_NO_BUNDLED to TRUE.

          Show
          Stefania Stefania added a comment - - edited You are correct, none of the versions with this fix have been released yet. The easiest workaround is, as you said, to downgrade Python. I'm pretty sure 2.7.11 should also be fine, since I was only able to reproduce this problem with 2.7.12. Installing version 3.5.0 or later of the Cassandra python driver is also an option, but there were some other minor issues that this patch fixed. Therefore, it is recommended to downgrade Python, if at all possible. If this is not possible, the driver can be installed by following these instructions . Further, cqlsh will only override its internal driver with a version installed elsewhere by setting the environment variable CQLSH_NO_BUNDLED to TRUE .
          Hide
          zouzias Anastasios Zouzias added a comment -

          I run into this problem yesterday.

          As of today, Cassandra versions 2.1.16, 2.2.8, 3.0.9, 3.8 (which solve this issue) are not released. A workaround is to downgrade python 2.7 to version 2.7.10 (tested for Cassandra 2.2.7)

          Show
          zouzias Anastasios Zouzias added a comment - I run into this problem yesterday. As of today, Cassandra versions 2.1.16, 2.2.8, 3.0.9, 3.8 (which solve this issue) are not released. A workaround is to downgrade python 2.7 to version 2.7.10 (tested for Cassandra 2.2.7)
          Hide
          Stefania Stefania added a comment -

          Thanks Paulo and Tyler. Committed to 2.1 as 3d3359eb95eebff48c4966e3e23f7efb4af0f535 and merged upwards. PR updated and merged.

          Show
          Stefania Stefania added a comment - Thanks Paulo and Tyler. Committed to 2.1 as 3d3359eb95eebff48c4966e3e23f7efb4af0f535 and merged upwards. PR updated and merged.
          Hide
          pauloricardomg Paulo Motta added a comment -

          Thanks for the input Tyler, I agree this is critical enough to go on 2.1. Marking as ready to commit.

          ps: Stefania, can you just include the excerpt above in the dtest PR? Thanks!

          Show
          pauloricardomg Paulo Motta added a comment - Thanks for the input Tyler, I agree this is critical enough to go on 2.1. Marking as ready to commit. ps: Stefania, can you just include the excerpt above in the dtest PR? Thanks!
          Hide
          thobbs Tyler Hobbs added a comment -

          The only final nit is that I'm still a bit unsure if this meets the critical requirement to go on 2.1 though, WDYT Stefania? In particular I'm not sure at which point we stop supporting python upgrades. Tyler Hobbs Would you have any suggestion on this?

          I just hit this problem yesterday after upgrading to the latest Ubuntu. The workarounds are not simple – you have to upgrade the bundled python driver, and if you try to upgrade it to the latest python driver release, you need to also modify cqlsh to remove references to PagedResult. Given that this completely breaks cqlsh in 2.1 on a common Linux distro and the workaround sucks, I would argue that this is just critical enough to be included in 2.1.

          Show
          thobbs Tyler Hobbs added a comment - The only final nit is that I'm still a bit unsure if this meets the critical requirement to go on 2.1 though, WDYT Stefania? In particular I'm not sure at which point we stop supporting python upgrades. Tyler Hobbs Would you have any suggestion on this? I just hit this problem yesterday after upgrading to the latest Ubuntu. The workarounds are not simple – you have to upgrade the bundled python driver, and if you try to upgrade it to the latest python driver release, you need to also modify cqlsh to remove references to PagedResult . Given that this completely breaks cqlsh in 2.1 on a common Linux distro and the workaround sucks, I would argue that this is just critical enough to be included in 2.1.
          Hide
          pauloricardomg Paulo Motta added a comment -

          During debugging I noticed that failed attempts were always sent to child process no. 0. This change ensures that failed attempts are sent to the child process following the one that failed.

          ah OK, now it makes sense, thanks for the explanation!

          Thanks for the manual tests. Perhaps we can open a ticket to create a test once byteman is integrated with dtests?

          Sure, I created CASSANDRA-12272 for this matter and assigned to DS Test Eng so it's not forgotten (hope it's OK Philip Thompson )

          OK, I think you've already removed the known failures in you commit. I will close those tickets when we commit this.

          The multiplexer 3 x 100 run looks good for trunk, but on 2.1 there are a few test runs failing with 21 Jul 2016 09:39:25 [node1] Missing: ['127.0.0.2 is now [dead|DOWN]']:, but it seems to be some environmental problem unrelated to this particular test so I guess we're safe to remove the flaky annotations and close CASSANDRA-11999/CASSANDRA-10884.

          Can you please include the excerpt below on your dtest PR to hopefully increase resilience to this?

          diff --git a/cqlsh_tests/cqlsh_tests.py b/cqlsh_tests/cqlsh_tests.py
          index 721d81f..c9f03c8 100644
          --- a/cqlsh_tests/cqlsh_tests.py
          +++ b/cqlsh_tests/cqlsh_tests.py
          @@ -1468,7 +1468,7 @@ Tracing session:""")
                   @jira_ticket CASSANDRA-9689
                   """
                   self.cluster.populate(3)
          -        self.cluster.start(wait_for_binary_proto=True)
          +        self.cluster.start(wait_for_binary_proto=True, wait_other_notice=True)
           
                   node1, node2, node3 = self.cluster.nodelist()
                   node2.stop(wait_other_notice=True)
          

          I've extracted the change and put it on a separate branch. Details are on CASSANDRA-11979, I've put you as reviewer.

          Sounds good, will follow-up there.

          Here are the up-to-date branches and latest tests results:

          Final patch and test results look good. The only final nit is that I'm still a bit unsure if this meets the critical requirement to go on 2.1 though, WDYT Stefania? In particular I'm not sure at which point we stop supporting python upgrades. Tyler Hobbs Would you have any suggestion on this?

          Show
          pauloricardomg Paulo Motta added a comment - During debugging I noticed that failed attempts were always sent to child process no. 0. This change ensures that failed attempts are sent to the child process following the one that failed. ah OK, now it makes sense, thanks for the explanation! Thanks for the manual tests. Perhaps we can open a ticket to create a test once byteman is integrated with dtests? Sure, I created CASSANDRA-12272 for this matter and assigned to DS Test Eng so it's not forgotten (hope it's OK Philip Thompson ) OK, I think you've already removed the known failures in you commit. I will close those tickets when we commit this. The multiplexer 3 x 100 run looks good for trunk , but on 2.1 there are a few test runs failing with 21 Jul 2016 09:39:25 [node1] Missing: ['127.0.0.2 is now [dead|DOWN]']: , but it seems to be some environmental problem unrelated to this particular test so I guess we're safe to remove the flaky annotations and close CASSANDRA-11999 / CASSANDRA-10884 . Can you please include the excerpt below on your dtest PR to hopefully increase resilience to this? diff --git a/cqlsh_tests/cqlsh_tests.py b/cqlsh_tests/cqlsh_tests.py index 721d81f..c9f03c8 100644 --- a/cqlsh_tests/cqlsh_tests.py +++ b/cqlsh_tests/cqlsh_tests.py @@ -1468,7 +1468,7 @@ Tracing session:""") @jira_ticket CASSANDRA-9689 """ self.cluster.populate(3) - self.cluster.start(wait_for_binary_proto=True) + self.cluster.start(wait_for_binary_proto=True, wait_other_notice=True) node1, node2, node3 = self.cluster.nodelist() node2.stop(wait_other_notice=True) I've extracted the change and put it on a separate branch. Details are on CASSANDRA-11979 , I've put you as reviewer. Sounds good, will follow-up there. Here are the up-to-date branches and latest tests results: Final patch and test results look good. The only final nit is that I'm still a bit unsure if this meets the critical requirement to go on 2.1 though, WDYT Stefania ? In particular I'm not sure at which point we stop supporting python upgrades. Tyler Hobbs Would you have any suggestion on this?
          Hide
          Stefania Stefania added a comment -

          I've created the dtest pull request here.

          Show
          Stefania Stefania added a comment - I've created the dtest pull request here .
          Hide
          Stefania Stefania added a comment -

          Thanks for the review.

          The 2.1 patch looks, can you just update to use the 2.7.2 zip now that #606 got in?

          Done.

          Similarly, can you regenerate the 2.2+ cassandra-driver libs after #617 was merged?

          Done.

          I didn't get this change, is this related to this patch? won't prev_worker_no=-1 and thus i=0 always?

          Sorry I should have mentioned this. During debugging I noticed that failed attempts were always sent to child process no. 0. This change ensures that failed attempts are sent to the child process following the one that failed. The worker no. is updated in the for loop and set to the worker the range is sent to. So if we process the range twice, we should send it to the next worker no.

          +1 to dtest and cdc-related changes (afaic)

          Thanks, they should be straightforward.

          The original intent of test_refresh_schema_on_timeout_error on CASSANDRA-9689 was to make sure a newly created keyspace/table will show up if there is a down node during the DDL statement, but since down nodes no longer causes schema mismatches after PYTHON-531 the schema mismatch assertions are no longer necessary (even though we still need to keep the --request-timeout option on 2.1 dtest to avoid flakiness - see CASSANDRA-10686), so I renamed the test to test_update_schema_with_down_node. Here is the dtest branch with these changes.

          OK, thanks for the explanation.

          I also added a new commit with the following changes:
          Refactor perform_simple_statement to always try to reload the schema if there's a mismatch in order to cover both CASSANDRA-9689 and CASSANDRA-10686.
          Remove the schema mismatch check on startup, since this is no longer necessary after PYTHON-303.
          Update the schema mismatch warning message

          Your changes LGTM, I've incorporated them into my branches.

          It would be nice to add a dtest to verify that the schema mismatch warning is being print on a proper schema mismatch but I didn't find a simple way to induce a schema mismatch without adding a special test flag on C* which is not an ideal solution. I tested manually by disabling the schema_version update on the system.peers table, and the warning is being printed correctly. We could probably achieve that easily with byteman but since that is not yet integrated with dtests let's maybe leave it for another ticket.

          Thanks for the manual tests. Perhaps we can open a ticket to create a test once byteman is integrated with dtests?

          Submitted multiplexer 100x runs for 2.1 and trunk so we can remove the @known_failure annotations and close CASSANDRA-11999 and CASSANDRA-10884.

          OK, I think you've already removed the known failures in you commit. I will close those tickets when we commit this.

          It would be nice if you could maybe extract CASSANDRA-11979 to a separate commit to improve traceability?

          I've extracted the change and put it on a separate branch. Details are on CASSANDRA-11979, I've put you as reviewer.

          Here are the up-to-date branches and latest tests results:

          2.1 2.2 3.0 3.9 trunk
          patch patch patch patch patch
          dtest dtest dtest dtest dtest
          Show
          Stefania Stefania added a comment - Thanks for the review. The 2.1 patch looks, can you just update to use the 2.7.2 zip now that #606 got in? Done. Similarly, can you regenerate the 2.2+ cassandra-driver libs after #617 was merged? Done. I didn't get this change, is this related to this patch? won't prev_worker_no=-1 and thus i=0 always? Sorry I should have mentioned this. During debugging I noticed that failed attempts were always sent to child process no. 0. This change ensures that failed attempts are sent to the child process following the one that failed. The worker no. is updated in the for loop and set to the worker the range is sent to. So if we process the range twice, we should send it to the next worker no. +1 to dtest and cdc-related changes (afaic) Thanks, they should be straightforward. The original intent of test_refresh_schema_on_timeout_error on CASSANDRA-9689 was to make sure a newly created keyspace/table will show up if there is a down node during the DDL statement, but since down nodes no longer causes schema mismatches after PYTHON-531 the schema mismatch assertions are no longer necessary (even though we still need to keep the --request-timeout option on 2.1 dtest to avoid flakiness - see CASSANDRA-10686 ), so I renamed the test to test_update_schema_with_down_node. Here is the dtest branch with these changes. OK, thanks for the explanation. I also added a new commit with the following changes: Refactor perform_simple_statement to always try to reload the schema if there's a mismatch in order to cover both CASSANDRA-9689 and CASSANDRA-10686 . Remove the schema mismatch check on startup, since this is no longer necessary after PYTHON-303. Update the schema mismatch warning message Your changes LGTM, I've incorporated them into my branches. It would be nice to add a dtest to verify that the schema mismatch warning is being print on a proper schema mismatch but I didn't find a simple way to induce a schema mismatch without adding a special test flag on C* which is not an ideal solution. I tested manually by disabling the schema_version update on the system.peers table, and the warning is being printed correctly. We could probably achieve that easily with byteman but since that is not yet integrated with dtests let's maybe leave it for another ticket. Thanks for the manual tests. Perhaps we can open a ticket to create a test once byteman is integrated with dtests? Submitted multiplexer 100x runs for 2.1 and trunk so we can remove the @known_failure annotations and close CASSANDRA-11999 and CASSANDRA-10884 . OK, I think you've already removed the known failures in you commit. I will close those tickets when we commit this. It would be nice if you could maybe extract CASSANDRA-11979 to a separate commit to improve traceability? I've extracted the change and put it on a separate branch. Details are on CASSANDRA-11979 , I've put you as reviewer. – Here are the up-to-date branches and latest tests results: 2.1 2.2 3.0 3.9 trunk patch patch patch patch patch dtest dtest dtest dtest dtest
          Hide
          pauloricardomg Paulo Motta added a comment -

          Thanks for the patch and detailed explanation! Overall it looks good, see follow-up:

          • The 2.1 patch looks, can you just update to use the 2.7.2 zip now that #606 got in?
          • Similarly, can you regenerate the 2.2+ cassandra-driver libs after #617 was merged?
          • I didn't get this change, is this related to this patch? won't prev_worker_no=-1 and thus i=0 always?
          • +1 to dtest and cdc-related changes (afaic)
          • The original intent of test_refresh_schema_on_timeout_error on CASSANDRA-9689 was to make sure a newly created keyspace/table will show up if there is a down node during the DDL statement, but since down nodes no longer causes schema mismatches after PYTHON-531 the schema mismatch assertions are no longer necessary (even though we still need to keep the --request-timeout option on 2.1 dtest to avoid flakiness - see CASSANDRA-10686), so I renamed the test to test_update_schema_with_down_node. Here is the dtest branch with these changes.
          • I also added a new commit with the following changes:
          • It would be nice to add a dtest to verify that the schema mismatch warning is being print on a proper schema mismatch but I didn't find a simple way to induce a schema mismatch without adding a special test flag on C* which is not an ideal solution. I tested manually by disabling the schema_version update on the system.peers table, and the warning is being printed correctly. We could probably achieve that easily with byteman but since that is not yet integrated with dtests let's maybe leave it for another ticket.
          • Submitted multiplexer 100x runs for 2.1 and trunk so we can remove the @known_failure annotations and close CASSANDRA-11999 and CASSANDRA-10884.
          • It would be nice if you could maybe extract CASSANDRA-11979 to a separate commit to improve traceability?

          Submitted tests with updates below:

          2.1 2.2 3.0 3.9 trunk dtest
          branch branch branch branch branch branch
          dtest dtest dtest dtest dtest
          Show
          pauloricardomg Paulo Motta added a comment - Thanks for the patch and detailed explanation! Overall it looks good, see follow-up: The 2.1 patch looks, can you just update to use the 2.7.2 zip now that #606 got in? Similarly, can you regenerate the 2.2+ cassandra-driver libs after #617 was merged? I didn't get this change , is this related to this patch? won't prev_worker_no=-1 and thus i=0 always? +1 to dtest and cdc-related changes (afaic) The original intent of test_refresh_schema_on_timeout_error on CASSANDRA-9689 was to make sure a newly created keyspace/table will show up if there is a down node during the DDL statement, but since down nodes no longer causes schema mismatches after PYTHON-531 the schema mismatch assertions are no longer necessary (even though we still need to keep the --request-timeout option on 2.1 dtest to avoid flakiness - see CASSANDRA-10686 ), so I renamed the test to test_update_schema_with_down_node . Here is the dtest branch with these changes. I also added a new commit with the following changes: Refactor perform_simple_statement to always try to reload the schema if there's a mismatch in order to cover both CASSANDRA-9689 and CASSANDRA-10686 . Remove the schema mismatch check on startup, since this is no longer necessary after PYTHON-303 . Update the schema mismatch warning message It would be nice to add a dtest to verify that the schema mismatch warning is being print on a proper schema mismatch but I didn't find a simple way to induce a schema mismatch without adding a special test flag on C* which is not an ideal solution. I tested manually by disabling the schema_version update on the system.peers table, and the warning is being printed correctly. We could probably achieve that easily with byteman but since that is not yet integrated with dtests let's maybe leave it for another ticket. Submitted multiplexer 100x runs for 2.1 and trunk so we can remove the @known_failure annotations and close CASSANDRA-11999 and CASSANDRA-10884 . It would be nice if you could maybe extract CASSANDRA-11979 to a separate commit to improve traceability? Submitted tests with updates below: 2.1 2.2 3.0 3.9 trunk dtest branch branch branch branch branch branch dtest dtest dtest dtest dtest
          Hide
          Stefania Stefania added a comment -

          I've rebased the 3.9 and trunk branches again, I noticed a (most likely) unrelated failure yesterday on 3.9. It basically failed to read the prompt even if we can see it in the debug messages. I could not reproduce it locally (run it 30 times in a loop), so I've added an additional debug message to the 3.9+ branches in case this failure happens again.

          Show
          Stefania Stefania added a comment - I've rebased the 3.9 and trunk branches again, I noticed a (most likely) unrelated failure yesterday on 3.9. It basically failed to read the prompt even if we can see it in the debug messages. I could not reproduce it locally (run it 30 times in a loop), so I've added an additional debug message to the 3.9+ branches in case this failure happens again.
          Hide
          2012ecs41@smvdu.ac.in Abhyuday P Singh added a comment -

          Thank you, it resolved the issue for now.

          Show
          2012ecs41@smvdu.ac.in Abhyuday P Singh added a comment - Thank you, it resolved the issue for now.
          Hide
          Stefania Stefania added a comment -

          Not yet. When this is fixed the resolution of this ticket becomes "Resolved" and the status becomes "Fixed". You find the corresponding release versions to download in the fix versions. Meanwhile, you can either downgrade Python to 2.7.11 OR install an up-to-date Cassandra Python driver and disable the cqlsh embedded driver by setting the CQLSH_NO_BUNDLED environment variable to TRUE before launching clqsh, see comments above for more details.

          Show
          Stefania Stefania added a comment - Not yet. When this is fixed the resolution of this ticket becomes "Resolved" and the status becomes "Fixed". You find the corresponding release versions to download in the fix versions. Meanwhile, you can either downgrade Python to 2.7.11 OR install an up-to-date Cassandra Python driver and disable the cqlsh embedded driver by setting the CQLSH_NO_BUNDLED environment variable to TRUE before launching clqsh, see comments above for more details.
          Hide
          2012ecs41@smvdu.ac.in Abhyuday P Singh added a comment -

          I am getting the same error.
          have this been fixed?

          Show
          2012ecs41@smvdu.ac.in Abhyuday P Singh added a comment - I am getting the same error. have this been fixed?
          Hide
          Stefania Stefania added a comment -

          Paulo Motta, welcome back! Here is a recap to help you catch up more quickly:

          • The 2.1 patch is just a driver upgrade to a custom version that only fixes the problem with python 2.7.12 (which is trivial in itself)
          • The 2.2 patch merges cleanly to 3.0 and there is a trivial conflict to 3.9. copyutil.py also contains the fix for CASSANDRA-11979.
          • The 3.9 patch contains an additional commit for CDC changes
          • The 3.9 patch merges cleanly to trunk
          • There are also dtest changes here, of which the most notable changes are for CDC in 3.9+ and in cqlsh_tests.py test_refresh_schema_on_timeout_error, where I removed not only the operation timed out exception but also the warning on schema mismatch, since it is not true that DOWN nodes will report a schema mismatch, see get_schema_mismatches in _cassandra/cluster.py. In view of this we should perhaps also change the warning message in cqlsh refresh_schema_metadata_best_effort and close CASSANDRA-11999 once this is committed.
          2.1 2.2 3.0 3.9 trunk
          patch patch patch patch patch
          dtest dtest dtest dtest dtest
          Show
          Stefania Stefania added a comment - Paulo Motta , welcome back! Here is a recap to help you catch up more quickly: The 2.1 patch is just a driver upgrade to a custom version that only fixes the problem with python 2.7.12 (which is trivial in itself) The 2.2 patch merges cleanly to 3.0 and there is a trivial conflict to 3.9. copyutil.py also contains the fix for CASSANDRA-11979 . The 3.9 patch contains an additional commit for CDC changes The 3.9 patch merges cleanly to trunk There are also dtest changes here , of which the most notable changes are for CDC in 3.9+ and in cqlsh_tests.py test_refresh_schema_on_timeout_error , where I removed not only the operation timed out exception but also the warning on schema mismatch, since it is not true that DOWN nodes will report a schema mismatch, see get_schema_mismatches in _cassandra/cluster.py . In view of this we should perhaps also change the warning message in cqlsh refresh_schema_metadata_best_effort and close CASSANDRA-11999 once this is committed. 2.1 2.2 3.0 3.9 trunk patch patch patch patch patch dtest dtest dtest dtest dtest
          Hide
          Stefania Stefania added a comment -

          The reason why future.is_schema_agreed is always true is probably caused by the issue described in this pull reqeust.

          Show
          Stefania Stefania added a comment - The reason why future.is_schema_agreed is always true is probably caused by the issue described in this pull reqeust .
          Hide
          Stefania Stefania added a comment -

          Test results are good now, provided the Netty upgrade is rolled back from 3.9 / trunk (CASSANDRA-12072).

          We just need the confirmation of Paulo Motta re. test_refresh_schema_on_timeout_error and a reviewer, cc. Joshua McKenzie.

          Show
          Stefania Stefania added a comment - Test results are good now, provided the Netty upgrade is rolled back from 3.9 / trunk ( CASSANDRA-12072 ). We just need the confirmation of Paulo Motta re. test_refresh_schema_on_timeout_error and a reviewer, cc. Joshua McKenzie .
          Hide
          Stefania Stefania added a comment -

          No problem, thanks for confirming!

          Show
          Stefania Stefania added a comment - No problem, thanks for confirming!
          Hide
          aholmber Adam Holmberg added a comment -

          +1 on that fix. The driver previously optimistically marked hosts up even if they were not connected by the load balancing policy and known to be up. is not False is the right way to do this. Sorry I didn't think to mention that.

          Show
          aholmber Adam Holmberg added a comment - +1 on that fix. The driver previously optimistically marked hosts up even if they were not connected by the load balancing policy and known to be up. is not False is the right way to do this. Sorry I didn't think to mention that.
          Hide
          Stefania Stefania added a comment -

          Failures on 3.9 / trunk were caused by CDC changes or connection errors. The CDC changes are fixed. The connection errors also happen on unpatched branches and I believe they are being investigated separately (Netty problems).

          Show
          Stefania Stefania added a comment - Failures on 3.9 / trunk were caused by CDC changes or connection errors. The CDC changes are fixed. The connection errors also happen on unpatched branches and I believe they are being investigated separately (Netty problems).
          Hide
          Stefania Stefania added a comment -

          Fixed one more failure, caused by the fact that, after calling shell.get_ring, the replicas are no longer marked as up (is_up is false), is this expected Adam Holmberg? I fixed it like this, can we do better?

          I've also implemented CASSANDRA-11979 here whilst I was looking for the culprit of the failure above.

          There are plenty more failures on 3.9 and trunk, they seem to happen also with the old driver version but I will take a look anyway.

          Show
          Stefania Stefania added a comment - Fixed one more failure , caused by the fact that, after calling shell.get_ring , the replicas are no longer marked as up ( is_up is false), is this expected Adam Holmberg ? I fixed it like this , can we do better? I've also implemented CASSANDRA-11979 here whilst I was looking for the culprit of the failure above. There are plenty more failures on 3.9 and trunk, they seem to happen also with the old driver version but I will take a look anyway.
          Hide
          Stefania Stefania added a comment -

          2.1 tests are fine but in 2.2+ we have some behavioral changes in the driver that cause problems, mostly related to utf-8 encoding and PYTHON-458 / CASSANDRA-10686.

          Some fixes went into the patches linked above, whilst the fixes in the dtest code are here. I've relaunched the tests.

          Paulo Motta, given that PYTHON-458 is now available, are we still expecting an operation timeout and schema mismatch warning in test_refresh_schema_on_timeout_error? I added the following in cqlsh perform_simple_stement but future.is_schema_agreed is always true (I think the driver refreshes the metadata before returning the statement result):

          future = self.session.execute_async(statement, trace=self.tracing_enabled)
          result = future.result()
          
          if not future.is_schema_agreed:
              self.refresh_schema_metadata_best_effort()
          
          Show
          Stefania Stefania added a comment - 2.1 tests are fine but in 2.2+ we have some behavioral changes in the driver that cause problems, mostly related to utf-8 encoding and PYTHON-458 / CASSANDRA-10686 . Some fixes went into the patches linked above, whilst the fixes in the dtest code are here . I've relaunched the tests. Paulo Motta , given that PYTHON-458 is now available, are we still expecting an operation timeout and schema mismatch warning in test_refresh_schema_on_timeout_error ? I added the following in cqlsh perform_simple_stement but future.is_schema_agreed is always true (I think the driver refreshes the metadata before returning the statement result): future = self.session.execute_async(statement, trace=self.tracing_enabled) result = future .result() if not future .is_schema_agreed: self.refresh_schema_metadata_best_effort()
          Hide
          Stefania Stefania added a comment -

          I've reproduced the problem with python 2.7.12 and cassandra 2.1+. I've also verified that the latest python driver, 3.5.0, fixes the problem for 2.2+ whilst for 2.1 I've created a pull request here to add the fix to the driver 2.7.2 branch.

          I've launched the cqlsh tests with the updated driver:

          2.1 2.2 3.0 3.9 trunk
          patch patch patch patch patch
          dtest dtest dtest dtest dtest

          I'm hoping that the Unicode problems I previously observed with the driver version 3.4 are now resolved. If so we can commit, otherwise I'll bump the urgency of this ticket given that that problem now occurs with a released python version.

          Show
          Stefania Stefania added a comment - I've reproduced the problem with python 2.7.12 and cassandra 2.1+. I've also verified that the latest python driver, 3.5.0, fixes the problem for 2.2+ whilst for 2.1 I've created a pull request here to add the fix to the driver 2.7.2 branch. I've launched the cqlsh tests with the updated driver: 2.1 2.2 3.0 3.9 trunk patch patch patch patch patch dtest dtest dtest dtest dtest I'm hoping that the Unicode problems I previously observed with the driver version 3.4 are now resolved. If so we can commit, otherwise I'll bump the urgency of this ticket given that that problem now occurs with a released python version.
          Hide
          nicerobot nicerobot added a comment - - edited

          Just to clarify, it is due to (in my DSE 4.8.6):

          DSE/resources/cassandra/lib/cassandra-driver-internal-only-2.7.2.zip/cassandra-driver-2.7.2/cassandra/cluster.py:2147
                  self_weakref = weakref.ref(self, callback=partial(_clear_watcher, weakref.proxy(connection)))
          

          Specifically, callback= is what is producing that error.

          https://hg.python.org/cpython/raw-file/v2.7.12/Misc/NEWS

          • Issue #17765: weakref.ref() no longer silently ignores keyword arguments.
            Patch by Georg Brandl.
          Show
          nicerobot nicerobot added a comment - - edited Just to clarify, it is due to (in my DSE 4.8.6): DSE/resources/cassandra/lib/cassandra-driver-internal-only-2.7.2.zip/cassandra-driver-2.7.2/cassandra/cluster.py:2147 self_weakref = weakref.ref(self, callback=partial(_clear_watcher, weakref.proxy(connection))) Specifically, callback= is what is producing that error. https://hg.python.org/cpython/raw-file/v2.7.12/Misc/NEWS Issue #17765: weakref.ref() no longer silently ignores keyword arguments. Patch by Georg Brandl.
          Hide
          bric3 Brice Dutheil added a comment -

          For reference the issue mentionned here is also present on version 2.x of cassandra. I just upgraded my python environment to 2.7.12 for other reasons. And cqlsh fails to connect with the same error :

          $ cqlsh
          Connection error: ('Unable to connect to any servers', {'127.0.0.1': TypeError('ref() does not take keyword arguments',)})
          
          $ uname -a
          Darwin ulysse 15.5.0 Darwin Kernel Version 15.5.0: Tue Apr 19 18:36:36 PDT 2016; root:xnu-3248.50.21~8/RELEASE_X86_64 x86_64
          $ python --version
          Python 2.7.12
          $ cassandra -v
          2.1.13
          $ cqlsh --version
          cqlsh 5.0.1
          
          Show
          bric3 Brice Dutheil added a comment - For reference the issue mentionned here is also present on version 2.x of cassandra. I just upgraded my python environment to 2.7.12 for other reasons. And cqlsh fails to connect with the same error : $ cqlsh Connection error: ('Unable to connect to any servers', {'127.0.0.1': TypeError('ref() does not take keyword arguments',)}) $ uname -a Darwin ulysse 15.5.0 Darwin Kernel Version 15.5.0: Tue Apr 19 18:36:36 PDT 2016; root:xnu-3248.50.21~8/RELEASE_X86_64 x86_64 $ python --version Python 2.7.12 $ cassandra -v 2.1.13 $ cqlsh --version cqlsh 5.0.1
          Hide
          Stefania Stefania added a comment -

          Thank you for confirming that the patch works Andrew Madison, that's really great to know!

          You don't need to set CQLSH_NO_BUNDLED if you are running the 3.7 patch, because the patch already bundles the 3.4.0 driver. You only need to set CQLSH_NO_BUNDLED if you want to keep on running an unpatched cqlsh that uses the 3.4.0 driver installed on your machine, rather than the older bundled driver.

          Show
          Stefania Stefania added a comment - Thank you for confirming that the patch works Andrew Madison , that's really great to know! You don't need to set CQLSH_NO_BUNDLED if you are running the 3.7 patch, because the patch already bundles the 3.4.0 driver. You only need to set CQLSH_NO_BUNDLED if you want to keep on running an unpatched cqlsh that uses the 3.4.0 driver installed on your machine, rather than the older bundled driver.
          Hide
          uskratos Andrew Madison added a comment -

          Thank you Stefania. I deployed the 3.7 patch and set CQLSH_NO_BUNDLED to TRUE. That fixed the problem and was able to connect from cqlsh

          export CQLSH_NO_BUNDLED=TRUE
          $ cqlsh -u cassandra
          Password:
          Connected to DBA Cassandra Cluster at odbasandbox1:9042.
          [cqlsh 5.0.1 | Cassandra 3.0.6 | CQL spec 3.4.0 | Native protocol v4]

          Show
          uskratos Andrew Madison added a comment - Thank you Stefania. I deployed the 3.7 patch and set CQLSH_NO_BUNDLED to TRUE. That fixed the problem and was able to connect from cqlsh export CQLSH_NO_BUNDLED=TRUE $ cqlsh -u cassandra Password: Connected to DBA Cassandra Cluster at odbasandbox1:9042. [cqlsh 5.0.1 | Cassandra 3.0.6 | CQL spec 3.4.0 | Native protocol v4]
          Hide
          Stefania Stefania added a comment -

          Unfortunately I cannot reproduce the issue with 2.7.11 built from source. We need to either get the source code from the Python master branch or assume that the problem is fixed since I'd rather not mess up the Debian packages on my laptop. Alternatively, Andrew Madison would you be able to confirm if the problem is fixed when running cqlsh with CQLSH_NO_BUNDLED set and the latest driver version?

          There are 5 failing tests, they will need fixing before proceeding. They seem caused by Unicode errors. I'll try and resume as soon as I can.

          Show
          Stefania Stefania added a comment - Unfortunately I cannot reproduce the issue with 2.7.11 built from source. We need to either get the source code from the Python master branch or assume that the problem is fixed since I'd rather not mess up the Debian packages on my laptop. Alternatively, Andrew Madison would you be able to confirm if the problem is fixed when running cqlsh with CQLSH_NO_BUNDLED set and the latest driver version? There are 5 failing tests , they will need fixing before proceeding. They seem caused by Unicode errors. I'll try and resume as soon as I can.
          Hide
          Stefania Stefania added a comment -

          I've updated the bundled driver and started the cqlsh tests for 3.0:

          3.0 3.7 trunk
          patch patch patch
          dtest dtest dtest

          If the 3.0 tests are fine I will also run the 3.7 and trunk tests.

          I'm also going to update my python version to 2.7.11 to make sure cqlsh now runs.

          Show
          Stefania Stefania added a comment - I've updated the bundled driver and started the cqlsh tests for 3.0: 3.0 3.7 trunk patch patch patch dtest dtest dtest If the 3.0 tests are fine I will also run the 3.7 and trunk tests. I'm also going to update my python version to 2.7.11 to make sure cqlsh now runs.
          Hide
          Stefania Stefania added a comment -

          cqlsh ships with an embedded python driver and that will only be updated in a future Cassandra release (watch this ticket for the exact fix version).

          If you have an up-to-date driver installed, you can disable the embedded driver by setting the environment variable CQLSH_NO_BUNDLED to any non empty string, for example export CQLSH_NO_BUNDLED=true.

          Show
          Stefania Stefania added a comment - cqlsh ships with an embedded python driver and that will only be updated in a future Cassandra release (watch this ticket for the exact fix version). If you have an up-to-date driver installed, you can disable the embedded driver by setting the environment variable CQLSH_NO_BUNDLED to any non empty string, for example export CQLSH_NO_BUNDLED=true .
          Hide
          uskratos Andrew Madison added a comment -

          Thank you Adam. I downloaded version 3.4.0 from https://github.com/datastax/python-driver/tree/3.4.0 and unfortunately the error still occurs.

          cassandra.cluster.NoHostAvailable: ('Unable to connect to any servers',

          {'10.248.33.96': TypeError('ref() does not take keyword arguments',), '10.2.51.2': TypeError('ref() does not take keyword arguments',)}

          )

          Show
          uskratos Andrew Madison added a comment - Thank you Adam. I downloaded version 3.4.0 from https://github.com/datastax/python-driver/tree/3.4.0 and unfortunately the error still occurs. cassandra.cluster.NoHostAvailable: ('Unable to connect to any servers', {'10.248.33.96': TypeError('ref() does not take keyword arguments',), '10.2.51.2': TypeError('ref() does not take keyword arguments',)} )
          Hide
          aholmber Adam Holmberg added a comment -

          FYI this is resolved in driver 3.4.0, which was released today.

          Show
          aholmber Adam Holmberg added a comment - FYI this is resolved in driver 3.4.0, which was released today.
          Hide
          aholmber Adam Holmberg added a comment -

          No issues with released version of Python. This is a known change with dev versions:
          https://github.com/datastax/python-driver/pull/585

          Show
          aholmber Adam Holmberg added a comment - No issues with released version of Python. This is a known change with dev versions: https://github.com/datastax/python-driver/pull/585
          Hide
          iamaleksey Aleksey Yeschenko added a comment -

          Adam Holmberg Are there any known issues with Python driver on 2.7.11+?

          Show
          iamaleksey Aleksey Yeschenko added a comment - Adam Holmberg Are there any known issues with Python driver on 2.7.11+?

            People

            • Assignee:
              Stefania Stefania
              Reporter:
              uskratos Andrew Madison
              Reviewer:
              Paulo Motta
            • Votes:
              2 Vote for this issue
              Watchers:
              20 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development