cpp-rust-driver test parity with original cpp-driver

[roydahan]

We would like the cpp-rust-driver to have all the original cpp driver tests running as part of CI.
This is an umbrella issue (epic) to include all the current & future tasks to get a working running CI and update here about the status of it.
(e.g. How many tests we have that are not passing (if any)).

Tasks

Beta Give feedback

1. Make sure we have all tests from original cpp driver (let's write here the amount of tests)
2. Enable (if not already) all tests to run as part of CI --> how many failing compared to original driver
3. Start openning issues for failing tests / classes / missing functionality
4. CI: some tests are unnecessarily skipped when run against Scylla #144
   +0
   

https://docs.google.com/spreadsheets/d/1CuXBS9CkBpwIoLmPOXPPS3_V-mMffnt_EWGJjNPDKgA/edit?usp=sharing

[wprzytula]

IIRC, some Datastax tests require some functionality besides the API defined by cassandra.h, whereas cpp-rust-driver only supports that API. Can you confirm that @Lorak-mmk?

[Lorak-mmk]

There is additional testing API
https://github.com/scylladb/cpp-driver/blob/master/src/testing.hpp
https://github.com/scylladb/cpp-driver/blob/master/src/testing.cpp

but it should not be a problem - we can implement it or change the tests to avoid using it (or modify it).

Regarding the issue: We already run integration tests, with both Scylla and C* - or at least the subset that works.
You can find the list of used tests in CI yaml: https://github.com/scylladb/cpp-rust-driver/blob/master/.github/workflows/cassandra.yml

[roydahan]

Yes, I know we run some.
First, we would like to run all and get a mapping of the amount of tests failing compared to original cpp driver and start mapping also missing functionality.

In this epic, we will collect the task items to enable/disable relevant tests and track the status till we reach parity.

[roydahan]

@muzarski can you please update your progress here?

[muzarski]

@muzarski can you please update your progress here?

The tests needed to be enabled to reach parity with cpp-driver:

• AsyncTests.Integration_Cassandra_Close - not a suite, but a single test. It seems like cpp-driver awaits all futures when session object is dropped (when futures are not explicitly awaited by the user). It's not mentioned in the documentation, but the test behaviour strongly suggests that this is the case. This needs to be implemented in cpp-rust-driver as well. Currently it causes segmentation faults.
• ControlConnectionTests - requires implementing at least these API functions:
  • cass_cluster_set_num_threads_io - we marked it as deprecated in the spreadsheet. Will need to adjust the test to not use it
  • cass_cluster_set_constant_reconnect - not implemented in rust-driver
  • cass_cluster_set_local_address - not implemented in rust-driver
• ControlConnection[Two/Three/Four]NodeClusterTests - same as above
• HeartbeatTests.Integration_Cassandra_HeartbeatFailed - not a suite, but a single test. Requires cass_session_get_metrics. For this, need to investigate what kind of metrics rust-driver collects and whether we need to adjust them there.
• MetricsTests - requires cass_session_get_metrics, cass_cluster_set_core_connections_per_host, cass_cluster_set_num_threads_io, cass_cluster_set_core_connections_per_host
• LatencyAwareTest - from my investigation, it only requires adjusting the test to logs emitted by rust-driver, or if the logs are missing, adding them to rust-driver. The test searches for some pattern in driver logs and makes an assertion based on them.
• ExecutionProfileTest/DCExecutionProfileTest - both require whitelist/blacklist filtering. I thing this is an important task and should be prioritized.
• DisconnectedNullStringApiArgsTest - again, whitelist/blacklist filtering
• ControlConnectionSingleNodeDataCentersClusterTests - there is only one test and it fails. What happens is that we open a session with dc-aware load balancing policy with an invalid datacenter. The test expects that we do not open control connection to any node, because of the invalid datacenter. There is a discrepancy between cpp-rust-driver and rust-driver in this matter. cpp-driver makes use of load balancing policy for control connections, while rust-driver does not. I'm not sure whether we want to mimic cpp-driver's behaviour here.
• PreparedTests.Integration_Cassandra_PreparedIDUnchangedDuringReprepare - not a suite, but a single test. Same as LatencyAwareTest, requires to investigate the logs emitted by rust-driver
• PreparedMetadataTests.Integration_Cassandra_AlterDoesntUpdateColumnCount - not a suite, but a single test. It seems that cpp-driver caches prepared statement's result metadata by default. It's configurable optional in rust-driver. This test expects that the result will contain an outdated metadata after schema alter. In our case, the metadata is up to date, since we do not cache it by default, and return a metadata provided by the server after statement execution.
• SchemaAgreementTests - again, depends on driver's logs
• ServerSideFailureTests - two of the tests require UDF support, not focusing on that. 1 test passes, and last one depends on driver's logs (so again, a small thing)
• SessionTest - require at least cass_session_get_metrics and cass_cluster_set_host_listener_callback (required rust-driver support)
• SetKeyspaceTests- all tests require cass_statement_set_keyspace. This needs to be implemented in rust-driver. Also, there is a comment by Wojciech in the spreadsheet: cpp driver states that only with CQL v5 does this really USE KEYSPACE for a statement; for lower protocol version it is only used for unprepared statements to know the keyspace and hence the replication strategy for load balancing.
• StatementTests - require cass_statement_set_host and cass_future_coordinator. There is an issue for this in rust-driver (Enable enforcing coordinator for request scylla-rust-driver#1031). I think I could start working on this. The deserialization and QueryResult API refactor is finished, so there should be no conflicts between the changes introduced by me and Wojciech.
• StatementNoClusterTests - same as above
• TimestampTests - 2 tests passing, 3 failing - these require client side timestamp generators. There is an issue in rust-driver Implement timestamp generators scylla-rust-driver#1032
• UseKeyspaceCaseSensitiveTests.Integration_Cassandra_ConnectWithKeyspace - not a suite, but a single test. Probably a small bug in cass_session_connect_keyspace regarding keyspace case-sensitivity.

Remaining test suites are not enabled by cpp-driver. The list of disabled test suites and specific tests in cpp-driver:

• AuthenticationTests
• ConsistencyTwoNodeClusterTests.Integration_Cassandra_SimpleEachQuorum
• ControlConnectionTests.Integration_Cassandra_TopologyChange
• ControlConnectionTwoNodeClusterTests.Integration_Cassandra_Reconnection
• CustomPayloadTests
• DbaasTests
• DcAwarePolicyTest.Integration_Cassandra_UsedHostsRemoteDc
• ExecutionProfileTest.Integration_Cassandra_RequestTimeout
• ExecutionProfileTest.Integration_Cassandra_SpeculativeExecutionPolicy
• MetricsTests.Integration_Cassandra_SpeculativeExecutionRequests
• MetricsTests.Integration_Cassandra_StatsConnections
• PreparedTests.Integration_Cassandra_PreparedIDUnchangedDuringReprepare
• ServerSideFailureTests.Integration_Cassandra_Warning
• ServerSideFailureTests.Integration_Cassandra_ErrorFunctionFailure
• ServerSideFailureTests.Integration_Cassandra_ErrorFunctionAlreadyExists
• SessionTest.Integration_Cassandra_ExternalHostListener
• SchemaMetadataTest
• SchemaNullStringApiArgsTest
• SpeculativeExecutionTests
• SslTests
• SslClientAuthenticationTests