YDB Server 24.2.8

Long domain name (#9108)

YDB Server 24.1.18

Merge stable-24-1-16-analytics into stable-24-1 (#5596)

Co-authored-by: Vitalii Gridnev <gridnevvvit@gmail.com>
Co-authored-by: Andrey Neporada <aneporada@ydb.tech>
Co-authored-by: niksaveliev <nik@saveliev.me>
Co-authored-by: Sergey Veselov <sergeyveselov@ydb.tech>
Co-authored-by: alexnick88 <alexnick@ydb.tech>
Co-authored-by: Ilnaz Nizametdinov <ilnaz@ydb.tech>
Co-authored-by: Iuliia Sidorina <yulia@ydb.tech>
Co-authored-by: kungurtsev <kungasc@ydb.tech>
Co-authored-by: Nikolay Shestakov <tesseract@ydb.tech>
Co-authored-by: azevaykin <145343289+azevaykin@users.noreply.github.com>
Co-authored-by: DimasKovas <34828390+DimasKovas@users.noreply.github.com>
Co-authored-by: ijon <ijon@ydb.tech>
Co-authored-by: vporyadke <zalyalov@ydb.tech>
Co-authored-by: kruall <kruall@ydb.tech>
Co-authored-by: ivanmorozov333 <ivanmorozov@ydb.tech>
Co-authored-by: Alexander Rutkovsky <alexvru@ydb.tech>
Co-authored-by: ildar-khisambeev <ikhis@ydb.tech>
Co-authored-by: nsofya <nsofya@ydb.tech>
Co-authored-by: nsofya <nsofya@yandex.ru>
Co-authored-by: Sofya Novozhilova <nsofya@qavm-4bd1829b.qemu>
Co-authored-by: Олег <150132506+iddqdex@users.noreply.github.com>
Co-authored-by: Ivan Morozov <ivanmorozov@localhost.localdomain>
Co-authored-by: Andrei Rykov <arykov@ydb.tech>
Co-authored-by: qyryq <qyryq@ydb.tech>
Co-authored-by: Daniil Cherednik <dcherednik@ydb.tech>
Co-authored-by: Aleksei Borzenkov <snaury@ydb.tech>

YDB Server 23.4.11

Ydb stable 23-4-11

x-stable-origin-commit: 9cc0616f07997e08c92266a30796bc4ff25aea66

YDB CLI 2.8.0

Released on January 12, 2024. To update to version 2.8.0, select the Downloads section.

Features:

• Added configuration management commands for the cluster ydb admin config and ydb admin volatile-config.

• Added support for loading PostgreSQL-compatible data types by ydb import file csv|tsv|json command. Only for row-oriented tables.

• Added support for directory load from an S3-compatible storage in the ydb import s3 command. Currently only available on Linux and Mac OS.

• Added support for outputting the results of ydb table query execute, ydb yql and ydb scripting yql commands in the Apache Parquet format.

• In the ydb workload commands, the --executer option has been added, which allows to specify which type of queries to use.

• Added a column with median benchmark execution time in the statistics table of the ydb workload clickbench command.

• (Experimental) Added the generic request type to the ydb table query execute command, allowing to perform DDL and DML operations, return with arbitrarily-sized results and support for MVCC. The command uses an experimental API, compatibility is not guaranteed.

• (Experimental) In the ydb table query explain command, the --collect-diagnostics option has been added to collect query diagnostics and save it to a file. The command uses an experimental API, compatibility is not guaranteed.

Bug fixes:

• Fixed an error displaying tables in pretty format with Unicode characters.

• Fixed an error substituting the wrong primary key in the command ydb tools pg-convert.

Full Changelog: CLI_2.7.0...CLI_2.8.0

YDB Server 23.3.17

Bugfixes:

• Fix InFlightTxCount become negative during split/merge
• Allow compatibility between non-stable versions of different applications
• Various stability improvements (segfault, handle balancer and boot queue events in StateInit, data race when reporting issue counters, memory leak in TRopeArena)
• CPU overutilization in case of numerous topic consumers
• Fix import multiple converted csv columns
• Fix mem table merge when writing over committed deltas in tablets
• Check errors during read_table_scan and export
• Throttle datashard log messages

Features:

• Enable ANSI settings for generic query
• DML logging
• Request unit trailing header for ScanQuery/ScanYql sctipts
• Grpc KV api
• Change default metering mode of topics to request units
• Pqv0 support on server side
• Allow to maintain unavailable nodes in tablets checker
• Allow to reboot unavailable node
• Add Lazy Histogram for grpc request

YDB CLI 2.7.0

Released on October 23, 2023. To update to version 2.7.0, select the Downloads section.

Features:

• Added the ydb tools pg-convert command, which prepares a dump obtained by the pg_dump utility for loading into the YDB postgres-compatible layer.

• Added the ydb workload query load testing command, which loads the database with script execution queries in multiple threads.

• Added a command ydb scheme permissions list to list permissions.

• In the commands ydb table query execute, ydb table query explain, ydb yql, and ydb scripting yql, the --flame-graph option has been added, specifying the path to the file in which you need to save the visualization of query execution statistics.

• Special commands in the interactive query execution mode are now case-insensitive.

• Added validation for special commands and their parameters.

• Added table reading in the scenario with transactions in the command ydb workload transfer topic-to-table run.

• Added the --commit-messages option to the command ydb workload transfer topic-to-table run, specifying the number of messages in a single transaction.

• Added the options --only-table-in-tx and --only-topic-in-tx in the command ydb workload transfer topic-to-table run, specifying restrictions on the types of queries in a single transaction.

• Added new columns Select time and Upsert time in the statistics table in the command ydb workload transfer topic-to-table run.

Bug fixes:

• Fixed an error when loading an empty JSON list by commands: ydb table query execute, ydb scripting yql and ydb yql.

Full Changelog: CLI_2.6.0...CLI_2.7.0

YDB Server 23.3.13

Release notes: https://ydb.tech/en/docs/changelog-server#23-3

YDB CLI 2.6.0

Released on September 7, 2023. To update to version 2.6.0, select the Downloads section.

Features:

• Added --path option to ydb workload tpch run, which contains the path to the directory with tables created by the ydb workload tpch init command.

• Added ydb workload transfer topic-to-table run command, which loads the database with read requests from topics and write requests to the table.

• Added the option --consumer-prefix in the commands ydb workload topic init, ydb workload topic run read|full, specifying prefixes of consumer names.

• Added the --partition-ids option in the ydb topic read command, which specifies a comma-separated list of topic partition identifiers to read from.

• Added support for CSV and TSV parameter formats in YQL query execution commands.

• The interactive mode of query execution has been redesigned. Added new interactive mode specific commands: SET, EXPLAIN, EXPLAIN AST. Added saving history between CLI launches and auto-completion of YQL queries.

• Added the command ydb config info, which outputs the current connection parameters without connecting to the database.

• Added the command ydb workload kv run mixed, which loads the database with write and read requests.

• The --percentile option in the ydb workload topic run write|read|full commands can now take floating point values.

• The default values for the --seconds and --warmup options in the ydb workload topic run write|read|full commands have been increased to 60 seconds and 5 seconds, respectively.

• Changed the default value for the --supported-codecs option to RAW in the ydb topic create and ydb topic consumer add commands.

Bug fixes:

• Fixed string loss when loading with the ydb import file json command.

• Fixed ignored statistics during the warm-up of commands ydb workload topic run write|read|full.

• Fixed incomplete statistics output in the ydb scripting yql and ydb yql commands.

• Fixed incorrect output of progress bar in ydb tools dump and ydb tools restore commands.

• Fixed loading large files with the header in the ydb import file csv|tsv command.

• Fixed hanging of the ydb tools restore --import-data command.

• Fixed error Unknown value Rejected when executing the ydb operation list build index command.

Full Changelog: CLI_2.5.0...CLI_2.6.0

YDB Server 23.2.12

Functionality:

• (Experimental) Implemented visibility of own changes. With this feature enabled you can read changed values from the current transaction, which has not been committed yet. This functionality also allows multiple modifying operations in one transaction on a table with secondary indexes. To enable this feature add enable_kqp_immediate_effects: true under table_service_config section into configuration file.
• (Experimental) Implemented read iterators. This feature allows to separate reads and computations. Read iterators allow datashards to increase read queries throughput. To enable this feature add enable_kqp_data_query_source_read: true under table_service_config section into configuration file.

Embedded UI:

• Navigation improvements:
  • Diagnostics and Development mode switches are moved to the left panel.
  • Every page has breadcrumbs.
  • Storage groups and nodes info are moved from left buttons to tabs on the database page.
• Query history and saved queries are moved to tabs over the text editor area in query editor.
• Info tab for scheme objects displays parameters using terms from CREATE or ALTER statements.
• Added column tables support.

Performance:

• For scan queries, you can now effectively search for individual rows using a primary key or secondary indexes. This can bring you a substantial performance gain in many cases. Similarly to regular queries, you need to explicitly specify its name in the query text using the VIEW keyword to use a secondary index.
• (Experimental) Added an option to give control of the system tablets of the database (SchemeShard, Coordinators, Mediators, SysViewProcessor) to its own Hive instead of the root Hive, and do so immediately upon creating a new database. Without this flag, the system tablets of the new database are created in the root Hive, which can negatively impact its load. Enabling this flag makes databases completely isolated in terms of load, that may be particularly relevant for installations, consisting from a roughly hundred or more databases. To enable this feature add alter_database_create_hive_first: true under feature_flags section into configuration file.

Bug fixes:

• Fixed a bug in the autoconfiguration of the actor system, resulting in all the load being placed on the system pool.
• Fixed a bug that caused full scanning when searching by prefix of the primary key using LIKE.
• Fixed bugs when interacting with datashard followers.
• Fixed bugs when working with memory in column tables.
• Fixed a bug in processing conditions for immediate transactions.
• Fixed a bug in the operation of iterator-based reads on datasharrd followers.
• Fixed a bug that caused cascading reinstallation of data delivery sessions to asynchronous indexes.
• Fixed bugs in the optimizer for scanning queries.
• Fixed a bug in the incorrect calculation of storage consumption by Hive after expanding the database.
• Fixed a bug that caused operations to hang on non-existent iterators.
• Fixed bugs when reading a range on a NOT NULL column.
• Fixed a bug in the replication of VDisks.
• Fixed a bug in the work of the run_interval option in TTL.

Full Changelog: 23.1.26...23.2.12

YDB CLI 2.5.0

Released on June 20, 2023. To update to version 2.5.0, select the Downloads section.

Features

• For the ydb import file command, a parameter --timeout has been added that specifies the time within which the operation should be performed on the server.
• Added a progress bar in commands ydb scheme rmdir --recursive and ydb import file.
• Added the command ydb workload kv run read-rows, which loads the database with requests to read rows using a new experimental API call ReadRows (implemented only in the main branch), which performs faster key reading than select.
• New parameters --warmup-time, --percentile, --topic have been added to the ydb workload topic, setting the test warm-up time, the percentile in the statistics output and the topic name, respectively.
• Added the ydb workload tpch command to run the TPC-H benchmark.
• Added the --ordered flag in the command ydb tools dump, which preserves the order by primary key in tables.

Performance

• The data loading speed in the ydb import file command has been increased by adding parallel loading. The number of threads is set by the new parameter --threads.
• A performance of the ydb import file json command has been increased by reducing the number of data copies.

Full Changelog: CLI_2.4.0...CLI_2.5.0