Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Regression main 20.11 with volatile off #11843

Open
naspirato opened this issue Nov 21, 2024 · 4 comments
Open

Regression main 20.11 with volatile off #11843

naspirato opened this issue Nov 21, 2024 · 4 comments
Assignees
@alexvru
Labels
performance

Comments

@naspirato
Copy link
Collaborator

naspirato commented Nov 21, 2024 •
edited
Loading

Uploading logs.zip…
REGRESSION: 156655/201000: -22.1%

кластер кластер Ветка Конфигурация Начало Тестирования дата коммита Результат 1 Результат 2
perf4_3dc_in_1dc_CI perf4 origin/main default_volatile_off 20.11.2024 11:20:38 20.11.2024 09:24:56 REGRESSION: 156655/201000: -22.1% REGRESSION: 6000/2300: 160.9%
perf4_3dc_in_1dc_CI perf4 origin/main latency_volatile_off 20.11.2024 13:35:37 20.11.2024 09:24:56 REGRESSION: 152447/153500: -0.7% REGRESSION: 416/180: 131.1%
мониторинг Ветка Конфигурация Дата Коммит Время warehouses tpmc efficiency latency
perf4 origin/main default_volatile_off 20.11.2024 2cf848f3 09:24:56 16 000 156 655 76,13 6 000
perf4 origin/main default_volatile_off 20.11.2024 5b45c749 00:26:58 16 000 201 202 97,78 1 000

2024-11-20T15:26:19.784975Z :INTERCONNECT NOTICE: Proxy [50002:7439363654807384853:2050] [node 3] ICP32 transit to hold-by-error state Explanation# outgoing handshake Peer# ydb-qa-01-klg-014.ydb.yandex.net(2a02:6b8:c41:16:0:5a59:d300:a2d:19001) Couldn't connect to any resolved address LastSessionDieTime# 2024-11-20T14:14:13.134972Z
2024-11-20T15:26:19.784988Z :INTERCONNECT NOTICE: Proxy [50002:7439363654807384851:2048] [node 1] ICP25 outgoing handshake failed, temporary: 1 explanation: outgoing handshake Peer# ydb-qa-01-klg-012.ydb.yandex.net(2a02:6b8:c41:16:0:5a59:d300:847:19001) Couldn't connect to any resolved address incoming: [0:0:0] held: no
2024-11-20T15:26:19.784999Z :INTERCONNECT NOTICE: Proxy [50002:7439363654807384851:2048] [node 1] ICP32 transit to hold-by-error state Explanation# outgoing handshake Peer# ydb-qa-01-klg-012.ydb.yandex.net(2a02:6b8:c41:16:0:5a59:d300:847:19001) Couldn't connect to any resolved address LastSessionDieTime# 2024-11-20T14:15:57.158909Z
2024-11-20T15:26:20.785480Z :INTERCONNECT_NETWORK NOTICE: [50002 <-> 3] network-related error occured on handshake: outgoing handshake Peer# ydb-qa-01-klg-014.ydb.yandex.net(2a02:6b8:c41:16:0:5a59:d300:a2d:19001) Couldn't connect to any resolved address
2024-11-20T15:26:20.785517Z :INTERCONNECT NOTICE: Proxy [50002:7439363654807384853:2050] [node 3] ICP25 outgoing handshake failed, temporary: 1 explanation: outgoing handshake Peer# ydb-qa-01-klg-014.ydb.yandex.net(2a02:6b8:c41:16:0:5a59:d300:a2d:19001) Couldn't connect to any resolved address incoming: [0:0:0] held: no
2024-11-20T15:26:20.785536Z :INTERCONNECT NOTICE: Proxy [50002:7439363654807384853:2050] [node 3] ICP32 transit to hold-by-error state Explanation# outgoing handshake Peer# ydb-qa-01-klg-014.ydb.yandex.net(2a02:6b8:c41:16:0:5a59:d300:a2d:19001) Couldn't connect to any resolved address LastSessionDieTime# 2024-11-20T14:14:13.134972Z
2024-11-20T15:26:20.785545Z :INTERCONNECT_NETWORK NOTICE: [50002 <-> 1] network-related error occured on handshake: outgoing handshake Peer# ydb-qa-01-klg-012.ydb.yandex.net(2a02:6b8:c41:16:0:5a59:d300:847:19001) Couldn't connect to any resolved address
2024-11-20T15:26:20.785555Z :INTERCONNECT NOTICE: Proxy [50002:7439363654807384851:2048] [node 1] ICP25 outgoing handshake failed, temporary: 1 explanation: outgoing handshake Peer# ydb-qa-01-klg-012.ydb.yandex.net(2a02:6b8:c41:16:0:5a59:d300:847:19001) Couldn't connect to any resolved address incoming: [0:0:0] held: no
2024-11-20T15:26:20.785567Z :INTERCONNECT NOTICE: Proxy [50002:7439363654807384851:2048] [node 1] ICP32 transit to hold-by-error state Explanation# outgoing handshake Peer# ydb-qa-01-klg-012.ydb.yandex.net(2a02:6b8:c41:16:0:5a59:d300:847:19001) Couldn't connect to any resolved address LastSessionDieTime# 2024-11-20T14:15:57.158909Z

![image.png](/ajax/v2/attachments/227193163?inline=true =x600)

git log --graph --oneline --pretty=format:'%H - %ad %s %an' 5b45c74^..2cf848f
e1004307b2b619c0263c6d7379acf7f4
@naspirato
Copy link
Collaborator Author

naspirato commented Nov 21, 2024 •
edited
Loading

2cf848f - perf2 http://ydb-vla-dev04-005.search.yandex.net:8765/monitoring/cluster/nodes - падает на запуске
912022f - perf5 http://ydb-qa-01-klg-018.ydb.yandex.net:8765/monitoring/cluster/nodes - падает на загрузке
5876bb1 - perf4 http://ydb-qa-01-klg-012.ydb.yandex.net:8765/monitoring/cluster/nodes - одна нода мертва
8357e1c - perf4 http://ydb-qa-01-klg-012.ydb.yandex.net:8765/monitoring/cluster/nodes - ok
509152a -perf5 http://ydb-qa-01-klg-018.ydb.yandex.net:8765/monitoring/cluster/nodes - ok
5b45c74 - perf2 http://ydb-vla-dev04-005.search.yandex.net:8765/monitoring/cluster/nodes - ok

@naspirato
Copy link
Collaborator Author

naspirato commented Nov 22, 2024 •
edited
Loading

Похоже, что проблемы начались с 5876bb1

cluster_monitoring git_branch run_type run_start_datetime git_sha git_commit_timestamp warehouses tpmC efficiency newOrder90p
perf4 origin/main default_volatile_off 21.11.2024 00:06:38 41baadd 20.11.2024 20:40:43 16000 179243 87.11 5250
perf4 origin/main default_volatile_off 20.11.2024 11:20:38 2cf848f 20.11.2024 09:24:56 16000 156655 76.13 6000
perf4 origin/main default_volatile_off 21.11.2024 22:40:57 5876bb1 20.11.2024 08:50:38 16000 167170 81.25 5750
perf4 origin/main default_volatile_off 21.11.2024 13:24:48 8357e1c 20.11.2024 08:34:10 16000 201097 97.73 512

Нода отваливается после загрузки данных

: 1 Version: (PathId: [OwnerId: 72075186224037889, LocalPathId: 1], Version: 22) DomainId: [OwnerId: 72057594046678944, LocalPathId: 2] AbandonedSchemeShards: there are 0 elements }
2024-11-21T21:11:03.744907Z :SCHEME_BOARD_REPLICA NOTICE: [3:7439842311472782035:2059] Update description: path# /Root/db1, pathId# [OwnerId: 72075186224037889, LocalPathId: 1], deletion# false
2024-11-21T21:11:03.744993Z :SCHEME_BOARD_REPLICA NOTICE: [3:7439842311472782035:2059] Update description: path# /Root/db1/customer, pathId# [OwnerId: 72075186224037889, LocalPathId: 7], deletion# false
2024-11-21T21:11:03.875610Z :SCHEME_BOARD_REPLICA NOTICE: [3:7439842311472782035:2059] Update description: path# /Root/db1/warehouse, pathId# [OwnerId: 72075186224037889, LocalPathId: 2], deletion# false
2024-11-21T21:11:03.902914Z :SCHEME_BOARD_REPLICA NOTICE: [3:7439842311472782035:2059] Update description: path# /Root/db1/district, pathId# [OwnerId: 72075186224037889, LocalPathId: 3], deletion# false
2024-11-21T21:11:03.907075Z :SCHEME_BOARD_REPLICA NOTICE: [3:7439842311472782035:2059] Update description: path# /Root/db1/new_order, pathId# [OwnerId: 72075186224037889, LocalPathId: 4], deletion# false
2024-11-21T21:11:03.907098Z :SCHEME_BOARD_REPLICA NOTICE: [3:7439842311472782035:2059] Update description: path# /Root/db1/oorder, pathId# [OwnerId: 72075186224037889, LocalPathId: 5], deletion# false
2024-11-21T21:11:03.916087Z :SCHEME_BOARD_REPLICA NOTICE: [3:7439842311472782035:2059] Update description: path# /Root/db1/item, pathId# [OwnerId: 72075186224037889, LocalPathId: 6], deletion# false
2024-11-21T21:11:03.934898Z :SCHEME_BOARD_REPLICA NOTICE: [3:7439842311472782035:2059] Update description: path# /Root/db1/customer, pathId# [OwnerId: 72075186224037889, LocalPathId: 7], deletion# false
2024-11-21T21:11:35.531100Z :INTERCONNECT_NETWORK NOTICE: [3 <-> 50013] socket: Connection reset by peer
2024-11-21T21:11:35.535275Z :INTERCONNECT NOTICE: Proxy [3:7439842844048740777:2085] [node 50013] ICP25 outgoing handshake failed, temporary: 2 explanation: outgoing handshake Peer# ydb-qa-01-klg-012.ydb.yandex.net(2a02:6b8:c41:16:0:5a59:d300:847:19007) Peer rejected session continuation handshake incoming: [0:0:0] held: no
2024-11-21T21:11:35.535319Z :PIPE_SERVER ERROR: [72057594046447618] NodeDisconnected NodeId# 50013
2024-11-21T21:11:35.535326Z :PIPE_SERVER ERROR: [72057594046447617] NodeDisconnected NodeId# 50013
2024-11-21T21:11:35.535327Z :PIPE_SERVER ERROR: [72057594046447619] NodeDisconnected NodeId# 50013
2024-11-21T21:11:35.535330Z :PIPE_SERVER ERROR: [72057594037936131] NodeDisconnected NodeId# 50013
2024-11-21T21:11:35.535335Z :HIVE WARN: HIVE#72057594037968897 Handle TEvInterconnect::TEvNodeDisconnected, NodeId 50013
2024-11-21T21:11:35.535342Z :PIPE_SERVER ERROR: [72057594037936129] NodeDisconnected NodeId# 50013
2024-11-21T21:11:35.535349Z :PIPE_SERVER ERROR: [72057594046678944] NodeDisconnected NodeId# 50013
2024-11-21T21:11:35.535350Z :PIPE_SERVER ERROR: [72057594037936129] NodeDisconnected NodeId# 50013

@naspirato
Copy link
Collaborator Author

@alexvru cluster http://ydb-qa-01-klg-012.ydb.yandex.net:8765
logs
live node
down node

@naspirato
Copy link
Collaborator Author

Исправилось после реверта

#11891 (comment)

cluster_monitoring repository git_branch run_type run_start_datetime git_sha git_commit_timestamp duration sec warehouses tpmC efficiency newOrder90p
perf4 alexvru/ydb origin/main issue_11843 22.11.2024 15:49:34 f437d65 22.11.2024 13:50:28 7200 16000 201699 98.03 512

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@alexvru alexvru

Labels
performance
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@alexvru @naspirato