Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: [drop_packets/drop_packets.py][DNX] 'RX_DRP' drop counter was not incremented on Port Channels #20873

Open
kenneth-arista opened this issue Nov 18, 2024 · 3 comments
Labels
BRCM regression Triaged this issue has been triaged

Comments

@kenneth-arista
Copy link
Contributor

Issue Description

Tracking failures seen in drop_packets/drop_packets.py on T2 and T2-min

Results you see

Failed: 'RX_DRP' drop counter was not incremented on iface Ethernet40. DUT RX_DRP == [0, 0]; Sent == 1000

Results you expected to see

The test should pass

Is it platform specific

broadcom

Relevant log output

def _check_drops_on_dut():
            return packets_count in _get_drops_across_all_duthosts()

        if not wait_until(25, 1, 0, _check_drops_on_dut):
            # We were seeing a few more drop counters than expected, so we are allowing a small margin of error
            # The max number of unexpected drop we see equals to the number of vlan members in t0 topology
            duthost = duthosts.frontend_nodes[0]
            mg_facts = duthost.minigraph_facts(host=duthost.hostname)["ansible_facts"]
            DROP_MARGIN = 0 if mg_facts['minigraph_vlans'] else 10
            for vlan in mg_facts['minigraph_vlans']:
                DROP_MARGIN += len(mg_facts['minigraph_vlans'][vlan]['members'])
            logger.info(f"The DROP_MARGIN is {DROP_MARGIN}")
            actual_drop = _get_drops_across_all_duthosts()
            for drop in actual_drop:
                if drop >= packets_count and drop <= packets_count + DROP_MARGIN:
                    logger.warning("Actual drops {} exceeded expected drops {} on iface {}\n".format(
                        actual_drop, packets_count, dut_iface))
                    break
            else:
                fail_msg = "'{}' drop counter was not incremented on iface {}. DUT {} == {}; Sent == {}".format(
                    column_key, dut_iface, column_key, actual_drop, packets_count)
>               pytest.fail(fail_msg)
E               Failed: 'RX_DRP' drop counter was not incremented on iface Ethernet40. DUT RX_DRP == [0, 0]; Sent == 1000

DROP_MARGIN = 10
_check_drops_on_dut = <function verify_drop_counters.<locals>._check_drops_on_dut at 0x7f55d4d10280>
_get_drops_across_all_duthosts = <function verify_drop_counters.<locals>._get_drops_across_all_duthosts at 0x7f55d4d109d0>
actual_drop = [0, 0]
asic_index = None
column_key = 'RX_DRP'
drop       = 0
dut_iface  = 'Ethernet40'
duthost    = <MultiAsicSonicHost nfc420-7>
duthosts   = [<MultiAsicSonicHost nfc420-7>, <MultiAsicSonicHost nfc420-8>, <MultiAsicSonicHost nfc420>]
fail_msg   = "'RX_DRP' drop counter was not incremented on iface Ethernet40. DUT RX_DRP == [0, 0]; Sent == 1000"
get_cnt_cli_cmd = 'portstat -j '
mg_facts   = {'deployment_id': '1', 'dhcp_servers': [], 'dhcpv6_servers': [], 'forced_mgmt_routes': ['10.64.0.0/14', '10.80.0.0/12', '172.16.0.0/12', '10.224.0.0/11', '10.100.248.0/22', '10.100.252.0/22', ...], ...}
packets_count = 1000

common/helpers/drop_counters/drop_counters.py:134: Failed


### Output of `show version`

_No response_

### Attach files (if any)

_No response_
@kenneth-arista kenneth-arista changed the title [Bug]: [drop_packets/drop_packets.py] 'RX_DRP' drop counter was not incremented on iface [Bug]: [drop_packets/drop_packets.py] 'RX_DRP' drop counter was not incremented on Port Channels Nov 20, 2024
@kenneth-arista kenneth-arista changed the title [Bug]: [drop_packets/drop_packets.py] 'RX_DRP' drop counter was not incremented on Port Channels [Bug]: [drop_packets/drop_packets.py][DNX] 'RX_DRP' drop counter was not incremented on Port Channels Nov 20, 2024
@kenneth-arista
Copy link
Contributor Author

This problem is seen on only Port Channels on DNX platforms. We see the issue regardless if port-channel members are on the same or different cores.

Opened Broadcom CS case: CS00012379105

@kenneth-arista
Copy link
Contributor Author

@arlakshm @vmittal-msft @ysmanman @arista-nwolfe for awareness

@rlhui rlhui transferred this issue from sonic-net/sonic-mgmt Nov 20, 2024
@rlhui rlhui added BRCM regression Triaged this issue has been triaged labels Nov 20, 2024
@kenneth-arista
Copy link
Contributor Author

kenneth-arista commented Nov 20, 2024

Potentially related to sonic-net/sonic-mgmt#15051 (corresponding Brcm case CS00012373986)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
BRCM regression Triaged this issue has been triaged
Projects
Status: No status
Development

No branches or pull requests

2 participants