Pin system conda version to 23.7.3 (#4752) (#4753)

This causes following issues with M1 runners: 

https://github.com/pytorch/audio/actions/runs/6969147299/job/18977954592
Caused by system conda version being 23.10.

Conda and Conda build versions need to be advanced togeather. This makes
sure our system conda version is the same as environment conda version
we create.

.github/actions/setup-binary-builds/action.yml

@@ -109,6 +109,10 @@ runs:
           set -euxo pipefail
           CONDA_ENV="${RUNNER_TEMP}/conda_environment_${GITHUB_RUN_ID}"
 
+          # Both conda-build and conda need to be advanced togeather.
+          # This will make system conda use required value.
+          conda install --yes conda=23.7.3
+
           if [[ "${PYTHON_VERSION}" = "3.11" ]]; then
 
             if [[ "${PACKAGE_TYPE:-}" == "conda" ]]; then