Upgrade to Spark 3.5.4

[jacobsalway]

Purpose of this PR

Upgrade to Spark 3.5.4

Change Category

• Bugfix (non-breaking change which fixes an issue)
• Feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that could affect existing functionality)
• Documentation update

Rationale

Checklist

• I have conducted a self-review of my own code.
• I have updated documentation accordingly.
• I have added tests that prove my changes are effective or that my feature works.
• Existing unit tests pass locally with my changes.

Additional Notes

[google-oss-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from jacobsalway. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[jacobsalway]

We can add another row for 3.5.4 later depending on whether this ends up in a minor or patch release.

[ChenYi015]

What about bumping the Spark version to v3.5.4 when we are going to release the next minor version i.e. v2.2.0?

[jacobsalway]

Sure, I'll put this back into draft and mark it as frozen so we can take it up in the next minor release.

[jacobsalway]

/assign @ChenYi015

[jacobsalway]

/lifecycle frozen

Holding for next minor release.

[google-oss-prow]

@jacobsalway: The lifecycle/frozen label cannot be applied to Pull Requests.

In response to this:

/lifecycle frozen

Holding for next minor release.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[jacobsalway]

/hold for next minor release.