Does TCA need a benchmarking suite?

[tgrapperon]

It can be sometimes difficult to assess how much changes to the core library may improve or deteriorate performance.
I think it would be constructive if a common benchmarking suite for TCA were available. This kind of setup requires a lot of boilerplate, and it will help assessing effects of even minor changes. It would also permit to better understand the weaknesses of the library in specific areas.

At first sight, I think that one should benchmark:

• States with many derived states
• States deeply nested
• Actions with a lot of cases
• Actions deeply nested
• Reducers with many combined components
• Deep pullbacks of reducers
• Reducers producing many side effects
• Reducers producing many synchronous effects

It would also be great to be able to benchmark an arbitrary branch against the latest release, but I don't know how to setup this with SPM.

I also don't know what would be the best way to distribute this benchmarking suite.

What do you think about this?

[nikitamounier]

Completely agree. I was looking to test how using a Deque instead of an Array in the Store implementation would affect performance, but didn't know where to start. Having a benchmarking suite would make TCA more robust, and would lower the barrier of entry for those who want to contribute to TCA who may have great ideas.

I think we can take inspiration from swift-collections and its swift-collections-benchmark, which they made into its own separate library.

[stephencelis]

This would be great! If you've explored this yet, please share any preliminary results.

I also don't know what would be the best way to distribute this benchmarking suite.

As long as we support SPM's tools version 5.2, we should be able to include the benchmarks directly in this repository. Dependents shouldn't see libraries like swift-benchmark come through if they are merely internal to swift-composable-architecture.

[tgrapperon]

After more tests, it seems that only one version of the TCA package can exist in the project. As soon as some local variant of TCA is added, it instantly overrides any other library/package using depending on TCA.

I'm now leaning toward using an Xcode project with a build script building independently both libraries and linking the products in the test target. Xcode projects are more versatile than SPM because they allow build scripts and shared code in arbitrary folders, which is helping greatly in our case. This would furthermore guarantee that both libraries are built in release mode.

[stephencelis]

@tgrapperon We sketched out a super basic idea of how we could have benchmarks in the main repo here:

https://github.com/pointfreeco/swift-composable-architecture/compare/benchmarks

It uses swift-benchmark because we're familiar with it. We're not quite sure how to use XCTest for measuring performance, especially since tests are generally compiled for DEBUG mode. Even if we had a separate test target that ran in RELEASE we're not quite sure how to measure things over time or integrate with CI, since reference data is per machine. It probably can be done, we just have no idea where to start 😅 The time spent, CPU cycles, memory used, etc., all sound compelling, though! Do you have any examples of this in the wild for us to check out?

swift-benchmark does currently require that we manually compare results, but if this ever gets merged we could potentially use it:

google/swift-benchmark#85

[tgrapperon]

@stephencelis Great!, I'll look at it.
I've just published a proof of concept benchmark branch using XCTest.
Two frameworks sharing the same test code are built, one using TCA 0.20.0 (the local implementation), and another using TCA 0.5.0 (to exhibit greater discrepancies, but ideally the latest TCA release). Theses frameworks are built in release mode by a build script, so this aspect is guaranteed. Unfortunately, I'm currently nuking the build products each time, so it takes quite some time to build (twice). There are probably better way to achieve this.

Still unfortunately, one can't measure two things in a single test, so one can't condition the failure of the local measure according to the results of the reference implementation in one test only. This might be possible if all reference implementation measures are realized in a first pass, their results stored in some file (like snapshot-testing), and these results then used to compare local implementation measures. But I'm not an expert in this area.

From my short experience with the topic, the hardest is to manage two distinct implementations in an ergonomic way.
Google's benchmarks is fine and I guess it would be a great candidate.

In my benchmark branch, you only need to open the workspace. The package is here to host the benchmarks, but it is never built (the TCA it's using is only there for code completion). The source code of this package is imported by the two frameworks, one using the local TCA, and the other a reference implementation. Unfortunately, both frameworks need to be updated if some new files are added to the package's source. Modification inside existing files are automatically accounted.
These two frameworks are not exposing TCA and have a similar API. They are imported in the Benchmark app, and its tests are performing the measures.

For example, this is the difference between TCA 0.20.0 (local) and TCA 0.5.0 (reference) for a very simple single action reduction. The current implementation is twice as fast!

I'll look at your branch and I'll think about generating a few types to benchmark heavier scenarios.

[EDIT: I'm now thinking of a much simpler way to arrange things, where a build script manages two packages (instead of frameworks, with automatic code sharing), please let me know if there is some interest going farther in this direction.]

[tgrapperon]

Hey @stephencelis! I've completely refactored the code from my branch. It is way simpler and only use SPM and XCFrameworks. I've added a README in the Benchmarks package, but to sum up:

• You write your benchmarks in the Benchmarks module.

• You benchmark both implementations by simply executing Benchmarks tests.

• If you changed a benchmark's implementation, you simply have to call "sh build-all.sh" that will rebuilt both implementation with the current benchmarking suite.

• If you changed TCA's local implementation, you need to call "sh build-local.sh" to rebuild the local implementation of the benchmarks.

• If you changed which benchmarks are run or what you're measuring, you can directly run the tests without rebuilding the frameworks.

You can also directly call the benchmarks from Benchmarks in tests. It will use the local implementation, in debug mode most likely. This is very convenient when setting up new benchmarks as compilation is faster. This runs surprisingly faster than both frameworks routes for my simple benchmark (but I guess they pay the cost of their interface). Otherwise, both local and reference implementations as XCFrameworks are benchmarked in a strictly equal footing, which is ultimately what's matter I guess. In order to compensate the potential interface effect, I think that benchmarks should take a noticeable amount of time to run.

I guess this setup is also adaptable to Google's swift-benchmarks. I even think that the same code can be used, and both XCTest and swift-benchmarks can potentially coexist.

Unfortunately, the logs from testing are quite verbose. It would be nice if someone had a way to parse them...

It also need swift-package 5.3 (to import the binary frameworks), but I don't think that's an issue with regard to TCA as Benchmarks are living in a separate package in my branch.

That being said, I feel that the user experience is now acceptable and I'll start implementing proper benchmarks.
Please let me know what you think.

[tgrapperon]

Hey @stephencelis! I've updated my branch to support Google's swift-benchmark. Because benchmarks are closed functions in their own module, they can be called from XCTests or swift-benchmark effortlessly.

Once you clone the repo, you need to execute sh build-all.sh to build both implementations, then:

• swift run -c release benchmarks to run benchmarks using swift-benchmark
• swift test -c release to run benchmarks using XCTest.

A framework-less variant (similar to what you did) is benchmarked at the same time, as a proof of concept.

I've also added a simple case with pullbacks. I've used codegen (a playground is in /Internal) and I trimmed the results from 50 to 10 cases/fields/reducers a posteriori, to get faster compile time.

The PR on swift-benchmarks looks like to be only a python script that parses the logs and compares the results. I guess something similar can be realised easily in Swift.

In the current state of the repo, it compares local results with TCA 0.19.0.

I'll pause for now, waiting from feedback from the community.