Using Swift snapshot testing with Xcode Cloud

June 20, 2023

swift-snapshot-testing by Point-Free is an excellent modern approach to Swift snapshot (screenshot) testing. It doesn’t work out of the box with Xcode Cloud. This post is about how I made it work correctly with Xcode Cloud.

TL;DR: gist with the code.

Background

snapshot-testing default behavior assumes that the environment running the tests has access to the source repository where the reference snapshots are kept. This is a fine assumption when developing and testing locally and also in many CI environments, but not so in Xcode Cloud. Xcode Cloud is designed so that building and executing the tests are two discrete steps that happen in different environments, and the test runner does not have access to the source code, except the ci_scripts folder that seems to get special treatment.

When you set up snapshot-testing with its minimal default configuration and test locally, everything works. When you push your code and tests to Xcode Cloud and run them, the tests will fail because they do not find reference screenshots. You will see the error about snapshot-testing saving new screenshots, but of course the environment is cleaned with every run, so next run will have the same failure.

Many people have devised clever solutions about hacking the screenshots to be saved into ci_scripts, so that they get transferred across environments. This felt too hacky for me, so I thought about a better way to do this. Turns out that there is one: you can package the screenshots into the test bundle.

Storing snapshots into the test bundle

I asked about this in WWDC 2023 Ask Apple session, and got this helpful comment.

I had of course previously used resources bundled in tests, but hadn’t thought too much about it in the context of snapshot testing. It makes sense - the test bundle is cleanly transferred to the test runner in Xcode Cloud, and can use the snapshots that are bundled there.

With this in mind, I set myself the following design goals for my solution.

Minimal deviation from standard snapshot-testing behavior. I like what snapshot-testing does out of the box: for example, storing the snapshots in __Snapshots__ folder adjacent to the test code. Whatever I do for Xcode Cloud should be a thin clean layer on top of this.

Should work with both SPM test targets as well as top-level “xcodeproj” unit test targets. I have tests, including snapshot tests, that are run and built in both of these ways. Both kinds of snapshot tests should run correctly in Xcode Cloud.

Support testing multiple locales. I specifically test with multiple locales to assure that my localization is correct.

Minimal impact to call site. The snapshot testing API from the test functions should change as little as possible. When you build the tests, you shouldn’t need to know about Xcode Cloud.

Easy to understand and maintain. I don’t want to think too much about this. Whatever I do should be resilient and remain working with minimal maintenance. This is why I am doing this blog post and have comments in the code–mostly for my own future self.

So here’s my solution to make swift-snapshot-testing work correctly in Xcode Cloud in three easy steps.

Step 1: get comfortable with snapshot-testing locally

Before talking about Xcode Cloud, youy should have snapshot-testing set up and working locally. My solution builds on its standard behavior of storing snapshots next to your unit test classes in __Snapshots__ folder. It’s fine if you customize it, you’ll just need to accordingly customize the next steps then.

If you now run your tests in Xcode Cloud, you should see snapshot-tests failing with “no snapshot found, storing a new one.”

Step 2: bundle your snapshots into the test bundle

This step will vary depending on if you are working with SPM targets or xcodeproj app target.

For SPM, you just add the copy step into Package.swift in your test target:

.testTarget(
  name: "MyViewTests",
  path: "If/There/Is/Custom/Path",
  resources: [
    .copy("__Snapshots__")
  ]
)

For xcodeproj app target, you should add the folder containing the snapshots into the test target: make sure the “Target membership” has the test target checked for the folder.

You may first attempt to add multiple __Snapshots__ folders from different tests into the test bundle. When you do this, Xcode will return this kind of build error.

It doesn’t like multiple __Snapshots__ folders being added into the same test target. I thought that it would be smart and consolidate the contents of those folders into a single top-level __Snapshots__, but it doesn’t do that.

The solution is to not add the __Snapshots__ folders to the test target, but add the folders under them, one level down. The names of those folders should match the names of your test classes and are hopefully unique across your project. I did this by removing references to the __Snapshots__ folder from my Xcode project, and dragging the lower-level folders with snapshots into the project instead, which I added as members of the test target.

At the end of this step, everything looks like before: your snapshot tests continue to run locally and fail in Xcode Cloud. However, if you inspect your xctest bundles in DerivedData, you should see the screenshots bundled in them for both SPM targets and xcodeproj targets. In the first case, screenshots are in __Snapshots__ folder, and in the second, they are directly contained in the test resources.

Here’s what the folders look like for one of my SPM targets. The xctest target looks like this.

If you inspect the Library_SettingsTest.bundle that contains the resources, you should see the screenshots.

It will look similar for the app target, except that the xctest is inside your app target Plugins, and there are no intermediate Library_something.bundle and __Snapshots__ folders.

By this point, you should see that the screenshots are present in your test bundle. Let’s now make the snapshot testing library pick them up, which will make the tests to run correctly in Xcode Cloud.

Step 3: instruct snapshot-testing to use screenshots from the test bundle

Here is my test function that implements this.

The core idea is that the test runner checks whether a screenshot is present the test bundle. When true (which is the case for Xcode Cloud when everything is set up correctly), the runner uses screenshots inside the bundle. When false (which may be the case if you have written a test locally but not yet recorded snapshots), the runner falls back to snapshot-testing default behavior of using a folder adjacent to the test code, auto-recording the snapshot if needed.

When calling this function from SPM tests, use this:

assertSnapshot(
  view: someSwiftUIView,
  testBundleResourceURL: Bundle.module.resourceURL!
)

When calling in top-level xcodeproj app target tests, use this:

assertSnapshot(
  view: sut,
  testBundleResourceURL: Bundle(for: type(of: self)).resourceURL!
)

The testBundleResourceURL parameter is a bit unwiedly, but I couldn’t think of a better way to pass the information cleanly and correctly over to the function from both contexts.

I’ve written it up as a gist rather than a SPM package, because the requirements may vary, and this is currently implemented for only one device and fixed view size. If you are comfortable with snapshot-testing, you can surely adjust this to your own needs.

One other thing you need to do when using my solution is to adjust your Xcode workflow to use just “iPhone 14 Pro” instead of “Recommended iPhones”.

If you have done everything above correctly and followed the above steps and I didn’t mistype anything, you should now see your snapshot tests passing in Xcode Cloud. 🎉