How to build multiple apps from the same codebase, fast

Michał Laskowski

December 09, 2020


This article may not be helpful to iOS developers working on a single app. It is an explanation, and documentation, for teams that are building multiple versions or flavors of their apps; or multiple apps that have common internal frameworks.

Viacom’s case

We sometimes call our codebase a ‘platform’. For business needs, we are building different flavors of our apps for multiple brands and markets, for both iOS and tvOS.

Our apps are not monolithic targets but consist of several smaller projects. This kind of setup is often called multi-project setup, or modular architecture. Its main goal is to create boundaries between parts of code, to create abstractions between them. This often leads to faster incremental builds because compiler doesn’t have to recompile a module if it hasn’t changed. Also, compiler can optimize for a smaller module, one at a time, instead of trying to process one gigantic project.

Most of our projects also live in the same repository, in one .xcworkspace file, to achieve fastest integration of features in any library, and not bother with versioning of all of ours internal dependencies.

For a long time, our setup was using a standard Fastlane’s build_ios_app (aka gym) action, and we would rebuild every target with cleaned derived data.
By default, this also clears all frameworks that are products of all the projects inside a workspace. It was fine, until a moment when we had to build so many iOS and tvOS apps, that the Continuous Integration and Delivery of our apps was taking too much time on our Mac Mini nodes, or we hade to parallelize our builds so heavily, that we were occupying most of the nodes, and all other jobs had to wait for our post-merge integration and delivery job to finish first.

We attempted to find a solution a few times already, but our research was never fruitful.

Attempt no. 1 - do not clear Derived Data?

Derived Data folder is a special one, commonly known as the-one-to-clear when your code should compile but somehow doesn’t want to. Often, it really helps. The purpose of this folder is to store indexing results, logs, but most importantly Build Products. You can refer to other articles, https://vojtastavik.com/2018/09/02/what-is-inside-derived-data-xcode/ for additional insights.

But on a CI server, you could just clear it at the beginning of your operations, and hope that xcodebuild will be smart enough to reuse as much as possible. Right?

Well, no. At least not with Xcode 11.1, as of time of writing. Let’s follow what happens when you execute something like this in Terminal:

rm -rf derivedData && xcodebuild -workspace YourWorkspace.xcworkspace -scheme MyApp1 -configuration Debug -destination 'generic/platform=iOS' -derivedDataPath ./derivedData clean build | xcpretty

It should build target MyApp1 (under scheme MyApp1), in Debug configuration, and all derived data should go ./derivedData folder.

ls -laG derivedData/Build/Products/Debug-iphoneos/ gives a nice list of all the frameworks, among app and dSyms files, it created.

You can find there frameworks or static libraries, dSyms created from your projects, also object and .swiftmodules created from internal Swift Packages.

Let’s try that again, without clean operation on a second target.

xcodebuild -workspace PlayPlex-iOS/PlayPlex.xcworkspace -scheme MyApp2 -configuration Debug -destination 'generic/platform=iOS' -derivedDataPath ./derivedData build | xcpretty

As it turns out it works. Ok, great, but not so great for the article. Can we end it here? As you probably already see, there is more in it, so yeah, no…
One caveat. You do not build apps with build action. You need to archive them. (and later export signed package from the archive, but Fastlane’s gym is so nice it will do it for you in one step. This is a univeral step, so let’s skip this part).
Ok, one more time, but with archive.

rm -rf derivedData && xcodebuild -workspace YourWorkspace.xcworkspace -scheme MyApp1 -configuration Debug -destination 'generic/platform=iOS' -derivedDataPath ./derivedData clean archive | xcpretty

This time path derivedData/Build/Intermediates.noindex/ArchiveIntermediates/MyApp1/BuildProductsPath/Debug-iphoneos/ holds all the frameworks MyApp1.app, but if you look carefully all frameworks are all symbolic links to respective files under derivedData/Build/Intermediates.noindex/ArchiveIntermediates/MyApp1/IntermediateBuildFilesPath/UninstalledProducts/iphoneos/.

And if you build a second target, it will create a separate derivedData/Build/Intermediates.noindex/ArchiveIntermediates/MyApp2 folder, to keep all its files there. And it will not find anything and reuse from the previous build, in order to save time or CPU cycles.

Attempt no. 2 - build projects first, and do not clear Derived Data?

What if we build all shared frameworks first, where would they go? Would Xcode be able to find them in some other folder, if the frameworks were not build with archive action. How to even build multiple frameworks with one xcodebuild command?

Answer is ‘Find Implicit Dependencies’. This a special settings for a scheme, and it is enabled by default. What it means, is that if you link any framework or library that is a result of another project in your workspace, Xcode will build that project first. This is why you don’t need to provide all your workspace dependencies in Build Phases -> Dependencies tab.

Using this, you can build your frameworks with one command. If your frameworks do not depend on each other, you can try to create an Aggregate target, or create another framework with no sources that links all the others.
Either way, you can build them with:

xcodebuild -workspace YourWorkspace.xcworkspace -scheme iOSFrameworks -configuration Debug -destination 'generic/platform=iOS' -derivedDataPath ./derivedData clean build | xcpretty

And you will find all frameworks in derivedData/Build/Products/Debug-iphoneos/ folder.

Will archive pick them up? Sadly, no, it will build everything from scratch.

Attempt no. 3 - build projects first, disable implicit dependencies?

And of course do not clear derived data, like before. We can use the mechanism mentioned in a previous step to disable implicit dependencies, and force xcodebuild to build only code that belongs directly to the app’s target, and go directly to link and embed phase.
Could it work? Unfortunately, no. It will just throw errors about not being able to find frameworks you are trying to import.

Anyway, how to disable implicit dependencies for CI only? Shared app scheme is just an XML file, one that can be edited if you know the format (you will know it if you just open it in any text editor). Example with Fastlane and fastlane-plugin-xml_editor:

private_lane :disable_implicit_dependencies do |options| # could be a Fastlane action too
  schema = options[:schema]
  schema_path = "#{options[:project_path]}/xcshareddata/xcschemes/#{schema}.xcscheme"
  UI.message("Disabling buildImplicitDependencies at: #{schema_path}")
  xml_editor(path_to_xml_file: schema_path,
             xml_path: "/Scheme/BuildAction/@buildImplicitDependencies",
             new_value: "NO")
end

Configuration Build Dir - a Deus ex machina

Sometime over the winter holidays, I was searching for details for my personal project. It is done in a totally different technology but I was searching for info on different compile flags. Somehow, and I don’t really remember where, I have found info about CONFIGURATION_BUILD_DIR. I only made a note to try it out later. I would like to give credit where it is due, while I am not 100% certain, it could be this article about Kotlin Multiplatform, which gives 404 now.

What this flag controls, is the folder where all products will be stored. It seems it is affecting both build phase, and archive phase, which is a missing piece to make previous attempts work.

The flag can be applied to xcodebuild, but be sure to provide a full path, not a relative one. Providing relative one will cause every project to treat it as a relative to its own location and it will create frameworks all over the place in your workspace.

Now, we can build common frameworks first:

xcodebuild -workspace YourWorkspace.xcworkspace -scheme iOSFrameworks -configuration Debug -destination 'generic/platform=iOS' -derivedDataPath ./derivedData CONFIGURATION_BUILD_DIR=/Users/you/your_path clean build | xcpretty

And the app after:

xcodebuild -workspace YourWorkspace.xcworkspace -scheme MyApp1 -configuration Debug -destination 'generic/platform=iOS' CONFIGURATION_BUILD_DIR=/Users/you/your_path clean archive | xcpretty

And in Fastlane it goes best like this (modify to your liking and setup):

lane :build_common_frameworks do |options|
  platform = (options[:platform] || "iOS").downcase
  scheme = platform == "tvos" ? "tvOSFrameworks" : "iOSFrameworks"
  configuration = options[:configuration] || "Debug"
  
  configuration_build_dir = options[:configuration_build_dir] || configuration_build_dir_from(platform, configuration)
  FileUtils.rm_rf(configuration_build_dir)

  xcargs = "CONFIGURATION_BUILD_DIR='#{configuration_build_dir}'"
  xcargs = xcargs + " BITCODE_GENERATION_MODE=bitcode" if platform == "tvos"

  UI.message("Build common frameworks, scheme: #{scheme}, configuration: #{configuration}")
  xcodebuild(workspace: "YourWorkspace.xcworkspace",
             scheme: scheme,
             configuration: configuration,
             destination: "generic/platform=#{platform}",
             clean: false,
             build: true,
             derivedDataPath: "./derivedData",
             xcargs: xcargs)
end

def configuration_build_dir_from(platform, configuration)
  platform = platform.downcase
  configuration = configuration.gsub(/\s/,'').downcase
  File.expand_path(File.join(Dir.pwd, "..", "conf_build_dir_#{platform}_#{configuration}"))
end

# Requirement! run :build_common_frameworks first. It does not build dependencies
lane :build_app do |options|
  platform = options[:platform] || "iOS"
  project_path = platform == 'iOS' ? 'iOSApps.xcodeproj' : 'tvOSApps.xcodeproj'
  disable_implicit_dependencies(project_path: project_path, schema: options[:scheme])
  # passing for gym in :build lane, to use prebuilt dependencies
  configuration_build_dir = configuration_build_dir_from(platform, options[:configuration])
  
  xcargs = "CONFIGURATION_BUILD_DIR='#{configuration_build_dir}'}'"
  
  build_ios_app(
	  scheme: options[:scheme],
	  configuration: options[:configuration],
	  <any other options yoy need>,
	  xcargs: xcargs
  )
end

Why do we have a lane called build_common_frameworks and a separate lane for apps with disabled implicit dependencies? With disabled implicit dependencies, we are sure we will not start rebuilding frameworks for every app on a CI server. All frameworks should be created first explicitly. If some are missing, the job will fail. Also, we can built each app, via build_app, in a stage that will be visible as a separate step on a CI like Jenkins. If one specific app fails, you will see that easily.

One caveat - BITCODE_GENERATION_MODE=bitcode is appended to xcargs for tvOS, because by default build phase does not create Bitcode in frameworks. If you would compile frameworks without it, your tvOS target will fail to archive during linking phase.

Second caveat - dSyms. Debug symbols are generated once for common frameworks which is a great benefit. It is not a standard though, and if using tools like Firebase Crashlytics, you would need to check if the platform can handle sharing dSyms between apps, or if you need to upload it multiple times.

There are also app-specific dSyms, for files attached directly to the apps target. In this setup, they are stored in the same folder CONFIGURATION_BUILD_DIR. Fastlane’s lane_context[SharedValues::DSYM_OUTPUT_PATH] will not work because dsyms will not be in a standard location that build_ios_app expects it to be.
There is a way to make Fastlane find dsyms to populate lane_context[SharedValues::DSYM_OUTPUT_PATH]:

xcargs = xcargs + " DWARF_DSYM_FOLDER_PATH='#{ENV["CONFIGURATION_BUILD_DIR"]}/#{ENV["APP_SCHEME"]}.app'" unless ENV["PLATFORM"] == 'tvos',

but as a side effect this will also copy dSyms into .ipa archive, which will be rejected by Apple during upload to AppStoreConnect. If you need to reference those debug symbols, you can find and zip them yourself with an additional command:

Fastlane::Actions.sh(%(cd "#{ENV["CONFIGURATION_BUILD_DIR"]}" && zip -r "#{dsym_zip}" . -i '#{ENV['APP_SCHEME']}.app.dSYM/*'))

Why go through all this trouble?

First, to achieve shorter build times when building multiple similar apps and/or minimise the amount of used nodes. In our case, it means more nodes that can do CI for pull requests, but for other companies it can mean saved dollars that would be spent on CI services by having to enroll in higher-priced subscription plans.
For us, common frameworks take 8 minutes to build from scratch on some older Mac Minis we have dedicated to this process. Each app then takes only around 2 minutes to build, most of the time spent on signing the .ipa file. Normally, one app would build in around 10 minutes. As you see there is no difference for one app, but if you need to build ~15 of them from the same codebase, the difference is really noticeable.

Second, it is great to understand how the build system works. You never know when it will come in handy. Your builds can stop working any time, with any Xcode update :) (anyone also had projects using custom pre and post actions in Cocoapod’s Podfile? And then it breaking with newer Xcode? Good times…)

Third, it may be interesting for you and maybe you would like to venture more into the world of CI/CD instead of moving pixels left and right, or more REST calls. It seems there are not so many developers that do coding and CI/CD setup. It is sometimes like an uncharted territory, which you may like to explore. And then you can always write an article about it :)

Are there other ways?

Highly possible, almost certain.

For sure, the one proposed in this article has a great benefit - it requires no changes to the natural and default setup we have in Xcode workspace and projects, so that developers can still use Xcode to build apps, and expect reliable builds on our CI system later. On CI, we are still using the same setup, just disable implicit dependencies and build them explicitly before any app.
This gives great confidence when we need to change anything in project’s settings - if it builds in Xcode, it should also build with xcodebuild without further adjustments.