iOS Continuous Integration Guide

This guide will help you set up a server to test, build, and deploy your iOS apps automatically.

Here at Casa, we build solutions to help simplify the use of cryptocurrencies like Bitcoin. Due to the nature of our products, it is extremely important for all code that gets introduced to be peer reviewed. This alleviates any possibility of insecure code finding its way into our products.

We move quickly and we ship product constantly. Shipping code quickly introduces risks. It’s important to put in place safeguards to protect developers from accidentally introducing bugs. That’s why we recommend Continuous Integration.

What Is Continuous Integration and Why Use It?

Continuous Integration (CI) is an automated way for you to test code and push builds for testing and App Store deployment. It gives us several amazing benefits:

• Automated tests are run by a central server before every build is pushed. Writing and maintaining a test suite for your app may feel tedious, but in the long run you will have more confidence that your code won’t contain bugs. You can also push more regularly due to a lower reliance on manual testing, which can be very time consuming.
• We can guard against malicious code injection. This is important if you are working in a team environment and a developer may have something to gain by injecting code that, for example, logs a user’s private keys. A good CI process can ensure that every commit is peer reviewed before it is merged. Because only the CI server has access to the certificates needed to push to the app store, only code that has been merged through peer review will be compiled and pushed to the store.
• It gives us verifiable code. By using automated tagging, we can ensure our GitHub repo is tagged correctly for every build and that the exact commit hash of the build is embedded into our app. This gives us 100% confidence in our ability to identify what code is running where.
• It saves time. Pushing builds out constantly can become very time consuming. A good CI will take care of all this for you. Automated testing also reduces the need for time consuming manual testing.

Strategy Overview

There are two main recommended paths toward Continuous Integration for iOS. The path supported and recommended by Apple relies on Xcode Server, which allows you to configure and run ‘Bots’ to execute tests within your project. However, at the time of writing (Xcode 10.1), these integration bots do not support pushing the builds to the app store and I have had no luck working around this restriction.

Here is a link to Apple’s guide to CI with Xcode Server: https://developer.apple.com/library/archive/documentation/IDEs/Conceptual/xcode_guide-continuous_integration/index.html

The second path, outlined below, involves using a Jenkins server to handle task management and a library called Fastlane that helps simplify all the building, signing and uploading of the builds. Jenkins is listening for new events and then launching our builds while Fastlane is actually performing the heavy lifting and managing all of the command line utilities to perform the actions that we need locally on the server.

Jenkins will be set up with two tasks:

• The second task will look for changes to the master branch and, when found, run tests and build and deploy our staging and production targets.

Fastlane will be configured with a few simple jobs to make this possible:

• Run our automated UI tests. This will launch the app UI and click through all of our key work flows in the app.
• Run our automated unit and integration tests. Our unit tests will test specific key methods at a function level, allowing us to test stuff like signing a transaction with a given private key. Our integration tests will run against the development API and perform a set of API requests to make sure that our results are as expected.
• A set of 4 jobs exist to automate the building and uploading of each of our 4 targets: Prototype, Dev, Staging and Prod. Each of these targets have separate app store profiles, allowing each to be available and testable via TestFlight.

Deployments

Now, to deploy a new Prototype and Dev build, we need to create a Pull Request from our feature branch into Development and merge it after approval. The Jenkins server will detect this change and launch a new Project-Dev task, which will run tests and push to the store.

To push to staging / prod, we need to PR from Development into Master. This will launch a new Project-Prod task that will run tests and push to the TestFlight.

Note: In addition to pushing to the correct git branch, you must also ensure that for each target, the app store listing has a new version pending with correct new version number in the target. This is done by logging into appstoreconnect.apple.com, going to the given app, and selecting add new version. The version found in the info.plist for the given target in Xcode must be updated to match this version.

To check on the status of a build, you can log in to Jenkins from a remote computer by opening the IP address of the remote machine at port 8080 in any web browser (http://your.ip.address:8080). Once logged in, you may view the status of any previous jobs, view the console from any running jobs, and manually launch a new job.

Step-by-step Setup:

Access Remote Server (via macstadium.com)

Create a new account at macstadium.com. This will give you access to a fresh Mac Mini that is all set up for remote access.

To access the machine remotely, you may use Apple’s ‘Screen Sharing’ software and enter the IP address for the new machine as found in the Subscriptions tab in macstadium. The username will be administrator and the default password will be provided through their portal. Once logged in, change this default password.

Next, make sure the following are installed:

• the latest version of Mac OS (currently Mojave)
• Xcode (from the App Store)
• Xcode command line utilities

xcode-select --install

• Ruby Version Manager

\curl -sSL https://get.rvm.io| bash -s stable --ruby

• Homebrew

/usr/bin/ruby -e “$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

• CocoaPods

gem install cocoapods

• Fastlane

sudo gem install fastlane -NV

Install Jenkins Server

While logged in as the administrator, install Jenkins. While there is a dmg installer available, I was unable to get Jenkins set up with the installer and instead installed using Homebrew.

brew install jenkins

When installing, you only need a minimal set of plugins, so don’t install with all the defaults. Instead, look through the plugins and only install ones that look useful, like GitHub.

Once the installer is run, you have to unlock it. Do this by running jenkinsfrom the command line to launch Jenkins. Then, from a web browser, navigate to localhost:8080 to open the Jenkins application, which will direct you to access a local password file to unlock it.

Next, you’ll want to create a new Jenkins User. Open the Jenkins console in a web browser on the remote machine and hit Jenkins → Manage Jenkins → Manage Users → Add New. Create a new user and assign them a password.

Close the command line console to close Jenkins. Note that by default the workspace will be set at ~/.jenkins. You may wish to rename it, mv ~/.jenkins ~/jenkinsto make the workspace more visible.

Configure Jenkins to Launch on Boot

In the current configuration, Jenkins will only run when the administrator is logged in and must be manually started by the user. We’ll want Jenkins to launch without requiring a login every time the computer boots .

To help make this task a bit easier, I enlisted help from a program called LaunchControl, available here for $10:
http://www.soma-zone.com/LaunchControl/

Install, purchase, and open LaunchControl on the remote computer, and let’s create a new Global Daemon.

On the top-left of the screen, you’ll see a drop down with User Agents selected. Change that to Global Daemons and select File→New.

We’ll configure this job as shown below:

• Rename the File in the left hand pane to Jenkins
• The Program to Run should be set to /usr/local/bin/jenkins
• Drag and drop a Standard Output object from the right pane to configure your logging output location
• Drag and drop an environment variable object from the right pane to set up your environment variables. You can select Import from Shell to import your path. We must also configure our Jenkins home path and set our language to UTF8 as shown. Note that the default Jenkins home path is ~/.jenkins, but i manually moved my Jenkins files to ~/jenkins.
• Drag and drop the username object from the right pane and set it to run as Administrator and the group ‘wheel’.
• Make sure Run at Load is checked.
• Save this job, then right click it and ‘Load it’. This will launch the job and make sure it launches on boot from here on out. After loading, the job should show as ‘running’ and Jenkins should be accessible at localhost:8080 from the remote machine, or from any outside machine at http://[ipaddress]:8080.

Integrate Fastlane

We must now setup Fastlane in our local Xcode project.

From the project directory, on your mac, run

sudo gem install fastlane -NVfastlane init

(this assumes you already have ruby installed on your local machine. If you don’t, see above to install rvm and ruby)

You’ll also want to add these lines into your ~/.bash_profile

export LC_ALL=en_US.UTF-8export LANG=en_US.UTF-8

This will generate all the Fastlane files and integrate it into your project. You’ll likely want to add fastlane/fastfile into your project so you can edit it in Xcode and then add some of the Fastlane files to your .gitignore:

fastlane/report.xmlfastlane/Preview.htmlfastlane/screenshotsfastlane/test_output

You may also want to follow the instructions here to add a gemfile and lock the versioning of Fastlane: https://docs.fastlane.tools/getting-started/ios/setup/

Now in Xcode we can create our Fastlane tasks. Open the fastfile generated by Fastlane and add some tasks:

desc "Push a new proto dev build to the App Store"    lane :release_dev do				        set_info_plist_value(path: "Project/Info-Proto.plist", key: "commit_hash", value: last_git_commit[:commit_hash] )

build_app(workspace: "Project.xcworkspace", scheme: "Project-Dev")

upload_to_app_store(skip_metadata: true, skip_screenshots: true, app_identifier: "com.Dev", username: "user_name@apple.com")    end

desc "Run automated tests"    lane :test do        run_tests(workspace: "Project.xcworkspace",           devices: ["iPhone 6"],           scheme: "Project-Dev", clean: true)    end

desc "Run automated ui tests"    lane :test_ui do        run_tests(workspace: "Project.xcworkspace",        devices: ["iPhone 6"],        scheme: "Project-Proto", clean: true)    end

Note the set_info_plist command is going to take the commit hash of the current build that we are compiling and add it into our project at build time so we can access it and display it in code within our project.

Now commit this code and push it up to the server.

git checkout -b 'fastlane'git add .git commit -am 'add fastlane'

Push this new branch and merge it into dev.

Configure GitHub Webhooks

Generate a new secret key for the webhook. Open the Jenkins console from a web browser and go to Manage Jenkins → Configure Jenkins → GitHub. If you do not see the GitHub section ensure the GitHub plugin is installed.

Select Advanced Settings and you’ll see a drop-down for Shared Secret. Select ‘add’ and follow the steps to generate and store a new shared secret.

We need to configure GitHub to send webhooks for our repository to Jenkins. Go to github.com and navigate to the repo settings → webhooks → add webhook

• Set the URL: http://[server ip]:8080/github-webhook/NOTE THE TRAILING / !!
• Set content type to application/json
• Add the shared secret we generated in Jenkins
• Specify custom events and select pushes and pull requests

Configure Jenkins to Pull from GitHub

From the remote server, we need to add an SSH key to the admin user and add the key to our GitHub repo as a deploy key. Go to the repo settings page, under deploy keys and follow this guide to set up a new SSH key to the server and add it as a deploy key. Note that this deploy key must have write access in order to push tags to GitHub to tag our builds.

https://developer.github.com/v3/guides/managing-deploy-keys/#deploy-keys

Configure Jenkins Tasks

Now we need to get our project compiling and running tests on the server.

Log in to the remote server via screen sharing. Open Jenkins via localhost:8080.

Select ‘New Item’ and create a free style project. Call it Project-Dev.

• Give it a general description
• Under general — github project, link the http url of your repo: https://github.com/Org/project/
• Under source code management, select git and add a new repository
• Under repository add the SSH url like so: [git@github.com](<mailto:git@github.com>):Org/repo.git
• Under credentials, select ‘add new’ and add a new key secured with the Jenkins Credentials Provider.
• Select SSH username and private key and enter the username and private key generated in the previous steps.
• Select Advanced Details and add a name for the repo: Org/repo
• Under branches to build, add a new branch and enter the name of the development branch dev.
• Under Build Triggers, select “GitHub hook trigger for GITScm polling”
• Under Build, add a new Build Step and enter in the commands to execute the build:

pod repo update
pod install
fastlane test_ui
fastlane test
fastlane release_dev

We’ll want to then create a task for production releases. You can do this by simply copying the settings from the dev build, but change the names of the branches and the fastlane targets to execute.

Configure Email Notifications

To have Jenkins send emails after every build, first create a new gmail account exclusively for sending these build summary emails.

Next, configure Jenkins with a default email configuration

Go to Configure Jenkins — Manage Configuration

Under Extended E-mail Notification:

• Set smtp server: smtp.gmail.com
• Click Advanced
• Click Use SMTP Authentication
• Enter the username and password for your gmail account
• Check Use SSL
• Set smtp port: 465
• Set Default Reply-To email: you@company.co
• For Default recipients, add any developers that want to be emailed for EVERY action
• Under Default smtp suffix add your mail server suffix (@whatever.com)

Scroll up to Jenkins Location

• Set Jenkins Location to your external server URL.
• Set your System Admin e-mail address to the address you want to display in your emails

Now configure your tasks:

• Go to your build task configuration
• Under Post-build Actions, add a new Editable Build Notification
• Click advanced settings
• Under attach build log, select Attach Build Log
• Under Triggers, remove the default mail trigger
• Now add a new trigger for Always to send out an email to the default recipient list after the build runs
• Add a second trigger for Before Build to send to the default recipient list before a build is run
• Add a final trigger for Success. Under recipients, press advanced and add the email addresses for any additional mailing lists that would like to be notified after successful builds, like ‘eng@company.com’

Install Certificates and Provision Profiles

This should now compile our project, but builds cannot successfully upload, because the server does not have the certificates required to sign our code and to push the builds to the app store.

Certificate and provisioning profile management in iOS can be quite a pain, but here are the basic steps to get this working.

• Log in to the apple developer portal at developer.apple.com/ios
• Create a new app ID to match the project’s bundle in Xcode if it hasn’t been created yet;.
• Create a new private key to store on the server by opening keychain access on the server, then keychain assistant → request new certificate from a certificate authority.
• On the iOS developer portal, go to manage certificates → Add New and create a new App Store distribution certificate for the specific app ID. You’ll have to upload the signing request you generated in the previous step. Download and open that certificate on the server to load it into the keychain.
• Go to provisioning profiles and add a new distribution profile for App Store distribution. Select the certificate we generated previously. Download and open that profile on the Server to load it into Xcode.
• On your local machine, download those same files from the developer portal. The private key will not be installed on your local machine since it was generated remotely. This is good since we only want the server to have access to sign builds.
• Open Xcode on your local machine and make sure ‘automatically manage code signing’ is unchecked. Then select the certificate and provisioning profiles you downloaded under the release version of the target.
• Do the above for each project target that you want to upload to the app store.
• Commit and push these changes to git.

The project is now set up to use our new profiles and the profiles are stored on the admin keychain on the server, but we need to move them to the system keychain so that the Jenkins server can access them without the admin user being signed in.

On the server, open keychain access. From the Login keychain, open the certificates tab and find the certificates you downloaded previously.

Copy those certificates by clicking them to expand them and highlighting both the certificate and the associated private key. Then select Edit→ Copy. Then Paste those into the System keychain.

App Store Access

The last thing we need is access to upload the build to the app store. We’ll create a new user in appstore connect and add that user’s credentials to the server’s keychain.

• Go to appstoreconnect.apple.com → Users → Add New
• Add a new user with developer access assigned to the specific apps that you want the server to be able to push to. You’ll need to confirm the email address and assign a set of security questions.
• You’ll also need to accept the developer terms of use. Do this by navigating to developer.apple.com/ios and logging in with that user account. It will then prompt you to accept the developer terms of use.
• Now we need to add these credentials to the keychain on the server.Note: I was unable to actually add this to the keychain manually. However, Fastlane has a tool that will add this for us. We we are going to use this tool running with Jenkins so that we can be sure it gets stored to the system keychain instead of the admin user keychain, otherwise Jenkins won’t be able to access it.
• Go to the Jenkins console.
• Add a new Jenkins Task.
• Leave everything blank, but add a new build script.
• In the build script enter this, replacing the two passwords and user name:

echo ‘[server admin password]’ | sudo -S fastlane fastlane-credentials add --username [itunes username] --password [itunes password]

• Now run this task. It will save the password to your keychain.
• Delete the script out of the task, save, and then delete the entire task to remove traces of the passwords.

Enable Server Side Build Versioning

Let’s allow the CI system to increment our build numbers for us. That way, when we push new code, we won’t also have to increment the build numbers manually. Note that if we try to push builds to the app store with the same build number, the upload will already exist, and hence the build will fail.

First, we need to make sure our build number versioning is set to use whole numbers. If we are using something like 1.0.0.1, this system won’t work since we will have to do a bunch of parsing to successfully increment the build number.

Now we just add these ‘lanes’ to Fastlane in our fastfile.

lane :build_number_prod do        version_number = get_version_number(:target => "Project-Prod")        last_build_dev = latest_testflight_build_number(:app_identifier => "com.Proto", :version => version_number, :initial_build_number => 0, username: "you@apple.com")

last_build_prod = latest_testflight_build_number(:app_identifier => "com.project.Staging", :version => version_number, :initial_build_number => 0, username: "you@apple.com")

last_build = last_build_dev > last_build_prod ? last_build_dev : last_build_prod

set_info_plist_value(path: "Project/Info-Staging.plist", key: "CFBundleVersion", value:  (last_build + 1).to_s)

set_info_plist_value(path: "Project/Info-Prod.plist", key: "CFBundleVersion", value:  (last_build + 1).to_s)

#tag our new build        tag = version_number.to_s + "." + (last_build + 1).to_s        if git_tag_exists tag: tag            tag = tag + "-" + Time.now.to_i.to_s        end        git_add_tag tag: tag        push_git_tags, remote: "Org/repo"    end

lane :build_number_dev do        version_number = get_version_number(:target => "Project-Dev")        last_build_dev = latest_testflight_build_number(:app_identifier => "com.Proto", :version => version_number, :initial_build_number => 0, username: "you@apple.com")

last_build_prod = latest_testflight_build_number(:app_identifier => "com.Staging", :version => version_number, :initial_build_number => 0, username: "you@company.com")

last_build = last_build_dev > last_build_prod ? last_build_dev : last_build_prod

set_info_plist_value(path: "Project/Info-Dev.plist", key: "CFBundleVersion", value:  (last_build + 1).to_s)

set_info_plist_value(path: "Project/Info-Proto.plist", key: "CFBundleVersion", value:  (last_build + 1).to_s)

#tag our new build        tag = version_number.to_s + "." + (last_build + 1).to_s        if git_tag_exists tag: tag            tag = tag + "-" + Time.now.to_i.to_s        end        add_git_tag tag: tag        push_git_tags, remote: "Org/repo"    end

These tasks will essentially check TestFlight for the latest build number that has been pushed for the current version, then increment our build accordingly. It will then add a unique tag for this build in GitHub.

Once these lanes are created, modify your Jenkins tasks and add the commands fastlane build_number_devto the dev build before fastlane release_proto

and add fastlane build_number_prodto the prod build before fastlane release_staging

Locking It Down

The server is currently a little too easy to access. We need to turn off VNC / remote desktop access by default and require logins to come into the machine via SSH instead, where the SSH key is a hardware authentication device like a YubiKey.

Restrict SSH logins and require all logins to use a Hardware Token from our YubiKey:

• SSH into the server
• Add any Yubikey public keys for devices that need access to the file ~/.ssh/authorized_keys
• Prevent people from logging in with passwords, and only with public key based authentication by editing the /etc/ssh/sshd_config file and set:

PermitRootLogin=prohibit-password
PubkeyAuthentication=yes
ChallengeResponseAuthentication=no
PasswordAuthentication=no

• Restart the machine and and login via SSH to confirm all is working well. The login should not prompt you for a password and should instead require your Yubikey.

Now log in to the server via Screen Sharing (VNC) and create these scripts to start and stop VNC. Save them in the administrators’ home folder and make sure they have executable permissions.

~/start_vnc.sh

#!/bin/bash

sudo /System/Library/CoreServices/RemoteManagement/ARDAgent.app/Contents/Resources/kickstart -activate -configure -allowAccessFor -specifiedUsers

sudo /System/Library/CoreServices/RemoteManagement/ARDAgent.app/Contents/Resources/kickstart -configure -users administrator -access -on -privs -all -setmenuextra -menuextra no -restart -agent

~/stop_vnc.sh

#!/bin/bash

sudo /System/Library/CoreServices/RemoteManagement/ARDAgent.app/Contents/Resources/kickstart -deactivate -configure -access -off

Now we’ll want to run the stop_vnc command on boot, so open launch control and add a new Global Daemon job.

• Configure the job to run your new stop vnc script by setting the executable to run to /Users/administrator/stop_vnc.sh
• Make sure ‘run on load’ is checked

Now load the job and start it. This should kill your VNC connection. To launch a new one, just log in over SSH and run the start_vnc.sh command.

Conclusion

Now, after a long setup process, we have a very nice CI server all set up and ready to go.

• You can now push development builds simply by merging into your dev branch and production & staging builds by merging dev into your master branch.
• You are sure that your test suite will run before any build is pushed and developers will get notified of build failures.
• Every build will be tagged in GitHub automatically.
• You can add a message in your app to display the exact build and commit hash of the current build.
• You can ensure that builds are only uploaded by the CI server and that only peer reviewed code makes it into production.

With these goals accomplished you may now do what you came here to do: crank out code!