Create iOS Unity build pipelines on AWS with Jenkins and EC2 Mac instances

[amurleopard]

Create iOS Unity build pipelines on AWS with Jenkins and EC2 Mac instances

Every mobile game, whether it is a multiplayer or standalone game, must build binaries. Developers of high budget, high profile AAA and AA games tend to lean towards managing a build farm, whereas independent developers (indies) may rely on local or third-party solutions. Managing compute infrastructure is a time consuming and ongoing task for many companies and developers, especially in relation to iOS builds, which require applications to be signed before submission to the App Store. Given the overhead required to run and maintain a local build environment, it rarely makes sense to maintain physical build farms for many teams.

Amazon EC2 Mac instances launched at re:Invent 2020, introduced macOS as a supported instance type for the first time. Later in 2022, Apple M1-based Mac instances were released, enabling Apple silicon in the cloud alongside the x86 architecture. Today, EC2 Mac instances are being leveraged by thousands of customers today to build and run their macOS workloads on the AWS cloud. By using EC2 Mac instances in combination with EC2 Spot instances and Jenkins CI/CD, Unity developers were able to build scalable, cost efficient, and fast iOS pipelines for building, testing, signing, and publishing to the App Store.

In this blog post, we discover how to build a scalable and cost-efficient iOS build pipeline on AWS that can be deployed within hours.

Overview of solution

This solution is possible due to Unity`s ability to split a build process into two parts:

• Xcode Project preparation: a first phase involving processing compute-intensive images, videos, music, and compiling additional assets.

• Xcode Build, packaging, and signature: the second and final phase of building an iOS app which must be done using macOS, an EC2 Mac instance in our solution.

The high-level architecture looks like the following:

We will build upon these concepts and will use EC2 Spot instances for the compute-heavy phase of asset compilation in conjunction with the EC2 Autoscaling feature. We will also use an EC2 Mac instance to complete the second, final Xcode build step. For the whole pipeline, orchestration is done with Jenkins CI/CD. We will also use the ability of the Unity engine to be Dockerized to implement scalability and flexibility.

Services and dependencies used in our solution are shown on the following diagram:

Key pieces

In this blog I will not explain the solution step by step; instead, I will emphasize key pieces of the solution, which you can setup on your own. You can follow the whole procedure by going through the workshop here, a repository with code and a sample project can be found here.

Networking template

To setup the whole solution, I have created the following CloudFormation template. It prepares the environment by creating a VPC, subnets, a Jenkins Manager installation and an EC2 bastion host. To deploy the stack in us-east-2 (Ohio) region, click the button:

The stack incorporates VPC with two availability zones (AZs), and with two private and two public subnets. To follow best practices, we will place Jenkins Manager and execution nodes within the private subnet. The stack also sets up a bastion host in order to access resources in the private subnet.

Internal resources will use NAT gateway to reach the internet in two availability zones to reduce inter-AZ traffic costs.

Unity container and build server

The demonstration repository includes a sample Unity project to build. The Unity engine can be containerized—the Linux-based solution uses Docker to pull a specific Unity version from a repository, set up a license, and build the Xcode project. Since the process is compute-heavy, the types of instance used in this case is a C5.4xlarge instance, which are compute-optimized instances.

Unity part that needs to be legally validated

In order to build a Unity project and create an Xcode project in a production environment, you will need a license. For that, Unity provides several options: in our example, we will use “Unity Build Server” licenses. (You can read more here.) Here are some AWS cloud-specific steps to implement in order to ensure the Build Server is setup in resilient way.

1. Build server installations maintain a number of “seats” or licenses which are loaned to workers and released after the build process is done; however, the build server is bound to number of cores on the instance, as well as the network interface’s MAC address. This means that once you “bind” seats to an instance of Build Server and you need to launch it on a new instance, if the number of cores or network interface do not match, you will lose your seats. In order to avoid such a problem, you’ll need to provision an additional ENI, attach it to the instance, and use its MAC address to bind Build Server to that ENI’s address. Once that is done, you can launch new instances from AMIs with Build server, attach the ENI, and your licenses are secure and assigned.

2. To ensure that the process is automated, set up an autoscaling group (ASG) with the Build Server AMI— a single instance—and set it to attach the specific ENI to any new instance each time it is launched.

3. For a multi-AZ setup, you need two separate build servers with their own distinct seat groups.

4. You can expose Build Server via AWS Service Endpoint by setting up a Network Load Balancer in front of the Build Server instance’s ASG and providing a link to the endpoint to your consumers.

5. Unity editor within workers need to be configured in order to connect to Build server. This can be achieved by providing configuration file each time a Docker container is started via Jenkinsfile. Configuration might look like the following:

{
    "licensingServiceBaseUrl": "\{http(s)://\{server dns name}}",
    "enableEntitlementLicensing": true,
    "enableFloatingApi": true,
    "clientConnectTimeoutSec": 5,
    "clientHandshakeTimeoutSec": 10
}

and the file should be saved as '/usr/share/unity3d/config/services-config.json' of the container. The License server DNS name can be stored in Secrets Manager.

End of Unity Part

EC2 Mac and Secrets Manager

In order to build and sign iOS applications, we need an EC2 Mac instance. To launch an Amazon EC2 Mac instance, you must first allocate a dedicated host in Amazon EC2. A dedicated host is a physical server that is wholly allocated for your use. Please keep in mind that currently EC2 Mac dedicated hosts cannot be released earlier than 24 hours after being launched. There is no restriction on how often you can launch an EC2 Mac instance on a dedicated host, however. Depending on the architecture (mac1 for x86, mac2 for Apple silicon), you will need different dedicated hosts.

Once the instance is launched, you will need to enable VNC to connect to it via the graphical user interface (GUI). That process is described here.

EC2 Mac instances use most of the same tools that you have already been using with other EC2 instances. These instances live in the VPC, support IAM, support user data, and can boot from EBS volumes, so you can create golden AMIs with all the required software installed; for example, Xcode. EC2 Mac instances can be configured by Systems Manager, for example to install patches. It is also integrated with Cloudwatch for logs. Basically, treat it the same as any EC2 instance that you need to configure and then use as a part of the unattended CI/CD pipeline to run your builds.

In our case, the instance also needs to have Java and Xcode installed, as well as an IAM role to call to other services, such as Amazon Secrets Manager.

Build signing and AWS Secrets Manager

To sign the build for the App Store, we need to have a signing certificate and a provisioning profile. It is possible to also generate an application package that can later be signed by another certificate. This is common for companies that want to separate test environments’ certificates from their production environment.

The development certificate, its password, and the provisioning profile can be stored in AWS Secrets Manager, a great way to save your secrets and have secure access to resources.

To set up a temporary keychain with signature files, you can use either Fastlane or create the temporary keychain manually.

Linux worker instances

In order for Linux workers to be able to run build instructions from Jenkins, several features have to be enabled for Linux worker instances:

• Docker engine and Java must be installed

• The attached EBS storage has to be large enough to incorporate the docker images

• The instance’s security group should allow communication from Jenkins Manager node port 22 (Manager node uses ssh to connect to the worker and setup agent)

Jenkins setup

Jenkins uses a manager node to orchestrate builds. Build jobs will run either on EC2 Mac or on EC2 Spot instances managed by the EC2 Fleet plugin. Jenkins manager stores configurations, provides a user interface, and orchestrates build jobs. It can also store build artifacts, however I find it’s better to use Amazon S3 as it provides virtually unlimited storage, and can easily be used from within Jenkins pipeline.

In order to orchestrate Linux build instances, Jenkins needs the following plugins:

The Docker and Docker pipeline plugins allow us to run docker pipeline steps. These are required to launch Unity containers and run the first part of the build within.

The EC2 Fleet plugin allows for simple integration with Autoscaling groups, significantly reducing the overhead of launching new instances, starting a Jenkins agent, and running a build. The EC2 Fleet plugin is decoupled from the Auto Scaling group, which allows for full control of the kinds of instances to be launched. This way, we can utilize the full power of Spot instances as temporary workers, thus significantly reducing costs.

In order to orchestrate Mac instances, the instance has to have port 22 enabled as well as Java installed. You can add the instance manually, or launch it via CloudFormation or Terraform and use the self-registration method described here.

Please note that currently dynamic provisioning of EC2 Mac instances via Auto Scaling groups is not possible due to the minimum 24 hour reservation time for which an EC2 Mac dedicated host has to be reserved. However, we can use several Jenkins executors on a single EC2 Mac instance:

Also, labels can be used in order to separate kinds of workers within a pipeline. Note that on a screenshot above I use label “mac”.

Build pipeline and running a build

Every Jenkins pipeline can be described using a Jenkinsfile file. It is a YAML-formatted document which describes all the steps for the pipeline. You can read more here. I already have such a file stored in my repository. The file contents are following:

pipeline {
    agent none

    environment {
        UNITY_PROJECT_DIR='UnityProjectSample'
        IMAGE='unityci/editor'
        UNITY_VERSION='2021.3.6f1-ios-1.0'
        // Build parameters
        UNITY_LICENSE_FILE='UNITY_LICENSE_FILE'
        PROVISIONING_PROFILE_NAME='UnityBuildSample-profile'
        // secret from Secrets Manager
        TEAM_ID_KEY='TEAM_ID'
        LICENSE_SERVER_ENDPOINT='LICENSE_SERVER_ENDPOINT'
        SIGNING_CERT='SIGNING_CERT'
        SIGNING_CERT_PRIV_KEY='SIGNING_CERT_PRIV_KEY'
        SIGNING_CERT_PRIV_KEY_PASSPHRASE='SIGNING_CERT_PRIV_KEY_PASSPHRASE'
        APPLE_WWDR_CERT='APPLE_WWDR_CERT'
        PROVISIONING_PROFILE='PROVISIONING_PROFILE'
    }

    stages {
        stage('build Unity project on spot') {
            agent {
                docker {
                    image 'unityci/editor:2021.3.6f1-ios-1.0'
                    args '-u root:root'
                }
            }
            steps {
                // install stuff for Unity, build xcode project, archive the result
                sh '''
                    printenv
                    echo "===Installing stuff for unity"
                    apt-get update
                    apt-get install -y curl unzip zip
                    curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o
                    "awscliv2.zip"
                    unzip -o awscliv2.zip
                    ./aws/install
                    apt-get install sudo
                    # Following section can be uncommented if Unity Build server is used
                    # just to push it through
                    # sudo mkdir -p /usr/share/unity3d/config/
                    # endpoint=`aws secretsmanager get-secret-value \
                    # --secret-id $LICENSE_SERVER_ENDPOINT --output text --query
                    # 'SecretString' | cut -d '"' -f4`
                    # configfile='\{
                    # "licensingServiceBaseUrl": "'$endpoint'",
                    # "enableEntitlementLicensing": true,
                    # "enableFloatingApi": true,
                    # "clientConnectTimeoutSec": 5,
                    # "clientHandshakeTimeoutSec": 10
                    # }'
                    # Copying Unity .ulf license file from S3 to container
                    # aws s3 cp "s3://$\{S3_BUCKET}/Unity_2021.3.6f1-ios-1.0.ulf"
                    # "/root/.local/share/unity3d/Unity/Unity_lic.ulf"
                    # mkdir -p "/root/.local/share/unity3d/Unity"
                    # aws secretsmanager get-secret-value --secret-id $UNITY_LICENSE_FILE
                    # --output text --query SecretBinary |
                    # base64 -d > "/root/.local/share/unity3d/Unity/Unity_lic.ulf"
                    # echo "===Building Xcode project"
                    # We also pull in additional repository with actual Unity Project.
                    # We have several configuration files for our build configuration
                    # You can find those in UnityProjectSample folder
                    rm nodulus -rf
                    git clone https://github.com/Hyperparticle/nodulus.git
                    cp -nR nodulus/* UnityProjectSample/
                    cd $UNITY_PROJECT_DIR
                    mkdir -p ./iOSProj
                    mkdir -p ./Build/iosBuild
                    xvfb-run --auto-servernum --server-args='-screen 0 640x480x24' \
                        /opt/unity/Editor/Unity \
                        -quit \
                        -batchmode \
                        -nographics \
                        -executeMethod ExportTool.ExportXcodeProject \
                        -buildTarget iOS \
                        -customBuildTarget iOS \
                        -customBuildName iosBuild \
                        -customBuildPath ./Build/iosBuild \
                        -logFile /dev/stdout
                    echo "===Zipping Xcode project"
                    zip -r iOSProj iOSProj
                    '''
                    // pick up archive xcode project
                    dir("$\{env.UNITY_PROJECT_DIR}") {
                        stash includes: 'iOSProj.zip', name: 'xcode-project'
                    }
                }
                post {
                    always {
                        sh "chmod -R 777 ."
                    }
                }
            }
            stage('build and sign iOS app on mac')\{
                // we don't need the source code for this stage
                options {
                    skipDefaultCheckout()
                }
                agent {
                    label "mac"
                }
                environment {
                    HOME_FOLDER='/Users/jenkins'
                    PROJECT_FOLDER='iOSProj'
                }
                steps {
                    unstash 'xcode-project'
                    sh '''
                    pwd
                    ls -l
                    # Remove old project and unpack a new one
                    rm -rf $\{PROJECT_FOLDER}
                    unzip iOSProj.zip
                    '''

                    // create export options file
                    writeFile file: "$\{env.PROJECT_FOLDER}/ExportOptions.plist", text: """
                    <?xml version="1.0" encoding="utf-8"?>
                    <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN"
                    "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
                    <plist version="1.0">
                        <dict>
                            <key>signingStyle</key>
                            <string>manual</string>
                        </dict>
                    </plist>
                """

                sh '''
                PATH=$PATH:/usr/local/bin
                cd $\{PROJECT_FOLDER}
                # Update project settings
                # sed -i "" 's|^#!/bin/sh|#!/bin/bash|' MapFileParser.sh
                # extra backslash for groovy
                TEAM_ID=`aws secretsmanager get-secret-value \
                    --secret-id $TEAM_ID_KEY --output text --query 'SecretString' | cut -d '"' -f4`
                # extra backslash for groovy
                sed -i "" "s/DEVELOPMENT_TEAM = \\"\\"/DEVELOPMENT_TEAM = $TEAM_ID/g" Unity-iPhone.xcodeproj/project.pbxproj
                #############################################
                # setup certificates in a temporary keychain
                #############################################

                echo "===Setting up a temporary keychain"
                pwd
                # Unique keychain ID
                MY_KEYCHAIN="temp.keychain.`uuidgen`"
                MY_KEYCHAIN_PASSWORD="secret"
                security create-keychain -p "$MY_KEYCHAIN_PASSWORD" "$MY_KEYCHAIN"
                # Append the temporary keychain to the user search list
                # double backslash for groovy
                security list-keychains -d user -s "$MY_KEYCHAIN" $(security list-keychains -d user | sed s/\\"//g)
                # Output user keychain search list for debug
                security list-keychains -d user
                # Disable lock timeout (set to "no timeout")
                security set-keychain-settings "$MY_KEYCHAIN"
                # Unlock keychain
                security unlock-keychain -p "$MY_KEYCHAIN_PASSWORD" "$MY_KEYCHAIN"
                echo "===Importing certs"
                # Import certs to a keychain; bash process substitution doesn't work with security for some reason
                aws secretsmanager get-secret-value --secret-id $SIGNING_CERT --output text --query SecretBinary | base64 -d -o /tmp/cert && security -v import /tmp/cert -k "$MY_KEYCHAIN" -T "/usr/bin/codesign"
                rm /tmp/cert
                PASSPHRASE=`aws secretsmanager get-secret-value \
                    --secret-id $SIGNING_CERT_PRIV_KEY_PASSPHRASE --output text --query 'SecretString' | cut -d '"' -f4`
                aws secretsmanager get-secret-value --secret-id $SIGNING_CERT_PRIV_KEY --output text --query SecretBinary |
base64 -d -o /tmp/priv.p12 &&
                security -v import /tmp/priv.p12 -k "$MY_KEYCHAIN" -P "$PASSPHRASE" -t priv -T "/usr/bin/codesign"
                rm /tmp/priv.p12; PASSPHRASE=''
                #aws secretsmanager get-secret-value --secret-id $APPLE_WWDR_CERT --output text --query SecretBinary | \
                # base64 -d -o /tmp/cert &&
                # security -v import /tmp/cert -k "$MY_KEYCHAIN"
                # rm /tmp/cert
                # Dump keychain for debug
                security dump-keychain "$MY_KEYCHAIN"
                # Set partition list (ACL) for a key
                security set-key-partition-list -S apple-tool:,apple:,codesign: -s -k $MY_KEYCHAIN_PASSWORD $MY_KEYCHAIN
                # Get signing identity for xcodebuild command
                security find-identity -v -p codesigning $MY_KEYCHAIN
                # double backslash for groovy
                CODE_SIGN_IDENTITY=`security find-identity -v -p codesigning $MY_KEYCHAIN | awk '/ *1\\)/ \{print $2}'`
                echo code signing identity is $CODE_SIGN_IDENTITY
                security default-keychain -s $MY_KEYCHAIN
                #############################################
                # setup provisioning profile
                #############################################
                echo ===setting up a provisioning profile
                pwd
                # # if the provisioning profile already exists, don't overwrite
                # PROV_PROFILE_FILENAME="$\{HOME}/Library/MobileDevice/Provisioning Profiles/$\{PROVISIONING_PROFILE_NAME}.mobileprovision"
                # if [ ! -f "$PROV_PROFILE_FILENAME" ]; then
                # aws secretsmanager get-secret-value --secret-id $PROVISIONING_PROFILE --output text --query SecretBinary | \
                # base64 -d -o "$\{PROV_PROFILE_FILENAME}"
                # fi
                # # lock, since multiple jobs can use the same provisioning profile
                # if [ -f "$\{PROV_PROFILE_FILENAME}.lock" ]; then
                # n=`cat "$\{PROV_PROFILE_FILENAME}.lock"`
                # n=$((n+1))
                # else
                # n=1
                # fi
                # echo $n > "$\{PROV_PROFILE_FILENAME}.lock"
                #############################################
                # Build
                #############################################
                echo ===Building
                pwd
                # xcodebuild -scheme Unity-iPhone -sdk iphoneos -configuration AppStoreDistribution archive -archivePath "$PWD/build/Unity-iPhone.xcarchive" CODE_SIGN_STYLE="Manual" PROVISIONING_PROFILE_SPECIFIER_APP="$PROVISIONING_PROFILE_NAME" CODE_SIGN_IDENTITY=$CODE_SIGN_IDENTITY OTHER_CODE_SIGN_FLAGS="--keychain=$MY_KEYCHAIN" -UseModernBuildSystem=0
                xcodebuild -scheme Unity-iPhone -sdk iphoneos -configuration AppStoreDistribution archive -archivePath "$PWD/build/Unity-iPhone.xcarchive" CODE_SIGN_STYLE="Manual" CODE_SIGN_IDENTITY=$CODE_SIGN_IDENTITY OTHER_CODE_SIGN_FLAGS="--keychain=$MY_KEYCHAIN" -UseModernBuildSystem=0 CODE_SIGNING_REQUIRED=NO CODE_SIGNING_ALLOWED=NO
                # Generate ipa
                echo ===Exporting ipa
                pwd
                # xcodebuild -exportArchive -archivePath "$PWD/build/Unity-iPhone.xcarchive" -exportOptionsPlist ExportOptions.plist -exportPath "$PWD/build"
                #############################################
                # Upload
                #############################################
                # Upload to S3
                # /usr/local/bin/aws s3 cp ./build/*.ipa s3://$\{S3_BUCKET}/
                #############################################
                # Cleanup
                #############################################
                # Delete keychain - should be moved to a post step, but this would require a global variable or smth
                security delete-keychain "$MY_KEYCHAIN"
                # Delete a provisioning profile if no jobs use it anymore
                n=0
                if [ -f "$\{PROV_PROFILE_FILENAME}.lock" ]; then
                n=`cat "$\{PROV_PROFILE_FILENAME}.lock"`
                n=$((n-1))
                echo $n > "$\{PROV_PROFILE_FILENAME}.lock"
                fi
                if [ "$n" -le "0" ]; then
                rm -f "$\{PROV_PROFILE_FILENAME}"
                rm -f "$\{PROV_PROFILE_FILENAME}.lock"
                fi
                '''
            }
            post {
                always {
                    sh '''
                    #############################################
                    # cleanup
                    #############################################
                    zip -r iOSProj/build/Unity-iPhone.zip iOSProj/build/Unity-iPhone.xcarchive
                    rm -rf iOSProj/build/Unity-iPhone.xcarchive
                    '''
                    archiveArtifacts artifacts: '**/Unity-iPhone.zip', onlyIfSuccessful: true, caseSensitive: false
                }
            }
        }
    }
    post {
        success {
            echo 'Success ^_^'
        }
        failure {
            echo 'Failed :('
        }
    }
}

The key pieces of the file:

• agent – describes which instances the build should run on. In my case label “mac” is used to separate EC2 Mac instances from Spot instances

• environment – describes environment variables used by the build

• stage(‘Name’) – describes separate stage

• docker – describes docker image that is pulled and is later used to run Unity build in

• xvfb-run /opt/unity/Editor/Unity – runs unity editor in headless mode within a container

• security create-keychain – creates private keychain to store secrets like signing certificate. The keychain is later deleted.

• To pass Artifacts between stages, Jenkins` stash function is used

And once done, the basic pipeline should look like the following:

Final architecture

Once the all the key pieces mentioned in this post are assembled, the final picture is as following:

Also refer to the full diagram with additional details explained. For this diagram, we assume that code and Docker images are located within the AWS account as well, to reduce data transfer charges and improve latency.

Main solution benefits and costs

There are several factors that are important to consider when building this solution:

1. Unity version control.
   This solution allows for flexible control of which Unity version is used for the build. By simply using tagging for Docker images, the pipeline can run different versions of Unity without changing the configuration of the pipeline in general.
   

2. Flexible Xcode version control
   By utilizing AMI images for EC2 Mac, it is possible to build a library of iOS and preinstalled Xcode versions to quickly launch on EC2 Mac hosts. This process can be further automated by using tools like Packer or EC2 Image Builder to create AMIs for different versions of environments.
   

3. Cost benefits when using Spot instances and using less Mac instances
   Since this solution implements a split-build approach, it allows us to take 30 to 70% of the computing time from EC2 Mac instances. This allows for better parallelization of builds and reduces time required by the EC2 Mac instance to process the code, resulting in much faster builds in general. Spot instances are also used instead of on-demand instances. Thus, depending on a build, the approach can reduce the cost by around 30-40%.
   

4. It is also possible to setup “layers” of EC2 Mac instances by using several Macs for development and production builds separately, this allows for secure separation of environments.

5. Automatization of the pipeline via versioned Jenkinsfiles and Amazon CloudFormation templates allows for consistent and controllable approach to build environments.

Conclusion

This post explains key pieces of the of cost-effective Unity build pipeline. It utilizes a mechanism of separation of the build to Linux instances and EC2 Mac instances. The compute-heavy part can be done on cost-efficient Spot instances, which reduces load to Mac instances and allows for more parallel builds at a time. We went through Unity and iOS build environments, key elements, licensing, workers and overall CI/CD process automation with Jenkins.

This approach has already been adopted by our Game tech clients: https://aws.amazon.com/ec2/instance-types/mac/customers/ - Riot Games, Pokemon Company and others. The pipelines speed being improved up to 400% (Pokemon Company), Improving management time (Riot games) and reduced complexity (Jamcity).

We will be speaking more on this topic at the Jenkins Contributor Summit, on September 27 at DevOps World in Orlando, Florida. Hope to see you there!

Suggested tags: Amazon Game Development, Amazon EC2 Mac, AWS for Games, AWS game development, AWS Game Tech, Unity

https://www.jenkins.io/blog/2022/09/14/create-iOS-unity-build-pipelines-on-AWS-with-Jenkins-and-EC2-Mac-instances/