Azure DevOps

Terminology

STOP

Please be sure that your laptop setup is current. At a minimum, please do the following before proceeding:

brew update
brew upgrade

Though probably not required, sometimes it is helpful to update the gcloud-cli cask. Before doing so, use the Privileges tool to request administrator privileges on your laptop. The step below for updating the gcloud-cli cask may fail without doing this first. If prompted for your password (or Touch ID), provide that accordingly.

brew upgrade --cask gcloud-cli

Then, ensure that your grip-assets repository clone is current. This is vital for ensuring that the correct set of builds are created with accurate initiation details.

Getting Started

Before starting any builds, you must know what build number to use. We have no automated way to track the build number, and the way we utilize it can be confusing. You can infer a likely build number in one of three ways:

1. Look at the Azure pipeline for App Store Connect uploads and filter for the most recent successful qa-uat builds. Filtering for “Banno Dev Bank” is likely to give the best information.
2. Install Apple’s TestFlight app and get access to our internal TestFlight builds. Similar to the above option, you would load one of the internal apps, such as Banno Dev Bank, and check what is in parentheses after the app version. (You ought to be able to give yourself access to TestFlight apps, but you can also ask for help in the #team-ios channel in Slack.)
3. Ask for the current build number in the #team-ios channel in Slack. If unsure, this is always the best option.

One way or another, checking Banno Dev Bank builds currently provides what is likely to be the most accurate information.

IMPORTANT: The azdo-build and azdo-deploy commands can override the build number value but not the app version value. These commands read CFBundleShortVersionString from Banno/Info.plist using the current state of the local repository clone. The build number can come from the command line, but if it is not specified, then the commands use the value of CFBundleVersion, also from Banno/Info.plist. What this means is that it is important to run the commands from the same branch that will be built on Azure DevOps. An attempt has been made to strike a balance between flexibility and ease of use. Not requiring the app version and build number makes the command easier to run, and it reduces the potential for mistakes to some degree. Being able to override the build number in this way provides a means to make builds for TestFlight or production without having to go through the PR process. Every so often, the release engineers have a need to do this as part of a production release. If a problem is identified with an IPA that has been uploaded to App Store Connect but not yet submitted for review, another IPA can be built with an incremented build number to supersede the one with a problem. All of that can happen without a PR, release approval, etc.

Institution Groups

The grip-assets repository defines seven groups that contain institution app details needed for building the iOS and Android apps:

Build

With an understating of the above, we can now start builds. Google Cloud Secret Manager has the secrets that are needed for our tools and for Fastlane to authenticate with App Store Connect. Basically, what we do locally to access Google Cloud Secret Manager is required because Azure DevOps has no access to Google Cloud Secret Manager.

NOTE: These instructions are for making builds from the qa-uat group in the grip-assets repository.

Getting Started

Before doing anything else, the following must be done:

1. Pull the latest changes to your clone of the grip-assets repository.
2. Pull the latest changes to your clone of the Banno environments repository.
3. Open a Terminal (or iTerm) shell. All commands described from here to the end must be run from this shell.

From the directory where you saved your azdo-config.sh file, run the following command:

source ./azdo-config.sh

Test Google Cloud Secret Manager Access

As it may have been a while since you last accessed Google Cloud Secret Manager, the following steps provide a means to check on that. This part is completely optional and uses a different GCP access interface than the azdo-build and azdo-deploy commands.

To begin, run the following command:

gcloud auth login

This will open a web browser and guide you through Google Cloud authentication. Be sure to use your jackhenry.com Google ID.

The access to Google Cloud secrets can be confirmed by running the following from the root directory of the banno-ios repository clone:

./Build/apple-creds 5db40b7a-ee76-42b3-acb9-eb7045011c2a

If that prints out the username and password for one of our Apple IDs (such as appledev@banno.com), then Google Cloud secrets are accessible. If an error is reported where the solution is unclear, then please ask for help in the #team-ios Slack channel.

NOTE: The gcloud tool used by our apple-creds script may report issues with the version of Python that is installed. Depending on when you installed the gcloud-cli cask using Homebrew, it is possible that the local state is out of date. Running the following may help (and may even be recommended by the output from the gcloud command):

gcloud components reinstall

Starting Builds

Next, change directories to the root of the banno-ios repository clone. Then, run the following command:

./Build/azdo-build \
--build-number <build-number> \
--branch <branch-to-build> \
--group qa-uat

If you prefer the short version of command line arguments, the following is equivalent:

./Build/azdo-build \
-b <build-number> \
-B <branch-to-build> \
-g qa-uat

The value for <build-number> is based on what was determined using the above recommendations in the “Getting Started” section.

The value for <branch> is based on which branch should be built. This depends on whether a code freeze is in effect. If a code freeze is in effect, then the current release branch should be built. Otherwise, the develop branch should be built.

For example, the command to make build 1224 from develop would be the following:

./Build/azdo-build \
--build-number 1224 \
--branch develop \
--group qa-uat

After running this command, check the Banno Mobile iOS Build pipeline on Azure DevOps to confirm that all builds completed successfully. It takes approximately 30–45 minutes for each build to complete.

TIP: If you would like to save yourself some clicks in a web browser, add the --verbose 1 command line argument to the above. Then, the azdo-build utility will print the URL for the build that was started. It can look like this:

> ./Build/azdo-build --build-number 1224 --branch develop --group qa-uat --verbose 1
Starting build for Banno Dev Bank (5db40b7a-ee76-42b3-acb9-eb7045011c2a) ... success!
Build URL: https://dev.azure.com/JHABanno/ae823cfe-9afc-4995-be74-eaa84e340a6b/_build/results?buildId=166197

TIP: The --verbose command line parameter avlue can be changed to increase the amount of information log output:

./Build/azdo-build -v 2
./Build/azdo-build -v 3
./Build/azdo-build -v 4

If one or more builds fail, look at the Azure DevOps pipeline logs to try to determine what went wrong. Most of the time, a failure is due to a transient issue, and retrying will be sufficient. One or more specific institution IDs can be retried by adding a space-separated list at the end of the azdo-build command line:

./Build/azdo-build \
--build-number <build-number> \
--branch <branch-to-build> \
--group qa-uat <institution-id1> <institution-id2> ...

The default timeout is 75 minutes. If builds take approximately 75 minutes to fail, this may mean that a 2FA session for the Apple ID stored in Google Cloud Secret Manager has expired. In that case, the build will be stuck waiting for manual entry of a 2FA code. A key indicator is that the log output on Azure DevOps will show that the build is stopped at this message:

Please enter the 6 digit code:

You will need to reach out in the #org-mobile-deployment Slack channel so that one of the release engineers can refresh the 2FA sessions in Google Cloud Secret Manager.

TIP: To get more information about the azdo-build command, run the following command:

./Build/azdo-build --help

TIP: If builds fail to start, verify the following:

• The branch name is correct.
• Opening a GCP tunnel succeeded.
• Your Azure DevOps personal access token has not expired.

The azdo-build utility should report meaningful output when a build fails to restart. To get more log output, run the command again with the -v 2 (or even -v 4) command line option as described in an earlier tip.

Deploy

Once all the builds complete, the next step is to upload (deploy) the IPAs to App Store Connect. From the banno-ios directory, run the following command:

./Build/azdo-deploy \
--build-number <build-number> \
--branch <branch-that-was-built> \
--group qa-uat

This is the same command that was run above to start IPA builds except that build has been replaced by deploy.

TIP: If the last command run in the terminal was using azdo-build to start IPA builds, then the deployment can be started by running this command:

^build^deploy^

Once the azdo-deploy command completes, check the Banno Mobile iOS Deploy pipeline on Azure DevOps to confirm that all deployments to App Store Connect completed. The deployment step usually takes less than ten minutes to complete. Once an IPA is uploaded to App Store Connect, everything else is automated on Apple’s end.

TIP: If you would like to save yourself some clicks in a web browser, add the --verbose 1 command line argument to the above (just the same as with azdo-build). Then, the azdo-deploy utility will print the URL for the build that was started. It can look like this:

> ./Build/azdo-deploy --build-number 1224 --branch develop --group qa-uat --verbose 1
Starting deploy for Banno Dev Bank (5db40b7a-ee76-42b3-acb9-eb7045011c2a) ... success!
Build URL: https://dev.azure.com/JHABanno/ae823cfe-9afc-4995-be74-eaa84e340a6b/_build/results?buildId=166199

If an upload to App Store Connect fails, look at the Azure DevOps logs to try to determine what went wrong. Most of the time, a failure is due to a transient issue with App Store Connect, and retrying will be sufficient. One or more specific institution IDs can be retried by adding a space-separated list at the end of the azdo-deploy command line:

./Build/azdo-deploy \
--build-number <build-number> \
--branch <branch-that-was-build> \
--group qa-uat <institution-id1> <institution-id2> ...

After all the IPAs have been uploaded, then it is helpful to let people know via Slack about the forthcoming availability. Post the following message in the #org-mobile-release-testing channel in Slack:

iOS TestFlight build <app-version> (<build-number>) will be available soon.

Then, post the following message in the #team-ios channel:

@testers Today's TestFlight builds (XYZ) were made from `develop`, commit `1e054223e997a0437c9cf86df10cf1d6e0b5e41f`

That must be adjusted accordingly to use the real build number, branch name, and commit SHA. You can get the commit SHA from Git or from Azure DevOps by loading the details for one of the completed IPA builds.

TIP: To get more information about the azdo-deploy command, run the following command:

./Build/azdo-deploy --help

Build Number

The build number complexity is something that we put on ourselves. Strictly speaking, we could reset the CFBundleVersion value to 1 every time we change the CFBundleShortVersionString value. For unknown reasons, however, we have chosen to have a monotonically increasing integer value for CFBundleVersion that (currently) has to be tracked manually. Things get particularly confusing when a hot-fix release or a version-only release is needed. For example, here are some actual full version numbers that happened chronologically:

• 3.5.0 (1210)
• 3.4.1 (1211)
• 3.5.0 (1212)
• 3.5.0 (1213)
• 3.4.2 (1214)
• 3.5.0 (1215)

As noted above, azdo-build and azdo-deploy get the value for CFBundleVersion from the --build-number command line option. If that command line option is not specified, then both commands use what is currently in Banno/Info.plist This affords convenience and flexibility when one of the release engineers needs to replace a build that was uploaded to App Store Connect but not submitted for App Store review.

Having to specify the build number manually is also a way that things can go wrong if a mistake is made when choosing the build number. Most of the time, nothing bad will happen. A build number may get “burned” unnecessarily. The azdo-build utility also checks Artifactory and App Store Connect to avoid trying to reuse a build number, but it does so by creating an Artifactory URL that combines the current app version with the build number. Even this underscores how CFBundleVersion is really coupled to CFBundleShortVersionString and not something that has to keep increasing across app versions.

Sometimes, it may be necessary to replace one or more builds on Artifactory that has not yet been uploaded to App Store Connect. This can be achieved by adding the --force option to the azdo-build command line:

./Build/azdo-build \
--build-number <build-number> \
--branch <branch-to-build> \
--group qa-uat \
--force <institution-id1> <institution-id2> ...

TIP: Just because an IPA has been uploaded to Artifactory does not mean that the build number has to be incremented. Until an IPA is uploaded to App Store Connect, nothing is fixed in place. Apple will, of course, validate details of an uploaded IPA and reject those that contain invalid metadata (among many other automatic checks).

Local

IMPORTANT: Running a build locally will execute apply_institution.py, thus changing details about the project file, entitlements, etc. It is recommended that testing be done with a Git worktree or a separate repository clone. While the changes can be undone easily, it’s still a bit risky to apply institution details to a repository that is used for day-to-day development.

MORE IMPORTANT: Local keychain modifications have to occur so that Fastlane match can do the code signing work. In the past, this has been known to alter the keychain setup so that the Login keychain is no longer the default. Changes have been made to avoid this, but it may be worth doing the following before getting started:

cp ~/Library/Keychains/login.keychain-db ~/Library/Keychains/login.keychain-db.bak

This creates a backup of the Login keychain database should anything go awry. Being able to restore the Login keychain is a lot easier than having to get a Jack Henry laptop back into good graces with the company portal.

Should it be necessary to restore the login keychain, the following will do the trick:

mv ~/Library/Keychains/login.keychain-db.bak ~/Library/Keychains/login.keychain-db
security default-keychain -s ~/Library/Keychains/login.keychain-db

Setup

Creating an IPA requires the use of Fastlane. Presently, the instructions for getting everything installed up are in the mobile-deployment repository with all the Fastlane-based tooling.

Build

Running a local IPA build requires all the same setup as running an IPA build on Azure DevOps. The difference, of course, is that the work will happen on your local machine. All the log output and both products (the IPA and the dSYM archive) will be available afterward.

The banno-ios repository has a command called build-ipa in the Build folder. This command replicates what is in the azure-pipelines.yml pipeline configuration (also known as the Banno Mobile iOS Build pipeline) up to the point of uploading the IPA and dSYM archive to Artifactory. The intent is twofold:

1. Make it easier to test build pipeline changes.
2. Make it easier to diagnose build failures.

To get started, run the following from the banno-ios directory:

./Build/build_ipa.py --help

The output shows the command line options with and explanation for each. Of particular interest is the --disable-xcpretty option. Use of xcpretty on Azure DevOps is often what makes diagnosing build failures so difficult. With the build-ipa command, xcpretty will be used by default, thus replicating what is done on Azure DevOps. Adding the --disable-xcpretty option to the command line will make all the build details available, often times providing the exact explanation for a failure.

Next, get the password that is needed for SSH access to our organization GitHub repositories. This not-so-secret password can be seen in the variables for the Banno Mobile iOS Build pipeline on Azure DevOps. Another option is to ask in the #team-ios channel in Slack.

Next, choose the institution ID to build. Here are some common ones with the associated group from grip-assets:

Finally, run the following:

./Build/build_ipa.py --group <group> --github-password <ssh-password-from-above> --institution-id <unique-id>

The value for <group> depends on which institution ID has been chosen. Add --disable-xcpretty to the above to see more detailed output.

If the build runs to completion, there will be two output files:

1. A file ending in .ipa: The IPA that was built.
2. A file ending in .dSYM.zip: The dSYM archive that was built.

Both files can be unzipped to examine the contents. The .ipa file has all the interesting stuff, of course.

Whether the IPA creation succeeds or not, a file called ipa.log will also be produced. This file will contain the full log output with ANSI escape sequences stripped for readability. Having such a file is a nice alternative to running build_ipa.py in a script shell where the ANSI escape sequences tend to get in the way of reading the log output.

Clean-up

To undo changes made to the local repository clone, run the following from the root directory of the banno-ios repository:

git checkout -- .

It is also safe to remove ipa.log, the produced .ipa file, and the produced .dSYM.zip file.