Salesforce Lifecycle and Tooling: CircleCI and Salesforce DX02 Feb 2018
This is the first in a series looking at setting up a Salesforce project with a full suite of modern software engineering tools and services.
As a test-bed, I’m using a project called septaTrains, a transit tracker application for Lightning. The application is built with Lightning components and uses Apex to call various REST APIs to display information about commuter trains in the Philadelphia area.
Set Up Infrastructure
If you’re not working with an existing Salesforce DX project, start by setting up the Dev Hub, project structure, and Git repository.
Every paid Salesforce instance can have the Dev Hub functionality activated. If you don’t have access to a paid Salesforce instance or cannot activate Dev Hub, create a new Dev Hub trial account. Trial accounts are good for 30 days, after which they’re deleted. Our CI setup is largely resilient to Dev Hub changes, with minor setup needed but no commits.
Create an SFDX project and connect to your Dev Hub:
$ sfdx force:project:create --projectname proj $ sfdx force:auth:web:login -a DevHub -d
If your Dev Hub isn’t on
login.salesforce.com, change the parameter “sfdcLoginUrl” in the
sfdx-project.json configuration file appropriately.
Set up your Git repository:
$ cd proj $ git init .
Create your Git repository on GitHub and copy its URL. Configure the local repository to communicate with GitHub and commit the project structure:
$ git remote add origin <GITHUB_URL> $ git add force-app && git commit -m "Initial commit"
You may also wish to add a
.gitignore file to ensure that support files, like the
.sfdx directory, aren’t inadvertently added to Git.
Set Up JWT Authentication
If you’re working with an existing Salesforce DX/Git project, start here.
CircleCI will need to talk to the Dev Hub at commit time to create a new SFDX scratch org and run tests. To make this possible, we need to authorize CircleCI with the JWT flow, which doesn’t involve any user interaction or require storage of user credentials.
We’ll create a certificate, which will be stored in a Connected App in our Dev Hub, and an associated private key, which we’ll store in encrypted form in our repository and use to authenticate.
Create the key pair and server certificate as described in the SFDX Developer Guide.
$ openssl genrsa -des3 -passout pass:x -out server.pass.key 2048 $ openssl rsa -passin pass:x -in server.pass.key -out server.key $ rm server.pass.key $ openssl req -new -key server.key -out server.csr $ openssl x509 -req -sha256 -days 365 -in server.csr -signkey server.key -out server.crt $ rm server.csr
Generate a symmetric encryption key using your password manager of choice and save it securely - you’ll need it now, to encrypt the server private key, and later, to decrypt the key during CI.
Move the server private key to an
assets directory at the project root:
$ mkdir assets $ mv server.key assets
Encrypt the server private key (replacing
$KEY with your symmetric encryption key):
$ openssl aes-256-cbc -k $KEY -in assets/server.key -out assets/server.key.enc -e
Remove the unencrypted private key:
$ rm assets/server.key
assets directory and encrypted private key to Git:
$ git add assets $ git commit -m "Add encrypted key"`
Build Connected App
Set up a Connected App for the JWT authorization flow, as in the SFDX Developer Guide, by completing the following steps.
- In Lightning Setup, access App Manager and select “New Connected App”.
- Populate an arbitrary application name and your email address.
- Choose “Enable OAuth Settings”.
http://localhost:1717/OauthRedirectas the callback URL
- Select “Use Digital Signatures” and upload the
server.crtfile generated in the last step.
- File the
server.crtfile away somewhere safe, especially if you’re ever going to set up the CLI flow with a different org. Don’t commit it to Git.
- Add the “Access and manage your data”, “Perform requests on your behalf at any time”, and “Provide access to your data via the Web” OAuth scopes.
- Save the connected app.
- When the page finishes reloading, copy the Consumer Key and stash it securely (don’t commit it to Git).
- Click “Manage”, then “Edit Policies”.
- Set Permitted Users under “OAuth policies” to “Admin approved users are pre-authorized”.
- Click “Save”, then “Manage Profiles” (or Permission Sets) and add the Profile or Permission Set assigned to the user you’d like to connect CircleCI under. If you’re using a Dev Hub trial, add the System Administrator profile.
Here’s what the Connected App will look like at the end of the process.
Set up CircleCI for Authentication
CircleCI itself must be configured to talk to the Dev Hub using the private key before continuous integration can begin. We’ll do this using CircleCI Environment Variables, which are stored securely and aren’t part of the repository. Follow these steps to complete the CircleCI setup:
- Add your project to CircleCI after authenticating with GitHub (Projects->Add Project).
- In the Settings for your project in CircleCI, choose Environment Variables.
- Add the following environment variables:
- CONSUMERKEY, with the consumer key from your Connected App in Salesforce.
- KEY, with the value of the key used to encrypt
- USERNAME, with the user name of your Dev Hub user account.
Since CircleCI won’t allow you to access the values of these variables later, make sure to retain
$KEY in a secure store, like a password manager.
Add the Project to Codecov.io
Tracking test coverage on a commit-by-commit basis is a nice extra to add to our continuous integration and testing flow. Codecov.io provides full support for Apex and Salesforce DX development. If you want to push and track coverage statistics, login to Codecov via GitHub and add the project repository. A section in our
config.yml, below, will update the statistics on each build.
If you don’t want to track code coverage, omit this step, and remove the corresponding section from
config.yml for SFDX Deploys and Test Runs
We’ll need a
config.yml file to tell CircleCI how to build and test our project. We’re using CircleCI 2.0.
First, create the CircleCI directory and YAML file:
$ mkdir .circleci && touch .circleci/config.yml
Next, construct your
config.yml file. The
config.yml for septaTrains is designed as a template, which you’re welcome to copy and modify. It is broken down section-by-section below.
.circleci to Git and push:
$ git add .circleci $ git commit -m "Add CircleCI integration" $ git push
The project builds in SFDX on CircleCI, and if all goes well, you get a green checkmark in your commit log in GitHub.
Congratulations - you have a fully-fledged CI solution in place!
CircleCI will monitor commits to the repository and immediately test, and provide feedback on, every single commit, using and disposing a fresh scratch org to make sure code can be cleanly deployed and tested at all times. You’ll get immediate visibility into build problems, regressions, and (optionally) test coverage, helping to streamline the development process and raise issues early.
If you later need to alter the Dev Hub org for the project, simply create a new Dev Hub user account, ensure that a Connected App is in place in that org (you can use the same certificate), and update the
USERNAME variables on CircleCI. The remainder of the setup can remain constant.
config.yml for CircleCI
config.yml builds each commit in a new scratch org using SFDX and runs all Apex tests, storing test results in CircleCI. After completing the build, the scratch org is thrown away. It also includes optional functionality to monitor code coverage.
version: 2 jobs: build: docker: - image: circleci/node:latest
We’re using CircleCI 2.0 with the latest Node.js image as our base. This makes it easy to install SFDX from the command line.
steps: - checkout - restore_cache: keys: - sfdx-version-41-local
The caching structure is one area where you may wish to make changes. We’re using a constant cache key,
sfdx-version-41-local, to force CircleCI to retain SFDX version 41. SFDX’s
force:apex:test:run semantics are documented to be changing after version 41, and we don’t want our CI flow to suddenly break because Node installed a newer version of SFDX. Your project may demand a more sophisticated caching strategy, particularly if you have other dependencies.
- run: name: Install Salesforce DX command: | openssl aes-256-cbc -k $KEY -in assets/server.key.enc -out assets/server.key -d export SFDX_AUTOUPDATE_DISABLE=true export SFDX_USE_GENERIC_UNIX_KEYCHAIN=true export SFDX_DOMAIN_RETRY=300 npm install sfdx-cli node_modules/sfdx-cli/bin/run --version node_modules/sfdx-cli/bin/run plugins --core - save_cache: key: sfdx-version-41-local paths: - node_modules
First, we’ll install SFDX using
npm and decrypt our server key using the
$KEY environment variable we previously set up. As noted above, if this is the first run, we’ll save our SFDX infrastructure under the constant cache key.
Perform CI Build
- run: name: Create Scratch Org command: | node_modules/sfdx-cli/bin/run force:auth:jwt:grant --clientid $CONSUMERKEY --jwtkeyfile assets/server.key --username $USERNAME --setdefaultdevhubusername -a DevHub node_modules/sfdx-cli/bin/run force:org:create -v DevHub -s -f config/project-scratch-def.json -a scratch
We need a fresh scratch org for this CI run. We authenticate to our dev hub using the JWT flow, which requires no user interaction and uses the key file we just decrypted. We ask for a new scratch org, which is created synchronously.
- run: name: Remove Server Key when: always command: | rm assets/server.key
We’ll use a
when: always step to make sure that the unencrypted version of the server key is deleted immediately after use.
- run: name: Push Source command: | node_modules/sfdx-cli/bin/run force:source:push -u scratch - run: name: Run Apex Tests command: | mkdir ~/apex_tests node_modules/sfdx-cli/bin/run force:apex:test:run -u scratch -c -r human -d ~/apex_tests
Next, we push our source up to the new scratch org. Any compilation errors will be caught at this step and will abort the process.
We ask SFDX to run all Apex tests. With a
-r argument, this command is run synchronously in SFDX v.41, which we’ve preferentially cached. When a new version of SFDX is released, we’ll update
config.yml accordingly. Despite
-r human, SFDX will provide a JUnit XML file that CircleCI can interpret, as well as a human-compatible readout. Note that we’re also supplying
-c to request a code coverage JSON file.
Teardown, Store Artifacts and Test Results
- run: name: Push to Codecov.io command: | cp ~/apex_tests/test-result-codecoverage.json . bash <(curl -s https://codecov.io/bash)
We’re using codecov.io for code coverage metrics because they support the SFDX code coverage file format. This section is easily separable if you don’t want to track code coverage.
- run: name: Clean Up when: always command: | node_modules/sfdx-cli/bin/run force:org:delete -u scratch -p rm ~/apex_tests/*.txt ~/apex_tests/test-result-7*.json
To avoid having scratch orgs pile up and hit our limit, we immediately enqueue our new scratch org for deletion. We also remove the redundant test files created by SFDX (it stores several similar or identical formats) so that our
store_test_results step doesn’t grab them and double-count our test metrics in CircleCI.
- store_artifacts: path: ~/apex_tests - store_test_results: path: ~/apex_tests
Finally, we store the test results and code coverage artifacts using CircleCI’s native support.
- The sfdx-travisci example repository from Salesforce.
- “Wire It All Together” from the Continuous Integration with Salesforce DX Trailhead module.
- CircleCI 2.0 Docs
- The following sections from the SFDX Developer Guide: