Build Robust & Production Quality Applications - Lesson 6: Continuous Delivery

Continuous Delivery

Excerpt from Tealeaf:

Continuous Delivery (CD) or sometimes known as Continuous Deployment goes one step beyond CI to automatically deploy features when the new code passes the continuous integration phase. Continuous Delivery encourages small and incremental software updates over big and infrequent releases to shorten the feedback loop and fix bugs earlier.

Let's look at an example development workflow that has both CI and CD enabled, based on the Github Flow process:

** we pull the latest code from Github ** we create a new feature branch and develop a new feature ** after we finish the feature, we push it to a branch with the same name on Github ** we create a PR from this branch to the staging branch. ** we wait for the the CI server to ensure all tests pass. ** we allow the CI server to automatically deploy the code from the staging branch to our staging server ** we perform sanity tests on our staging server ** we create a PR from the staging branch to the master branch on Github ** this will trigger another round of integration and if it passes, the CI server will automatically deploy the code to the production server.

If adopted by everyone in the development team, this process will make sure our master branch is always in sync with the production server and new features are also always build on top of what's on the production server.

Assignment: Set up Continuous Delivery with Circle CI

For this assignment, you are going to enable Continuous Delivery with Circle CI.

Here are the steps:

create a staging branch locally if you don't have it yet. in Circle CI, follow Project settings for your project and select Heroku Deployment under Continuous Deployment configure the Heroku API key. (Get it from your Heroku account page) associate Heroku SSH key with your Circle account so Circle can have the authority to deploy to Heroku on your behalf. create a circle.yml file in your projects root directory and make sure your adjust productionappname & stagingappname to your own app name. Here is an example of a circle.yml file that you can use:

machine:
  ruby:
    version: 2.1.5
deployment:
  production:
    branch: master
    commands:
      - heroku maintenance:on --app production_app_name
      - heroku pg:backups capture --app production_app_name
      - git push git@heroku.com:production_app_name.git $CIRCLE_SHA1:refs/heads/master
      - heroku run rake db:migrate --app production_app_name
      - heroku maintenance:off --app production_app_name
  staging:
    branch: staging
    commands:
      - heroku maintenance:on --app staging_app_name
      - git push git@heroku.com:staging_app_name.git $CIRCLE_SHA1:refs/heads/master
      - heroku run rake db:migrate --app staging_app_name
      - heroku maintenance:off --app staging_app_name
Note: you should change the Ruby version to the version you're using for this project.

This code should be pretty self explanatory - this allows Circle to monitor your staging branch and deploy to the staging server, and monitor your master branch to deploy to the production server. It'll run migrations for you and for the production server, it also automatically backs up the database before a deploy.

Notifications Options

By default we will be notified by email on every status change of our builds. But receiving emails all the time build fails or passes might be cumbersome and there are also good alternatives.

For Mac users - there is a free application CCMenu by ThoughtWorks which we install and it will stick to our upper bar. In order to make it fully run - we will need to generate a CircleCI API key (Project settings -> API keys) and http address of build. See the setup info guide here. Great feature of this app is that it can play sounds based on build's status and we are able to customize it to our needs.

Another great feature most CIs offer are Embedded Status Badges. These are especially useful for open source projects. They look nice in README.md usually placed in GitHub repo page giving out the information about build being successful or not.

Build Robust & Production Quality Applications - Lesson 6: Continuous Intgeration

Continuous Integration

According to Martin Fowler's blog, Continuous Integration is Continuous Integration is a software development practice where members of a team integrate their work frequently, usually each person integrates at least daily - leading to multiple integrations per day. Each integration is verified by an automated build (including test) to detect integration errors as quickly as possible. Many teams find that this approach leads to significantly reduced integration problems and allows a team to develop cohesive software more rapidly. This article is a quick overview of Continuous Integration summarizing the technique and its current usage.

According to Tealeaf...

The problem that CI solves is to force integration on a regular basis. For example, you may have just finished a feature on my_feature branch and have had all the tests pass locally. You open a PR and your fellow coworkers are happy with the way the code looks. Github shows the PR can be automatically merged without conflict - so you do that, merge it back to the master branch and ready to deploy the feature!

The danger of this workflow is that while you are working on your feature, someone else could have just finished their feature and pushed to the master branch to Github, and your new feature never integrated with this new piece of code.

A Continuous Integration solution would watch your repository and force integration (running the entire test suite on the CI server) whenever necessary.

Once we have our CI server set up to monitor our repository, it'll pull the latest code and run the entire test suite every time we push code to any branch on Github (when you work with a CI server, you want to make sure that every push should always make the tests pass) and notify you on the results. The CI server also runs every time we merge our pull request to the master branch to ensure integration.

CI server helps us catch integration errors before they reach to production, and the earlier we catch errors, the less costly we can fix them.

A CI solution doesn't eliminate the necessity to run your tests locally - your will lose friends quickly if you constantly push code that "breaks the build" and have everyone notified. However, in cases of long running test suites, it is acceptable to run only the tests pertaining to the new feature built, and have the CI server to run the entire test suite to catch any potential regression. Most modern CI services allow you to run the tests in parallel (with a paid plan, typically) so it could run your tests several times faster.

Circle CI

Set Up Continuous Integration with Circle CI

Circle CI is a popular Continuous Integration service. Set up your project with Circle CI is pretty simple - link and authorize your Github account, then add the repository that you want to enable CI. By default, Circle CI runs on every push and PR merge.

Build Robust & Production Quality Applications - Lesson 6: Simple Deployment Pipeline

Set-up Production Error Monitoring

Regardless how well we have tested our application, there will always be errors on the production server. To get notified of production errors so you can respond to them, you need to set up an error monitoring service in production.

Install Sentry.

You also need to install the "sentry-raven" gem in your app.

Setup

After creating an account online and adding your project to Sentry, you will be given a DSN that should go into your config/initializers folder.

# in Rails, this might be in config/initializers/sentry.rb
Raven.configure do |config|
  config.dsn = 'http://public:secret@example.com/project-id'
end

Build Robust & Production Quality Applications - Lesson 6: Simple Deployment Pipeline

Continuous Delivery

Usually the first stage of a deployment pipeline will do any compilation and provide binaries for later stages. Later stages may include manual checks, such as any tests that can't be automated. Stages can be automatic, or require human authorization to proceed, they may be parallelized over many machines to speed up the build. Deploying into production is usually the final stage in a pipeline.

Continuous Delivery is a software development discipline where you build software in such a way that the software can be released to production at any time.

You’re doing continuous delivery when: [1]

*Your software is deployable throughout its lifecycle *Your team prioritizes keeping the software deployable over working on new features *Anybody can get fast, automated feedback on the production readiness of their systems any time somebody makes a change to them *You can perform push-button deployments of any version of the software to any environment on demand

Deployment Pipeline

A deployment pipeline is a way to deal with this by breaking up your build into stages. Each stage provides increasing confidence, usually at the cost of extra time. Early stages can find most problems yielding faster feedback, while later stages provide slower and more through probing.

More broadly the deployment pipeline's job is to detect any changes that will lead to problems in production.

Simple Approach to the Deployment Pipeline

Deployment pipelines can become very complex - typically the larger the project is and the more teams are involved, the more complicated the automatic deployment process is. For our purpose, we are going to adopt a simple but effective deployment pipeline for solo dev or small team situations.

1) Follow the Github Flow process, and after you complete a feature, merge the pull request back to the master branch.

2) Run your entire test suite. This is very important and you should always do this before pushing code to Github. The master branch should always have working code. Developers call this "don't break the build".

3) Deploy your code to the staging environment and test on your staging server manually to make sure that the new feature works. Again, your staging server should be as close to your production server as possible.

4) Deploy code to production. You want to deploy code to production as often as possible to get feedback as fast as possible. If something breaks, it's much easier to fix it with your mind still fresh with the feature you worked on. In some teams this step is out of your control and there are strict production deployment times. In that situation you'd go back to step 1 and build your next feature.

When you deploy to the staging server or the production server, effectively what you are doing is to push code from your local master branch to the corresponding Heroku server's master branch.

Why is it important to also push/deploy from your local master branch? The problem with deploy from a feature branch is that you miss the step of integration with other developers. For example, while you work on your feature, I might have push out my feature as well. If you deploy from your branch even if it does work you didn’t test if your feature would integrate with the feature that I just deployed, AFTER you branched off. By forcing everyone to integrate with master first, now every time you push, you HAVE to integrate with everyone else’ push at that moment. If someone pushed code after you branched off, this would force you to do a pull first on master, then integrate with it, otherwise you can’t push.

Note that you should never deploy to production before you go back home or go to sleep. Make sure you can be around your computer for some time to fix anything that could happen after a production deploy. Your error monitoring app will alert you with emails if they do happen.

In summary, our simple deployment pipeline process is run entire test suite locally -> deploy to staging and test -> deploy to production. This is probably good enough for solo developers or small teams. We'll talk about Continuous Integration and Continuous Deployment with more automation in the next several topics.

Build Robust & Production Quality Applications - Lesson 6: Staging Server & Production Servers

Follow this tutorial on Heroku to understand the difference between staging and prodcution

A few things I'd like to add:

1) Make staging server's configuration and data as close to the production server as possible - after all, the purpose of having the staging server is to be able to test out new features in a "production like" environment.

If you production app is already launched, you can periodically take database dumps from the production server to populate the staging server. You could automate this as well with either a replication service, or if you are on Heroku, use Heroku's follower DB (comes with a cost) https://devcenter.heroku.com/articles/heroku-postgres-follower-databases

2) You want to be mindful on email sending on staging - staging server will actually send out emails. You want to make sure that you do not spam your users while testing out features on the staging server. This can be achieved by adding a conditional in your mailer to see if the environment is staging, and if it is, set email recipients to administrators (such as yourself). This way, you can still test and verify that emails are sent out, but you don't spam users.

3) Your staging server can (and typically should) have a different config itself - you just need to create a staging.rb file in config/environments directory, and with it you can specify the exact settings you want on there (for example, turning on asset pipeline or not, sending out email or not, charging credit card for real or not, etc). On your Heroku staging server, make sure to set RACKENV and RAILSENV to staging, then it'll use that config instead.