Applying system upgrades

Automated package update policy

Cloud 66 aims to make it easier to build immutable infrastructure. Building servers and applications from scratch is much better than modifying existing server configurations and tinkering with settings until things start to work.

When a new server is provisioned we automatically install the latest versions of packages. After the server is created we only auto-install packages that are marked as security updates. Cloud 66 doesn’t (typically) automatically update other packages because it doesn’t want to risk disrupting your existing application. However, depending on your configuration, we may update your packages each time you redeploy your app (see below).

Controlling package updates

You can add custom Linux (Ubuntu) packages to your application using Deploy Hooks (read our full guide here.) Depending on how you’ve set them up, Deploy Hooks can either install packages only when a new server is created, or on every (re)deploy (or both).

Managing high-risk updates

If you’re concerned about a major update to one of the packages you depend on, our first recommendation is to build a new application and redirect your traffic to the new application using your Failover Address.

If rebuilding is impractical or impossible, there are three approaches to dealing with the issue:

Note

Some package updates require server-reboot. When scaling up we restart your new servers automatically to ensure everything works correctly. Alternatively, you can reboot your servers manually or via the Toolbelt.

Upgrade package types

Security updates

In the event of a security vulnerability on any of the components we deploy on the servers, we aim to be as quick as possible to roll out the recommended patches.

Ubuntu

To manually trigger security upgrades:

  1. Open the Application Overview from your Dashboard
  2. Clicking on the Build / Deploy button and choose Deploy with Options
  3. Click on the More Options tab
  4. Check Apply Security Upgrades and Yes, reboot my servers if required
  5. Click Run Now

This will perform operating system security package upgrades and also set up unattended upgrades for your application. Unattended upgrades will automatically check for and install the latest Ubuntu security packages on a daily basis.

Note that some security packages may require a server restart. We don’t automatically restart your server, and it is at your discretion to do so.

If the file /var/run/reboot-required exists, that means your server requires a restart. To see which packages contributed to the requirement for a restart, read /var/run/reboot-required.pkgs.

Passenger

The recommended way to upgrade your passenger to the latest one is to deploy to a new web server and drop the old one, so the scaled up one will automatically have the latest version supported by Cloud 66.

Docker and Weave

It is best to keep your Docker and Weave versions up to date as they are released quite frequently with bug/security fixes.

  1. Update your manifest file (Configuration Files -> Manifest.yml) and change the Docker and Weave version to the latest ones.

  2. Click on Deploy and choose Deploy with options
  3. Go to the More options tab and tick the Apply Docker upgrades check box.

Warning

Upgrading in-place involves downtime as the Docker engine and local files are all upgraded. To maintain zero down-time you should clone your application and use failover groups to switch to the new instance.

About manual upgrades

If you need to upgrade any part of your application the best course of action is always to build a new server. However, if that is not desirable or possible, you can always perform manual upgrades.

We detect the version of all the components we have configured or deployed on your servers on a regular basis and after each deployment. If you upgrade any part of your application manually, the upgrade will be detected by Cloud 66. This helps with the future automated upgrades.

Troubleshooting package upgrades

Although there are an almost infinite number of issues that could occur with updating different packages, we have summarised some common issues:

Deployment failed: Unable to install package(s)

The most common reason for this error is that the version of one or more components specified in your manifest.yml and/or Docker file conflicts with other components - particularly with the version of the underlying operating system (Ubuntu).

If you are trying to upgrade an existing application - particularly one that had been running for 12 months or longer - be sure to check both your Manifest file and (where appropriate) your Docker file. You can view and edit these files via your Dashboard or directly in your git repo.

If you have explicitly specified a version for any component in either of these files, Cloud 66 will attempt to use that version on the assumption that you have specified it for a good reason (such as backward compatibility).

“Failed to Fetch … 404 Not Found” errors

The most common cause of these kinds of errors is some kind of interruption in network connectivity - typically between your cloud provider and the package repository. Such interruptions are not uncommon, and typically resolve themselves quickly. Wait a few minutes and try again. If the problem persists:

  1. Check you can reach the server yourself (via SSH)
  2. If you can access the machine, you should attempt to run the package upgrade manually via ssh (using the standard apt-get install <package> command)
  3. If you’re still getting 404s you should check that your cloud provider’s firewall isn’t blocking access to the package repo (and/or Cloud 66’s IP range).

“Unable to find expected entry” errors

If your application depends on older or legacy packages, these packages (or the repos that support them) can sometimes cease to exist, particularly when an older version of an operating system reaches “end of life” (EOL). There are sometimes manual workarounds for this kind of things, but in general we recommend upgrading to packages that are being actively maintained.

“…has no installation candidate” errors

This typically only occurs if you’re using your own (“registered”) servers and running an unsupported operating system and/or distribution. We only support the two most recent major releases of Ubuntu. You can check what we currently support here. The best plan here is to reinstall a supported version of Ubuntu.