Upgrade Calico on OpenStack

Calico package update

This page describes how to upgrade to v3.32 from Calico v3.0 or later. The procedure varies by Linux distribution.

• Upgrading an OpenStack cluster based on CentOS

• Upgrading an OpenStack cluster based on Ubuntu

If you are upgrading from a version earlier than v3.32, you must also run a one-time policy data migration after the package upgrade; see Migrating policy data below.

note

Do not use older versions of calicoctl after the upgrade. This may result in unexpected behavior and data.

Upgrading an OpenStack cluster based on CentOS

1. On all nodes, change the location of the Calico packages to point to the v3.32 repo:

   sudo sed -i 's/calico-X.X/calico-Y.Y/g' /etc/yum.repos.d/calico.repo

   Replace X.X in the above command with the version you're upgrading from (must be v3.0 or later). Replace Y.Y with the version of the release you're upgrading to. Example: if you are upgrading from v3.1 to v3.5, replace X.X with 3.1 and replace Y.Y with 3.5.

2. On all compute nodes, update packages:

   sudo yum update

   We recommend upgrading the whole distribution as shown here. In case you prefer to upgrade particular packages only, those needed for a Calico compute node are the following.

   • calico-common
   • calico-compute
   • calico-dhcp-agent
   • calico-felix
   • dnsmasq
   • networking-calico
   • openstack-neutron
   • openstack-nova-api
   • openstack-nova-compute
     
     

3. Use the following command on the compute nodes to confirm that Felix has upgraded to v3.32.

   calico-felix --version

   It should return v3.32.

4. On all compute nodes, make sure the datastore type is etcdv3 in /etc/calico/felix.cfg:

   DatastoreType = etcdv3

   If you need to change the EtcdEndpoints address (e.g. because you've installed a new etcdv3 cluster rather than upgrading your existing etcdv2 cluster), you should update the EtcdEndpoints addresses in /etcd/calico/felix.cfg at this point.

5. On all control nodes, update packages:

   sudo yum update

   We recommend upgrading the whole distribution as shown here. In case you prefer to upgrade particular packages only, those needed for a Calico control node are the following.

   • calico-common
   • calico-control
   • networking-calico
   • openstack-neutron
     
     

6. On all control nodes, restart neutron-server:

   sudo systemctl restart neutron-server

7. If you ran calico-upgrade earlier to migrate non-openstack data, on the control node run:

   calico-upgrade complete

8. Remove any existing calicoctl instances and install the new calicoctl.

9. If you are upgrading from a version earlier than v3.32, run the policy data migration.

10. Congratulations! You have upgraded to Calico v3.32.

Upgrading an OpenStack cluster based on Ubuntu

1. On all nodes, change the location of the Calico packages to point to the v3.32 repo:

   sudo bash -c 'cat > /etc/apt/sources.list.d/project-calico-calico-X_X-trusty.list' << EOF
   deb http://ppa.launchpad.net/project-calico/calico-X.X/ubuntu trusty main
   # deb-src http://ppa.launchpad.net/project-calico/calico-X.X/ubuntu trusty main
   EOF

   Replace X_X and X.X with the version you're upgrading to. Example: if you're upgrading to v3.5, replace X_X with 3_5 and replace X.X with 3.5. Also replace trusty with the code name of your Ubuntu version.

2. On all compute nodes, update packages:

   sudo apt-get update
   sudo apt-get install calico-compute calico-felix calico-common \
                        networking-calico calico-dhcp-agent
   

3. Use the following command on the compute nodes to confirm that Felix has upgraded to v3.32.

   calico-felix --version

   It should return v3.32.

4. On all compute nodes, make sure the datastore type is etcdv3 in /etc/calico/felix.cfg:

   DatastoreType = etcdv3

   If you need to change the EtcdEndpoints address (e.g. because you've installed a new etcdv3 cluster rather than upgrading your existing etcdv2 cluster), you should update the EtcdEndpoints addresses in /etcd/calico/felix.cfg at this point.

5. On all control nodes, update packages:

   sudo apt-get update
   sudo apt-get install calico-control calico-common networking-calico

6. On all control nodes, restart neutron-server:

   sudo service neutron-server restart

7. If you ran calico-upgrade earlier to migrate non-openstack data, on the control node run:

   calico-upgrade complete

8. Remove any existing calicoctl instances and install the new calicoctl.

9. If you are upgrading from a version earlier than v3.32, run the policy data migration.

10. Congratulations! You have upgraded to Calico v3.32.

Migrating policy data when upgrading from earlier than v3.32

In Calico v3.32, the naming convention for policies stored in the etcd datastore changed: policies in the default tier are now stored under their plain name, whereas Calico v3.29.4 through v3.31.x stored them under a tier-prefixed name such as default.my-policy. On Kubernetes clusters this data is migrated automatically by kube-controllers. OpenStack clusters do not run kube-controllers as a service, so the same migration must be run once, by hand, after upgrading.

If you skip this step, policies created before the upgrade misbehave in the following ways, until each one is migrated or recreated:

• calicoctl get, calicoctl delete, and similar commands only find such a policy under its stored name, e.g. calicoctl get globalnetworkpolicy default.my-policy; using the plain name reports that the resource does not exist.

• Re-applying such a policy under its plain name creates a second, independent policy object instead of updating the existing one. Both objects are then live policy until one of them is deleted.

• The %n and %p specifiers in Felix's LogPrefix configuration expand to the stored name, so log entries for pre-upgrade policies include the default. prefix while entries for newly written policies do not.

Before you begin

• Upgrade every node first. All Felix instances (compute nodes) and all neutron-server instances (control nodes) must be running Calico v3.32 or later before you run the migration. On a Kubernetes cluster this ordering is verified automatically; on OpenStack it is your responsibility.

• Resolve any duplicate policies. If policies have already been re-applied since the upgrade, you may have duplicate pairs - the same policy name listed twice by calicoctl get globalnetworkpolicy, one showing the tier default and one with an empty tier column. For each pair, compare the two objects:

  calicoctl get globalnetworkpolicy default.my-policy -o yaml
  calicoctl get globalnetworkpolicy my-policy -o yaml

  and, once you have confirmed that the plain-named object has the spec you want, delete the legacy one:

  calicoctl delete globalnetworkpolicy default.my-policy

  If you don't resolve a duplicate pair yourself, the migration keeps the plain-named object and deletes the legacy one.

Run the migration

note

The openstackmigrations mode requires the v3.32.2 (or later) image; it is not present in v3.32.0 or v3.32.1.

Run the following on any machine that has Docker and can reach your etcd cluster (for example, a control node). Adjust ETCD_ENDPOINTS for your deployment:

docker run --rm --net=host \
    -e DATASTORE_TYPE=etcdv3 \
    -e ETCD_ENDPOINTS=http://127.0.0.1:2379 \
    -e ENABLED_CONTROLLERS=openstackmigrations \
    calico/kube-controllers:v3.32.1

If your etcd requires TLS, also mount the certificates and pass the usual etcd TLS environment variables:

docker run --rm --net=host \
    -v /etc/calico/certs:/etc/calico/certs:ro \
    -e DATASTORE_TYPE=etcdv3 \
    -e ETCD_ENDPOINTS=https://etcd1:2379,https://etcd2:2379 \
    -e ETCD_CA_CERT_FILE=/etc/calico/certs/ca.pem \
    -e ETCD_CERT_FILE=/etc/calico/certs/client.pem \
    -e ETCD_KEY_FILE=/etc/calico/certs/client-key.pem \
    -e ENABLED_CONTROLLERS=openstackmigrations \
    calico/kube-controllers:v3.32.1

If your etcd requires username/password authentication, add the credentials to either of the above commands with two further environment variables:

    -e ETCD_USERNAME=<username> \
    -e ETCD_PASSWORD=<password> \

(To keep the password out of your shell history and the process's visible environment, you can put the ETCD_PASSWORD=... line in a root-readable file and pass it with --env-file instead.)

The command migrates each affected policy by writing it back under its plain name and removing the legacy entry, logging a line per policy. When everything is migrated it logs

Policy name migration complete; no further work pending

followed by

All OpenStack datastore migrations are complete

and exits with status 0. The migration only rewrites policies that need it, so it is safe to run again; a second run completes immediately, reporting zero migrated policies.

note

Each migrated policy briefly exists under both its old and new names, and its object UID changes. The policy's rules are unchanged and remain enforced throughout, though Felix's log prefixes and chain names for the policy switch to the plain-name form as the migration lands on each compute node.

Verify

Pick a policy that was created before the upgrade and confirm that it is now accessible by its plain name:

calicoctl get globalnetworkpolicy my-policy

and that the legacy name is gone:

calicoctl get globalnetworkpolicy default.my-policy

The second command should report that the resource does not exist.