Automating cloud resources with Python and Pulumi: Basic provisioning

Combining imperative and declarative cloud automation with Python and Pulumi

Jarle Bjørgeengen

Jarle Bjørgeengen

Chief Product Officer

Efficient consumption of cloud services is all about automating the up and down-scaling of resources according to the ever-changing needs. The list of tools, their properties and their fitness for the purpose can be daunting.

On top of that comes the changing landscape in licensing and subscription, sometimes with astonishing effects on your existing cloud strategy, recently exemplified by Hashicorp’s unexpected license change to the Business Source License (BSL). In this landscape, we must always be prepared to adapt and change, thus, it is good to have acquaintance with both old established tools and some alternatives. This post is the first in a series on how to utilize Pulumi to automate service consumption towards the Safespring cloud APIs.

Prerequisites

API access

The APIs are firewalled, so either you need to operate from an instance (jumphost) which is already in a Safespring data center (then firewall openings are already in place). If not you must send an email to support@safespring.com and ask to whitelist the source IP of the host/CIDR you will be using Pulumi from. The command curl ifconfig.me will tell you your public IP as seen by the API firewall. This is useful especially if you access the API through NAT. Also, please tell us if the address(es) changes so we can open the new one and close the old one.

Setting up Pulumi

We recommend the manual installation for Linux because we advise against piping content from the Internet directly to a shell. Of course, the manual installation is pretty easy to automate according to your security polices too.

When having a working Pulumi executable on your local machine (in this case a Ubuntu 22.04 jump host), you must log in to the Pulumi SaaS in order to store track records of your resources and their history. In a later blog post, we will see how to change the stack storage to a Safespring S3 bucket, thus, running a standalone setup of Pulumi.

ubuntu@demo-jumphost:~$ pulumi login
Manage your Pulumi stacks by logging in.
Run `pulumi login --help` for alternative login options.
Enter your access token from https://app.pulumi.com/account/tokens
    or hit <ENTER> to log in using your browser                   :


  Welcome to Pulumi!

  Pulumi helps you create, deploy, and manage infrastructure on any cloud using
  your favorite language. You can get started today with Pulumi at:

      https://www.pulumi.com/docs/get-started/

  Tip: Resources you create with Pulumi are given unique names (a randomly
  generated suffix) by default. To learn more about auto-naming or customizing resource
  names see https://www.pulumi.com/docs/intro/concepts/resources/#autonaming.


Logged in to pulumi.com as JarleB (https://app.pulumi.com/JarleB)
ubuntu@demo-jumphost:~$

Before proceeding, we need to install a dependency for the Pulumi OpenStack Python project template:

$ sudo apt update && sudo apt install python3.8-venv

Then create a new Pulumi project and search for OpenStack and choose openstack-python:

ubuntu@demo-jumphost:~/pulumi$ pulumi new
Please choose a template (37/220 shown):
 opens  [Use arrows to move, type to filter]
  openstack-go                       A minimal OpenStack Go Pulumi program
  openstack-javascript               A minimal OpenStack JavaScript Pulumi program
> openstack-python                   A minimal OpenStack Python Pulumi program
  openstack-typescript               A minimal OpenStack TypeScript Pulumi program
  openstack-yaml                     A minimal OpenStack Pulumi YAML program


project name: (pulum) pulumi-demo
project description: (A minimal OpenStack Python Pulumi program)
Created project 'pulumi-demo'

Please enter your desired stack name.
To create a stack in an organization, use the format <org-name>/<stack-name> (e.g. `acmecorp/dev`).
stack name: (dev)
Created stack 'dev'

Installing dependencies...

Creating virtual environment...
Finished creating virtual environment
Updating pip, setuptools, and wheel in virtual environment...
Collecting pip
(...)

Next, we need to export the necessary environment variables and/or clouds.yaml config file. For now, the simplest way is to log in to the web GUI of OpenStack (Horizon) and download the OpenStack rcfile from the menu Project -> API access -> Download OpenStack RC file.

Then source that file:

ubuntu@demo-jumphost:~/pulumi$ source ~/.sandbox.safespring.com-openrc.sh
Please enter your OpenStack Password for project sandbox.safespring.com as user jarle@safespring.com:
ubuntu@demo-jumphost:~/pulumi$

You can also use application credentials if you don’t want to expose your personal password in the environment.

At this moment, we are ready to start creating resources by means of a Pulumi program written in Python. However, Pulumi programs need to know some parameters to maintain resources. The easiest way of quickly getting this information is to use the OpenStack CLI. OpenStack CLI already uses the sourced environment as configuration so we only need to install the OpenStack CLI in order to use the openstack command for this purpose.

ubuntu@demo-jumphost:~/pulumi$ sudo apt-get install virtualenvwrapper
source /usr/share/virtualenvwrapper/virtualenvwrapper.sh
mkvirtualenv oscli
(oscli) ubuntu@demo-jumphost:~/pulumi$ pip install --upgrade piping
(oscli) ubuntu@demo-jumphost:~/pulumi$ pip install python-openstackclient python-neutronclient
(oscli) ubuntu@demo-jumphost:~/pulumi$ openstack token issue
+------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field      | Value                                                                                                                                                                                   |
+------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| expires    | 2023-08-25T18:54:44+0000
(...)

The token issue commend verifies that the OpenStack CLI is configured correctly towards the OpenStack API. Pulumi will use the same configuration.

Creating an instance with Python/Pulumi

After setting Pulumi with the OpenStack template, we see a file called __main__.py in the Pulumi working directory. This is sample file created by the templating process.

"""An OpenStack Python Pulumi program"""

import pulumi
from pulumi_openstack import compute

# Create an OpenStack resource (Compute Instance)
instance = compute.Instance('test',
	flavor_name='s1-2',
	image_name='Ubuntu 16.04')

# Export the IP of the instance
pulumi.export('instance_ip', instance.access_ip_v4)

In order to create an instance in Safespring, we just need to substitute the example parameters with values that match the Safespring platform. You can use the OpenStack CLI to get this information.

To get a list of available flavors, use:

(oscli) ubuntu@demo-jumphost:~/pulumi$ openstack flavor list
+--------------------------------------+----------------+-------+------+-----------+-------+-----------+
| ID                                   | Name           |   RAM | Disk | Ephemeral | VCPUs | Is Public |
+--------------------------------------+----------------+-------+------+-----------+-------+-----------+
| 02e49e88-dfd3-4a41-b6f6-5c95ef8364cf | b2.c16r32      | 32768 |    0 |         0 |    16 | True      |
| 117ccf62-1758-4300-ab1b-b5ba78948346 | l2.c8r16.100   | 16384 |  100 |         0 |     8 | True      |
| 121bfee4-8d70-4668-81b6-0a755e022fd1 | l2.c16r32.500  | 32768 |  500 |         0 |    16 | True      |
| 1a58ba25-8444-4fb9-a2a8-4f8027cd0f59 | l2.c8r16.1000  | 16384 | 1000 |         0 |     8 | True      |
| 1da5b6f6-d722-4e83-b507-2d4ea6d3ca7d | l2.c16r32.1000 | 32768 | 1000 |         0 |    16 | True      |
| 52d06354-6cd0-42fa-a7e8-2998647db65d | l2.c2r4.1000   |  4096 | 1000 |         0 |     2 | True      |
| 52f4bc1c-973b-409e-aea1-23f7e2d14b27 | b2.c2r4        |  4096 |    0 |         0 |     2 | True      |
| 5748eadf-a45f-41f3-ba4e-dd03973ceed3 | b2.c1r4        |  4096 |    0 |         0 |     1 | True      |
| 601cf092-db6d-4faa-b456-0e8613a0c9dc | l2.c2r4.500    |  4096 |  500 |         0 |     2 | True      |
| 62502c5c-9441-4546-8859-c243a506da31 | b2.c2r8        |  8192 |    0 |         0 |     2 | True      |
| 79a8fc06-7385-490f-86b7-4daf178b6590 | l2.c16r32.100  | 32768 |  100 |         0 |    16 | True      |
| 7cade287-87a1-4bdf-a92b-a4208101895d | b2.c1r2        |  2048 |    0 |         0 |     1 | True      |
| 8574373f-b266-400c-80b1-49027c97bdcb | l2.c32r64.1000 | 65536 | 1000 |         0 |    32 | True      |
| 8f84ceab-89c1-4dfb-9ef6-97504475bd3a | l2.c4r8.500    |  8192 |  500 |         0 |     4 | True      |
| 9268de17-2d5b-4885-bc53-155f093aed6d | l2.c2r4.100    |  4096 |  100 |         0 |     2 | True      |
| a697753c-12ef-4abf-8c1d-f3bef761ffb7 | l2.c4r8.100    |  8192 |  100 |         0 |     4 | True      |
| b10d4f41-6ca4-4dae-8fec-7580cdd2a1dd | b2.c8r32       | 32768 |    0 |         0 |     8 | True      |
| b4d75d91-f3b4-4ad1-b859-0306064856d8 | l2.c8r16.500   | 16384 |  500 |         0 |     8 | True      |
| d2fc99a7-85da-49dd-9725-6670086a1aa9 | b2.c16r64      | 65536 |    0 |         0 |    16 | True      |
| e91ff4b7-cf9e-4d95-8374-aa3b1d765200 | l2.c4r8.1000   |  8192 | 1000 |         0 |     4 | True      |
| eb1d6bec-60ab-4a6b-95e9-313e33dd6712 | b2.c4r8        |  8192 |    0 |         0 |     4 | True      |
| f448fae2-135d-4865-a8d3-8306cc1a119e | b2.c4r16       | 16384 |    0 |         0 |     4 | True      |
| f578ce2f-2a60-4803-8274-a4b92a44a227 | b2.c8r16       | 16384 |    0 |         0 |     8 | True      |
+--------------------------------------+----------------+-------+------+-----------+-------+-----------+
(oscli) ubuntu@demo-jumphost:~/pulumi$

To get a list of available images, use:

(oscli) ubuntu@demo-jumphost:~/pulumi$ openstack image list
+--------------------------------------+------------------------------------------------+--------+
| ID                                   | Name                                           | Status |
+--------------------------------------+------------------------------------------------+--------+
| d007510b-908a-44a5-a1a5-b2dc8e751260 | debian-10                                      | active |
| f2ef69eb-2856-4319-95f5-902f43fccef8 | debian-11                                      | active |
| a7394047-8f79-4b2c-92aa-d4a818ef42c2 | debian-12                                      | active |
| ee81e161-d04c-40c0-a848-582750f9903b | ubuntu-18.04                                   | active |
| cfc0ca97-780b-4fa5-aa87-44e4f41a0766 | ubuntu-20.04                                   | active |
| aac74808-9dba-4f49-a530-70a23b4163f3 | ubuntu-22.04                                   | active |
+--------------------------------------+------------------------------------------------+--------+
(oscli) ubuntu@demo-jumphost:~/pulumi$

You can also upload your own image.

And finally for networks:

(oscli) ubuntu@demo-jumphost:~/pulumi$ openstack network list
+--------------------------------------+------------------+----------------------------------------------------------------------------+
| ID                                   | Name             | Subnets                                                                    |
+--------------------------------------+------------------+----------------------------------------------------------------------------+
| 14ff54e0-80e4-492b-a54a-8c4d4097ed8f | default          | 1ae2aebe-542a-410a-8fea-bf1f941d6d6a, 34489b94-634a-45cd-bac9-61deea3daf5a |
| 33dc493f-f4d5-4ab4-bf8e-43bee3faf3ef | public           | 368db41d-c77f-4759-8113-d702818702fd, 5d1e4008-7a1a-4c88-9b0c-7d0faf54a9d8 |
| 67892ac3-1dcd-4bba-bd60-28b5d037f6ff | private          | 059d94a0-0fc1-40dd-9814-eb00571c6a4d, 6ff36feb-deb9-4cc0-aa09-8007006988bb |
+--------------------------------------+------------------+----------------------------------------------------------------------------+

So we choose to change the __main__.py file like this:

"""An OpenStack Python Pulumi program"""

import pulumi
from pulumi_openstack import compute

# Create an OpenStack resource (Compute Instance)
instance = compute.Instance('pulumi-demo',
	flavor_name='l2.c2r4.100',
	networks=[{"name": "public"}],
	image_name='ubuntu-22.04')

# Export the IP of the instance
pulumi.export('instance_ip', instance.access_ip_v4)

Note that we only attach one network even though it is technically possible to attach several networks with the networks parameter being a list data type. This is because in the Safespring platform, there is no need for attaching multiple interfaces; it could even create instability and problems. To read about why this is the case, please read the blog post about Safespring’s network model

Finally, we can run pulumi up in order to build the resource graph, and subsequently apply that graph to the OpenStack API with Pulumi.

(oscli) ubuntu@demo-jumphost:~/pulumi$

(oscli) ubuntu@demo-jumphost:~/pulumi$ pulumi up
Previewing update (dev)

View in Browser (Ctrl+O): https://app.pulumi.com/JarleB/pulumi-demo/dev/previews/ec8d68fc-15a6-4dd3-8add-0b5076170bfd

     Type                           Name             Plan
     pulumi:pulumi:Stack            pulumi-demo-dev
 +   └─ openstack:compute:Instance  pulumi-demo      create


Outputs:
  + instance_ip: output<string>

Resources:
    + 1 to create
    1 unchanged

Do you want to perform this update? yes
Updating (dev)

View in Browser (Ctrl+O): https://app.pulumi.com/JarleB/pulumi-demo/dev/updates/3

     Type                           Name             Status
     pulumi:pulumi:Stack            pulumi-demo-dev
 +   └─ openstack:compute:Instance  pulumi-demo      created (15s)


Outputs:
  + instance_ip: "212.162.146.151"

Resources:
    + 1 created
    1 unchanged

Duration: 17s

(oscli) ubuntu@demo-jumphost:~/pulumi$

So, let’s see if the instances was created by using the OpenStack CLI:

(oscli) ubuntu@demo-jumphost:~/pulumi$ openstack server list |grep pulu
| 94593df2-eae9-40cf-bfba-7f8078bae970 | pulumi-demo-0d3a61e                   | ACTIVE  | public=212.162.146.151, 2a09:d400:0:1::1d9 | ubuntu-22.04             | l2.c2r4.100  |
(oscli) ubuntu@demo-jumphost:~/pulumi$

And sure, it was! Note, that since we did not specify a name, Pulumi created one for us using pulumi-demo as prefix and a random string as suffix.

If we care about the name of the instance, we can just add it to the Pulumi program like this:

"""An OpenStack Python Pulumi program"""

import pulumi
from pulumi_openstack import compute

# Create an OpenStack resource (Compute Instance)
instance = compute.Instance('pulumi-demo',
    name = 'pulumi-demo',              # < ---- here
	flavor_name='l2.c2r4.100',
	networks=[{"name": "public"}],
	image_name='ubuntu-22.04')


# Export the IP of the instance
pulumi.export('instance_ip', instance.access_ip_v4)

And then we apply the change:

(oscli) ubuntu@demo-jumphost:~/pulumi$ pulumi up
Previewing update (dev)

View in Browser (Ctrl+O): https://app.pulumi.com/JarleB/pulumi-demo/dev/previews/7a022268-94e2-4f79-9b20-9f0010563b57

     Type                           Name             Plan       Info
     pulumi:pulumi:Stack            pulumi-demo-dev
 ~   └─ openstack:compute:Instance  pulumi-demo      update     [diff: ~__defaults,name]


Resources:
    ~ 1 to update
    1 unchanged

Do you want to perform this update? details
  pulumi:pulumi:Stack: (same)
    [urn=urn:pulumi:dev::pulumi-demo::pulumi:pulumi:Stack::pulumi-demo-dev]
    ~ openstack:compute/instance:Instance: (update)
        [id=94593df2-eae9-40cf-bfba-7f8078bae970]
        [urn=urn:pulumi:dev::pulumi-demo::openstack:compute/instance:Instance::pulumi-demo]
        [provider=urn:pulumi:dev::pulumi-demo::pulumi:providers:openstack::default_3_13_3::4cf9816e-9eb8-447f-9c15-4d5614b3c329]
      ~ name             : "pulumi-demo-0d3a61e" => "pulumi-demo"

Do you want to perform this update? yes
Updating (dev)

View in Browser (Ctrl+O): https://app.pulumi.com/JarleB/pulumi-demo/dev/updates/5

     Type                           Name             Status           Info
     pulumi:pulumi:Stack            pulumi-demo-dev
 ~   └─ openstack:compute:Instance  pulumi-demo      updated (2s)     [diff: ~__defaults,name]


Outputs:
    instance_ip: "212.162.146.151"

Resources:
    ~ 1 updated
    1 unchanged

Duration: 4s

(oscli) ubuntu@demo-jumphost:~/pulumi$

And now the name was changed:

(oscli) ubuntu@demo-jumphost:~/pulumi$ openstack server list |grep pulu
| 94593df2-eae9-40cf-bfba-7f8078bae970 | pulumi-demo                           | ACTIVE  | public=212.162.146.151, 2a09:d400:0:1::1d9 | ubuntu-22.04             | l2.c2r4.100  |
(oscli) ubuntu@demo-jumphost:~/pulumi$

Connecting to the instance

In order to access the instance, though, we need to open up some ports using security groups and rules.

Let’s add this code to the Pulumi python program:

"""An OpenStack Python Pulumi program"""

import pulumi
from pulumi_openstack import compute
from pulumi_openstack import networking

sg = networking.SecGroup('pulumi-sg',
        name = 'pulumi-sg')

ssh_rule = networking.SecGroupRule("pulumi-ssh-ingress",
    direction="ingress",
    ethertype="IPv4",
    protocol="tcp",
    port_range_min=22,
    port_range_max=22,
    remote_ip_prefix="0.0.0.0/0",
    security_group_id=sg.id)

instance = compute.Instance('pulumi-demo',
        name = 'pulumi-demo',
        flavor_name='l2.c2r4.100',
        networks=[{"name": "public"}],
        security_groups=[sg.name],           # <- New parameter for security group membership
        image_name='ubuntu-22.04')

And then run pulumi up again:

(oscli) ubuntu@demo-jumphost:~/pulumi$ pulumi up
Previewing update (dev)

View in Browser (Ctrl+O): https://app.pulumi.com/JarleB/pulumi-demo/dev/previews/5ee8bfa0-7cf3-4e54-9c38-534de190f776

     Type                                  Name                Plan       Info
     pulumi:pulumi:Stack                   pulumi-demo-dev
 +   ├─ openstack:networking:SecGroup      pulumi-sg           create
 ~   ├─ openstack:compute:Instance         pulumi-demo         update     [diff: ~securityGroups]
 +   └─ openstack:networking:SecGroupRule  pulumi-ssh-ingress  create


Resources:
    + 2 to create
    ~ 1 to update
    3 changes. 1 unchanged

Do you want to perform this update? yes
Updating (dev)

View in Browser (Ctrl+O): https://app.pulumi.com/JarleB/pulumi-demo/dev/updates/17

     Type                                  Name                Status              Info
     pulumi:pulumi:Stack                   pulumi-demo-dev
 +   ├─ openstack:networking:SecGroup      pulumi-sg           created (1s)
 +   ├─ openstack:networking:SecGroupRule  pulumi-ssh-ingress  created (0.57s)
 ~   └─ openstack:compute:Instance         pulumi-demo         updated (4s)        [diff: ~securityGroups]


Resources:
    + 2 created
    ~ 1 updated
    3 changes. 1 unchanged

Duration: 8s

(oscli) ubuntu@demo-jumphost:~/pulumi$

Now we can connect on port 22:

(oscli) ubuntu@demo-jumphost:~/pulumi$ openstack server list |grep pul
| be203992-22ce-4f60-900b-d3b47b39262f | pulumi-demo                           | ACTIVE  | public=212.162.147.112, 2a09:d400:0:1::321 | ubuntu-22.04             | l2.c2r4.100  |

(oscli) ubuntu@demo-jumphost:~/pulumi$ nc -w 1 212.162.147.112 22
SSH-2.0-OpenSSH_8.9p1 Ubuntu-3
(oscli) ubuntu@demo-jumphost:~/pulumi$

Summary

We have seen that we can use Python (or other imperative programming languages) to deploy and maintain infrastructure resources in a similar fashion to Terraform. One thing to be aware of, though, is the mix between declarative and imperative approaches. In some sense, the Pulumi program behaves like a sequential recipe of tasks even though it eventually builds a similar resource graph (state) as Terraform. This means that if we change the order of events in the program by placing the instance creation before the security group creation, it will fail because the definition of the security group, that the instance will be a member of, is not yet defined in the program sequence. This is different in Terraform, where the ordering of the code statements is completely irrelevant.

In the next blog post, we will be scaling things up and separating the program configuration data out into yaml-files.

References