Automating cloud resources with Python and Pulumi: Splitting config and code

Splitting the config data into a yaml file as input to the Pulumi program

Jarle Bjørgeengen

Cloud Architect

In infrastructure code (and other code too) it is a good approach to separate the program logic from it's input data (configuration). That way, in order to change the state of our infrastructure, we only need to change the input data and not the program unless the logic of the program changes.

In the previous blog post we went through basic setup of Pulum with the Python template for using it to manage Openstack resources in Safespring. This a good starting point to understand the basics of how one can use Python together with Pulumi to declaratively manage infrastructure resources without having to write all resource graph management from the ground and up, which of course also would be possible with Python or any modern programming language for that matter.

One problem with the first example is that the configuration (instance name, flavor, network and so on) is embedded alongside the Python code.

While that approach serves as a nice self contained example it quickly becomes both error prone and tedious having to change the Python program every time a new object (an instance for instance ;-)) should be added, changed, reconfigured or removed.

If the configuration was stored outside the program, and loaded into it upon execution, for example in the most ubiquitous “human friendly data serialization language” in IT today: YAML, then, that would be an improvement over the initial approach, right ?


Reading the instance configuration from a YAML file

Consider the following Python code:

"""An OpenStack Python Pulumi program"""

import pulumi
from pulumi_openstack import compute
from pulumi_openstack import networking
from ruamel.yaml import YAML
import os.path

# Configure the behavior for the yaml module
yaml.default_flow_style = False

# Load config data from YAML file representation
# into Python dictionary representation

config_data_file = "pulumi-config.yaml"
if os.path.isfile(config_data_file):
  fh = open(config_data_file, "r")
  config_dict = yaml.load(fh)
  print(f'The file {config_data_file} does not exist!')

instances = {}
for instance in config_dict:
  instance = compute.Instance(instance['name'],
  name = instance['name'],
	flavor_name = instance['flavor'],
	networks = [{"name": instance['network']}],
	image_name = instance["image"])

In this example we have taken the same minimal set of parameters needed to define an instance, but instead of specifying the parameters in the code we read the from a dictionary which again is coming from de-serialising data from the pulumi-config.yaml file.

And the pulumi-config.yaml file looks like this:

- name: pulumi-snipp
  flavor: l2.c2r4.100
  image: ubuntu-22.04
  network: default
- name: pulumi-snapp
  flavor: l2.c2r4.500
  image: ubuntu-22.04
  network: public

So now we can just run pulumi up and iterate over the list of instances in the YAML file to converge desired into state to actual state ? Well, first we actually need to update the virtualenv the the Pulumi program uses in order to make use of the ruamel.yaml module. To make sure the change persist across when replicating the setup in other places (a pipeline for instance) we should add the ruamel.yaml module to the requirements.txt file and then run venv/bin/pip install -r requirements.txt in order to update the installed Python libraries according to the requirements.

Now we can apply the desired state by:

(oscli) ubuntu@demo-jumphost:~/pulumi$ pulumi up
Previewing update (dev)

View in Browser (Ctrl+O):

     Type                           Name             Plan
     pulumi:pulumi:Stack            pulumi-demo-dev
 +   ├─ openstack:compute:Instance  pulumi-snapp     create
 +   └─ openstack:compute:Instance  pulumi-snipp     create

    + 2 to create
    1 unchanged

Do you want to perform this update? yes
Updating (dev)

View in Browser (Ctrl+O):

     Type                           Name             Status
     pulumi:pulumi:Stack            pulumi-demo-dev
 +   ├─ openstack:compute:Instance  pulumi-snapp     created (15s)
 +   └─ openstack:compute:Instance  pulumi-snipp     created (14s)

    + 2 created
    1 unchanged

Duration: 17s

(oscli) ubuntu@demo-jumphost:~/pulumi$

Let’s inspect what was created by using openstack CLI:

(oscli) ubuntu@demo-jumphost:~/pulumi$ openstack server list |grep pulu
| 48d1cb9f-d732-4684-82e8-aa89ca05c5b9 | pulumi-snapp                          | ACTIVE  | public=, 2a09:d400:0:1::2b1  | ubuntu-22.04             | l2.c2r4.500  |
| 5870d687-5aac-40b8-8f23-e54755e0fc62 | pulumi-snipp                          | ACTIVE  | default=, 2a09:d400:0:2::82      | ubuntu-22.04             | l2.c2r4.100  |
(oscli) ubuntu@demo-jumphost:~/pulumi$

Looks like Pulumi kept it’s promise.