In my current work, there is often the need to deploy a similar application stack in various configurations, to several environments. Each configuration may vary in terms of scale, uptime requirements and feature flagging. Due to a lot of flux in infrastructure set up, each environment is also not equivalent. On top of this, there are obviously financial requirements to run all of this as cheaply as possible. Kubernetes and helm templating are valuable tools in this situation, they allow us to create a configuration blueprint with the details abstracted in values.yaml files.

Use helm’s built in linter

Let’s start with the basics, helm provides a helm lint command which performs checks

• YAML syntax
• Template rendering
• Missing or misnamed required files
• Best practice violations

You can run this with your different values.yaml files to ensure that all your configurations are compliant.

It’s also a good idea to use the helm template command to actually check that helm is able to render your templates.

Parallels with front end templating

I like to compare helm templating with html templating tools like JSX. This allows front end developers to create reusable components usable throughout pages of a web application, A button component for example can have many states, primary, secondary, loading, disabled, light or dark mode.

Each state may also look different depending on the size/type of device your are browsing the site with. Each of these states represents differences in many parameters (font size, colour, gradient, opacity, border, padding, margin, width, height, etc). These complexities are abstracted away giving the consuming code the list of states to chose from, so that they can write code like this.

<button type="primary">Click Me!</button>

Under the hood of course many aspects of the CSS or HTML code will be impacted by the change of state so you often end up with different parts of the markup having conditionals on the same check.

const Button = (props) => {
    return (
        <button className="btn btn-primary {classesForState(props.state)}">
            {props.state == "loading" && <span><svg src="loading.svg" /></span>}
            {props.children}
            <span>
        </button>
    );
}

Just in this contrived example you already have 2 different things being controlled by the state property with 2 separate checks, the CSS classes and the presence of the loading icon.

This is quite similar to the situation you end up templating in YAML with helm. Consider an application that has optional persistent storage. You could quite easily imagine a boolean property in your values.yaml file called persistent. Under the hood this has many implications likely affecting different files.

• Conditional creation of a PersistentVolume resource
• Conditional creation of a PersistentVolumeClaim resource
• Conditional storage requests/limits in your Pod
• Adding a volumes block to your Pod
• Adding a volumesMount block to your Pod

That’s 5 separate if blocks that need to be in your templates.

Forgetting one of these blocks could cause your application to function incorrectly and in this case, even cause unexpected data loss. Rather than find these problems out post deployment we can use the output of helm template with specific values to ensure that the right manifests are generated before going anywhere near a kubernetes cluster.

Helm unit test

After talking about this problem with a colleague, they told me that his team use helm unit test for this. This is a simple helm plugin that allows us to assert on the output of helm templates using yaml tests.

A test for the case described above could look like this. Assuming you have your chart templates arranged as one file per resource:

test-chart
├── Chart.yaml
├── templates
│   ├── _helpers.tpl
│   ├── persistent-volume-claim.yaml
│   └── pod.yaml
└── values.yaml

You could add a test for the persistent volume and a similar one for the persistent volume claim

suite: persistent volume suite
templates:
  - persistent-volume.yaml
tests:
  - it: doesn't include when persistence is disabled
    set:
      persistent: false
    asserts:
      - hasDocuments:
          count: 0
  - it: includes when persistence is enabled
    set:
      persistent: true
    asserts:
      - containsDocument:
          kind: PersistentVolume
          apiVersion: v1

Then you could add another test for the pod

suite: pod suite
templates:
  - pod.yaml
tests:
  - it:
      Sets storages limits and no volumes are added when persistence is disabled
    set:
      persistent: false
    asserts:
      - notExists:
          path: spec.volumes
      - notExists:
          path: spec.containers[0].volumeMounts
      - equal:
          path: spec.containers[0].resources.requests.ephemeral-storage
          value: 500Mi
      - equal:
          path: spec.containers[0].resources.limits.ephemeral-storage
          value: 1Gi
  - it: Volume is added when persistence is enabled
    set:
      persistent: true
    asserts:
      - lengthEqual:
          paths:
            - spec.volumes
            - spec.containers[0].volumeMounts
          count: 1
      - exists:
          path: spec.volumes
      - exists:
          path: spec.containers[0].volumeMounts
      - notExists:
          path: spec.containers[0].resources.requests.ephemeral-storage
      - notExists:
          path: spec.containers[0].resources.limits.ephemeral-storage

Your chart directory should now look like this

test-chart
├── Chart.yaml
├── templates
│   ├── _helpers.tpl
│   ├── persistent-volume-claim.yaml
│   └── pod.yaml
├── tests
│   ├── persistent_volume_claim_test.yaml
│   ├── persistent_volume_test.yaml
│   └── pod_test.yaml
└── values.yaml

You can run these tests with a single docker command which should be simple to integrate into your CI configuration

docker run -t --rm -v $(pwd):/apps helmunittest/helm-unittest:3.19.0-1.0.3 test-chart

Now you have confidence that the templated output for persistent and non persistent configurations is as you expect. If someone removes one of your template conditionals, they will be warned by failing tests.

Native helm test

Unit tests are all well and good, but they don’t really confirm that your chart works correctly or even that your templated output contains valid kubernetes manifests. This is where helm’s native test feature comes in. It allows you to run checks on your chart after it’s been deployed to a cluster. If your chart is for a custom built application, this could be your integration test suite, but if it’s a deployment of some vendor application with custom configuration, this is also a great way to check that your configuration works as expected. I find this especially useful for things like proxy servers.

As a simple example, let’s say you’re deploying a proxy to handle TLS redirection, in nginx, that would be something like

server {
    listen 80 default_server;
    server_name _;
    return 301 https://$host$request_uri;
}

You could use something like hurl to check that http requests are indeed redirecting to their https alternatives. You can put a hurl script in a config map.

# templates/tests/proxy-tests-config-map.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: "proxy-test-requests"
  annotations:
    "helm.sh/hook": "pre-install,pre-upgrade"
    "helm.sh/hook-weight": "0"
    "helm.sh/hook-delete-policy": before-hook-creation
data:
  tests.hurl: |
    # Test https redirection
    GET http://my-proxy.my-namespace.svc/path
    HTTP 301
    [Asserts]
    header "Location" == "https://my-proxy.my-namespace.svc/path"

And then add a pod to run it

# templates/tests/proxy-tests-pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: proxy-tests
  annotations:
    "helm.sh/hook": "test"
    "helm.sh/hook-weight": "1"
    "helm.sh/hook-delete-policy": "before-hook-creation"
spec:
  containers:
    - name: hurl
      image: ghcr.io/orange-opensource/hurl:7.1.0
      command: ["hurl"]
      args: ["--test", "/tests/tests.hurl", "--verbose"]
      volumeMounts:
        - name: "proxy-test-requests"
          readOnly: true
          mountPath: /tests
  volumes:
    - name: "proxy-test-requests"
      configMap:
        name: "proxy-test-requests"
  restartPolicy: Never

You can also use this to perform other checks, the advantage is that you can run these checks in the same kubernetes namespace that you deployed to giving you real world network conditions for example.

You can run these right after your deploying your chart in your CI system.

  helm test hs-solr-migration-proxy \
    --logs

The --logs argument will output the test pod’s logs in the output of helm test so you can examine failures easily and without necessarily accessing the cluster yourself.

Generating documentation

It’s also important to have human friendly documentation for your charts so that consumers understand the various options available set in their values.yaml files, what the defaults are and what each option does. The helm-docs tool, parses your chart values and metadata to generate documentation in a README.md file. Without any additional effort it will create a table of all the options and their default values. You can add a description column by adding a comment above the parameter in your values.yaml file.

# -- Saves application data to a persistent volume surviving application restarts
persistent: true

helm-docs also supply a pre-commit configuration which you can use to automatically regenerate the documentation when the chart changes which helps keep it in sync.

Full pipeline

To summarise, if we combine all the things we’ve discussed in this post, your workflow for creating a robust helm chart might look like this.

1. Make chart changes annotating your values file with comments to be consumed by helm-docs
2. Check your chart validates with helm lint and helm tempate
3. Add unit tests using helm unit test
4. Add integration tests using helm’s native tests
5. Generate documentation using helm-docs
6. Run linting, unit tests in CI before publishing
7. Run integration tests immediately after your deployment.

I recently had to parse relatively large XML files (multi GB) to export product data to a 3rd party API using node. Although this may sound like a trivial task at first, I found a few interesting challenges along the way.

Library choice

xml-stream

I started with this library, although very old, it seemed to offer a good balance of streamable event based parsing without too much difficulty extracting the data. The library eventually wraps the c library libexpat so I expected good performance.

However I had issues with native code compatibility running it in AWS lambda.

/var/task/node_modules/node-expat/build/Release/node_expat.node: invalid ELF header

I came to the conclusion that some dependency of libexpat was not available in the lambda runtime or that some pre-built binary wasn’t compatible.

SAX

In the hope to resolve my native code compatibility issues, I turned to sax, a very popular pure javascript implementation written by issacs (creator of npm). This did indeed resolve my issues in AWS lambda, however with my first implementation, I found it a lot more memory intensive than xml-stream. I started to have out of memory issues especially when parsing the larger files (more on that later). It was also somewhat slower than xml-stream.

Rust + neon

This was a little out there and more out of curiosity for performance benchmarks. I had a little experience building a node js wrapper for a rust crate using neon. I wondered if I could wrap something like quick xml and get the best performance. It turned out my rust skills were not quite up to the task.

sax-wasm

After my adventures in rust, I came across sax-wasm. I wondered if I could benefit from near native performance without the compatibility issues by using something wasm based. My initial results were promising, however despite a very responsive and talented maintainer, it became clear that the library was a little young to be used in production code (lack of documentation, some bugs with cdata, updated versions that broke). I did eventually get my code to work, but decided that the more productive way forward was to stick with sax and optimise my shoddy XML parsing code rather than search for a silver bullet library.

Out of memory issues

Although I never stored the entire file in memory, my initial strategy was to keep an array of all the product objects I’d parsed in memory until the full file had been read. Once the file had been parsed, I then batched requests to the api until all the products had been imported.

To fix the memory issues I changed strategy and came up with a threshold. I would keep the parsed products in memory until that threshold was reached, then send them off to the api, then fill up the next batch until all the products were imported.

The sax library is an event emitter, you pass it an XML readable stream and it fires events when it sees various XML related entities. I decided to implement my own event emitter that would send events as the products were parsed.

Here’s a highly simplified version. with all the error handling and the vast majority of the parsing logic removed.

to parse this XML:

<products>
  <product>
    <name>Fancy Trainers</name>
  </product>
</products>

Use the following class

import sax from "sax";
import {EventEmitter} from "node:events";

class ProductParser extends EventEmitter {
  parse(readableStream) {
    const xml = sax.createStream(strict, options);
    let product = null;
    let currentTag = null;
    xml
      .on("opentag", (tag) => {
        if (tag.name === "product") {
          product = {};
        }
        currentTag = tag.name;
      })
      .on("text", (text) => {
        if (currentTag === "name" && product) {
          product.name = text;
        }
      })
      .on("closetag", (tag) => {
        if (tag.name === "product") {
          // product is done, notify new product
          this.emit("product", product);
          product = null;
        }
      })
      .on("end", () => {
        // file has finished parsing
        this.emit("end");
      });
    readable.pipe(xml);
  }
}

Then orchestrate like this

async function importCatalogue(xmlStream) {
  return new Promise((resolve) => {
    const THRESHOLD = 1000;
    const productParser = new ProductParser();
    let products = [];

    productParser
      .on("product", async (product) => {
        products.push(product);
        if (products.length >= THRESHOLD) {
          const batch = [...products];
          // this is the important step. Reassigning 'products' prevents storing all products in memory.
          products = [];
          await api.import(batch);
        }
      })
      .on("end", async () => {
        if (products.length) {
          await api.import(products);
        }
        resolve();
      })
      .parse(xmlStream);
  });
}

This strategy worked well, I observed the memory usage of my application increase on start up then flatten out while it was going through all of the products.

However, it led me to a subtle unpredictable bug…

Async event emitter handlers

Notice how the promise is resolved in the 'end' event handler? Well the problem is that if one of the api.import() calls is slow, we could resolve the promise before the products have imported.

When running node locally, this isn’t a such a big problem because node waits for any asynchronous activity before exiting. However AWS lambda simply waits for your handler to complete (in this case to resolve).

This led to some batches being prematurely cut off without an obvious error or explanation.

The fix I came up with, was to store the promises and await them in the end handler.

const pendingHandlers = [];
productParser
  .on('product', (product) => {
    pendingHandlers.push((async () {
      products.push(product);
      if (products.length >= THRESHOLD) {
        const batch = [...products];
        products = [];
        await api.import(batch);
      }
    })())
  })
  .on('end', async () => {
    // Ensure all pending requests have completed
    await Promise.all(pendingHandlers);
    if (products.length) {
      await api.import(products);
    }
    resolve();
  })
  .parse(xmlStream)

This worked pretty well, however I found another issue.

Requests being delayed and timing out

Although the requests were being sent out as parsing was going on, I found that these were taking a very long time to complete.

On top of this, although I saw some improvement on memory usage, it was still continuously increasing until the full file was parsed.

My explanation to this is that the XML parsing is flooding the node js event loop with events so many events, that the callbacks used during the api request end up very far behind in the queue.

One solution to this may have been to use worker threads to send the api requests however, I didn’t like the idea of going down the multi threaded route in javascript.

Thankfully I came across this github issue in sax.

It turns out that readable streams have pause() and resume() functions which allow you to stop piping data while you perform some operation, then continue once you’re finished. So I changed my code to.

productParser
  .on('product', (product) => {
    pendingHandlers.push((async () {
      products.push(product);
      if (products.length >= THRESHOLD) {
        const batch = [...products];
        products = [];
        // stop pipeing data while the api request is sent
        xmlStream.pause();
        await api.import(batch);
        // continue pipeing data
        xmlStream.resume();
      }
    })())
  })

Success!

I was now able to import large xml files without running out of memory, what took 7GB of memory initially was reduced to 2GB without increasing the time taken.

I bought a second hand SNES mini recently and have been rediscovering the fun of super mario world. I got inspired to make a bit of pixel art using the canvas api. I wanted to generate different marios.

I started out with this image I found online.

The image contains 13 colors. I carefully translated the image to a 2D array of characters, with each character mapping to a color. Don’t pay too much attention to the actual characters, they’re pretty much random.

const template = [
  ["w", "w", "w", "w", "w", "w", "w", "r", "r", "r", "r", "r", "w", "w", "w"],
  ["w", "w", "w", "w", "w", "r", "r", "p", "p", "y", "p", "p", "r", "w", "w"],
  ["w", "w", "w", "w", "r", "p", "p", "P", "g", "y", "w", "P", "r", "w", "w"],
  ["w", "w", "w", "r", "p", "P", "P", "b", "b", "b", "b", "b", "b", "w", "w"],
  ["w", "w", "r", "p", "P", "b", "b", "b", "b", "b", "b", "b", "b", "b", "w"],
  ["w", "r", "P", "P", "b", "b", "b", "b", "b", "b", "b", "b", "b", "b", "w"],
  ["w", "r", "P", "P", "b", "b", "c", "w", "w", "c", "w", "w", "b", "b", "w"],
  ["w", "c", "s", "b", "b", "c", "c", "w", "b", "s", "b", "w", "b", "b", "w"],
  ["w", "m", "s", "b", "b", "c", "s", "w", "b", "s", "b", "w", "s", "m", "w"],
  ["w", "m", "c", "b", "b", "b", "s", "s", "c", "c", "s", "s", "s", "m", "w"],
  ["w", "c", "c", "c", "b", "s", "s", "b", "m", "c", "c", "c", "c", "m", "w"],
  ["w", "w", "w", "c", "c", "c", "b", "b", "b", "b", "b", "b", "b", "b", "w"],
  ["w", "w", "w", "w", "c", "c", "s", "s", "b", "b", "b", "b", "b", "w", "w"],
  ["w", "w", "w", "w", "c", "c", "c", "c", "c", "c", "c", "c", "m", "w", "w"],
  ["w", "w", "w", "w", "w", "P", "P", "r", "r", "m", "m", "m", "w", "w", "w"],
  ["w", "w", "w", "r", "p", "p", "r", "P", "m", "w", "w", "w", "m", "w", "w"],
  ["w", "w", "r", "P", "p", "p", "p", "m", "w", "w", "w", "w", "w", "m", "w"],
  ["w", "w", "r", "P", "P", "p", "p", "m", "w", "w", "w", "w", "w", "m", "w"],
  ["w", "w", "n", "r", "P", "P", "p", "p", "m", "w", "w", "w", "m", "b", "w"],
  ["w", "w", "n", "n", "r", "P", "P", "r", "r", "m", "m", "m", "b", "b", "w"],
  ["w", "w", "n", "t", "n", "r", "r", "r", "a", "a", "a", "t", "n", "w", "w"],
  ["w", "w", "n", "t", "t", "t", "a", "a", "a", "a", "a", "t", "n", "w", "w"],
  ["w", "w", "w", "n", "t", "t", "t", "a", "a", "n", "t", "n", "w", "w", "w"],
  ["w", "w", "w", "n", "t", "t", "t", "t", "t", "n", "t", "n", "w", "w", "w"],
  ["w", "w", "w", "w", "n", "n", "n", "n", "n", "n", "n", "w", "w", "w", "w"],
  ["w", "w", "w", "w", "m", "m", "m", "m", "m", "b", "m", "w", "w", "w", "w"],
  ["w", "w", "w", "w", "w", "m", "m", "m", "m", "y", "b", "y", "w", "w", "w"],
];

To make each mario different I needed to generate 3 different shades of a random color to use for his shirt. I used rgb notation for this and added a helper function to generate 3 versions (light, medium and dark) of a color.

const rgb = (r, g, b) => `rgb(${r},${g},${b})`;
const getShades = (r, g, b) => {
  return {
    light: rgb(r, g, b),
    medium: rgb(r * 0.8, g * 0.8, b * 0.8),
    dark: rgb(r * 0.5, g * 0.5, b * 0.5),
  };
};

I then mapped these to the corresponding letters in the template (p, P and r) to give each mario some unique clothing.

You can see 10 generated marios below. If you refresh the page they will be different every time. This was quite a fun simple way to play with the canvas api and I may revisit it in the future.

In this post I will demonstrate this by deploying an aws lambda function with a function url, but the same technique can be used to deploy any infrastructure supported by localstack.

Prerequisites

• docker
• terraform (I’m using version 1.4.6)

1. Setting up the localstack environment

Create the following docker-compose.yaml file

version: '3.4'
services:
  localstack:
    image: localstack/localstack:2.0.2
    ports:
      - '4566:4566'
    expose:
      - '4566'
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

Note that we are mounting the docker socket as a volume. Localstack pulls external docker images in order to run the lambda runtime so this is required.

Now run this to start the container.

docker compose up -d

Creating the state bucket

In a real world scenario, for your staging and production environments, you’re going to want to share the terraform state so that deploys can be made from multiple machines easily. Terraform does not support different backend types for different environments, but we can use s3 to hold our state.

the localstack image ships with a local version of the aws cli which we can use to create the bucket.

docker compose exec localstack awslocal s3api create-bucket --bucket terraform-state

2. Setting up terraform

Create a directory to store your terraform files and add a main.tf

variable "use_localstack" {
  type    = string
  default = true
}

terraform {
  backend "s3" {}
}

locals {
  aws_settings = (
    var.use_localstack ?
    {
      access_key = "fake"
      secret_key = "fake"

      skip_credentials_validation = true
      skip_metadata_api_check     = true
      skip_requesting_account_id  = true
      s3_use_path_style           = true

      override_endpoint = "http://localhost:4566"
      profile           = null
    } :
    {
      access_key                  = null
      secret_key                  = null
      skip_credentials_validation = null
      skip_metadata_api_check     = null
      skip_requesting_account_id  = null
      s3_use_path_style           = null

      override_endpoint = null
    }
  )
  archive_file = "build.zip"
}


provider "aws" {
  access_key                  = local.aws_settings.access_key
  secret_key                  = local.aws_settings.secret_key
  region                      = "eu-west-1"
  s3_use_path_style           = local.aws_settings.s3_use_path_style
  skip_credentials_validation = local.aws_settings.skip_credentials_validation
  skip_metadata_api_check     = local.aws_settings.skip_metadata_api_check
  skip_requesting_account_id  = local.aws_settings.skip_requesting_account_id
  dynamic "endpoints" {
    for_each = local.aws_settings.override_endpoint[*]
    content {
      lambda   = endpoints.value
      iam      = endpoints.value
    }
  }
}

This set up allows us to easily switch between localstack and the real AWS using the use_localstack terraform variable. Note that by setting the various settings to null we are telling terraform to use the defaults. If you are using more aws services, you will need to override the endpoints like we’ve done for lambda and iam here.

Now that you have this set up, create a backend configuration for localstack in your terraform directory, create a file called local.s3.tfbackend pointing at the s3 bucket we made earlier.

The force_path_style property is required to work with s3 locally without additional hosts configuration.

region = "eu-west-1"
bucket = "terraform-state"
key = "my-app.tfstate"
endpoint = "http://localhost:4566"
sts_endpoint = "http://localhost:4566"
force_path_style = true

You should now be able to initialise terraform

terraform -chdir=./terraform init -backend-config=./local.s3.tfbackend

3. Creating your infrastructure

Now you can start defining infrastructure in your main.tf file.

In this example I’m creating a lambda function with a simple function url.

data "aws_iam_policy_document" "assume_role" {
  statement {
    effect = "Allow"

    principals {
      type        = "Service"
      identifiers = ["lambda.amazonaws.com"]
    }

    actions = ["sts:AssumeRole"]
  }
}

resource "aws_iam_role" "iam_for_lambda" {
  name               = "redirects_lambda_role"
  assume_role_policy = data.aws_iam_policy_document.assume_role.json
}

data "archive_file" "lambda" {
  type        = "zip"
  source_file = "../index.js"
  output_path = local.archive_file
}

resource "aws_lambda_function" "redirect_lambda" {
  filename      = local.archive_file
  function_name = "my-function"
  role          = aws_iam_role.iam_for_lambda.arn
  handler       = "index.main"

  source_code_hash = data.archive_file.lambda.output_base64sha256

  runtime = "nodejs18.x"

  environment {
    variables = {
      NODE_OPTIONS = "--enable-source-maps"
    }
  }
}

resource "aws_lambda_function_url" "lambda_url" {
  function_name      = aws_lambda_function.redirect_lambda.function_name
  authorization_type = "NONE"
}

This will deploy the infrastructure for a lambda function from a file called index.js in the root.

exports.main = async (event) => {
  return {
    statusCode: 200,
    headers: {
      'Content-Type': 'application/json'
    },
    body: {
      message: 'Hello World!'
    }
  };
};

Now run terraform plan and apply your changes

terraform -chdir=./terraform plan -out terraform.plan && \
terraform -chdir=./terraform apply terraform.plan

You should now have your lambda function deployed.

4. Invoking your function

To get the url for your lambda function, inspect the state

terraform -chdir=./terraform state show aws_lambda_function_url.lambda_url

This will return you information about the resource including the function_url

# aws_lambda_function_url.lambda_url:
resource "aws_lambda_function_url" "lambda_url" {
    authorization_type = "NONE"
    function_arn       = "arn:aws:lambda:eu-west-1:000000000000:function:my-function"
    function_name      = "my-function"
    function_url       = "http://e6iu1sghet0ujaq6757ukxsjo6h50t90.lambda-url.eu-west-1.localhost.localstack.cloud:4566/"
    id                 = "my-function"
    url_id             = "e6iu1sghet0ujaq6757ukxsjo6h50t90"
}

you can then call the function

curl -v http://e6iu1sghet0ujaq6757ukxsjo6h50t90.lambda-url.eu-west-1.localhost.localstack.cloud:4566/ 

⚠️ Status code issu️e ⚠️

As good as localstack is, it’s not a 100% true implementation of AWS. At time of writing there is an issue where function urls will always return a 200 http status even when the lambda code explicitly sets a different status code. In a real AWS environment, the proper status code is returned.

4. Going live in AWS

Now that you have your infra and production code tested locally you’re ready to go to a real AWS environment. In your terraform directory create a new backend configuration called production.s3.tfbackend

region = "eu-west-1"
bucket = "terraform-state-production"
key = "my-function.tfstate"

Ensure you’ve created the state bucket in aws first

Reinitialise terraform for this backend

terraform -chdir=./terraform init \
  -backend-config=./production.s3.tfbackend \
  -reconfigure

Deploy your infrastructure setting the use_localstack variable to false.

terraform -chdir=./terraform plan \
  -var 'use_localstack=false' \
  -out terraform.plan && \
terraform -chdir=./terraform apply \
  -var 'use_localstack=false' terraform.plan

You should now be able to repeat exactly the same steps as you made locally to run your function.

References

• Terraform override local provider for use with localstack
• Localstack terraform documentation

node 18 provides the handy fetch() function which we can use to make HTTP requests to a remote server. But what exactly does fetch do? This is not a question we often ask ourselves but it’s a good opportunity to step down a level and appreciate the networking required to give us such a function.

When talking about networking, you’ll often here people mention the Open Systems Interconnection (OSI) model. This is a way of breaking down networking technologies into layers

7. Application
6. Presentation
5. Session
4. Transport
3. Network
2. Data link
1. Physical 

HTTP is a layer 7 or application protocol. It provides a standard way for clients and servers to communicate over a layer 4 transport protocol. Most commonly, that transport protocol is TCP although more recent evolutions of HTTP explore using other transport protocols such as QUIC to improve some of the limitations of TCP. The scope of this article is to implement a fetch function for HTTP 1.1 (not HTTPS) connections over TCP IP only.

With that little bit of theory out of the way, lets get going.

The task

We’ll be looking at implementing this piece of code.

const response = await fetch('http://example.com');
console.log(await response.text());

URL parsing

The url module can be used to parse the url. In order to write our fetch function we need to split this string:

http://example.com/?foo=bar#hash

into it’s parts

URL parsing

This is exactly what the URL constructor does.

import {URL} from 'url';
const url = new URL('http://example.com/?foo=bar#hash');

URL {
  href: 'http://example.com/?foo=bar#hash',
  origin: 'http://example.com',
  protocol: 'http:',
  username: '',
  password: '',
  host: 'example.com',
  hostname: 'example.com',
  port: '',
  pathname: '/',
  search: '?foo=bar',
  searchParams: URLSearchParams { 'foo' => 'bar' },
  hash: '#hash'
}

Note that this gives us everything except the port which we need to default to 80.

DNS lookup

Now that we’ve split the url, in order to make an http request over TCP IP, we need to have the IP address of the server we’re trying to connect to. We don’t typically remember website’s based on IP address this is where DNS and the dns module comes in handy. It allows us to convert a host name to an ip address.

import {promises as dns} from 'node:dns';

const dnsRecord = await dns.lookup('example.com');
//dnsRecord.address -> 93.184.216.34
//dnsRecord.family -> 4

address is the ip address we looked up and family tells us it’s an IPV4 address as opposed to IPV6

Connecting over TCP IP

The net module allows us to make TCP connections. It needs the ip and the port. The createConnection function returns a Duplex stream.

import {createConnection} from 'net';
const exampleDotComIp = '93.184.216.34';

const connection = createConnection(
  {port: 80, host: exampleDotComIp},
  () => {
    console.log('connected!');
  }
);

If everything went well, the callback function was called to log we have created a connection to the remote server.

Building our HTTP request

Now that we have an open TCP connection to example.com, we need to send it some data to initiate an HTTP request.

HTTP requests follow the follow the following format

• method - a valid HTTP request method (eg GET or POST)
• path - the pathname from our parsed URL object.
• version - the version of HTTP we are using (1.1 in our case)
• headers - key value pairs of the request headers specified separated by a :
• request body GET requests don’t typically have a body but if we were sending a POST request, this would follow the headers.

At a minimum the server requires us to specify the host header as HTTP allows multiple hosts to reside on the same IP address.

Here’s the request we’re aiming for.

GET / HTTP/1.1
host: example.com

import {createConnection} from 'net';
const exampleDotComIp = '93.184.216.34';

const connection = createConnection(
  {port: 80, host: exampleDotComIp},
  () => {
    const request = 'GET / HTTP/1.1\n' +
      'host: example.com\n\n';
    connection.write(request);
  }
);

We just made our HTTP request. However this isn’t much use to us because we’re not handling the server’s response in any way.

Handling the HTTP response

Just like HTTP requests, responses from the server also get sent in a standard format and includes the following information.

• The HTTP version that the server is responding with
• The HTTP status code of the response to indicate how the request was processed
• The response headers in the same format as the request headers above
• The response body

A successful response will look something like this

HTTP/1.1 200 OK
response_header_key: response_header_value

<DOCTYPE html>
<html>.....

Let’s take a second to go back the code we’re trying to implement.

const response = await fetch('http://example.com');
console.log(await response.text());

Now the http response is potentially very large, maybe even infinitely large. In order for fetch to provide an api that can be useful in all scenarios, it doesn’t read the entire response before resolving. Instead, it reads the status line and the headers then returns control back to the client. This allows the client to inspect the status and headers before deciding what to do with the response body.

To start with let’s focus on the first line and resolve the fetch() promise.

Reading the response headers

Our client code is not overly concerned with the status or headers. Lets start by just resolving once the headers have been read. Now the HTTP spec tells us that the headers end once we receive a carriage return and line feed (CRLF) sequence. This is represented by the bodyStartMarker variable. When the TCP stream emits a new chunk of data, we look at what has already been received and check if we have a CRLF sequence. If we do, we pause the stream and callback to the consumer with a response object containing the text function. This in turn resolves the fetch promise leaving the decision of what to do next up to the consumer.

const readResponse = (connection, onHeaders) => {
  const bodyStartMarker = '\n\r';
  let rawHeaders = '';

  connection.on('data', (data) => {
    for (let char of data.toString('utf8')) {
      const previousChar = rawHeaders.slice(-1);
      if (`${previousChar}${char}` === bodyStartMarker) {
        connection.pause();
        onHeaders({
          async text() {}
        })
        continue;
      } else {
        rawHeaders += char;
      }
    }
  });
};

const fetch = async (url) => {
  // parseUrl();
  // getIpFromDNSLookup();
  return new Promise((resolve) => {
    const connection = createConnection(
      {port: port, host: ip},
      () => {
        // sendRequest();
        readResponse(connection resolve);
      }
    );
  });
}

You might think that we can simply listen for the connection’s end event to tell us when the response is finished however this is not the case. This is because the TCP connection remains open after the response is fully sent. This is actually a great thing because setting up a TCP connection is expensive. Keeping it open allows clients to reuse the same TCP connection for multiple HTTP requests.

There is therefore one response header that we do care about. The content-length header tells us how many bytes the body contains. This is how we know when the full body has been sent and we can resolve the text() promise Later on.

Once we’ve received the headers we can try to parse the header before calling back so we have it for later. I’ve done this here using a regexp with a named group.

const parseContentLength = (rawHeaders) => {
  const pattern = /content-length: (?<contentLength>\d+)/i;
  return Number(
    rawHeaders.match(pattern).groups.contentLength
  );
};

const readResponse = (connection, onHeaders) => {
  const bodyStartMarker = '\n\r';
  let contentLength;
  let rawHeaders = '';

  connection.on('data', (data) => {
    for (let char of data.toString('utf8')) {
      const previousChar = rawHeaders.slice(-1);
      contentLength = parseContentLength(rawHeaders);

      if (`${previousChar}${char}` === bodyStartMarker) {
        connection.pause();
        onHeaders({
          async text() {}
        })
        continue;
      } else {
        rawHeaders += char;
      }
    }
  });
};

Reading the response body

Now let’s focus on the next line of our client code.

console.log(await response.text());

In our current implementation this will print undefined. What we need to do is resume reading the rest of the response data when the text() method is called and resolve it when we’ve read the number of bytes specified in our contentLength.

Our data event handler needs to do something different once the response headers have been sent to start writing the body so let’s introduce a headersRead flag. let’s also introduce a new body variable to store the body.

Once the headers have been read let’s make the text() function resume reading the rest of the data.

Once we’ve read the number of bytes specified in our contentLength we resolve the text() promise.

const readResponse = (connection, onHeaders) => {
  const bodyStartMarker = '\n\r';
  let contentLength;
  let onBody = () => {};
  let rawHeaders = '';
  let body = '';
  let headersRead = false;

  connection.on('data', (data) => {
    for (let char of data.toString('utf8')) {
      if (!headersRead) {
        const previousChar = rawHeaders.slice(-1);
        if (`${previousChar}${char}` === bodyStartMarker) {
          contentLength = parseContentLength(rawHeaders);
          headersRead = true;
          connection.pause();
          onHeaders({
            async text() {
              connection.resume();
              return new Promise((resolve) => {
                onBody = resolve;
              })
            }
          })
          continue;
        }
        rawHeaders += char;
        continue;
      } else {
        body += char;
        if (body.length === contentLength) {
          connection.end(() => onBody(body));
        }
      }
    }
  });
};

⚠️ Warning happy path only

We now have a working implementation of our fetch function. It should work for any HTTP 1.1 server over TCP returning a text based body. However a lot could go wrong here that we would want to handle in a real worlds implementation.

• The URL being invalid
• Using an unsupported protocol
• Failed DNS lookups
• The TLS connection failing or being unexpectedly dropped
• No response body (eg in a 204 no content response)
• All types of bad response data (missing/invalid content-length, headers never properly closed, body ending before we expect, etc)

We also should avoid leaking TCP connections and close the connection once it’s no longer needed.

I won’t go into the implementation of these in this post but it’s important to note them.

As a closing note, here’s our final implementation.

import {URL} from 'url';
import {promises as dns} from 'node:dns';
import {createConnection} from 'net';

const parseContentLength = (rawHeaders) => {
  const pattern = /content-length: (?<contentLength>\d+)/i;
  return Number(
    rawHeaders.match(pattern).groups.contentLength
  );
};

const readResponse = (connection, onHeaders) => {
  const bodyStartMarker = '\n\r';
  let contentLength;
  let onBody = () => {};
  let rawHeaders = '';
  let body = '';
  let headersRead = false;

  connection.on('data', (data) => {
    for (let char of data.toString('utf8')) {
      if (!headersRead) {
        const previousChar = rawHeaders.slice(-1);
        if (`${previousChar}${char}` === bodyStartMarker) {
          contentLength = parseContentLength(rawHeaders);
          headersRead = true;
          connection.pause();
          onHeaders({
            async text() {
              connection.resume();
              return new Promise((resolve) => {
                onBody = resolve;
              })
            }
          })
          continue;
        }
        rawHeaders += char;
        continue;
      } else {
        body += char;
        if (body.length === contentLength) {
          connection.end(() => onBody(body));
        }
      }
    }
  });
};

export const fetch = async (urlString) => {
  const url = new URL(urlString);
  const dnsRecord = await dns.lookup(url.hostname);
  return new Promise((resolve) => {
    const connection = createConnection(
      {port: url.port || 80, host: dnsRecord.address},
      () => {
        const request = `GET ${url.pathname} HTTP/1.1` + '\n' +
          `host: ${url.hostname}` + '\n\n';
        connection.write(request);
        readResponse(connection, resolve);
      }
    );
  });
}

The official react website has a tutorial to teach people how to use react. The idea is to build a tic tac toe (or naughts and crosses) game. This game also recalls every move that was made and allows you to return to any previous state.

I wanted to show not only that this exercise can be implemented relatively easily in in vanilla js, but also that we can learn from the way react encourages us to write code to write plain js in a nice way.

A word of warning, this code is not optimized for performance, I’m certainly not advocating that this quick and dirty solution can replace react in a large production based application with a lot of components. Hopefully it does make the reader think about whether they can justify the 801 npm dependencies that come bundled into create-react-app (the starting point for the react tutorial).

Setting up the project

1. In a new directory create an index.html with the following contents. Note the defer attribute on the script tag. This will defer the execution of your script until the content has loaded. It’s a neat way out of having to do something like document.on('DOMContentLoaded', () => {/*your code*/}.

<!DOCTYPE HTML>
<html>
    <head>
        <meta http-equiv="content-type" content="text/html; charset=utf-8" />

        <title>Tic Tac Toe</title>
        <link rel="stylesheet" href="./style.css" title="" type="" />
    </head>
    <body>
        <div id="container"></div>
        <script defer src="./script.js"></script>
    </body>
</html>

1. Add a file called style.css with the same content that react give you.

2. Add a file called script.js with console.log('tic tac toe')

3. Most systems should come with python installed, run python -m http.server to serve the current directory over http and visit http://localhost:8000 in your browser. Inspect the javascript console and make sure you see tic tac toe being logged to make sure your script is wired up correctly.

Making the grid

When creating elements using vanilla javascript I like to make a small wrapper function to help to define my markup in a declarative way. This function does the job for this task.

const element = (tag, options) => {
  const el = document.createElement(tag);
  options.class && el.classList.add(options.class);
  el.innerText = options.innerText || '';
  (options.children || []).forEach(child => {
    el.appendChild(child);
  });
  Object.entries(options.listeners || {}).forEach(([event, listener]) => {
    el.addEventListener(event, listener);
  });
  return el;
};

You can then use this to build your UI in a similar way with functions to define components as you might with jsx. This creates a status line and a 3 by 3 grid. Clicking on a square will fill it with an X.

const Status = () => element('div', {
  innerText: 'Next player: X',
  class: 'status'
});

const Square = () => element('div', {
  class: 'square',
  listeners: {
    click: (evt) => {
      evt.target.innerText = 'X';
    }
  }
});

const Row = () => element('div', {
  class: 'board-row',
  children: Array.from(new Array(3), Square)
});

const render = () => {
  const game = document.getElementById('container');
  game.appendChild(Status());
  for (let i = 0; i < 3; i++) {
    game.appendChild(Row(i));
  }
}

Reacting to state changes

Something that react encourages us to do is to pull the state up as close to the top of the application as possible. Let’s create a state object to hold the current state of the grid and the status. This is where the Proxy class comes in handy. Proxy allows us to react to changes (or attempts to access) our state object. The second argument to the Proxy constructor specifies an event handler for whenever a property is set on our state object. In this case, we set the property, then re-render the entire UI with the new state.

const stateHandler = {
  set(obj, prop, value) {
    obj[prop] = value;
    render();
  }
};

const state = new Proxy(
  {
    status: 'Next player: X',
    grid: [
      ['', '', ''],
      ['', '', ''],
      ['', '', ''],
    ]
  },
  stateHandler
);

Now of course this is not very efficient as we may be rendering everything just to change a single square. However for such a small tree, this isn’t really noticeable to the user.

Here’s what our Square component and our render method look like at this point.

const Square = (rowIndex, squareIndex) => element('div', {
  class: 'square',
  innerText: state.grid[rowIndex][squareIndex],
  listeners: {
    click: () => {
      const newGrid = state.grid;
      newGrid[rowIndex][squareIndex] = 'X';
      // re-set the grid property on the state proxy
      state.grid = newGrid;
    }
  }
});

const render = () => {
  const game = document.getElementById('container');
  game.innerHTML = '';
  game.appendChild(Status());
  state.grid.forEach((_, rowIndex) => {
    game.appendChild(Row(rowIndex));
  });
}

render();

Notice how in the click handler we are re-assigning state.grid. This will cause our set handler to be called and thus re-render the entire grid with the updated value.

Finding a winner

This calculateWinner function goes over the grid and checks if any of the players have a complete row or column. It’s not really the point of this post but it’s included for completeness.

const players = ['X', '0'];
const rowWins = (player, rowIndex) => {
  return state.grid[rowIndex].every(square  => square === player);
}

const colWins = (player, colIndex) => {
  return state.grid.every(row => row[colIndex] === player)
}

const diagonalWin = (player) => {
  return state.grid.every((_, index) => {
    console.log(state.grid[index][state.grid.length - 1 - index]);
    return state.grid[index][state.grid.length - 1 - index] === player
  }) || state.grid.every((_, index) => {
    return state.grid[index][index] === player
  });
}

const isWinner = (rowIndex, colIndex, player) => {
  return rowWins(player, rowIndex) || colWins(player, colIndex);
}

const calculateWinner = () => {
  for(let rowIndex = 0; rowIndex < state.grid.length; rowIndex++) {
    for (let colIndex = 0; colIndex < state.grid[rowIndex].length; colIndex++) {
      const winner = players.find(player => isWinner(rowIndex, colIndex, player));
      if (winner) {
        return winner;
      }
    }
  }
  const diagonalWinner = players.find(diagonalWin);
  if (diagonalWinner) {
    return diagonalWinner;
  }
  return null;
}

This means that by changing our status property on the state to keep track of the currentPlayer instead, we can now change the status line to indicate the current player and alternate turns between 0 and X. We now have a working, playable tic tac toe implementation.

const state = new Proxy(
  {
    currentPlayer: 0,
    grid: [
      ['', '', ''],
      ['', '', ''],
      ['', '', ''],
    ],
    history: []
  },
  stateHandler
);

const Status = () => element('div', {
  innerText: calculateWinner() === null ?
    `Next player: ${players[state.currentPlayer % 2]}` :
    `Winner: ${winner}`,
  class: 'status'
});

const Square = (rowIndex, squareIndex) => element('div', {
  class: 'square',
  innerText: state.grid[rowIndex][squareIndex],
  listeners: {
    click: () => {
      const newGrid = state.grid;
      // switch who's go it is based on the current player
      newGrid[rowIndex][squareIndex] = players[state.currentPlayer % 2];
      state.grid = newGrid;
      // increment the current player
      state.currentPlayer += 1;
    }
  }
});

Implementing the history

The history can be implemented using the same idea. We’ll add a history property to our state object and assign it to an empty array. Every time a move is made, in the Square click handler, we will add the previous state of the grid to the history array. One thing we need to be mindful of here is to take a deep copy of the grid rather than adding a reference to the grid array in the history.

const gridCopy = state.grid.map(x => x.map(y => y));
state.history = [...state.history, gridCopy];

We can then implement some new components to render the history and allow users to go back in time to any previous move in the game.

const Move = (move) => element('li', {
  children: [
    element('button', {
      innerText: move === 0 ? 'Go to game start' : `Go to move #${move}`,
      listeners: {
        click: () => {
          state.grid = state.history[move];
          state.history = state.history.slice(0, move);
        }
      }
    })
  ]
});

const History = () => element('ol', {
  children: state.history.map((_, i) => Move(i))
});

Database schema migrations are a tough part of working with databases whether you let your framework (eg: rails, django) manage your schema or you choose to do it yourself. We opted for a simpler set of scripts that applied all the sql files in a ./db/migrations directory in order to build up the schema. The directory listing looks something like this:

000-baseline.sql
001-add-sample-table.sql
002-add-pack-table.sql

When you want to change/modify an object, you simply add a new sql file with the next number in the list to make your change.

Postgres offers an official docker image to do exactly this task. We place the following Dockerfile in our ./db directory.

FROM postgres:10-alpine
ADD ./migrations /docker-entrypoint-initdb.d/
ENV POSTGRES_DB your_database_name

This image creates a new postgresql server with a database called your_database_name and runs through the migrations in /docker-entrypoint-initdb.d/ sequentially.

For local development, we use docker-compose to test our application with a fresh copy of our database schema in a known state.

version: "3.4"
services:
  db:
    build:
      context: ./db
  tests:
    build:
        context: .
    depends_on:
      - db
    links:
      - db
    command: npm test

This runs our tests against the freshly created local database.

Initially, when we were ready to go to the test environment we manually ran the latest migration, deployed our application and ran our smoke tests against it. If that went well, we repeated the process in production.

This worked up to a point, however, as our application and database grew, migrations became more tricky. We needed to think carefully about how they affected existing data, about users of specific database functions expecting a certain signature and generally about how to avoid downtime during migrations.

If migrations are tricky and need to be run in stages manually, you will undoubtedly sooner or later end up with diverging schemas where what’s in git doesn’t match what’s in the test environment and what’s in production.

This happened to us after releasing some new database orientated features and caused a couple of bugs in production that were hard to track down and were due to migrations being partially or not applied at all between environments.

To avoid this from happening again, we added migra to our CI pipeline. Migra allows you to compare 2 database schemas and output the code required to make them identical. Think of it like a diff for database schemas. When we push a change to an environment, We compare the schema we’ve tested against in docker with the environment we’re targetting.

docker-compose run tests migra \
  --unsafe \
  --with-privileges \
  postgresql://$user:$password@$host:$port/$database \
  postgresql://postgres:password@db/$database

If they are not identical, the build fails and outputs the SQL code needed to bring the environment’s schema in line with what the code is expecting.

From there, we have the choice to run the output code in that environment or to structure the migrations differently to prevent data loss. Most of the time, the migration that was used to modify the docker image for the local tests can simply be used.

Once the migrations have been run in the target environment, the build can be rerun and will continue to deploy the application code now that the database schema is up to date. This ensures that our environments remain in sync with each other and what we are testing against.