Yet another adventure in automation land

In this article we’re going to be using these tools:

  • AWS — EC2 + ASG and some other things
  • AWS EKS for our k8s provider
  • CloudFormation for managing secrets
  • Terraform for managing clusters
  • Python as our “glue” language, plus some jinja2 and boto stuff
  • Rancher as a sort of middleware layer to help us coordinate everything

User Story

Let’s say we have a customer called “SeaView.” SeaView is a company which produces vehicles of all sorts from trucks, to trains and even consumer vehicles like mini-vans. SeaView has an IoT device in each vehicle that sends telemetry to the cloud:

  • Blob data…

Now let’s have some fun with code! In this part I’m using Python3 to automate the creation of our saved searches, visualizations and dashboards.

tl;dr: here’s the code to make this happen:

I’m intentionally leaving in debugging code here ( warts and all! ) in case someone needs to learn with me. It’s important to me that people see that this was a work in progress.

Here’s the workflow for what goes on here:

  • Create the search saved_objects. Saved objects are things used by Kibana in dashboards and search results.
  • Create a different type of saved_object called a visualization. …

This is a two-part series on how we created an automated, centralized logging system at Renovo using BanzaiCloud’s FluentD operator. Our layout for domains and deployments ends up being quite a bit more complex than the average installation.

We currently have 5 Renovo domains ( like prod, demo, etc… ) 1 developer domain, and a collection of customer domains that get increasingly more complex the closer they get to the 5g / WaveLength deployments.

Here I will lay out our journey to centralized logging in two chapters:

  • Flows: this will include how we setup ES + FluentD on k8s and…

In this episode we’ll talk about a project to connect GitLab CICD with Atlasssian Confluence with a little python in between.

  • Use a YAML file to define which projects should be checked.
  • Connect to GitLab, look for the most recent, successful pipeline with a coverage value, if the value exists, record the coverage percentage value.
  • Gather up all the project data into a table structure, send the table to a Confluence page.

What we wanted was a dashboard, of sorts to help us see where we are with our code coverage stats for relevant projects. The relevant projects in this…

I just started using AWS EKS and found a few little gotchas that have been causing issues with the monitoring system. The software stack I’m using here:

  • AWS::EKS ( 1.14 )
  • Prometheus operator ( prometheus-operator-8.5.14 )
  • metrics-server from latest

Installing the metrics server:

Once everything is up, I create a tunnel to the prom frontend:

kubectl port-forward --address -n monitoring prometheus-prometheus-operator-prometheus-0 9090 &

When I load it, the kube-proxy’s are all in a down state:

monitoring/prometheus-operator-kube-proxy/0 (0/6 up)

The fix is to patch the kube-proxy deployment as such:

kubectl get cm -n kube-system kube-proxy-config -o yaml | sed "s/metricsBindAddress…

soIn the previous article, I created “phase 1” of the keycloak script:

This is our basic work to frame up the idea and get a handle on the key components for a new software service. Now we want to expand on the idea by cleaning up the code and adding a couple of new requirements.

Here’s the final version of the code to view:

Our new requirements that we want to bake in:

  1. Reformat the code to make it easier and a little more reusable.
  2. Pull the authorization information from AWS::SecretsManager, where our secrets are stored.
  3. Open a file containing…

It is sometimes the case that I’ll run into a software package that I’ve never even heard of and need to do something amazing with it. Up until about a week ago I had never heard of a software package called “Keycloak.”

It’s basically a canned oauth solution, like AWS::Cognito. My job was to write some automation that could bootstrap this software to our needs installing some basic clients and roles. This type of thing is actually pretty standard.

What I mean by “this type of thing” is that we’re going from a POC to something more robust. The POC…

Something that I’ve always noticed ( and people have often asked for ) is a way to do simple infrastructure.

As an operations engineer, so that I can keep my infrastructure consistent, use a single manifest to create many of the same things.

CloudFormation makes this very easy, however, it’s also possible to do this with less evolved tools like Terraform. We can use Terraform ( tf ) and workspaces to do this, and add gitlab pipelines to help with the automation.

Here’s the basic workflow:

  • Create a basic VPC/Subnet layout which expresses a basic layout
  • Create some workspaces in…

Exploring the space at the intersection of technology and spirituality.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store