Basic Kubernetes for the SA

Understanding how Kubernetes works is all but essential for the modern System Administrator. In this guide, I'll walk you through containers, Docker, and then give you a basic understanding of how Kubernetes works. I'll assume an understanding of VMs as a baseline knowledge.

Everyone seems to start here, and it's a reasonable place to start. However, I will be going a little more in the weeds because it's important to understand how they differ as much as it's important to understand how they're similar.

At a very, very high level containers and VMs are quite similar. The basic idea is that you don't want to run every app on the same host machine for various reasons. Usually it's library conflicts, backup complexity, and overall management hassle. In a traditional VM world, you normally would create a VM to have it "accomplish a task" - usually host a web server for some app (or a few related apps) or run a SIEM tool or scanner. One VM for one task (or a few closely-related set of tasks). This same holds true for containers. One container is one application, task, etc.

Now let's climb down from our 50,000 foot view and start looking at the differences between containers and VMs. Most likely you've seen this picture (or something very similar) before, if you've ever looked into containers:

Let's break down what this actually means, since it's often a bit confusing for people that aren't as familiar with the Linux kernel.

As discussed in the "How a computer boots" section of Computers, the kernel is what people generally consider to be "the operating system" and that holds true here, as well. When this image references "OS" or "Operating System" it's referring to the Linux kernel. The main difference between a VM and a container is that a VM uses its own kernel while a container uses the kernel of the host it runs on.

This single, small fact entirely changes how VMs and containers operate. The kernel is, effectively, the interface between software and hardware, so something that "runs its own kernel" will expect there to be underlying hardware to interface with. This makes all the difference.

When you create a VM, you are emulating physical hardware for whatever operating system you install. A container, in contrast, uses the host's kernel and thus uses its host's hardware instead. It is also beholden to the host's quirks and other installed software that is not containerized.

Traditionally, you would fully emulate an entire hardware stack. A specific network card, a particular model of hard drive, etc. This means that the kernel (and the hypervisor) would need to deal with any bugs or quirks of each emulated hardware.

Paravirtualization uses virtualization-specific hardware that "tells" the guest VM that it is virtualized, eliminating vendor-specific bugs and often leading to performance improvements due to short-circuiting many code paths from guest VM to hypervisor.

Paravirtualized hardware uses **virtio** drivers, which are pre-installed into most Linux kernels. Windows, however, often requires the virtio drivers to be loaded and installed to function. Red Hat supplies an ISO with all the necessary drivers here if you need to use a Windows VM.

When people say "containers have much less overhead than VMs" this is what they mean. If you were to run a single app in a container, and that same app in a VM, the container would win out on resource usage because it's not hosting an entire kernel and OS on top of it. In practice, you can usually run many times the number of containers in your stack than you would otherwise be able to with a purely VM-oriented approach.

The downside of this approach is that containers do not have the same sort of access to their systems that VMs do. For example, a container cannot itself run privileged kernel functions whereas a VM can. Doubling down on this idea, the people who make the software that runs containers (Docker, Podman, LXC, etc) will also limit what the container can access and how many resources they can use by making use of things such as Linux namespaces and cgroups.

Docker is probably the most popular container engine available, as it's very easy to install and almost always "just works" right out of the box.

You can find a guide on installing it here:

The final, compiled container is a very standardized format and will work regardless of what container engine you use. You can run a container from "Docker Hub" made by a "Dockerfile" even if you use Podman, OpenShift, or whatever other software you selected.

Due to Docker's popularity, Dockerfiles are a very standard way to create a container image of your choosing. Dockerfiles are a way to describe what you want to do inside a container (how to create the container) and you can run a Docker-specific command to compile those containers into a format usable by any container engine.

To keep things simple, I will not be going into detail on how these work, but I will point out a few elements because they're handy to know when you want to put a container on a system.

I'll use the MariaDB 11.8-ubi Dockerfile for reference.

You'll notice the first line is FROM redhat/ubi9-minimal which tells us what kind of "OS" we're expecting to see. There's a few different popular ones used:

Generally alpine images are considered the smallest (in regards to disk space usage) but UBI images are also quite tiny. Small images are generally preferred, though obviously the smaller the image the fewer bits of software are installed in it, so they can be annoying to create and debug.

Down a bit farther, we get to VOLUME /var/lib/mysql which means the container expects /var/lib/mysql to be a volume that the container engine is giving it. This path is inside the container, since each container has its own little filesystem, so as the SA you should make note of this path and make sure it's mounted on the host in some way so you don't lose all your data every time it restarts.

Next, we see USER mysql which was created way up at the top with a RUN groupadd --gid 999 -r mysql && useradd -r -g mysql mysql --home-dir /var/lib/mysql --uid 999 - effectively, this means the container will run as UID 999 and GID 999, so any files it accesses should probably be owned by that UID and GID (this is important when doing things like mounting NFS volumes)

Finally, we see EXPOSE 3306 which tells us this container will expose the port 3306 and probably expects some form of communication to happen on it. We can "redirect" ports from container to host if needed. eg. if you run multiple MariaDB servers on a host you can redirect one of them to port 2560 so connecting to the host on that port connects to one of your MariaDB containers on port 3306.

Okay, so you've followed some tutorials and figured out that docker compose exists. You've got a few containers running in a VM with some volumes attached to them, which is great.

Kubernetes actually works very similar to docker compose, where you create a yaml file (or several yaml files) that define the behavior your want and then Docker (or Kubernetes, in this case) takes care of the rest.

Now: what happens when that VM restarts? All your containers go down while it reboots, yes? Well, that's not very highly-available of you, now is it? How can we solve this?

When people say "Kubernetes" they mean the Kubernetes "platform". Think of the word "Kubernetes" like the words "FTP" or "SSH". It's more of a protocol, RFC, or implementation guide with some backing code. You can't really "run" Kubernetes, same as you can't "run" SSH. You find an implementation of Kubernetes (popular ones include k3s or microk8s) and run that.

Kubernetes has the concept of "nodes" which are hosts (normally VMs) that run containers. There are two main types of nodes:

Largely, the control planes handle "container orchestration" for you and you don't need to worry much about it. You simply tell the cluster what kind of containers you'd like to run and it'll do the heavy lifting of figuring out which nodes will run what containers.

Now you may be thinking: This idea of running containers on whatever node happens to be available is nice and all, but there might be some problems with ports and volumes, right?

Yes. You would be correct. In a Docker environment on a single VM, you would simply add a volume or select a path on the host for the container to put its data. If the container moves from host to host, however, (such as in a Kubernetes cluster) that data would be lost. Similarly, how would the Kubernetes cluster know what ports need to be open on which hosts in order for things to communicate? Thankfully, there's a solution for everything.

But first, we need to break down the idea of a container (or application) into components.

A container (or any software, really) will have none, all, or some of these components:

Kubernetes provides ways to mess with all of these systems in almost any way you can imagine. Let's take the MariaDB container from earlier as an example. We'll go over the high-level ideas in each section before going into the actual implementation on an example app.

The MariaDB container puts the database data (the stateful data) in the path /var/lib/mysql. In Kubernetes, there exists the concept of a StorageClass which, as the name implies, allows you to create storage of some kind, using some sort of "storage provisioner" or "storage driver".

The default storage provisioner/driver in most Kubernetes implementations either 1) doesn't exist or 2) only creates storage "volumes" on the host the container is spun up on. A very simple-to-install "cloud-native" (ie. "designed for Kubernetes") storage driver is called Longhorn. It provides a storage driver that runs on every node, and copies data from node to node as needed. So, no matter where you container ends up running, it can always access the data it just had from any host. Longhorn provides a default StorageClass that is reasonably sane and does what you would expect, but, as always, you can add others or replace it as well.

Longhorn does not actually copy data in real time as the phrase "copy data from node to node as needed" would imply. Instead, it provides an iSCSI network which is load-balanced across the cluster that any node has access to. If the volume happens to live on the same node as the container, then great. If not, it simply uses the iSCSI network to communicate reads and writes to the nearest node that has the volume.

Longhorn stores these volumes in /var/lib/longhorn on every worker node in the cluster, so each worker node will need enough space in that directory to contain (potentially) every volume that every container would need - in a worst-case scenario where there's only one worker left in the cluster, anyway.

The default storage class does replicate data to 3 nodes, however, so if a node's Longhorn directory is corrupted or lost it can still rebuild the data from the other nodes. This process of "rebuilding" (or restoring data) is automatic, which is a commonality you will find between "cloud-native" applications.

Kubernetes provides three main mechanisms for this. These are, again, "pluggable" so they can be replaced with whatever third-party software you like best.

ClusterIP

A Service (as Kubernetes calls it) running with a ClusterIP is the default. Kubernetes will create a private IP address in an internal cluster-wide network, and will create a DNS record in an internal DNS server that points the name of the service to that IP.

Any container within the same namespace, regardless of which node it's on, can connect to this Service and any ports it hosts by simply connecting to <service-name>:<port> - eg. a Service named mariadb with port 3306 defined would be open as mysql://mariadb:3306 on any other container in the same namespace.

There is no way for a browser, app, or anything outside the cluster to directly communicate with a Service using a ClusterIP.

NodePort

A Service running in NodePort mode opens the specified port on every node in the cluster. Each node will then redirect traffic on that port to the container on whatever node happens to be currently hosting it.

The downside of this approach is that you can only have one app listening to this port at one time, across your entire cluster. eg. if you have two MariaDB containers only one of them can use this port at any given time. Sometimes this is fine, sometimes this is not.

LoadBalancer

A Service running in LoadBalancer mode will behave very similarly to an app running in NodePort mode, but instead of a port being opened on every node in the cluster a port is instead opened on an external service or IP address.

Depending on the load balancer being used and its configuration, this could mean all node IP addresses, but usually means some tertiary IP address somewhere. MetalLB is a common third-party implementation, but doesn't generally come installed by default in most Kubernetes implementations.

The downside of a LoadBalancer Service is that you will often need to configure whatever external load balancer you'll be using. The upside is that the configuration options between IPs, ports, and external services are effectively infinite.

Similar to a Docker container, some apps will allow configuration files to be mounted inside of them. To do this in Kubernetes, you would mount (map) a ConfigMap or possibly a Secret containing your configuration file to a particular file inside of the container. The idea is the same, just the implementation details change.

Most containers, as stated previously, will provide some level of configuration through environment variables. Often times this is easier, but it depends on the app. These are largely the same between Docker and Kubernetes, with the added benefit of Kubernetes offering Secrets which allow you to store sensitive environment variables outside of the cluster.

As per usual, many different types of software can provide different Secrets for Kubernetes (often called "secret controllers" or "secret operators"). My personal favorite is Infisical, but use whatever you like. Kubernetes offers a default implementation that you can manage via the command-line, but those are generally stored on-cluster so if you lose your cluster you lose your config keys.

In this case, Kubernetes provides a few different solutions. I'll list the three most common:

Pods

A pod is often called "the smallest unit Kubernetes will allow you to create" but I find that misleading. While technically true, that verbiage can lead to people conflating a Pod with the containers it runs.

When you create a file that runs your container, the "smallest" thing you can create is a Pod, which can actually house as many containers as you'd like. Most of the time it'll only be one container, but there are many situations that require more complex structures.

The downside of creating a container as a Pod is that once the container has stopped (for any reason!) it's done. The cluster will not attempt to spin it up again. This can be handy for one-and-done things like quick fixes or cron jobs (which are just fancy Pods that run on a schedule) but it's not useful for an app that you want constantly running.

Deployments

Enter: the Deployment. A deployment is, very simply, a pod that, if it terminates, restarts automatically. If your pod crashes, or gets moved, or whatever, it'll automatically start back up again as many times as you want. Usually, infinite.

A Deployment is the "standard" way to run a stateless app, but can be used for any kind of app or container. You can even scale deployments up or down dynamically (adding replicas) to allow for more traffic, and container version updates can be rolled across deployments with multiple replicas!

In k9s, names such as server-589574cb6b-wwh7x will be Deployments, since Deployment names are randomly-generated.

StatefulSets

Let's say you want to run a MariaDB instance, but you know that only having one database to connect to 1) isn't exactly highly-available and 2) is more prone to data corruption than two databases communicating together. The problem is that, if you decide to scale your MariaDB Deployment, you won't know the names of each of the servers so they can communicate with eachother without you needing to fiddle with configurations.

This is where a StatefulSet shines: It's effectively a deployment designed for stateful apps that rely on inter-communication. A DNS entry is created for each Pod (as per usual, even with a Deployment) but the names are static (ending in -0, -1, -2, etc) so it's extremely easy to configure inter-communication between apps.

A StatefulSet will also automatically create volumes for each of the containers entirely automatically, whereas a Deployment will not - those rely on you creating volumes yourself.

In k9s, names such as postgres-1, postgres-2, postgres-3, etc will be StatefulSets.

Volumes are often created in Kubernetes via a PersistentVolumeClaim, or PVC. You create a PVC, give it a name, and then tell your Deployment or StatefulSet to mount "the PVC with this name" to "this location".

With a Deployment, you create the PVC yourself and mount it. In a StatefulSet, you define some parameters for the PVC and it names and creates the PVC for each of the replicas you defined in the StatefulSet file.

Many Kubernetes implementations come with a default Ingress (reverse proxy) they prefer. Some use Traefik, some use Nginx, and I've heard of one or two that use HAProxy. All are - say it with me - pluggable, so if you don't like your current "ingress controller" then you can always swap it out for another one.

All of these controllers are often behind a LoadBalancer which they add to the cluster which redirects ports 80 and 443 to them - so no matter what node you point to with your web browser you can get to your web app containers that have Ingresses defined.

If you have a web app, for example, you can create an Ingress that uses the name my-app.local.net and the ingress controller will reduirect any web browser pointing at the cluster asking for the hostname my-app.local.net to the container the ingress belongs to.

Ingresses actually direct traffic to Services, but in the end you'll almost always be pointing a Service to a Deployment so it doesn't matter.

Okay, so we've looked at the high-level concepts of breaking a container down into its parts. How do we go from "concept" to "running in a cluster"?

We'll use Actual(Budget) as our real-world example, it meets all of our requirements and is a nice little start to self-hosted services as a self-hosted budget management app that integrates well with U.S. banking systems.

First, we need to look at the container and separate it into its constituant parts. After we do that, we can put it back together, piece by piece, on the cluster. So, what are the relevant parts of the container?

Well, they have a nice little "install on Docker" documentation page here so we don't have to go looking in the Dockerfile for all of the information we need (though we could do that as well).

From the documentation, we have the following parts that we need to piece together:

Okay, that's not too bad. Let's start with creating namespace and data volume.

apiVersion: v1 kind: Namespace metadata: name: actualbudgetnamespace.yaml

Okay, this one's pretty easy, but there's some critical things to know:

apiVersion

Kubernetes always requires an apiVersion which is a bit of misnomer. These are essentially "namespaces" for CustomResourceDefinitions or CRDs. For example, in my cluster I have a postgres system and a mariadb system installed which both provide a Backup CRD - but they don't conflict because the postgres Backup is deifned under postgresql.cnpg.io/v1 and the mariadb Backup is defined under k8s.mariadb.com/v1alpha1.

In this instance, Kubernetes itself provides Namespace so we just use v1 to say we want to use the default Kubernetes CRDs.

kind

This is the "kind" of CRD we're implementing in this file. In this case, we want to define a namespace so we use the Namespace kind.

metadata

This is where you name the thing you're wanting to define, as well as some other stuff that we won't worry about just yet.

metadata.name

This is pretty easy: The name of the thing you're defining. In this case, we want to create a namespace called "actualbudget" so that's what we name this Namespace.

In my cluster, I use Longhorn to create and manage volumes so data is always available to my containers regardless of what node they end up on.

apiVersion: v1 kind: PersistentVolumeClaim metadata: name: actual-server-data namespace: actualbudget spec: accessModes: - ReadWriteOnce resources: requests: storage: 2Gi storageClassName: clusteredactual-server-data-pvc.yaml

Let's break this down:

apiVersion

Similar to Namespace above, Kubernetes provides this resource so we just use v1.

kind

In this case, we want a PVC. The app stores data in /data but we need to mount that directory to a more permanent storage volume so we don't lose it. That's what a PVC does.

metadata.name

This is the name we'll give the the PVC. In this case, we named it actual-server-data - we'll need to remember this name for the Deployment later.

metadata.namespace

We want to create this PVC under the actualbudget namespace (in case have another PVC named actual-server-data somewhere else for some reason, they won't collide)

spec

Almost all CRDs will have a spec that allows you to define how they behave. This is sort of the "entry point" for any CRD that requires configuration.

spec.accessModes

PVCs have two common "access modes" available:

These names are, again, misnomers because they're not really "once" or "many".

The ReadWriteOnce access mode lets a single node mount and use the PVC at a time. The ReadWriteMany access mode allows multiple nodes to mount and use the PVC at a time.

So, why not always use RWX? Because it's inefficient and slow compared to RWO (in Longhorn) - there's much more contention and locking that needs to happen, even if you're only using one node.

spec.resources.requests.storage

Here we're defining the maximum size this PVC can grow to. You can always increase this value, but in order to decrease it you need to delete and re-create the PVC. Best to start small.

spec.storageClassName

I have a custom StorageClass named clustered which is a little more efficient and space-saving than the default longhorn storage class, but use whatever you want.

apiVersion: v1 kind: Service metadata: labels: app: actual-server name: actual-server namespace: actualbudget spec: ports: - name: "5006" port: 5006 targetPort: 5006 selector: app: actual-serveractual-server-service.yaml

apiVersion

Once again, Service is part of Kubernets so we only need v1 here.

kind

Here we're defining a Service. This means that we want Kubernetes to create an internal IP (remember: the default Service type is ClusteredIP) and we also want an internal DNS entry in this namespace that points actual-server to this Service's IP.

We'll attach this Service to a Deployment later, using the selector section.

metadata.labels

This allows us to label the Service so we can use it in a slector section in the Deployment later. We've chosen the key app and value actual-server for consistency but use whatever you want.

metadata.name

We're naming this Service actual-server - the internal DNS will automatically add this name and bind it to an internal IP address associated with it.

metadata.namespace

We want to make sure this stays in the actualbudget namespace so we don't have issues or clash later.

spec.ports

This is where we define what ports and protocols we're using. We're not doing any port translation (most often you won't) so we're matching the targetPort (the container's exposed port) to the defined port (what this Service will listen on) but, again, you can do anything you want here.

TCP is the default, but if needed we could add protocol: TCP to this port definition.

Also note that this is a list, so you can have many ports listening on this internal cluster-wise IP address if needed.

spec.selector

This is where we tell the Service to bind to anything (in this case a Deployment) with the label key app and label value actual-server.

apiVersion: apps/v1 kind: Deployment metadata: labels: app: actual-server name: actual-server namespace: actualbudget spec: replicas: 1 strategy: type: Recreate selector: matchLabels: app: actual-server template: metadata: labels: app: actual-server spec: containers: - image: actualbudget/actual-server:25.6.1-alpine name: actual-server env: - name: TZ value: Etc/UTC ports: - containerPort: 5006 protocol: TCP volumeMounts: - mountPath: /data name: actual-server-data livenessProbe: httpGet: path: / port: 5006 initialDelaySeconds: 120 periodSeconds: 30 timeoutSeconds: 10 failureThreshold: 3 volumes: - name: actual-server-data persistentVolumeClaim: claimName: actual-server-dataactual-server-deployment.yaml

There's a bit going on here, so we'll break it down again:

apiVersion

Here we're actually using apps/v1 but it's still defined by Kubernetes itself.

kind

We want this file to describe our Deployment so that's what we're going to use.

metadata.labels

Remember the Service? It wants to bind to anything with the key app and value actual-server so we're telling it to bind to this Deployment. That way, anything going to the internal IP and port defined in the Service will automatically direct itself toward this Deployment.

metadata.name

We're naming our Deployment actual-server for consistency.

metadata.namespace

We want this Deployment to be in the same namespace as everything else.

spec.replicas

This is a stateful app (using a PVC for data storage instead of an external service) so we do not want to have more than one instance running at a time. In some circumstances you can have multiple containers of an app running at once, and generally there's no reason not to when you can.

spec.strategy.type

The default "strategy" for handling eg. new versions in a Deployment is to do a "rolling update" (RollingUpdate) where one instance of the app will be updated and brought up at a time. Because our PVC can only ever be attached to one node at a time, this can cause problems where a new instance with a new version is spun up before the first (and only) instance is taken down, so the update stalls forever.

Ask me how I figured this out.

Here, we're telling Kubernetes that, for this Deployment, we want to take down all (one) of the existing instances before spinning up new versions.

spec.selector.matchLabels

Similar to Service we want to attach this Deployment to anything that matches the key app and value actual-server.

spec.template.metadata.labels

It's a bit annoying that you have to do this dance so many times in a file, but same as above. Attaching this Deployment to the Service.

spec.spec.volumes

All the way at the bottom of the file is where we actually attach our PVC we created earlier to this Deployment. Remember, we named this PVC actual-server-data so that's what the claimName uses. The name gives us the ability to change the PVC name when we use it in this file, but that's just confusing so we're keeping it the same.

You can also mount ConfigMaps and use emptyDirs here (for ramdisks), but that's all more advanced than we're going to do for now.

spec.spec.containers

Here is where we define the containers that will run, and in which order they will run. Each container has a few places you'll often want to change:

You can add an initContainers section just above the containers section (eg. spec.spec.initContainers) - this allows you to run containers before you run your main container. They'll use the same volumes defined in this Deployment, which allows you to do things like transform configuration files or wait for database servers to be online before moving on to the main show (the containers section).

apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: actual-server-ingress namespace: actualbudget annotations: cert-manager.io/cluster-issuer: "ipa" nginx.ingress.kubernetes.io/proxy-body-size: "1g" spec: ingressClassName: "nginx" rules: - host: budget.yggdrasil.home http: paths: - path: / pathType: Prefix backend: service: name: actual-server port: number: 5006 tls: - hosts: - budget.yggdrasil.home secretName: actual-server-tlsactual-server-ingress.yaml

I run RKE2, and the default Ingress is Nginx. k3s provides Traefik, and you can use/install whatever you'd like to your cluster.

Here's the breakdown:

apiVersion

Here we're using the Ingress CRD defined in networking.k8s.io/v1 which, while being a different namespace from the other stuff we've used, is still basically part of default Kubernetes.

kind

We want to define an Ingress, which tells the cluster to handle any traffic coming at it on port 80/443 and check the hostname we'll define later. If it matches, we're going to direct traffic to the Service we'll also be defining later.

metadata.name

We're naming this Ingress actual-server-ingress but it doesn't matter too much.

metadata.namespace

We definitely want this Ingress to be in the same namespace as everything else, or nothing will work.

metadata.annotations

"Annotations" are, once again, a bit of a misnomer in Kubernetes. You're not "annotating" - that's what labels do, really. In this case, the Nginx CRD looks for these annotations in any Ingress that uses it and configures itself accordingly. For this Ingress we want to be able to upload large files so we set the proxy-body-size to 1g.

We're also using a custom ClusterIssuer that I created with cert-manager. This issuer connects to my FreeIPA system via ACME and issues valid SSL certificates, so we don't need to worry about SSL or renewals. It's all handled for us.

spec.ingressClassName

Here we're telling the cluster that we want to use the nginx IngressClass, which, for my cluster, is the only one defined.

spec.rules

This is where we tell the Ingress the hostnames it should be listening for and where to direct the traffic. Note that this is a list, so you can define multiple endpoints here.

The short of this block of yaml is that we want people connecting to budget.yggdrasil.home to forward traffic to our actual-server Service on port 5006 (which, in turn forrwards to our actual-server Deployment on port 5006)

I've configured external-dns on my cluster to, once again, connect to my FreeIPA system and use ACME to modify DNS records as needed. When an Ingress is created it looks for the hosts defined within it and automatically created, deletes, and manages DNS records as needed.

spec.tls

Through the magic of a properly configured cluster-issuer we can automatically create and manage SSL certificates for any hosts that we define here. Note that this is, again, a list, but we're only after one DNS name. The secretName dosn't matter since we're not using the key anywhere, but if we needed a keyfile for eg. docker-mailserver we could mount this as a file for the container to use.

Okay, so let's send traffic from the browser on our local machine (far from the cluster) through the cluster and to this container. How does it look at each step?

First, we look up DNS and see where it points:

Looking good so far! Let's see if the SSL certs are working:

Also looking good! We have a valid cert at the end, there, and Kubernetes has labeled it as "ready" to use.

Okay, so what happens when we hit that endpoint? Well, first it hits the Ingress we defined earlier:

Looking good here, too. It's listening on all the worker nodes on 80/443 and recognizes that it's listening for budget.yggdrasil.home.

Okay, so it should be directing traffic to our Service which should have an internal, cluster-only IP address and DNs name. Let's check:

Excellent! We have our Service with its internal DNS name actual-server with a private IP that isn't part of my home network. It's also listening on port 5006/TCP which is great.

That Service will forward any traffic it gets to our Deployment since they should share the same selector so let's check that.

This all looks good, so let's see what happens when we go to it in a web browser:

It works!

Okay, it's been working for months, but you get the idea.

Now, go forth and build your deployments.

There's more to learn, like Helm charts and Kubernetes Operators, as well as GitOps systems like FluxCD or ArgoCD.

I'll leave it here, however, and let you explore with a few of my recommendations.

Start with microk8s. It's a one-node development cluster (runs on your machine) that's stupid easy to install, and if you break it you just bring it down and start up another cluster.

Grab this fork of it-tools. One of the tools available is a "Docker compose to Kubernetes" converter that can initially help you out and save you a lot of time later.

For things that use postgres, the cnpg operator is phenomenal and, once you figure out how operators work, is a total lifesaver. Note that the operator will drop built-in support for barman backups soon, and they recommend using the barman plugin (which isn't that different and there's a conversion guide).

For apps that use MariaDB, I would consider the MariaDB operator to be "adequate". Also supports backups, which are just-very-slightly simpler than postgres backups.

If you're in need of Redis for your app, try the Dragonfly operator. Just beware the whole "it'll fill up your PVC and then repeatedly crash" gotcha and use the --dbfilename= flag.

There's also a seaweedfs-operator if your app wants S3 storage and you don't already have MinIO or similar available (or don't want to use that).

cert-manager and external-dns will be lifesavers on a real cluster, and maybe trust-manager will help you as well (eg. if you're behind a corporate proxy) - Reflector, too, can be handy when you want those ConfigMaps containing all your certs to appear in every namespace.

Want a container that uses your GPU? Nvidia has an operator for that.

Kured is quite nice for handling automatic node reboots when needed.

Rancher (the folks behind k3s & RKE2) have a very helpful system-upgrade controller that updates the actual kubernetes software/implementation across your cluster (safely).