egg82

The Two Generals

egg82 — Sun, 13 Oct 2024 00:00:00 GMT

Recently I've taken to getting Starbucks on my way to work. I do this every fall/winter season; as soon as the Pumpkin Spice Lattes hit the shelves I, without fail, head straight to my local Starbucks. Then winter rolls in and pumpkin fades to eggnog and I start buying those lattes too. Thankfully it's only ever fall and winter but it's certainly not a cheap way to breakfast.This has become more of a daily thing for this fall since they opened a location pretty much directly on my way to work last year, and so I've finally started using their app with their star rewards system. I still don't quite know how it works and I may never fully figure it out because I just don't care that much, but I'm pretty adamant about not leaving money on the table when possible. If I'm going to be getting drinks anyway, I might as well sacrifice some privacy for the sake of a free $8 latte every week or so.Last week, as I was on my way out the door I went ahead and loaded $25 onto the Starbucks card as I usually do when I'm low. Doing it this way doubles the stars you get, and money/table.This time, however, something unusual happened; just as I hit the "done" button (or whatever it is) I happened to, at that moment, go out of range of my wifi and hop onto cell data. The waiting/loading/spinny bar seemed to go on for several minutes before it finally completed and brought me back to the main screen with my total Buck$$ Dollars (or whatever they're called). But instead of $25, I wound up with $50. And, checking my bank account, I confirmed that, in fact, I had two transactions for $25 each withdrawn.This reminded me of the "Two Generals Problem" because I believe that problem was likely the root cause of this behavior.In computer science, we have a few different "unsolved problems" that, as the name implies, are currently unsolved- and they aren't expected to be solved any time soon. One of those problems is "The Two Generals". It goes something like this:Two armies, each led by a general, are stationed on opposite sides of a valley. They both need to coordinate an attack to defeat a common enemy in the valley, but the only way to communicate is by sending a messenger through the dangerous terrain, where the message could easily be lost or intercepted. Both generals know that attacking alone would lead to disaster, so they need to coordinate.One general sends a messenger proposing to attack at dawn. The messenger crosses the valley and delivers the message, but now the second general needs to send a confirmation. When that messenger arrives back, the first general wonders, "How can I be sure they received my message and will actually attack?" Another confirmation needs to be sent to that confirmation. This back-and-forth could go on forever, with no way for either side to be completely sure the other will follow through, making coordinated action impossible.I assume that Starbucks pays their developers well, but, as I mentioned, this is an unsolved problem. When I moved from wifi to cellular data it was in a perfect moment where the transaction had gone through (somewhere) but wasn't confirmed. (probably on the app?) When I let the app continue on and do its thing, it assumed a communication failure and re-tried the transaction. Both were successful, and now I have an extra Reward$* (or whatever it's called).It's a fun little reminder that everyone's human and that developers suck at their jobs- and it's not even their fault that they do.Sometimes I have thoughts. That is all.

Sudo - A rant

egg82 — Mon, 13 Jan 2025 00:00:00 GMT

Recently, I was shown this article from David Both titled "Real sysadmins don't sudo"Now, I disagree with the article almost entirely. First off, my issues are with definitions: What is a "real" sysadmin? For that matter, what is a sysadmin? The article hints at a partial definition in the middle with "They lull the users—who are really System Administrators—into thinking that they are..." This tells me that the author thinks anyone with the capability to become root is a sysadmin, and a "real" sysadmin is, at a guess, someone who doesn't make mistakes. If that's the case, I've never met a real sysadmin in my life.But that's not even what sudo is designed for.I'll start from the top.Quote To do so is a gross misuse of the sudo command. Sudo is developed to be flexible and adaptable to all environments. If you feel like you're misusing sudo, that's on you (or your environment maintainer) to fix.Quote Part of being a system administrator and using your favorite tools is to use the tools we have correctly and to have them available without any restrictions. No, it is not. We (sysadmins) work within the constraints of our environment and corporate requirements such as security. We use the tools we have, not our favorites. Sudo is, first and foremost, a security tool. Its sole purpose in life is to limit access to a potentially dangerous system.Quote I have a particular dislike for how the sudo facility is being used in some distributions, especially because it is employed to limit and restrict access by people doing the work of system administration to the tools they need to perform their duties. I sort-of agree with this on some level. Distrbutions include sudo as standard for three reasons: People will blindly copy/paste commands from the internet and running into "sudo: command not found" is not good UX Some software requires it to function It's a sane system that dosn't interfere with the operations of a nominal OS Relying on sudo as a crutch to prevent you from damaging your systems means you're not using the system as it was designed, and it will fail you.Quote We don’t need the training wheels in order to do our jobs. In fact, they get in the way. This is just pure gatekeeping. The idea here is that a "real" sysadmin doesn't need "training wheels" which is.. an interesting take on many levels. I don't even know where to begin here.Quote This is supposed to make the user a little more careful about issuing commands that need elevated privileges such as adding and deleting users, deleting files that don’t belong to them, installing new software, and generally all of the tasks that are required to administer a modern Linux host. No, that is not why distros include it. See above.Quote Forcing sysadmins to use the sudo command as a preface to other commands is supposed to make working with Linux safer. Using sudo in the manner it is by these distros is, in my opinion, a horrible and ineffective attempt to provide novice sysadmins with a false sense of security. [...] Partially correct. When someone takes the time to configure it properly, this is how it works. Cyber teams love it, because it enforces one of the basic tenants of least privilege. That's what sudo is. Least privilege. That's the whole point. It does nothing more and nothing less.Quote So let’s be clear about this—these distributions expect the user to perform all of the tasks of system administration. They lull the users—who are really System Administrators—into thinking that they are somehow protected from the effects of doing anything bad because they must take this restrictive extra step to enter their own password in order to run the commands. No. See above.This man has clearly never tried to login as the root user on a workstation. Remember all of those really terrible browser sandbox escapes that happened over the last some odd years and keep happening? You know, where a user loads a website and they're now infected with some 0-day? You want to run that as root?VLC also won't run as root- it performs a check at startup for this very reason.This is a person who obviously still lives in the times of hooking your Windows XP box directly to the internet.Quote The reasoning behind this is that of the lazy sysadmin; allowing the users access to a command or two that requires elevated privileges and that they use constantly, many times per day, saves the SysAdmin a lot of requests from the users and eliminates the wait time that the users would otherwise experience. NO. It is, once again, a security tool designed to make security teams happy. Its entire purpose in life is to enforce least privilege.Sudo even provides logging and session replays for security teams to review later.Quote The sudo facility was never intended to be used as a gateway for commands issued by a sysadmin. Actually, that's exactly why it was designed.Quote Unix never says please. [...] We sysadmins login as root when we need to do work as root and we log out of our root sessions when we are done. More gatekeeping. Though I might (?) see what's going on here. The author is saying that users are sysadmins and that "real" sysadmins log in as root directly- therefore everyone is actually super cool and special and should just log in as root all the time, because who needs to be able to track who did what and when?Sysadmin learns this one weird trick. Security teams hate him!Okay, but really: As I mentioned before, sudo is a security tool meant to enforce least privilege and perform audit logging for security review later if needed.If you don't need these functions in your environment, that's perfectly fine. The system is flexible and designed to conform to your environment, even if that environment is effectively just a single user on their home laptop. By default, that's what sudo targets. If you want more, you'll have to configure it to do more.That said, planting your flag on the "I am a real sysadmin and therefore sudo is a constraint that nobody needs" hill is an odd choice and I don't think I would die thereThat's it. That's my thoughts.

One-dimensional

egg82 — Fri, 18 Oct 2024 00:00:00 GMT

When you're writing a character for a book, it's common to "develop" that character. Give it a name, a backstory, ideologies, and a reason for living the life that it does. Without those elements, the character would be called "one-dimensional" - flat. Not realistic.Why is that?Every person on this pale blue dot has a story. They have a family of some kind, living or dead. Maybe they have pets. Maybe they work at the grocery store, or maybe they're going to college- or, maybe, they're living at the local nursing home. It's easy to relegate the people you walk by in your local coffee shop to background characters. People that exist for the 15 minutes that it takes to get your order, and then you never see them again.There's a lot of people in the world. To them, you're the background character; there for 15 minutes and then gone.This happens, I think, because it wouldn't be reasonable to expect a single person to care about everything they do and everyone they see. We automatically filter out things we consider "normal". If we constantly, actively felt the sensation of the shoestring in our fingers as we tied our shoes every morning it would take hours to get them laced up and we'd feel terrible after.The same thing happens with people: we generalize and make assumptions because otherwise we wouldn't be able to see another human being without hours or days of agonizing over every aspect of their life that we can't even know. These generalizations are easier to remember, anyway. They make things simple, and simple is easy and safe.It's like a really nice three-to-four-syllable chant. Short, thus generalized, thus easy to remember. Things that are easy to remember stick with you long past the point that you think you've forgotten them. The chant itself? Maybe. The idea behind the chant? Not until the day you die. You may forget where the idea came from, but the idea is still there. And surely that means the idea came from you, right? And ideas that come from you are rarely ever challenged by you.But I digress.Warning Once again I'll be talking about current US politics. This time, explicitly.Once again I'll be talking about current US politics. This time, explicitly. I've been thinking about a few different things lately and some of them feel a little nonsensical. One of the best examples I've heard recently goes something like this: "The GOP's abortion ban is an attempt to control women's bodies." and, when asked why they would want this: "They lack this control in their own lives and need to exert it somehow." This, I think, conflates a few points for the sake of simplicity. Again, simplicity is powerful because it spreads ideas quickly. It's a great tool for making ideas portable, and I think that, at least to some degree, this has truth to it. These statements are accurate enough for now while still being memorable.That said..There's a difference between a leader of a group and the everyman of the same group. Cynically, the leader of a group doesn't need to believe everything they say. Their job is to lead the group in the direction they want to go, not to be a truth-teller. The everyman of the group, in contrast, needs to believe in the leader. Whether that's trust, fear, cold calculation, or something else, people don't follow a leader they can't even pretend to believe in.To bring this back to the GOP/abortion thing and put it simply: if you drew a Venn Diagram of the people that support a ban on abortion, people that want to control women's bodies, and people that lack control over their lives and feel the need to control others, I think you'd find a relatively normal-looking diagram with three mostly-separate circles.Flattening the opposition - "the other" - into a single dimension helps people understand the point you're trying to make. The problem is that a lot of people are trying to make a lot of points right now, and largely we see and remember this one-dimensional caricature of the "the other". Obviously this is not a one-sided thing: the "woke left" is a popular phrase for a reason. Same with "snowflake". Now we have "Russian troll" or "Tankie" to dismiss someone off-hand for stating outlandish things. All memorable phrases that turn a complex idea into a solid single-dimensional point. Like a sharp blade.I'll try something.When confronted by someone on your position in a debate, it's safer to hide behind a mask of complication and nuance. It's easier to tell people that you're pro-life and that it's because of deeply-held religious beliefs. Or that you're concerned about the morality and ethics of potentially killing a living being. When, in reality, you didn't think about it that hard and you don't need to. You listen to your leader and your leader says that today's bad thing is abortion. And, really, why should you care anyway? It's not something that affects you and you'd just prefer if everything went back to the way it was when life was great and everything was simple. Before women had any real say in things and trans people existed.This, however, is too much for regular conversation to a crowd or to someone who believes the same things you do. Not only is it inconvenient to think about a group of people like this, it's difficult to create and maintain this kind of image at all. It destroys the flow of conversation and any point that you'd want to make. You can't keep creating new tangents in your story. People will think you're nuts.When talking to another person on the opposing side the context changes. I think people forget this. We bring along our generalizations and assumptions built by our leaders and echo chambers and let that become our opinions of the other person. People are hard. Communication is hard. We don't need to make it harder by assuming a one-dimensional character on the receiving end of our words.Just remember that everyone is people too.

Blame

egg82 — Mon, 14 Oct 2024 00:00:00 GMT

There a lot of difficult things in life. Extreme sports, public speaking, waking up in the morning.. the list goes on. Among the most difficult things is admitting you're wrong. Maybe it's human nature: a need to always be correct about your worldview. Or, maybe, it's a social thing: being wrong makes you lesser than your peers, if only in the moment. Maybe a little of column A and B? Or maybe I'm wrong on both accounts.I just know that, at least for me, admitting I'm wrong about something is one of the most difficult things that I do every day. I don't think it's a profound statement to say this is a shared thing among, at least, most of the rest of us. It requires re-evaluating your assumptions and views and understanding that what you previously knew to be true is no longer the case- and then making the choice to accept this new truth counter to your beliefs seconds or minutes ago. And then you get to tell people you previously "lied" to that you were incorrect.It's exhausting, especially when you're doing it often. And, let's face it, if you're doing any sort of knowledge-based work you're probably doing it often. The easier and safer thing is to reject this new piece of information with any sort of logic that you can. It may not be right, but it sure is a lot kinder to the soul for a short time. Especially when you desperately need things to be simple and not your fault.Warning From this point on, although I won't be mentioning it specifically, it'll be fairly obvious that I'm talking about current US politics.From this point on, although I won't be mentioning it specifically, it'll be fairly obvious that I'm talking about current US politics. The problem really starts to show when you keep rejecting this new information because it doesn't fit your worldview. Slowly, over time, your understanding of things drifts from reality and, months or years later, you find yourself in a position where you need to find some way to reject or deflect almost everything in order to keep your ideas of the world contained inside their snowglobe without being broken or contaminated by complications like the real world. A simple scene that never changes.I get it. I really do. As I mentioned earlier, learning new things counter to your previous beliefs is difficult. I still don't understand the idea of someone taking on a new name or identity halfway through their life. What I do understand, however, is respecting someone enough to acknowledge their desire to be called by a new name. The thing I learned, here, is that I'll never fully understand everything and that's okay as long as I know that I lack the understanding and that respecting someone means compensating for that deficiency.Sometimes people recognize this in themselves, and sometimes they can empathize with others enough that they recognize this same trouble in them. What comes next depends on what you do with that knowledge.For example, if you tell someone that, yes, their idealized, simplified version of reality is the truth and that they're being lied to by everyone else.. Well, confirmation bias kicks in and accepting your words is much easier than poking holes in their own crystal palace. If you tell that same person that the problem isn't them- that there is an "other" that is causing all of their problems, they'll be grateful to you for pointing that out. For acknowledging their beliefs and helping them solidify their defense against what they're certain, now, is the real issue.It would be so simple to build your own little version of reality. The world is endlessly complex and everything from technology to communities change so often. I mean, we all do this anyway, even unintentionally. We can't know everything at every second and we can't empathize with everyone even if we could. We have to build our own worldview shaped by what we've seen and experienced. It would be impossible to do anything else.All we would have to do is lock in that worldview and assume that nothing changes and that we know everything we need to know already.It all sounds nice and simple. Everyone loves simple. Simple is easy. Simple allows us to focus on important issues and our daily routine, and lets us direct our anger towards "the other" - who is never the same "other" as yesterday but that doesn't really matter because it was never the point. The point is that the world never changes and that if anything is wrong it's not our fault. It's that everything really is easier than what everyone else makes it out to be. The solutions are simple: It's always "the other"'s fault and if we just got rid of "the other" everything would be fine.It really would be so easy, wouldn't it? To stay so safe and warm in your snowglobe.Maybe that's just people being human.

Tmpfs

egg82 — Thu, 12 Jun 2025 00:00:00 GMT

Okay, hear me out: tmpfs actually has some weird and surprising behavior sometimes and it's worth knowing how it actually works and what happens when it doesn't work or isn't used in the places you expect.For those not in the know, tmpfs is used when you want to mount a directory to memory. It's an entire filesystem, but directly in memory. This has the added effect that, when you reboot your Linux machine, it's cleared out. You'll very, very often find that your /tmp directory is a tmpfs filesystem. While lesser-known, it's the same story with /dev/shm.One would assume that a tmpfs filesystem would have the same amount of storage space as you have RAM, but that's not quite correct.By default, all tmpfs filesystems will have half of your available RAM set as their maximum size. If you need to manually set the size, however, you can use the size mount option. eg.mount -o size=1G tmpfs /mnt/test For an fstab entry, it would look like this:tmpfs /tmp tmpfs defaults,size=1G 0 0 A single tmpfs filesystem can have a maximum size of +. This means you can have a tmpfs system larger than your maximum RAM, but doing this will absolutely cause you massive problems if it ever fills up.Danger Be careful about mounting too many tmpfs filesystems or setting their size too large - especially if your RAM is limited. In short: anything you put in /tmp will stay there until it's manually deleted or cleaned up by a sysemd-tmpfiles. The contents of /tmp don't get wiped automatically on reboot unless you clear it out manually or set up a systemd-tmpfiles rule to do so on a regular basis.Not-so-temporary, eh? In hindsight, this should have been obvious for me, but I really only started figuring this out after working with CIS benchmarks and STIGs which tell you to use a "real" directory or volume for /tmp.To set up systemd-tmpfs to automatically clean out /tmp on boot (the behavior you'd expect) do the following steps: Create a file,/etc/tmpfiles.d/tmp.conf, and add the following line to it: D! /tmp 1777 root root 0 D!says two things: 1) that this is a directory (D) and 2) that it should only ever be acted on during boot (!) /tmpshould be obvious. It's the location we want to affect 1777sets the permissions for the directory - the standard for/tmp root rootdefines the owner user and group 0is the maximum age of the file before it's cleaned. It can be eg.1d,10m,2h, etc but we use0here to clean up everything Apply our custom ruleset immediately: systemd-tmpfiles --clean /etc/tmpfiles.d/tmp.conf Note This isn't strictly necessary because it only removes files on boot, but it's a good habit to get into when editing your tmpfiles configs.
And you're done! It's easy enough to configure (the man page helps a lot) but it's just not obvious that you should configure it.The systemd-tmpfiles service is quite flexible. If, for example, you'd like to clean out yesterday's files on a regular basis, you can do something like this: Edit/etc/tmpfiles.d/tmp.confand add the following line below the existing one: D /tmp 1777 root root 1d Note Ownership and permissions set here override the previous rule we had. Dindicates a directory, but this time without the added!so it runs when it needs to rather than only during boot The rest is the same, until.. 1donly removes files that are at least 1 day old so we don't accidentally clean out something that's being used Again, apply the custom ruleset immediately: systemd-tmpfiles --clean /etc/tmpfiles.d/tmp.conf Danger You can do this even with a tmpfs-mounted filesystem to help keep long-running machines clean. Now, not only will /tmp be completely cleared out on boot, but it'll also stay clean by managing itself.Good question! How regularly is "regularly", anyway? Well, since systemd-tmpfiles is part of systemd it's safe to assume it has a service or timer of some kind. A quick systemctl status systemd-tmpfiles confirms this:[root@ansible ~]# systemctl status systemd-tmpfiles systemd-tmpfiles-clean.service systemd-tmpfiles-clean.timer systemd-tmpfiles.service systemd-tmpfiles-setup-dev.service systemd-tmpfiles-setup.service And it looks like there is, indeed, a timer. Presumably specifically one for running the --clean operation.[root@ansible ~]# systemctl status systemd-tmpfiles-clean.timer ● systemd-tmpfiles-clean.timer - Daily Cleanup of Temporary Directories Loaded: loaded (/usr/lib/systemd/system/systemd-tmpfiles-clean.timer; static) Active: active (waiting) since Thu 2024-10-17 01:01:29 UTC; 52min ago Until: Thu 2024-10-17 01:01:29 UTC; 52min ago Trigger: Fri 2024-10-18 01:16:40 UTC; 23h left Triggers: ● systemd-tmpfiles-clean.service Docs: man:tmpfiles.d(5) man:systemd-tmpfiles(8) Oct 17 01:01:29 ansible.yggdrasil.home systemd[1]: Started Daily Cleanup of Temporary Directories. We can look at the /usr/lib/systemd/system/systemd-tmpfiles-clean.timer referenced here to see what that looks like:[root@ansible ~]# cat /usr/lib/systemd/system/systemd-tmpfiles-clean.timer # SPDX-License-Identifier: LGPL-2.1-or-later # # This file is part of systemd. # # systemd is free software; you can redistribute it and/or modify it # under the terms of the GNU Lesser General Public License as published by # the Free Software Foundation; either version 2.1 of the License, or # (at your option) any later version. [Unit] Description=Daily Cleanup of Temporary Directories Documentation=man:tmpfiles.d(5) man:systemd-tmpfiles(8) ConditionPathExists=!/etc/initrd-release [Timer] OnBootSec=15min OnUnitActiveSec=1d This tells us that it will do a first run 15 minutes after a system boot. From that point, it runs once a day.So, any of your tmpfiles.d files that tell systemd-tmpfiles to clean out old files will run after 15 minutes, and then from there every day. The exact time depends on when (the time of day) the system was booted.
Harkening back to the note on computers for a minute, here, I'll remind you that all data (eg. files) are just a long string of bytes. For permanent storage, those bytes are stored on mediums like disks. In order to use those bytes you first need to load them into memory. To work on a file, for instance, that file (or at least part of it) needs to be loaded into RAM.
Well, when your filesystem is entirely in RAM already, it's pretty easy for the computer to load it up. For systems like Plex, it's recommended to have the transcode directory be a ramdisk (a tmpfs filesystem) for this reason. It's fast. Like, really fast.Another great reason is why systems have a /tmp directory in the first place: it's nice to have some temporary storage place that everyone can use for various files. Even systemd uses it for some of its stuff. Here's a RHEL system that's been freshly booted up:
Ultimately it's up to you to decide when and how they're used and cleaned. Linux provides some reasonable defaults that most users are used to - so, personally, I'd stick with those. To each their own, though!

SELinux

egg82 — Thu, 12 Jun 2025 00:00:00 GMT

SELinux (Security-Enhanced Linux) was created by the NSA to add stronger access control to Linux. It uses Mandatory Access Control (MAC) to define what each program can access, which means stricter security controls than traditional user permissions.SELinux needs to be configured properly before it can be enforced on a system.SELinux configuration is controlled by the /etc/selinux/config file. The uncommented lines should look something like this:SELINUX=enforcing SELINUXTYPE=targeted SELINUX=enforcing: Ensures that SELinux runs in enforcing mode. SELINUXTYPE=targeted: The most common policy, targeting a specific set of services. You can check the current status of SELinux enforcement with the getenforce or sestatus commands.We should also ensure that the bootloader is configured to load SELinux policies. As a handy trick, instead of directly editing GRUB's config file, you can use grubby to manage GRUB command line options.grubby --update-kernel=ALL --args="selinux=1 security=selinux enforcing=1" If you ever need to remove the parameters again, you can run this:grubby --update-kernel=ALL --remove-args="selinux=1 security=selinux enforcing=1" In either case, remember to update grub:update-grub And, finally, reboot the system for changes to take effect.SELinux operates in three different modes. Each mode determines how SELinux enforces its policies on the system.In this mode, SELinux is completely turned off. No policies are enforced, and the system operates as if SELinux is not installed. This should be avoided unless absolutely necessary, as it removes the security benefits of SELinux.
In other words, please stop disabling SELinux. That doesn't mean disabling it can't be handy and you should never do it, however.I often find it handy to run a quick setenforce 0 to drop to permissive mode while I work on installing new software to make sure any issues I encounter are due to the software or my own incompetence rather than SELinux itself. After the install is done, I will set the system back with setenforce 1 and verify everything is still working. Then, I use the SETroubleshoot daemon to check for and correct any errors.To disable SELinux for the current boot, simply run setenforce 0. To disable it permanently, modify /etc/selinux/config:SELINUX=disabled This is the default and most secure mode. In enforcing mode, SELinux policies are actively applied, and any actions not explicitly allowed by a policy are blocked.To enable SELinux for the current boot, just run setenforce 1. To enable it permanently, modify /etc/selinux/config:SELINUX=enforcing This should be your configuration for the vast majority of your system's life. If it's not, please re-evaluate your configuration and try to understand why and what you can do to fix it.In permissive mode, SELinux policies are loaded, but violations are not enforced, but they are still logged. This is useful for troubleshooting and fine-tuning policies without disrupting the system.To set SELinux to permissive mode for the current boot, just run setenforce 0 and ensure your boot parameters have SELinux enabled. To set it to permissive permanently, modify /etc/selinux/config:SELINUX=permissive This would be a very strange thing to do permanently, but it is just a tiny bit more secure than disabled at least, so there's that.SELinux policies (rules) can be adjusted using a few key tools. These allow you to manage permissions, contexts, and make temporary or permanent changes to how the system behaves.
setsebool is used to enable or disable SELinux Booleans. Booleans are toggles for various common SELinux settings (based on the software you have installed) that allow you to adjust the behavior of specific services without needing to change entire policies.To check the status of booleans, use getsebool -a or semanage boolean -l. The latter command gives more information as to what, exactly, each boolean is supposed to allow or disallow.To enable a boolean for the current boot:setsebool httpd_can_network_connect on To enable a boolean permanently:setsebool -P httpd_can_network_connect on chcon (change context) is used to change the SELinux security context of files and directories. This is one of the more frequent commands you'll be running when fixing eg. file access denials on a system.For example, to set the context for a web directory for the current boot:chcon -R -t httpd_sys_content_t /var/www/html And to set the default context for that same directory:semanage fcontext -a -R -t httpd_sys_content_t /var/www/html Note Fcontext doesn't actually apply the changes until you run restorecon, detailed below. It's changing the default context, not setting the current context. When SELinux denies an action, it logs the denial in the audit logs. audit2allow is a tool that can analyze those logs and generate rules to allow the previously denied actions.First, search for denials related to a specific process or service:ausearch -c 'process_name' --raw | audit2allow -M my_policy For example, to handle denials for the hostname command:ausearch -c 'hostname' --raw | audit2allow -M my-hostname Then apply the generated policy:semodule -X 300 -i my-hostname.pp restorecon is used to restore the default SELinux security context for files and directories based on the current policy. This is especially useful after restoring files from a backup or moving them to a new location, or when changing the default context via the fcontext subcommand described above.To recursively restore contexts for a directory:restorecon -Rv /var/www/html When SELinux issues arise, knowing how to troubleshoot them effectively is essential. SELinux provides various tools to help identify and fix problems.Several command-line tools are useful for troubleshooting SELinux: sestatus: Check the current status of SELinux, including the mode and policy being used.getenforceis similar, but only provides the current mode. getsebool: View the status of SELinux booleans, which can control specific behaviors. semanage: Manage SELinux settings, including file contexts and booleans. sealert: A helpful command provided bysetroubleshoot. Gives you human-readable summaries of SELinux denials. setroubleshootd provides real-time notifications and human-readable explanations for SELinux denials. It can be extremely helpful when diagnosing issues. As its name implies, it runs as a daemon, usually under the setroubleshootd system service.To install setroubleshoot:dnf install setroubleshoot After installation, SELinux denials will be logged in /var/log/messages with a summary and recommended fix. For example, you might see:SELinux is preventing /usr/sbin/httpd from name_connect access on the tcp_socket port 80. setroubleshoot will often suggest solutions, such as running setsebool to adjust a boolean or using audit2allow to create a policy exception. You can see these alerts and fix suggestions in a nice GUI (or CLI depending on your setup) by running sealert.
Sometimes, eg. after restoring your system from a backup or making significant changes to the system, SELinux contexts can become out of sync. You can fix this by relabeling the filesystem.To trigger a full system relabel on the next boot, create the /.autorelabel file:touch /.autorelabel Then reboot the system, and SELinux will automatically relabel the entire filesystem based on the current policy. This is also useful because it can be much easier than running a lot of restorecon commands.

Memory Management

egg82 — Thu, 12 Jun 2025 00:00:00 GMT

It's important for an SA to know exactly how memory works and is accessed on a system. From physical hardware all the way down to a program requesting a few bytes of memory for whatever thing it needs.In the spirit of that idea, we'll (somewhat quickly) tackle each "big idea" in memory management today. There's plenty of interesting history and technical details that we'll skim, but the idea is to hammer the main concepts that will actually help you discover, understand, and diagnose problems you encounter.RAM simply stands for Random Access Memory. So named, because, as opposed to SAM (Sequential Access memory) such as tapes, data inside of a RAM chip can be accessed in any order. The most common type of RAM in a computer is called Dynamic RAM, or DRAM. I'll continue on, here, assuming this type, but once you start getting to the OS and software level the type of RAM stops mattering much at all.The more important part to remember for these systems is Double Data Rate (DDR). The original DDR RAM was double the speed / data rate of the previous Single Data Rate (SDR) RAM. Then we had DDR2, which doubled the speed from DDR, DDR3 which doubled again, and now DDR4 and DDR5. RAM is quite fast these days, but as we'll discuss below it's still far, far slower than the CPU. **DDR**: 200-400 MT/s **DDR2**: 400-1,066 MT/s **DDR3**: 800-2,133 MT/s **DDR4**: 1,600-3,200 MT/s **DDR5**: 3,200-8,400 MT/s So, not quite double each generation, but close.Modern RAM uses something called a Metal-Oxide Semiconductor - a type of Field-Effect Transistor (or MOSFET) - to store individual bits inside its chips. The physical characteristics of a MOSFET mean that it requires a constant power source to keep the bits where they need to be. Once removed from power, MOSFETs rapidly lose their charge and they can no longer hold information. This means that all MOSFET-based memory (all RAM, realistically) is volatile memory and completely powering off a computer will often clear out the bits stored in the physical chips of RAM.We're going to dive right into how an OS (Operating System) gives programs running under it some bytes located in physical memory. Because of some clever decisions made by clever software developers throughout modern history, the OS doesn't simply hand a program a location of some bytes it can use on a physical stick of RAM. Instead, it does something called memory mapping which turns the RAM's physical address into a virtual memory address that can be arbitrarily shuffled around if needed.Memory management, as it turns out, is a bit of a complex system full of history and landmines. It's actually very interesting, but I'll be glossing over a lot and lying here and there to get to a few points that are more useful to an SA.The OS does not directly map physical memory to virtual memory. It was, originally, a chipset on the motherboard called an Input-Output Memory Management Module (IOMMU) which did the task. Then, the functionality was added directly to the CPU on the Platform Controller Hub (PCH) which introduced the concepts of a northbridge and southbridge. Finally, the PCH was fully integrated to the main "body" of the CPU and removed altogether. This last step happened decades ago so you'll be unlikely to encounter a CPU with a PCH unless you're really into old tech.These days, MMUs perform extra tasks like handling the L1, L2, and L3 caches on the processor, integrating memory protection on-die, and sometimes handling bussing bytes around on the motherboard.So, memory mapping is still a physical component of the CPU, but the kernel has a lot of control over that process and direct access to that controller. The MMU handles translating physical memory to pages, but the kernel turns those pages into virtual memory and decides when to create those pages.So, there's a part of the CPU die called an MMU (or sometimes IOMMU). The MMU takes large chunks of the physical memory (called pages) and provides those blocks (again, pages) to the OS. When the OS updates those pages with new data, the MMU maps that data to physical addresses on the RAM itself.
The process of storing and retrieving data from the RAM sticks is, compared to the processor's built-in memory storage (called L1, L2, and L3 cache), extremely slow. The processor is hundreds of times faster than any motherboard signal and the LN caches on the processor are very small. The trick for the OS (and its programs), then, is to manage this data in such a way as to utilize physical RAM (and especially disk) as little as possible.We're not quite done yet, though. The OS now has "pages" of memory, but these are huge blocks of effectively raw bytes in memory that the OS needs to now divvy out to its running processes. And, as mentioned earlier, doing the simple thing of just giving its programs direct access to these pages is both risky and inefficient.
Those "huge blocks" are typically 4kb (4,096 bytes) in size. Through the use of a system called hugepages, however, they can be up to 1GB (!!) large.So, what does the OS do? It creates what's called "virtual memory" which chunks out these pages into various sizes as-needed and hands them out to programs on request. The OS can then 1) protect these virtual memory regions more effectively and 2) move these regions from page to page without worrying about affecting the underlying program. The program, then, just sees a large, contiguous block of memory whereas the bytes in this block are stored all over the physical RAM.
This process of creating virtual memory means we can do neat things like the Copy-on-Write (CoW) memory management system which is standard in the modern Linux kernel.When a process fork()s itself, it is expected that the forked copy of the process will have a copy of the parent process's memory, often called "memory sharing" or "shared memory" - but it is not the same concept as "shared memory" in /dev/shm.Older UNIX systems (think: Bell Labs) copied the memory of the parent process into a region the child process could access. Modern kernels will point the child process at the parent process's memory region, but when the child (or parent) process wants to modify the parent process's memory somewhere, a copy of those bytes are created for the child process.To clarify: If the parent modifies its own memory, a copy is made and the child sees the **old** bytes. If a child modifies the parent's memory, a copy is made and the child sees the **new** bytes but the parent's memory region remains unaffected.The end result of both systems is that the child process has "access" to the parent's memory but can only modify that memory for itself. The newer CoW is much, much more efficient, however.Swap is one of those things that many, many SAs get wrong. There's a lot of old and just plain incorrect information about how Swap or vm.swappiness works, and those misunderstandings cause real performance impacts on real machines.
There's an excellent write-up by Chris Down, one of the developers of cgroup v2, on the common misconceptions of SWAP. It's worth the read and even has a tl;dr, but I'll summarize and simplify here.As mentioned earlier, pages are handed to the OS by the MMU and the OS can then do with those pages as it pleases. Under the Linux kernel, there are different "types" of pages. There's pages related to caching metadata about files, pages for holding code for your running programs, pages for kernel-specific operations, and more. By far the most common page type, however, is the "anonymous" page which holds memory used by programs themselves.Anonymous pages are called anonymous because they have no backing and thus that memory is unreclaimable. But, what does it mean to have a backing for a page, and what is unreclaimable memory?For most "types" of pages, the OS automatically backs them by storing a copy somewhere on more permanent storage, like the disk. This allows for placing non-critical or not-recently-used pages on a slower, longer-term storage medium and freeing up valuable RAM real-estate for things that need faster access. This process of taking pages off of RAM and putting them onto disk instead is called reclaiming and those pages are thought to be reclaimable.The most obvious result of this reclamation system is to allow for hybernation, which means a much faster boot time because most of the OS-specific stuff can be loaded back into RAM quickly instead of being re-initialized. All modern Windows OSes (from 10 onwards) do this when you power them off.Now, anonymous pages have no backing like most of the other types. This means that their contents are purely in RAM and there are no other copies anywhere. If the OS decides that a program hasn't accessed something in a while (or code for destructors is infrequently called) it has no option to put that data onto disk to free up physical RAM for other things.This is where Swap space comes in. It's a section of disk that the OS can use to back anonymous pages. Adding Swap to a system allows the OS to be able to move much more data around and free up valuable RAM for data that needs to move fast. This gives the kernel much more of a fighting chance to make the overall system as fast and efficient as it can.So, as Chris puts it:Quote Swap is primarily a mechanism for equality of reclamation, not for emergency "extra memory". Swap is not what makes your application slow – entering overall memory contention is what makes your application slow. This system of reclamation combines with the idea of the MMU's pages to introduce a new concept: page faults.When something happens, like the OS requesting a page that the MMU doesn't see in physical memory or even just a program requesting more memory causing the OS to load a page that doesn't exist yet, the MMU creates a system interrupt called a "page fault" which tells the OS to do something.In the case of the former (called a minor fault, where the page was likely just moved from memory to disk), the OS simply requests a page at that location and loads the page from disk into that section of RAM. For the latter (a major fault, where the page never existed in the first place), the OS needs to request a page to be created at that location and hand it to the program.Hardware interrupts are similar to software interrupts: they're basically communicating information to the kernel/program by biting ankles in a specific way that the kernel or program now needs to interpret before it can do anything else. As you can imagine, it's not the most efficient process on the planet.Either way, having **many** page faults indicates a problem, and ignoring your problems often leads to bigger problems.And, finally, some very quick facts about about vm.swappiness: It's asysctlsetting (can be set in/etc/sysctl.d) It's a number from 0 to 200 (inclusive) - yes, 200 Itbiasesreclamation between anonymous pages and other types of pages vm.swappiness = 50would give you a ratio of 50:150 (1:3) for anonymous page reclamation and "other" page reclamation, respectively A value of 100 would give you an even 50/50 chance of reclaiming anonymous pages and all other (reclaimable) pages
vm.swappiness = 0is aspecial casewhich essentially disables anonymous page reclaiming,exceptit uses a different system to the standard bias system vm.swappiness = 1is the lowest you can go while still keeping the biasing system The default value is usually somewhere around 60 When a program is run (ELF for Linux, PE for Windows), parts of the executable (called segments) are loaded into various regions of memory. Machine code, data, etc are all taken from the executable's file on disk and loaded into RAM before the executable actually runs.When a program requests memory during runtime (eg. malloc or declaring a new variable) the OS either finds space in an existing page or raises a major page fault and tells the MMU to create a new page in RAM for the newly-allocated block of memory.Remember that the OS only gives the program virtual memory, a slice of an existing page, which is itself a slice of RAM.Depending on the type of memory allocation requested, the OS will put the newly-allocated memory in the program's heap or stack.Stack memory is often very temporary and is used in function calls. It is LIFO (Last In, First Out) and uses a stack pointer in a relatively small contiguous set of memory to allocate and deallocate very small portions of its memory. This concept of a "stack pointer" is what makes the stack extremely fast, but means it is limited to LIFO.Basically, think of the stack like a very tall "stack" (aha!) of small sections of memory. The program keeps adding to the top of the stack, expecting to eventually get to the bottom of it by constantly removing the topmost section. Except that, instead of doing all of the work of adding to and removing from the top of this stack, the program keeps a little slider next to the stack that points to the current working location.
The stack is used only to store scoped variables, function parameters, and return addresses (to return to the previous function when done).
Ben Eater has a fantastic video on how the stack works if interested.In the example below, the variable a and the parameter n are both stored on the stack, as is the return addresses of the previous function that called the test function, and any functions that called those functions all the way to the beginning of the program's entry point.void test(int n) { int a = 10; return; } A "stack overflow", then, is an exception raised when the program's stack memory has filled up - the stack pointer has run through the entire stack memory region. Stack is non-expandable memory so it can, eventually, become full. Because each item on the stack contains very little data it can hold a lot of items, but it's not infinite and something like an endless recursive function can demolish it.Each program has access to heap memory, which is both much larger than the stack and is "infinitely" expandable. Well, until you hit the system's total physical memory available. Then, you run into fun things like the kernel's OOM killer, detailed below.Unlike the stack, the heap is designed to persist the entire lifetime of the program. It's used to hold the actual contents of complex variables (eg. variables holding objects) and anything held in a malloc'd region.In the example below, the pointer to the data variable is stored on the stack, and the actual value of the malloc'd region of memory that the data variable represents is stored in the program's heap.int *data = (int *) malloc(sizeof(int) * 50); // Allocate heap if (data == NULL) { // Allocation failure } // Data now points to somewhere in heap free(data); // Deallocate heap Here's the main differences between the stack and the heap:kg-card-begin: htmlkg-card-end: htmlAnd, of course, we can't go through allocation and deallocation of memory without mentioning memory leaks - which are memory regions that weren't properly deallocated; meaning they take up space but nothing points at them, so they sit there being useless until the program ends or is killed. These happen, as mentioned, when heap memory is allocated (eg. malloc) without being deallocated later (eg. free). Stack memory is always allocated and deallocated automatically by the OS, so "memory leaks" in the stack aren't really possible. The most similar problem on a stack would be a "stack overflow", as explained above.GC (Garbage Collection) is a memory management mechanism where the language runtime (eg. JVM or Java Virtual Machine for Java) automatically deallocates memory that is no longer referenced by the program, preventing that memory region from being "leaked". Java, C#, and Python utilize garbage collection extensively.This process helps reduce issues like memory leaks by managing heap memory automatically. A common argument is that a GC can introduce latency as it finds and cleans unreferenced objects, though in practice that argument has been outdated for years (see: G1GC, ZGC, etc) and code in GC languages is often faster than similar code in C++ due to a VM's internal management and chunking of memory which is often much faster than requesting new memory from the OS via raw malloc calls.**Direct memory allocation**In languages like Java, direct memory allocation (eg. ByteBuffer.allocateDirect()) allocates memory outside the traditional VM heap for native I/O operations. These heap memory allocations are managed manually and can bypass the Garbage Collector entirely. They're often used for communicating with native libraries, although allocation of very large chunks of memory outside the VM can be much faster than a similar operation within the VM.The downside, of course, is that these implementations can lead to the same problems as non-GC languages: memory leaks.A buffer overflow occurs when a program writes more data to a buffer (a contiguous block of memory) than it can hold. The excess data can overwrite adjacent memory, leading to unpredictable behavior and vulnerabilities.Programs use buffers to handle data like strings or chunked file data. When a program does not check the size of incoming data before writing it to a buffer, an overflow can occur.Here's a simplified example in C:#include #include void vulnerableFunction(char *input) { char buffer[10]; // small buffer strcpy(buffer, input); // no bounds checking printf("Buffer contents: %s\n", buffer); } int main() { char largeInput[] = "ThisInputIsWayTooLargeForTheBuffer"; vulnerableFunction(largeInput); return 0; } In this example, strcpy copies data from input into buffer, but since buffer can only hold 10 characters, anything beyond that will overflow into adjacent memory.But why can this lead to exploitation? What does an attacker actually take advantage of, here?In the above example, buffer is allocated on the stack, which, as explained above, is the same section of memory that contains the previous (calling) function's return address.Here's what the stack would look like, in this case:kg-card-begin: htmlkg-card-end: htmlAn attacker would take advantage of the size of the buffer and any padding around it to inject shellcode (malicious code) directly onto the stack, and then overwrite the return address (again, directly on the stack) to point to this new shellcode.So, a buffer overflow attack would do something like this:kg-card-begin: htmlkg-card-end: htmlIn some cases this will cause the program to crash, but in others the overflow exploit and shellcode can be crafted in such a way as to allow the stack to continue without interruption.So, what happens when a program (or set of programs) has used all of the available physical RAM and the system has no more Swap space left?We'll talk about the Out-Of-Memory (OOM) killer, which is a system in the Linux kernel designed to, as a last resort, start killing off processes that it deems necessary for its own survival.When a process is created, it is given an OOM score, ranging from 0 to 1000, which is constantly updated by the kernel. You can see this score for each process by simply viewing the /proc//oom_score file associated with the process.The process with the highest score, then, is killed first. That death is followed by the second-highest, and so on.For example, a firefox process running as PID (process ID) 7159 would have the file /proc/7159/oom_score with the file contents 677, which is the process's OOM score.You can adjust this score by simply putting your "score adjustment" number into /proc//oom_score_adjeg. to reduce PID 7159's score of 677 by 200 points (resulting in a total score of 477), you would run echo -200 > /proc/7159/oom_score_adjYou can put a range of values from -1000 to 1000 into the oom_score_adj file. A resulting oom_score of 0 effectively disables OOM killing for that process.The exact heuristics of this score are a bit more complex, but the basic idea is that the kernel doesn't want to kill important things or things that don't use a lot of memory.
It isn't quite as simple as "highest score dies first" but for all intents and purposes that is effectively the case.From man procfs:The badness heuristic assigns a value to each candidate task ranging from 0 (never kill) to 1000 (always kill) to determine which process is targeted. The units are roughly a proportion along that range of allowed memory the process may allocate from, based on an estimation of its current memory and swap use. For example, if a task is using all allowed memory, its badness score will be 1000. If it is using half of its allowed memory, its score will be 500. There is an additional factor included in the badness score: root processes are given 3% extra memory over other tasks.Note that the man page references a "badness" score, which is the old name for "OOM score".Essentially the kernel over-allocates memory to all of its processes, and tracks how much memory each process uses. If you have a computer with 4GB physical RAM, it might allocate a total of 5GB to all of its running processes, expecting most processes will not use 100% of their allocated memory. If the system ends up in trouble and the OOM killer is required, it will look at processes nearing or at their total memory allocation first.Usually, these processes are the "problem children" and are consuming way too much or simply leaking memory, so the system works as intended. Usually.

FreeIPA

egg82 — Thu, 12 Jun 2025 00:00:00 GMT

FreeIPA/IdM provide centralized identity, authentication, and access management for Linux environments. LDAP, Kerberos, DNS, centralized RBAC, etc in an extremely easy-to-install package.FreeIPA is Red Hat's open-source version of IdM. They're nearly identical in every way, but here's a quick list of differences between the two for posterity:kg-card-begin: htmlkg-card-end: htmlFunctionally, the two are equivalent, and commands, LDIFs, files/dirs, etc will behave the same on both. FreeIPA is based off of389-DS, which used to be called Fedora Directory Server. This will be important to know when Googling around, since 389-DS isnotthe same as OpenLDAP.
FreeIPA usesDNAfor automatic ID allocation for users/groups. It can be2000-19999,796600000-796799999, or other ranges depending on your version, install parameters, etc. Basically, well outside the normal Linux range so it should be easy to spot LDAP users with anidor similar.
FreeIPA's CA/PKI is based off ofDogtag, which is widely used even outside of IPA. Recommend getting familiar withTomcattroubleshooting.
FreeIPA usesBIND(BIND 9) for DNS, which, if configured through FreeIPA, stores DNS records in LDAP, which is automatically synced between masters. It's extremely important that you are at least familiar with the "big picture" overall design of BIND and where its config files are located if you're using DNS. Honestly, it's pretty easy- just dnf install freeipa-server freeipa-server-dns freeipa-client and then ipa-server-installIf you prefer a guide, I followed this one the first time I ever did this. It's quite detailed.

Install FreeIPA Server on Rocky Linux 9 / AlmaLinux 9 | ComputingForGeeksIn this article we discuss in detail the installation and configuration of FreeIPA Server on Rocky Linux 9 / AlmaLinux 9 system. FreeIPA is popular and widelycomputingforgeeks.comThis can happen after a fresh install or upgrade. It's not totally uncommon. Check that your DNA range is actually set and set it if needed. Use the link below.
https://rcritten.wordpress.com/2015/01/05/freeipa-and-no-dna-range/I've encountered this a few times. Sometimes the autofs service isn't fully configured or it fails and doesn't restart. Try the following steps:Run the following on each of the affected client(s):ipa-client-automount --unattended sed -i '/^\[Service\]/a Restart=on-failure' /usr/lib/systemd/system/autofs.service sed -i '/^\[Service\]/a RestartSec=5' /usr/lib/systemd/system/autofs.service systemctl daemon-reload systemctl restart autofs You should be able to get to your mounted directories.This is the default behavior for FreeIPA, and you have to configure BIND to respond to queries. Edit/etc/named/ipa-ext.confand add the following: acl "trusted_network" { localnets; localhost; 10.0.0.0/8; 172.16.0.0/12; 192.168.0.0/16; } Edit/etc/named/ipa-options-ext.confand changednssec-validation yes;todnssec-validation no;or else things likenslookup google.comwill return aSRVFAIL. If you don't care about external domains, then keep DNSSEC enabled. Edit/etc/named/ipa-options-ext.confand add the following: allow-recursion { trusted_network; }; allow-query-cache { trusted_network; }; Restart thenamedservice:systemctl restart named Also make sure your firewall isn't blocking port 53.This happens (a lot, actually) - it's probably fine. Normally it happens after an upgrade but sometimes it just happens out of nowhere, for no reason. Automatic LDIF backups are created by FreeIPA and stored in /etc/dirsrv/slapd--/ as .ldif.bak files.Most often what's happened is that your main dse.ldif file has been zeroed out (for whatever reason) and simply overwriting that file with its backup copy and restarting your dirsrv@- service will do the trick.Warning Be VERY careful about overwriting files in this directory. For example, that backup LDIF file is your **one and only** backup copy for the current master. Worst-case scenario, you can often restore from a recent backup of your VM and re-initialize from another current/working replica. IPA is fairly tolerant of mis-matched internal stuff even if it's pretty far out.Speaking of which, sometimes restoring from the backup copy partially works but things are out-of-sync with the other systems. In that case..If your masters are out of sync (you see discrepancies in return values when running commands and/or in the web UI) and/or one of your replicas has been offline for a while (a week or more, or thereabouts) it's recommended to re-initialize it from a working replica.Simply run the following:kinit admin ipa-replica-manage re-initialize --from Pretty easy!Yep, this is a BIND limitation. Thankfully this is pretty easy to work around because BIND supports multi-line strings which work perfectly fine instead.In order to take your very long string (eg. DKIM record) and turn it into a multi-line string, simply add spaces where you want new lines to be. Yep, that's it! As long as each line is < 255 characters BIND will take it.For example, you have the DNS record (a string) abcdefgh..xyz - you can turn it into a multiline string by simply doing abc def ghi...xyz. In this example, the first line will be abc, the second line will be def, and so on. When looking at this record externally (through tools like dig) you will find that there's no spaces and it's all put together on a single line. When BIND responds to a DNS query with a multi-line string, it puts it all back together into one line for the client.So, really, multi-line strings are a construct of BIND and not the DNS system. A solution to a limitation by the software.This is going to sound really obvious, but: have you tried rebooting the master/replica that the system you're working on is communicating with?Sometimes funny things happen that reboots can solve.The following services are important to proper operation and break the most often: dirsrv@- you'll need to use your own domain name. Tab-complete should help you out. named / named-pkcs11 BIND server. Uses different service names/files depending on your PKCS11 setup. autofs Used for auto-mountingauto.directandauto.homefiles configured through FreeIPA httpd Web server for FreeIPA web UI. Doesn't usually break, but I've had to follow its trail once or twice. Useful logs can be found in the following locations: /var/log/ipa/ FreeIPA logs. Unless your upgrade or install went bad (which it sometimes does) they're not super useful, but if you run into that specific situation then they become invaluable /var/log/httpd/ More useful than you'd think. If your issue also affects the web UI, you can trigger then and then check the logs for more detail /var/log/pki/ & /var/log/tomcat/ Alotof data to sift through, but it's very often useful when troubleshooting PKI/CA things /var/log/sssd/
I've only had luck searching through these whenenabling debug in the sssd service, but when they're useful they're very useful Useful commands for troubleshooting: ipa-replica-manage Mostly useful for re-initializing a replica, but it's saved me many times ipa-server-upgrade Sometimes your update gets borked. You can re-try it after checking the logs ipa-getcert Check on any certs you generated "manually" - sometimes it'll let you know if/how it failed (if it did) ipactl Useful for restartingallIPA master services gracefully (and checking their status) - sometimes all you need to fix something is a quick service restart
It doesn't take a whole lot of setup, and once it's done you can use things like cert-manager for Kubernetes to automatically manage and renew certs that would be generated by your CA.The general steps, as follows: Ononeof your IPA master servers (this only needs to be done once, ever, per individual cluster) kinit admin tsig-keygen -a hmac-sha512 acme-update >> /etc/named/ipa-ext.conf ipa dnszone-mod example.com --dynamic-update=True --update-policy='grant acme-update wildcard * ANY;' ipa dnszone-mod my.domain.tld --dynamic-update=True --update-policy='grant acme-update wildcard * ANY;' ipa dnszone-mod test.me --dynamic-update=True --update-policy='grant acme-update wildcard * ANY;' # Basically, any DNS zone (internal domain) you wish to enable ACME for, run the dnszone-mod command. Get your zone key so you can add it to the other masters (if needed): cat /etc/named/ipa-ext.conf On each of theotherIPA masters (again, only needed once ever- just for each master in the domain) kinit admin ipa-acme-manage enable # Edit /etc/named/ipa-ext.conf and add the TSIG key lines (the acem-update block) generated on the first master systemctl restart named Then, you just have to make sure zone transfers are allowed. For each domain you have:kinit admin ipa dnszone-mod .. --dynamic-update=1 ldapmodify -Y GSSAPI << EOF dn: idnsname=..,cn=dns,dc=,dc= changetype: modify replace: idnsAllowTransfer idnsAllowTransfer: key keyname; - EOF eg. for my system:kinit admin ipa dnszone-mod yggdrasil.home. --dynamic-update=1 ldapmodify -Y GSSAPI << EOF dn: idnsname=yggdrasil.home.,cn=dns,dc=yggdrasil,dc=home changetype: modify replace: idnsAllowTransfer idnsAllowTransfer: key acme-update; - EOF Now you can use your new TSIG key for automatic ACME cert generation and renewal on any zone you configured!Warning Keep this TSIG key secret. Anyone with the key can easily generate valid certs for any hostname on your domain. It's fairly easy to do, and the IPA client will take care of auto-renewal for you.As an example, I'll use the cockpit web UI. Simply run the following commands:kinit admin semanage fcontext -a -t cert_t "/etc/cockpit/ws-certs.d(/.*)?" restorecon -FvR /etc/cockpit/ws-certs.d rm -f /etc/cockpit/ws-certs.d/$(hostname -f).cert rm -f /etc/cockpit/ws-certs.d/$(hostname -f).key ipa-getcert request -f /etc/cockpit/ws-certs.d/$(hostname -f).cert -k /etc/cockpit/ws-certs.d/$(hostname -f).key -D $(hostname -f) -K host/$(hostname -f) -m 0640 -o root:cockpit-ws -O root:root -M 0644 What we're doing, here, is getting a kerberos ticket for our admin user so we can create certs, setting up SELinux to allow cockpit (and the IPA client) to access these certs, removing the self-signed certs that cockpit generated, and finally using ipa-getcert to request a new certificate for us. $(hostname -f) is just a trick to get the FQDN of the current host and name our certs appropriately (and get the right ticket for the host).Pretty easy! You can do the same sort of thing for any application. The ipa-getcert command will generate a certificate file and a private key for you and put them in any directory you want with any owner and permissions you want.Replication is extremely easy through the web UI. Log in as an admin, go to IPA Server -> Topology -> Topology Graph and use the interactive graph to add and remove CA and domain (LDAP) replication agreements between systems. It's practically drag-and-drop!

freeipa-sam is an extremely useful tool in creating and managing service accounts. Did you know FreeIPA supports special service accounts? It does!
Managing automounts (home dirs, public mounts, etc) is also extremely easy in the web UI. Log in as an admin, head to Network Services -> Automount and Add maps named auto.direct, auto.home, and auto.master to get started. In your new auto.master mount, Add a new key named /- and use auto.direct as the mount information. Similarly, Add a key named /mnt/home and use auto.home as the mount information.

Warning Note that many people use /home instead of something like /mnt/home for their automount home directory. Although this often works, it is inadvisable as it can cause problems with programs, provisioning systems, etc that expect /home to be a local filesystem, and confuses local accounts with non-local ones which leads to more cognitive load when troubleshooting. When creating new mounts in your auto.direct entry, your key should be something like /mnt/public and mount information should be a string similar to an fstab entry or manual mount with all of the accompanying flags.After plenty of trial-and-error on my own systems, I've discovered the following flags work quite well on my home system where there's a LACP (dual-)10-Gbit fiber pipe between the NFS storage system and the VMs mounting the locations:kg-card-begin: htmlkg-card-end: htmlOptionally, you can include the following:kg-card-begin: htmlkg-card-end: htmlYour auto.direct keys and mount information would look something like this:
The auto.home key and mount information is a bit special, in that it uses some special characters. see below:
The key is listed as * for all users, and the NFS mount location has an & sybol in it, which is where the user's UID is placed when mounting.As an example- in this setup, home directories are mounted on each system at /mnt/home/ and the NFS export location being mounted is /mnt/misc/home/ on the remote storage server.
Log into the web UI and go to Authentication -> Certificates. Here, you can issue a new certificate with an appropriate CSR, download a certificate (downloading your root CA cert with serial number 1 is advised because it's quite useful to have on hand), and revoke or expire certs.There's a few configuration options/sections here, so w'll go through them in a logical order. Log into the FreeIPA web UI and go to Network Services -> DNS -> DNS Global Configuration. The forward policy for an internet-connected IPA should probably be "Forward first" and your global forwarders would be whatever DNS server (IP addresses) you want your upstream DNS to be.Note Don't forget to save! Now, head to Network Services -> DNS -> DNS Zones. Here's where you'll find your internal domains- both forward and reverse. Generally, it's all pretty self-explanatory. Take a poke around. It's really difficult to screw up your DNS here, as you'll get plenty of warnings and errors if you try.Just remember that zones always end in a dot (eg. my.domain.tld would be my.domain.tld.) - same with CNAME records and a few others like some MX records.
Warning Split-horizon DNS is officially unsupported and is a massive pain to maintain. It's recommended to have an entirely internal domain instead. Remember to add any new DNS zones in IPA Server -> Realm Domains if they aren't already there.
Officially, FreeIPA/IdM doesn't support Windows systems. They assume you'll be using a real Active Directory (AD) setup for that and hooking the two systems together.That said, you can definitely get Windows login and even Kerberos to work with FreeIPA. It just feels like it's held together with bubble gum and duct tape- and, in some ways, it is. That said, it works pretty well and you can do some fancy stuff if you configure it correctly.And it's free!
So, yes - you can, in fact, add a Windows system to FreeIPA. While FreeIPA isn't a replacement for AD, with the help of pGina fork you can get pretty close with emulating at least its main functionality as a Windows directory. You'll want to use the newer forked version because the original doesn't have the configuration options we need for the specific 389-DS settings that FreeIPA uses.
pGina isn't well-documented, so these settings are almost entirely trial-and-error. I've managed to get most everything working the way I think they should, but Kerberos is still a little off and I still haven't gotten SMB shares working with TrueNAS Scale when they require Kerberos tickets. Linux clients work fine with NFS, but Windows has problems with SMB. Go figure.The way I usually do things is with two groups: windows-admins for people that belong in the Administrators group, and windows-users for those that belong in the Users group. You can map any LDAP group to any local group(s) with pGina, though.
First, create your Windows 10 system however you feel you should - just makes ure you're using a version that includes GPO (eg. Pro). Most settings changes and hardening won't affect pGina. For this example, I used Ameliorated with the AtlasOS playbooks after a quick base install (and update) on a VM. It just makes Windows a little lighter and nicer to use. No ads, less telemetry, some optimizations, etc.
Download and install pGina fork and run the configuration UI, which will be in your start menu. The UI should look something like this:
Here, the only thing I suggest changing is the NTP servers to use your local systems or your preference of NTP servers. You can change the MOTD and icon here for a white-label treatment if you'd like.
Note If you decide to change the icon, make sure it's a BMP (bitmap) or PNG in RGB format or it won't show up on the login screen. In the "Plugin Selection" section of the UI, we'll select the following: KRB5 Authentication LDAP Authentication Authorization Gateway Local Machine Authentication Gateway
And, in the "Plugin Order" section, we'll set KRB5 before LDAP before Local Machine in each section.

Warning LDAP, at least, needs to be ordered above Local Machine so it can set groups correctly. Now, go back to the "Plugin Selection" so we can configure each plugin.Double-click KRB5 (or select it and click Configure... at the bottom) to configure it. There's only one option, so fill out your Kerberos Realm here. Again, Kerberos doesn't seem to work, but it doesn't hurt anything to have it enabled. Or not. Up to you.
Just remember to hit "Save" and not just close the dialog box after.Same with LDAP. Double-click to configure. This is one of the important ones, since it'll be doing most of the heavy-lifting.Most of this is self-explanatory, but here's my settings anyway:
Note For whatever reason, TLS with a valid root certificate doesn't seem to work here even though every other system I've used seems fine with it.
Make sure your Search DN and Search password are set appropriately. I used freeipa-sam to create credentials for this.Similarly, your bind section (under authentication) should be set correctly.For FreeIPA (again, based on 389-DS) your user accounts are loated under cn=users,cn=accounts,dc=,dc= and groups are under cn=groups,cn=accounts,dc=,dc=.For the attribute converter section, set whatever feels appropriate. I set email to Email and cn to Fullname.
The authorization section controls who is allowed to log in, and that's the only thing it does. Keep that in mind, here.Under the authorization section, set your default to deny and add any groups that you'd like to be able to log in to the "allow" list. For FreeIPA, your "path" will be cn=users,cn=accounts,dc=,dc= and your filter will be something like (memberOf=cn=windows-users,cn=groups,cn=accounts,dc=,dc=). Make sure the search is set to "Subtree" (the default is "Base") and you should be good to go.There's a lot of options for configuration, here, so feel free to play around with it.Here's an example:
The checkboxes here give you some helpful info about what they do when you hover over them. My selections here aren't the most secure, so decide what you want within your own org. Try checking "Deny when LDAP authentication fails" and unchecking "Allow when server is unreachable" for more secure settings.Next, the gateway section controls what local groups the LDAP users are added to. That's it- the only thing it does.The configuration here is similar to the authorization section. Here's an example of adding the windows-users LDAP group to the local Remote Desktop Users group:
Note Remember the "Subtree" selection from the dropdown. Then, hit "Save" and you're done!Note You **can** also enable the "Change Password" section, here, but I've never actually tested this functionality. The settings I've used previously are: userPassword -> SSHA1 (Salted SHA1) shadowLastChange -> Timestamp days since '70 Remember that, if you want to enable this feature, you'll need to check the appropriate box in the "Plugin Selection" of the main UI. This is where you'd put down your SMB home directory information. If you can get it working, then great, but I haven't figured it out with TrueNAS Scale myself yet. If you're curious, here are my settings:
Apply, then save your changes to close the pGina configuration UI.Optionally, for a cleaner login screen, you can perform the following steps: ClickStart, and then clickRun. In theOpenbox, typegpedit.msc, and then clickOK. UnderComputer Configuration, expandWindows Settings, expandSecurity Settings, expandLocal Policies, and then clickSecurity Options. In the right pane, double click the settingInteractive logon: Don't display last signed-inand enable the option And you're set! Reboot your system and you should be able to log in as any of your LDAP users in the windows-users group - or whatever you had set up.
If you're planning on using RDP, make sure to skip RDP pre-authentication as pGina doesn't support it. To do this, add the following like to your RDP file or connection config on your RDP client:enablecredsspsupport:i:0 Note This setting is only for Windows RDP clients. Linux RDP clients will work automatically and don't need this. Additionally, for RDP servers running on Windows 10, run these registry edits on the RDP server machine (once, as administrator) to disable preauthentication for any users logging into it:reg add “HKLM\SYSTEM\CurrentControlSet\Control\Terminal Server\WinStations\RDP-Tcp” /v SecurityLayer /t REG_DWORD /d 0 /f reg add “HKLM\SYSTEM\CurrentControlSet\Control\Terminal Server\WinStations\RDP-Tcp” /v UserAuthentication /t REG_DWORD /d 0 /f You should be greeted by the regular Windows login screen when running your RDP clients to it instead of being asked for credentials right away. This lets pGina take over the login from there.The downside to this setup is that even unauthenticated users can shutdown or reboot your machine from the login screen, so let's fix that: ClickStart, and then clickRun. In theOpenbox, typegpedit.msc, and then clickOK. UnderComputer Configuration, expandWindows Settings, expandSecurity Settings, expandLocal Policies, and then clickSecurity Options. In the right pane, double click the settingShutdown: Allow system to be shut down without having to log onand disable the option Again, reboot the system and you should see these changes take effect.I'll say again that I never managed to get Kerberos fully-functional, but I think this gude gets you fairly close: logins work but they don't add the account to local groups automatically (you need pGina for that, or you need to do it manually) and SMB to TrueNAS Scale (when requiring a Kerberos ticket, eg. SMB home dirs) still isn't functional for whatever reason. If you're interested, though..First, create a host for your Windows box in the IPA UI.
Then, SSH into your IPA master and run the following:kinit admin ipa-getkeytab -s .. -p host/.@. -e aes128-cts-hmac-sha1-96,aes256-cts-hmac-sha1-96,aes128-cts-hmac-sha256-128,aes256-cts-hmac-sha384-192 -k krb5.keytab.. -P eg. for my system:kinit admin ipa-getkeytab -s heimdall.yggdrasil.home -p host/[email protected] -e aes128-cts-hmac-sha1-96,aes256-cts-hmac-sha1-96,aes128-cts-hmac-sha256-128,aes256-cts-hmac-sha384-192 -k krb5.keytab.desktop-4p5252j.yggdrasil.home -P Set some password that you'll remember later when you log into the Windows machine to set it up. No need to copy the generated keytab file anywhere, though.Now, log into the Windows 10 system and open up a command prompt (or PS window) as an administrator:ksetup /setdomain . ksetup /addkdc . .. ksetup /addkpasswd . .. ksetup /setcomputerpassword ksetup /mapuser * * eg. for my system:ksetup /setdomain YGGDRASIL.HOME ksetup /addkdc YGGDRASIL.HOME alfodr.yggdrasil.home ksetup /addkpasswd YGGDRASIL.HOME alfodr.yggdrasil.home ksetup /setcomputerpassword no-this-isnt-the-actual-password-i-used ksetup /mapuser * * Warning If you don't do this in an administrator cmd/PS window you're going to get some errors like "Failed to set dns domain info: 0xc0000022" which is cryptic and hard to Google Now, close the command prompt and perform the following steps: ClickStart, and then clickRun. In theOpenbox, typegpedit.msc, and then clickOK. UnderComputer Configuration, expandAdministrative Settings, expandSystem, and then clickLogon. In the right pane, double click the settingAssign a default domain for logonand chooseEnabled. UnderOptions, set the default logon domain to.(eg.YGGDRASIL.HOME) and also: ClickStart, and then clickRun. In theOpenbox, typegpedit.msc, and then clickOK. UnderComputer Configuration, expandWindows Settings, expandSecurity Settings, expandLocal Policies, and then clickSecurity Options. In the right pane, double click the settingNetwork security: Configure encryption types allowed for Kerberosand selectAES128_HMAC_SHA1andAES256_HMAC_SHA1 Then reboot, and you should be able to log in as any FreeIPA user (since there's no real controls on user mapping or groups with this method).Users can log in, but they won't be admins or automatically assigned to any groups. You'll need to do that manually on the machine itself. On the upside, maybe this will somehow work with SMB shares with some extra fiddling? It's not something I tested thoroughly, but I couldn't immediately get a ticket that worked with my own home directory on TrueNAS with this method.Additionally, the user that's created with this method is a different user than what pGina will create, so using both login methods at the same time may not be the wisest choice unless you and everyone logging into that machine know what's going on: pGina will create a user called eg.amason Kerberos will create a user called. eg.amason.YGGDRASIL No, notamason.YGGDRASIL.HOME- you'd think it would be this, but it's not If you can find a way to reconcile the two users, maybe that works for you. I don't know if I see a purpose to that, however, unless somehow you can get Kerberos tickets working with pGina by doing this.

Computers

egg82 — Thu, 12 Jun 2025 00:00:00 GMT

Note This is a refresher for troubleshooting computers. Skip this if you’re pressed for time, but I recommend reading at some point. There’s the old adage that "we taught rocks to think." It’s not that far off, honestly. Computers are just a pile of sand (silicon) and metal coaxed into "thinking" through sheer wizardry. While the underlying magic like photolithography is fascinating, you don’t really need to know that for SA work. What is useful is knowing that all computers - whether they’re a laptop, a car’s ECU, or your smart fridge - work with the same basic components:Danger If you can't find these, it's probably not a computer. kg-card-begin: htmlkg-card-end: htmlComputers all share these things, and the easiest way to understand them is like this:Think of the CPU as the brain, RAM as its short-term memory (it forgets everything if you turn off the power), and storage as long-term memory where it writes down everything it needs to remember later.Note Heat is just the consequence of trying to make rocks think faster than nature intended. It's inefficiency at its finest. kg-card-begin: htmlgraph TD; CPU-->RAM; CPU-->Storage; CPU-->I/O; CPU-->Power; I/O-->GPIO; I/O-->Ethernet; I/O-->USB;kg-card-end: htmlLet's break this down a bit and give some more concrete examples.PC/LaptopThis one’s easy: You have your CPU, RAM, disk drives, power supply, GPU, and a motherboard that ties them all together. Every piece needs the others to function.Smartphones: Computers in Your PocketIn a smartphone, the CPU, RAM, GPU and other (usually external) components get smooshed together in a tiny chip because space and power efficiency are everything. It’s not that they magically made volatile memory disappear - it’s just closer to the CPU for speed. Apple may claim it's revolutionary, but it's really just convenient.
Raspberry Pi: The Transition DeviceThe humble Raspberry Pi is a nice halfway point between your standard desktop and a smartphone. It’s basically a DIY computer, but just as capable as your everyday devices.Note The Wi-Fi chip on the Raspberry Pi is just another CPU with its own RAM! Even your network interface has its own mini-computer going on. TVsModern TVs are just smartphones with a bigger screen and fewer apps (thankfully). Samsung didn’t get rid of CPUs or RAM - they just hid them really well.CarsCars today (especially EVs) are essentially computers on wheels. They even have their own network! No, your car doesn’t have a Cisco switch lurking in the trunk, but it uses simpler protocols like I2C and SPI to keep things running smoothly.Danger Cars post-1990 usually have several computers communicating internally. It’s like a mini data center under your hood. MotherboardsEven motherboards now have their own mini-computers to manage things like power-on and testing components. That BIOS update you did? It’s just code for the motherboard’s brain. Fancy motherboards may even have multiple chips doing different tasks, like managing overclocking.Quote "All computers are fundamentally the same set of components. They may move parts around, but no matter if you're troubleshooting a laptop, desktop, or even a smart fridge, **the system is still a computer.**" Probably you, explaining this to someone else. In summary, no matter what you’re working on, remember that all computers function using these same basic parts. The details might change, and the packaging might look fancy, but fundamentally, you’re just teaching rocks to think.I once spoke to a developer who was stressed about storing custom data on disk because no file format fit their exact needs. At first, it seemed silly to me, but then I realized a lot of very technical folks might not fully grasp the connection between data and storage.So, what is a file, really? What makes a zip file different from a log file or an executable binary?Let’s start with this: What is data?A byte is the smallest unit of data a computer can handle. It’s just a series of 1s and 0s arranged in a particular order.Note A byte can store 255 distinct values. To store more, we just add more bytes. Bytes form the foundation of how all data is stored:kg-card-begin: htmlgraph TD; Bytes -->|1s and 0s| Data_Types; Data_Types["Data Types"] -->|Numbers, Text| Files; Files -->|Containers| File_Formats; File_Formats["File Formats"] -->|Magic Bytes, Headers| Programs;kg-card-end: htmlYou can think of bytes like Lego bricks - they can be combined in endless ways to represent any idea, whether it’s text, numbers, or even images. The cool thing is, bytes are just ideas, stored and represented in a format that computers (and programmers) agree on.
Standards like UTF-8 for text or IEEE 754 for floating-point numbers are ways to make sure everyone agrees on how to organize those bytes. Without these, your computer might not know what to do with the data!So, now we know what bytes are. But what’s a file?A file is a container that stores your bytes in a specific sequence. The operating system knows where the file begins and ends, and everything in between is the byte-based data that represents the file creator’s ideas.Streams of bytesWhether it’s a hard disk, flash drive, CD, or even a network, it’s all just a long stream of bytes. You can make a perfect copy of any file or storage device by copying the bytes.When cloning storage devices, tools like dd perform their task by copying these bytes directly from one device to another, much like taking a snapshot of the data byte by byte.When you insert a data CD or connect a USB drive, the OS reads bytes in key locations to figure out how the device is structured. That’s it! It’s just bytes!
Quote "It's bytes all the way down." So, what makes a zip file different from an executable or a log file? The answer is simple: it’s all about how the developer decided to organize the bytes.File Signatures and Magic BytesMost file formats have something called a "signature" or "magic bytes" at the beginning. These are the first few bytes (at the very, very beginning of the file!) that identify the file type.
kg-card-begin: htmlkg-card-end: htmlYou can use the file command in Linux to inspect the magic bytes and find out what kind of file you’re dealing with.$ file myfile.zip myfile.zip: Zip archive data, at least v2.0 to extract Danger The .pdf.exe malware trick works because file extensions are mostly meaningless to the operating system. It looks at the first few bytes to decide what the file is, not the extension! Here’s where things get interesting: You can invent your own file format. If you’re writing a program, you can choose how the bytes are organized. As long as the program knows how to read and write the data consistently, you’re free to structure it however you want!The same goes for transmitting data over a network. You can send and receive bytes in any order, as long as both ends agree on the structure.For example, this could be your custom file format's structure if you so desire:kg-card-begin: htmlgraph LR; Header["File Header"] --> Data_Block_1["Data Block 1"] --> Data_Block_2["Data Block 2"]; Data_Block_2 --> Footer["End of File"];kg-card-end: html
Recommended reading: This rant on the zip file format because it's hilarious and gives some more insight on how these things are made.All file formats - PDFs, Word documents, executables - were invented by someone who decided, "This looks like it could work," and started writing code to interpret and organize bytes that way. That’s all there is to it. It’s just people putting ideas into bytes and agreeing on how to read them.In Summary: Bytes: The smallest data units, a series of1sand0s. Files: Containers that store sequences of bytes. File Formats: Structures that define how bytes are organized in a file (like a blueprint). Magic Bytes: Special sequences identifying file types, such asPKfor zip files. From here on out, we’ll be focusing on "standard" computers - PCs, servers, and laptops. While microcontrollers and ASICs are cool, they have a different set of ideas and problems that guide them.So, you’re sitting in front of your computer - off. What really happens when you press the power button?We’ll go step-by-step, but don’t worry, not too slowly.kg-card-begin: htmlgraph TD; A[Power Button] --> B[POST]; B --> C[Bootloader]; C --> D[Kernel]; D --> E[Init System]; E --> F[System Services]; F --> G[Desktop/CLI];kg-card-end: html
The power button is a momentary switch (engineering-speak for "button"). When you press it, it’s wired into the motherboard and signals the start of the boot process.The motherboard performs a POST. This is a self-check where it verifies that all hardware components are compatible and error-free, like making sure the CPU is seated correctly and not about to burst into flames.Note When someone says "the computer is POSTing," it means the motherboard is running these self-tests. They're just looking at a screen saying something like "POST" or "checking memory". Servers can take a while to go through this. Once the system passes POST, it searches for a bootable device - usually a hard drive, SSD, or even a remote device (for network boot scenarios like PXE).UEFI boot process The system looks for a device usingGPT format. It checks the device for a bootloader in theEFI System Partition (ESP). This is typically located in\EFI\BOOT\BOOTX64.EFI. For Linux systems, this path is often in/boot/efi/EFI/BOOT/BOOTX64.EFI.
If it finds a bootloader, it launches it. For Linux, this is oftenGRUB, the menu that lets you select a kernel version, adjust startup parameters, etc. BIOS boot process The system looks for a device usingMBR format. It searches the device for a bootloader right after theMBR magic bytes. The bootloader is then called. Again, in Linux, this is usuallyGRUB. Danger The bootloader’s sole job is to find and load the OS. Windows, macOS, or Linux - it’s all the same here.
The bootloader now looks for the Linux kernel - usually a file starting with the name vmlinuz, often found in /boot. It can pass startup parameters to the kernel, and then hands full control of the system over to it. The bootloader is done at this point.With the kernel in charge, it mounts the root filesystem (and only the root filesystem). From here, it: Initializes hardware and drivers. Sets up the core system environment. It’s all coming together now! Once everything is ready, the kernel hands control over to the init system - either systemd (on most modern Linux systems) or init/initd (if you're stuck in the past).From here, it’s all about system services starting up, and before you know it, you’re staring at your desktop.

Basic Kubernetes for the SA

egg82 — Sat, 14 Jun 2025 00:00:00 GMT

Understanding how Kubernetes works is all but essential for the modern System Administrator. In this guide, I'll walk you through containers, Docker, and then give you a basic understanding of how Kubernetes works. I'll assume an understanding of VMs as a baseline knowledge.Everyone seems to start here, and it's a reasonable place to start. However, I will be going a little more in the weeds because it's important to understand how they differ as much as it's important to understand how they're similar.At a very, very high level containers and VMs are quite similar. The basic idea is that you don't want to run every app on the same host machine for various reasons. Usually it's library conflicts, backup complexity, and overall management hassle. In a traditional VM world, you normally would create a VM to have it "accomplish a task" - usually host a web server for some app (or a few related apps) or run a SIEM tool or scanner. One VM for one task (or a few closely-related set of tasks). This same holds true for containers. One container is one application, task, etc.Now let's climb down from our 50,000 foot view and start looking at the differences between containers and VMs. Most likely you've seen this picture (or something very similar) before, if you've ever looked into containers:Let's break down what this actually means, since it's often a bit confusing for people that aren't as familiar with the Linux kernel.
As discussed in the "How a computer boots" section of Computers, the kernel is what people generally consider to be "the operating system" and that holds true here, as well. When this image references "OS" or "Operating System" it's referring to the Linux kernel. The main difference between a VM and a container is that a VM uses its own kernel while a container uses the kernel of the host it runs on.This single, small fact entirely changes how VMs and containers operate. The kernel is, effectively, the interface between software and hardware, so something that "runs its own kernel" will expect there to be underlying hardware to interface with. This makes all the difference.When you create a VM, you are emulating physical hardware for whatever operating system you install. A container, in contrast, uses the host's kernel and thus uses its host's hardware instead. It is also beholden to the host's quirks and other installed software that is not containerized.Traditionally, you would fully emulate an entire hardware stack. A specific network card, a particular model of hard drive, etc. This means that the kernel (and the hypervisor) would need to deal with any bugs or quirks of each emulated hardware.Paravirtualization uses virtualization-specific hardware that "tells" the guest VM that it is virtualized, eliminating vendor-specific bugs and often leading to performance improvements due to short-circuiting many code paths from guest VM to hypervisor.
Paravirtualized hardware uses **virtio** drivers, which are pre-installed into most Linux kernels. Windows, however, often requires the virtio drivers to be loaded and installed to function. Red Hat supplies an ISO with all the necessary drivers here if you need to use a Windows VM.When people say "containers have much less overhead than VMs" this is what they mean. If you were to run a single app in a container, and that same app in a VM, the container would win out on resource usage because it's not hosting an entire kernel and OS on top of it. In practice, you can usually run many times the number of containers in your stack than you would otherwise be able to with a purely VM-oriented approach.The downside of this approach is that containers do not have the same sort of access to their systems that VMs do. For example, a container cannot itself run privileged kernel functions whereas a VM can. Doubling down on this idea, the people who make the software that runs containers (Docker, Podman, LXC, etc) will also limit what the container can access and how many resources they can use by making use of things such as Linux namespaces and cgroups.Note Docker, Podman, LXC, and others are just different "Container Engines" (as shown in the image above) which provide the software foundation to run containers on a host. All made by different folks, so just pick your poison. Docker is probably the most popular container engine available, as it's very easy to install and almost always "just works" right out of the box.You can find a guide on installing it here:

InstallLearn how to choose the best method for you to install Docker Engine. This client-server application is available on Linux, Mac, Windows, and as a static binary.docs.docker.comThe final, compiled container is a very standardized format and will work regardless of what container engine you use. You can run a container from "Docker Hub" made by a "Dockerfile" even if you use Podman, OpenShift, or whatever other software you selected.Due to Docker's popularity, Dockerfiles are a very standard way to create a container image of your choosing. Dockerfiles are a way to describe what you want to do inside a container (how to create the container) and you can run a Docker-specific command to compile those containers into a format usable by any container engine.To keep things simple, I will not be going into detail on how these work, but I will point out a few elements because they're handy to know when you want to put a container on a system.
I'll use the MariaDB 11.8-ubi Dockerfile for reference.You'll notice the first line is FROM redhat/ubi9-minimal which tells us what kind of "OS" we're expecting to see. There's a few different popular ones used: Alpine, which is based on Android Debian/Ubuntu UBI, which is based on Red Hat Generally alpine images are considered the smallest (in regards to disk space usage) but UBI images are also quite tiny. Small images are generally preferred, though obviously the smaller the image the fewer bits of software are installed in it, so they can be annoying to create and debug.Down a bit farther, we get to VOLUME /var/lib/mysql which means the container expects /var/lib/mysql to be a volume that the container engine is giving it. This path is inside the container, since each container has its own little filesystem, so as the SA you should make note of this path and make sure it's mounted on the host in some way so you don't lose all your data every time it restarts.Next, we see USER mysql which was created way up at the top with a RUN groupadd --gid 999 -r mysql && useradd -r -g mysql mysql --home-dir /var/lib/mysql --uid 999 - effectively, this means the container will run as UID 999 and GID 999, so any files it accesses should probably be owned by that UID and GID (this is important when doing things like mounting NFS volumes)Finally, we see EXPOSE 3306 which tells us this container will expose the port 3306 and probably expects some form of communication to happen on it. We can "redirect" ports from container to host if needed. eg. if you run multiple MariaDB servers on a host you can redirect one of them to port 2560 so connecting to the host on that port connects to one of your MariaDB containers on port 3306.
Okay, so you've followed some tutorials and figured out that docker compose exists. You've got a few containers running in a VM with some volumes attached to them, which is great.Kubernetes actually works very similar to docker compose, where you create a yaml file (or several yaml files) that define the behavior your want and then Docker (or Kubernetes, in this case) takes care of the rest.Now: what happens when that VM restarts? All your containers go down while it reboots, yes? Well, that's not very highly-available of you, now is it? How can we solve this?Note Yes, Kubernetes is the result of some folks asking the question, "how can we take the idea of containers and turn it into a high-availability platform?" When people say "Kubernetes" they mean the Kubernetes "platform". Think of the word "Kubernetes" like the words "FTP" or "SSH". It's more of a protocol, RFC, or implementation guide with some backing code. You can't really "run" Kubernetes, same as you can't "run" SSH. You find an implementation of Kubernetes (popular ones include k3s or microk8s) and run that.Danger If you ever get the question "what is k9s?" in an interview, it's a trap. Although it's similarly-named to the other Kubernetes implementations, it's actually a TUI that allows you to see and interact with any kind of Kubernetes cluster. Definitely worth using. Kubernetes has the concept of "nodes" which are hosts (normally VMs) that run containers. There are two main types of nodes: Workers These do the main body of work, running all of your containers that you actually want the world to see These are generally pretty beefy VMs with plenty of CPU and RAM allocated Control planes (previously masters) These communicate with other control planes and worker nodes to tell the worker nodes what to do and when to do it, like what containers go on which workers These generally only host a few containers for running and maintaining the cluster, so they don't need to be huge
You want an odd number of these to avoid thesplit-brain problem, and you usually want more than one Note Red Hat provides OpenShift and VMWare provides Tanzu. Both of these systems abstract these types of nodes away and runs containers directly on the hypervisors. No VM "worker nodes" and the control plane is the hypervisor cluster itself. You still want an odd number of hypervisors in a cluster to avoid split-brain. Largely, the control planes handle "container orchestration" for you and you don't need to worry much about it. You simply tell the cluster what kind of containers you'd like to run and it'll do the heavy lifting of figuring out which nodes will run what containers.Now you may be thinking: This idea of running containers on whatever node happens to be available is nice and all, but there might be some problems with ports and volumes, right?Yes. You would be correct. In a Docker environment on a single VM, you would simply add a volume or select a path on the host for the container to put its data. If the container moves from host to host, however, (such as in a Kubernetes cluster) that data would be lost. Similarly, how would the Kubernetes cluster know what ports need to be open on which hosts in order for things to communicate? Thankfully, there's a solution for everything.But first, we need to break down the idea of a container (or application) into components.Danger You will do this "breaking down" process a lot, for every new application or container that comes your way. **This is important**. A container (or any software, really) will have none, all, or some of these components: A place to store data An open port to communicate Configuration files and/or environment variables that perform configuration The actual app/container itself Danger Containers are much more likely to provide configuration through environment variables, but not all do - and, sometimes, the better option is a file (if the software provides both). Kubernetes provides ways to mess with all of these systems in almost any way you can imagine. Let's take the MariaDB container from earlier as an example. We'll go over the high-level ideas in each section before going into the actual implementation on an example app.The MariaDB container puts the database data (the stateful data) in the path /var/lib/mysql. In Kubernetes, there exists the concept of a StorageClass which, as the name implies, allows you to create storage of some kind, using some sort of "storage provisioner" or "storage driver".Note Every one of these systems in Kubernetes is pluggable, so if you don't like your current storage driver you can simply replace it with another (or if you need multiple provisioners you can do that too).
The default storage provisioner/driver in most Kubernetes implementations either 1) doesn't exist or 2) only creates storage "volumes" on the host the container is spun up on. A very simple-to-install "cloud-native" (ie. "designed for Kubernetes") storage driver is called Longhorn. It provides a storage driver that runs on every node, and copies data from node to node as needed. So, no matter where you container ends up running, it can always access the data it just had from any host. Longhorn provides a default StorageClass that is reasonably sane and does what you would expect, but, as always, you can add others or replace it as well.Longhorn does not actually copy data in real time as the phrase "copy data from node to node as needed" would imply. Instead, it provides an iSCSI network which is load-balanced across the cluster that any node has access to. If the volume happens to live on the same node as the container, then great. If not, it simply uses the iSCSI network to communicate reads and writes to the nearest node that has the volume.Longhorn stores these volumes in /var/lib/longhorn on every worker node in the cluster, so each worker node will need enough space in that directory to contain (potentially) every volume that every container would need - in a worst-case scenario where there's only one worker left in the cluster, anyway.The default storage class does replicate data to 3 nodes, however, so if a node's Longhorn directory is corrupted or lost it can still rebuild the data from the other nodes. This process of "rebuilding" (or restoring data) is automatic, which is a commonality you will find between "cloud-native" applications.Kubernetes provides three main mechanisms for this. These are, again, "pluggable" so they can be replaced with whatever third-party software you like best. ClusterIP NodePort LoadBalancer ClusterIPA Service (as Kubernetes calls it) running with a ClusterIP is the default. Kubernetes will create a private IP address in an internal cluster-wide network, and will create a DNS record in an internal DNS server that points the name of the service to that IP.Any container within the same namespace, regardless of which node it's on, can connect to this Service and any ports it hosts by simply connecting to : - eg. a Service named mariadb with port 3306 defined would be open as mysql://mariadb:3306 on any other container in the same namespace.Danger Namespaces are important in a cluster. They allow you to have multiple mariadb:3306 services defined without having them clash. There is no way for a browser, app, or anything outside the cluster to directly communicate with a Service using a ClusterIP.NodePortA Service running in NodePort mode opens the specified port on every node in the cluster. Each node will then redirect traffic on that port to the container on whatever node happens to be currently hosting it.The downside of this approach is that you can only have one app listening to this port at one time, across your entire cluster. eg. if you have two MariaDB containers only one of them can use this port at any given time. Sometimes this is fine, sometimes this is not.LoadBalancerA Service running in LoadBalancer mode will behave very similarly to an app running in NodePort mode, but instead of a port being opened on every node in the cluster a port is instead opened on an external service or IP address.
Depending on the load balancer being used and its configuration, this could mean all node IP addresses, but usually means some tertiary IP address somewhere. MetalLB is a common third-party implementation, but doesn't generally come installed by default in most Kubernetes implementations.The downside of a LoadBalancer Service is that you will often need to configure whatever external load balancer you'll be using. The upside is that the configuration options between IPs, ports, and external services are effectively infinite.Similar to a Docker container, some apps will allow configuration files to be mounted inside of them. To do this in Kubernetes, you would mount (map) a ConfigMap or possibly a Secret containing your configuration file to a particular file inside of the container. The idea is the same, just the implementation details change.Most containers, as stated previously, will provide some level of configuration through environment variables. Often times this is easier, but it depends on the app. These are largely the same between Docker and Kubernetes, with the added benefit of Kubernetes offering Secrets which allow you to store sensitive environment variables outside of the cluster.
As per usual, many different types of software can provide different Secrets for Kubernetes (often called "secret controllers" or "secret operators"). My personal favorite is Infisical, but use whatever you like. Kubernetes offers a default implementation that you can manage via the command-line, but those are generally stored on-cluster so if you lose your cluster you lose your config keys.In this case, Kubernetes provides a few different solutions. I'll list the three most common: Pods Deployments StatefulSets PodsA pod is often called "the smallest unit Kubernetes will allow you to create" but I find that misleading. While technically true, that verbiage can lead to people conflating a Pod with the containers it runs.When you create a file that runs your container, the "smallest" thing you can create is a Pod, which can actually house as many containers as you'd like. Most of the time it'll only be one container, but there are many situations that require more complex structures.The downside of creating a container as a Pod is that once the container has stopped (for any reason!) it's done. The cluster will not attempt to spin it up again. This can be handy for one-and-done things like quick fixes or cron jobs (which are just fancy Pods that run on a schedule) but it's not useful for an app that you want constantly running.DeploymentsEnter: the Deployment. A deployment is, very simply, a pod that, if it terminates, restarts automatically. If your pod crashes, or gets moved, or whatever, it'll automatically start back up again as many times as you want. Usually, infinite.A Deployment is the "standard" way to run a stateless app, but can be used for any kind of app or container. You can even scale deployments up or down dynamically (adding replicas) to allow for more traffic, and container version updates can be rolled across deployments with multiple replicas!In k9s, names such as server-589574cb6b-wwh7x will be Deployments, since Deployment names are randomly-generated.StatefulSetsLet's say you want to run a MariaDB instance, but you know that only having one database to connect to 1) isn't exactly highly-available and 2) is more prone to data corruption than two databases communicating together. The problem is that, if you decide to scale your MariaDB Deployment, you won't know the names of each of the servers so they can communicate with eachother without you needing to fiddle with configurations.This is where a StatefulSet shines: It's effectively a deployment designed for stateful apps that rely on inter-communication. A DNS entry is created for each Pod (as per usual, even with a Deployment) but the names are static (ending in -0, -1, -2, etc) so it's extremely easy to configure inter-communication between apps.A StatefulSet will also automatically create volumes for each of the containers entirely automatically, whereas a Deployment will not - those rely on you creating volumes yourself.In k9s, names such as postgres-1, postgres-2, postgres-3, etc will be StatefulSets.Volumes are often created in Kubernetes via a PersistentVolumeClaim, or PVC. You create a PVC, give it a name, and then tell your Deployment or StatefulSet to mount "the PVC with this name" to "this location".With a Deployment, you create the PVC yourself and mount it. In a StatefulSet, you define some parameters for the PVC and it names and creates the PVC for each of the replicas you defined in the StatefulSet file.Many Kubernetes implementations come with a default Ingress (reverse proxy) they prefer. Some use Traefik, some use Nginx, and I've heard of one or two that use HAProxy. All are - say it with me - pluggable, so if you don't like your current "ingress controller" then you can always swap it out for another one.All of these controllers are often behind a LoadBalancer which they add to the cluster which redirects ports 80 and 443 to them - so no matter what node you point to with your web browser you can get to your web app containers that have Ingresses defined.If you have a web app, for example, you can create an Ingress that uses the name my-app.local.net and the ingress controller will reduirect any web browser pointing at the cluster asking for the hostname my-app.local.net to the container the ingress belongs to.Ingresses actually direct traffic to Services, but in the end you'll almost always be pointing a Service to a Deployment so it doesn't matter.Okay, so we've looked at the high-level concepts of breaking a container down into its parts. How do we go from "concept" to "running in a cluster"?
We'll use Actual(Budget) as our real-world example, it meets all of our requirements and is a nice little start to self-hosted services as a self-hosted budget management app that integrates well with U.S. banking systems.First, we need to look at the container and separate it into its constituant parts. After we do that, we can put it back together, piece by piece, on the cluster. So, what are the relevant parts of the container?
Well, they have a nice little "install on Docker" documentation page here so we don't have to go looking in the Dockerfile for all of the information we need (though we could do that as well).From the documentation, we have the following parts that we need to piece together: Port 5006/TCP, which is the web UI /data(inside the container) which stores the database files and other persistent data Finally, the actual server itself Okay, that's not too bad. Let's start with creating namespace and data volume.Danger Remember: Namespaces are good and it's helpful to use them to logically separate your apps so they don't conflict on DNS names or ports. apiVersion: v1 kind: Namespace metadata: name: actualbudgetnamespace.yamlOkay, this one's pretty easy, but there's some critical things to know:apiVersionKubernetes always requires an apiVersion which is a bit of misnomer. These are essentially "namespaces" for CustomResourceDefinitions or CRDs. For example, in my cluster I have a postgres system and a mariadb system installed which both provide a Backup CRD - but they don't conflict because the postgres Backup is deifned under postgresql.cnpg.io/v1 and the mariadb Backup is defined under k8s.mariadb.com/v1alpha1.In this instance, Kubernetes itself provides Namespace so we just use v1 to say we want to use the default Kubernetes CRDs.kindThis is the "kind" of CRD we're implementing in this file. In this case, we want to define a namespace so we use the Namespace kind.metadataThis is where you name the thing you're wanting to define, as well as some other stuff that we won't worry about just yet.metadata.nameThis is pretty easy: The name of the thing you're defining. In this case, we want to create a namespace called "actualbudget" so that's what we name this Namespace.In my cluster, I use Longhorn to create and manage volumes so data is always available to my containers regardless of what node they end up on.apiVersion: v1 kind: PersistentVolumeClaim metadata: name: actual-server-data namespace: actualbudget spec: accessModes: - ReadWriteOnce resources: requests: storage: 2Gi storageClassName: clusteredactual-server-data-pvc.yamlLet's break this down:apiVersionSimilar to Namespace above, Kubernetes provides this resource so we just use v1.kindIn this case, we want a PVC. The app stores data in /data but we need to mount that directory to a more permanent storage volume so we don't lose it. That's what a PVC does.Note Remember: a PVC is just a volume with a name that we can reference later, in the Deployment for this app. metadata.nameThis is the name we'll give the the PVC. In this case, we named it actual-server-data - we'll need to remember this name for the Deployment later.metadata.namespaceWe want to create this PVC under the actualbudget namespace (in case have another PVC named actual-server-data somewhere else for some reason, they won't collide)specAlmost all CRDs will have a spec that allows you to define how they behave. This is sort of the "entry point" for any CRD that requires configuration.spec.accessModesPVCs have two common "access modes" available: ReadWriteOnce (RWO) ReadWriteMany (RWX) These names are, again, misnomers because they're not really "once" or "many".The ReadWriteOnce access mode lets a single node mount and use the PVC at a time. The ReadWriteMany access mode allows multiple nodes to mount and use the PVC at a time.So, why not always use RWX? Because it's inefficient and slow compared to RWO (in Longhorn) - there's much more contention and locking that needs to happen, even if you're only using one node.spec.resources.requests.storageHere we're defining the maximum size this PVC can grow to. You can always increase this value, but in order to decrease it you need to delete and re-create the PVC. Best to start small.spec.storageClassNameI have a custom StorageClass named clustered which is a little more efficient and space-saving than the default longhorn storage class, but use whatever you want.apiVersion: v1 kind: Service metadata: labels: app: actual-server name: actual-server namespace: actualbudget spec: ports: - name: "5006" port: 5006 targetPort: 5006 selector: app: actual-serveractual-server-service.yamlapiVersionOnce again, Service is part of Kubernets so we only need v1 here.kindHere we're defining a Service. This means that we want Kubernetes to create an internal IP (remember: the default Service type is ClusteredIP) and we also want an internal DNS entry in this namespace that points actual-server to this Service's IP.We'll attach this Service to a Deployment later, using the selector section.metadata.labelsThis allows us to label the Service so we can use it in a slector section in the Deployment later. We've chosen the key app and value actual-server for consistency but use whatever you want.metadata.nameWe're naming this Service actual-server - the internal DNS will automatically add this name and bind it to an internal IP address associated with it.metadata.namespaceWe want to make sure this stays in the actualbudget namespace so we don't have issues or clash later.spec.portsThis is where we define what ports and protocols we're using. We're not doing any port translation (most often you won't) so we're matching the targetPort (the container's exposed port) to the defined port (what this Service will listen on) but, again, you can do anything you want here.TCP is the default, but if needed we could add protocol: TCP to this port definition.Also note that this is a list, so you can have many ports listening on this internal cluster-wise IP address if needed.spec.selectorThis is where we tell the Service to bind to anything (in this case a Deployment) with the label key app and label value actual-server.apiVersion: apps/v1 kind: Deployment metadata: labels: app: actual-server name: actual-server namespace: actualbudget spec: replicas: 1 strategy: type: Recreate selector: matchLabels: app: actual-server template: metadata: labels: app: actual-server spec: containers: - image: actualbudget/actual-server:25.6.1-alpine name: actual-server env: - name: TZ value: Etc/UTC ports: - containerPort: 5006 protocol: TCP volumeMounts: - mountPath: /data name: actual-server-data livenessProbe: httpGet: path: / port: 5006 initialDelaySeconds: 120 periodSeconds: 30 timeoutSeconds: 10 failureThreshold: 3 volumes: - name: actual-server-data persistentVolumeClaim: claimName: actual-server-dataactual-server-deployment.yamlThere's a bit going on here, so we'll break it down again:apiVersionHere we're actually using apps/v1 but it's still defined by Kubernetes itself.kindWe want this file to describe our Deployment so that's what we're going to use.metadata.labelsRemember the Service? It wants to bind to anything with the key app and value actual-server so we're telling it to bind to this Deployment. That way, anything going to the internal IP and port defined in the Service will automatically direct itself toward this Deployment.metadata.nameWe're naming our Deployment actual-server for consistency.metadata.namespaceWe want this Deployment to be in the same namespace as everything else.spec.replicasThis is a stateful app (using a PVC for data storage instead of an external service) so we do not want to have more than one instance running at a time. In some circumstances you can have multiple containers of an app running at once, and generally there's no reason not to when you can.spec.strategy.typeThe default "strategy" for handling eg. new versions in a Deployment is to do a "rolling update" (RollingUpdate) where one instance of the app will be updated and brought up at a time. Because our PVC can only ever be attached to one node at a time, this can cause problems where a new instance with a new version is spun up before the first (and only) instance is taken down, so the update stalls forever.Ask me how I figured this out.Here, we're telling Kubernetes that, for this Deployment, we want to take down all (one) of the existing instances before spinning up new versions.spec.selector.matchLabelsSimilar to Service we want to attach this Deployment to anything that matches the key app and value actual-server.spec.template.metadata.labelsIt's a bit annoying that you have to do this dance so many times in a file, but same as above. Attaching this Deployment to the Service.spec.spec.volumesAll the way at the bottom of the file is where we actually attach our PVC we created earlier to this Deployment. Remember, we named this PVC actual-server-data so that's what the claimName uses. The name gives us the ability to change the PVC name when we use it in this file, but that's just confusing so we're keeping it the same.You can also mount ConfigMaps and use emptyDirs here (for ramdisks), but that's all more advanced than we're going to do for now.spec.spec.containersHere is where we define the containers that will run, and in which order they will run. Each container has a few places you'll often want to change: image The actual image you'll be pulling from Docker Hub, Github, Quay, etc name The name of the container (makes CLI work a little easier when debugging) env Any environment variables you'd like to configure. You can do cool things like insert Secrets here ports If the container has any exposed ports (eg. in our case5006) we'll want to define them here volumeMounts This is where we finally map our PVC to a directory inside of the container (and ConfigMaps, emptyDirs, etc) livenessProbe This is a neat little trick that Kubernetes picked up from Docker, where it'll constantly do an http request to "this container" at whatever port and endpoint you specify, and if it fails too often it'll kill and restart the container You can add an initContainers section just above the containers section (eg. spec.spec.initContainers) - this allows you to run containers before you run your main container. They'll use the same volumes defined in this Deployment, which allows you to do things like transform configuration files or wait for database servers to be online before moving on to the main show (the containers section).apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: actual-server-ingress namespace: actualbudget annotations: cert-manager.io/cluster-issuer: "ipa" nginx.ingress.kubernetes.io/proxy-body-size: "1g" spec: ingressClassName: "nginx" rules: - host: budget.yggdrasil.home http: paths: - path: / pathType: Prefix backend: service: name: actual-server port: number: 5006 tls: - hosts: - budget.yggdrasil.home secretName: actual-server-tlsactual-server-ingress.yamlI run RKE2, and the default Ingress is Nginx. k3s provides Traefik, and you can use/install whatever you'd like to your cluster.Here's the breakdown:apiVersionHere we're using the Ingress CRD defined in networking.k8s.io/v1 which, while being a different namespace from the other stuff we've used, is still basically part of default Kubernetes.kindWe want to define an Ingress, which tells the cluster to handle any traffic coming at it on port 80/443 and check the hostname we'll define later. If it matches, we're going to direct traffic to the Service we'll also be defining later.metadata.nameWe're naming this Ingress actual-server-ingress but it doesn't matter too much.metadata.namespaceWe definitely want this Ingress to be in the same namespace as everything else, or nothing will work.metadata.annotations"Annotations" are, once again, a bit of a misnomer in Kubernetes. You're not "annotating" - that's what labels do, really. In this case, the Nginx CRD looks for these annotations in any Ingress that uses it and configures itself accordingly. For this Ingress we want to be able to upload large files so we set the proxy-body-size to 1g.We're also using a custom ClusterIssuer that I created with cert-manager. This issuer connects to my FreeIPA system via ACME and issues valid SSL certificates, so we don't need to worry about SSL or renewals. It's all handled for us.spec.ingressClassNameHere we're telling the cluster that we want to use the nginx IngressClass, which, for my cluster, is the only one defined.spec.rulesThis is where we tell the Ingress the hostnames it should be listening for and where to direct the traffic. Note that this is a list, so you can define multiple endpoints here.The short of this block of yaml is that we want people connecting to budget.yggdrasil.home to forward traffic to our actual-server Service on port 5006 (which, in turn forrwards to our actual-server Deployment on port 5006)I've configured external-dns on my cluster to, once again, connect to my FreeIPA system and use ACME to modify DNS records as needed. When an Ingress is created it looks for the hosts defined within it and automatically created, deletes, and manages DNS records as needed.spec.tlsThrough the magic of a properly configured cluster-issuer we can automatically create and manage SSL certificates for any hosts that we define here. Note that this is, again, a list, but we're only after one DNS name. The secretName dosn't matter since we're not using the key anywhere, but if we needed a keyfile for eg. docker-mailserver we could mount this as a file for the container to use.Okay, so let's send traffic from the browser on our local machine (far from the cluster) through the cluster and to this container. How does it look at each step?First, we look up DNS and see where it points:

Looking good so far! Let's see if the SSL certs are working:

Also looking good! We have a valid cert at the end, there, and Kubernetes has labeled it as "ready" to use.Okay, so what happens when we hit that endpoint? Well, first it hits the Ingress we defined earlier:
Looking good here, too. It's listening on all the worker nodes on 80/443 and recognizes that it's listening for budget.yggdrasil.home.Okay, so it should be directing traffic to our Service which should have an internal, cluster-only IP address and DNs name. Let's check:
Excellent! We have our Service with its internal DNS name actual-server with a private IP that isn't part of my home network. It's also listening on port 5006/TCP which is great.That Service will forward any traffic it gets to our Deployment since they should share the same selector so let's check that.
This all looks good, so let's see what happens when we go to it in a web browser:
It works!Okay, it's been working for months, but you get the idea.Now, go forth and build your deployments.
There's more to learn, like Helm charts and Kubernetes Operators, as well as GitOps systems like FluxCD or ArgoCD.I'll leave it here, however, and let you explore with a few of my recommendations.
Start with microk8s. It's a one-node development cluster (runs on your machine) that's stupid easy to install, and if you break it you just bring it down and start up another cluster.
Grab this fork of it-tools. One of the tools available is a "Docker compose to Kubernetes" converter that can initially help you out and save you a lot of time later.
For things that use postgres, the cnpg operator is phenomenal and, once you figure out how operators work, is a total lifesaver. Note that the operator will drop built-in support for barman backups soon, and they recommend using the barman plugin (which isn't that different and there's a conversion guide).
For apps that use MariaDB, I would consider the MariaDB operator to be "adequate". Also supports backups, which are just-very-slightly simpler than postgres backups.
If you're in need of Redis for your app, try the Dragonfly operator. Just beware the whole "it'll fill up your PVC and then repeatedly crash" gotcha and use the --dbfilename= flag.
There's also a seaweedfs-operator if your app wants S3 storage and you don't already have MinIO or similar available (or don't want to use that).
cert-manager and external-dns will be lifesavers on a real cluster, and maybe trust-manager will help you as well (eg. if you're behind a corporate proxy) - Reflector, too, can be handy when you want those ConfigMaps containing all your certs to appear in every namespace.
Want a container that uses your GPU? Nvidia has an operator for that.
Kured is quite nice for handling automatic node reboots when needed.
Rancher (the folks behind k3s & RKE2) have a very helpful system-upgrade controller that updates the actual kubernetes software/implementation across your cluster (safely).

Bash for SAs

egg82 — Thu, 12 Jun 2025 00:00:00 GMT

The simple answer? Pretty much any Unix-based system is going to have some version of Bash on it. Even languages such as python aren't guaranteed because not every system uses systemd and thus doesn't require it (eg. AIX) - but you'll be hard-pressed to find a system without bash in it.Each new version of bash adds new features and changes or removes others. It's important to know the capabilities of all of the systems you plan on running your bash scripts on.I took the liberty of summarizing the most notable changes in each version of bash. The full list can be found here.kg-card-begin: htmlkg-card-end: htmlkg-card-begin: htmlkg-card-end: htmlkg-card-begin: htmlkg-card-end: htmlkg-card-begin: htmlkg-card-end: htmlkg-card-begin: htmlkg-card-end: htmlkg-card-begin: htmlkg-card-end: htmlkg-card-begin: htmlkg-card-end: htmlkg-card-begin: htmlkg-card-end: htmlkg-card-begin: htmlkg-card-end: htmlkg-card-begin: htmlkg-card-end: htmlkg-card-begin: htmlkg-card-end: htmlkg-card-begin: htmlkg-card-end: htmlkg-card-begin: htmlkg-card-end: htmlkg-card-begin: htmlkg-card-end: htmlkg-card-begin: htmlkg-card-end: htmlkg-card-begin: htmlkg-card-end: htmlkg-card-begin: htmlkg-card-end: htmlThis is intended as a cheatsheet and crash-course for SAs looking to automate things with Bash, so I won't be going into great detail about any particular thing. If you're an SA, you know how to use Google. I hope.Bash scripts and the command-line are effectively the same thing. You can paste an entire script into your terminal and it will run fine, and vice-versa.When I write bash scripts, I'll usually test each line I'm unsure of on the machine or machines on which I plan to run the script- as long as the line doesn't actually make any changes to the OS that I can't easily undo.The main thing to note is the shebang at the top of a script file (#!/bin/bash etc) - it's usually recommended to use #!/usr/bin/env bash for portability reasons but there's a lot of history and debate online around anything that you can put into a script and its "portability" - so, using #!/bin/bash, as long as it works for your use-case, is perfectly fine.Variables can be set for the current shell by simply doing the following:variable_1="x" VARIABLE_2="y" You can set variables for the current shell and any subprocesses it spawns by doing the following instead:export variable_1="x" export VARIABLE_2="y" or, if the variable was already set somewhere else:export variable_1 export VARIABLE_2 To use variables in your script later, it's usually recommended to use the "${}" syntax.eg.echo "${variable_1}" this allows for fancy things like replacements directly in the variable and also avoids problems like shell expansions.Sometimes, though, you really need to do weird tuff like run yum or dnf based on the OS:if command -v dnf; then pkg_cmd="dnf" else pkg_cmd="yum" fi $pkg_command -y update The above script would run dnf -y update if the system has dnf, and yum -y update if not.The basic syntax for an if/else if/else statement is:if condition-1; then do-thing-1 elif condition-2; then do-thing-2 elif condition-3; then do-thing-3 else do-thing-4 fi Return codes in Bash indicate the success or failure of a command. A return code of 0 generally means success, while any non-zero code indicates an error. You can access the return code of the previous command using $?.eg.ls /nonexistent_directory if [[ $? -ne 0 ]]; then echo "Command failed." fi You can use command -v or which to check if a command is available:if command -v cat &>/dev/null; then echo "cat is installed." else echo "cat is not installed." fi Note that this works for Bash builtins (eg. timeout) as well as "real" commands located on the system.The $() syntax is used for command substitution in Bash. It allows you to capture the output of a command and use it as part of another command or assign it to a variable.eg.date_output=$(date) echo "The current date and time is: $date_output" This is equivalent to the older backtick syntax, but $() is preferred as it is more readable and can be nested easily.The older syntax would look like this:date_output=`date` echo "The current date and time is: $date_output" Combining the previous two sections, a handy trick I use sometimes is ls -l $(which cat) or vi $(which custom-command) to quickly find and edit a particular script registered as a command (eg. for Bash scripts located in ~/.local/bin)Almost every Unix-based distro will support os-release, which is really just a file located at /etc/os-release.For example, to determine if a system is RHEL-based or Debian-based, you can use the ID_LIKE variable located in it:ID_LIKE=$(grep ^ID_LIKE /etc/os-release | awk -F'[="]' '{print $3}') or, even better:source /etc/os-release Sourcing the file will put all of the content into variables like NAME, VERSION, ID, ID_LIKE, etc for use in your script.Then, you can do something like the following:if [[ "${ID_LIKE}" == *rhel* ]]; then echo "RHEL OS!" elif [[ "${ID_LIKE}" == *debian* ]]; then echo "Debian OS!" fi Strings in Bash can be enclosed with single (') or double (") quotes. Single quotes preserve literal value, while double quotes allow variable interpolation.name="John" echo 'Hello, $name' # Output: Hello, $name echo "Hello, $name" # Output: Hello, John Bash allows simple arithmetic using $((...)) syntax:num1=5 num2=10 sum=$((num1 + num2)) echo $sum # Output: 15 These operators are used for numeric comparisons in conditional expressions: -eq: Equal -ne: Not equal -gt: Greater than -lt: Less than eg.if [[ num1 -gt 2 ]]; then echo "$num1 > 2" else echo "2 > $num1" fi -n and -z are used to test strings: -nchecks if a string is non-empty. -zchecks if a string is empty. eg.if [[ -n "$MY_VAR" ]]; then echo "MY_VAR is not empty." fi if [[ -z "$MY_VAR" ]]; then echo "MY_VAR is empty." fi Bash supports indexed arrays and (from version 4) associative arrays:# Indexed Array my_array=("apple" "banana" "cherry") echo ${my_array[1]} # Output: banana # Associative Array declare -A fruits fruits["yellow"]="banana" echo ${fruits["yellow"]} # Output: banana [is a synonym fortestand is used for basic conditional tests. [[is a Bash-specific keyword that supports more complex conditions and pattern matching. No brackets are used for simple commands or variable assignment. eg.if command -v cat; then echo "Using no brackets." fi if [ "$var" = "value" ]; then echo "Using single bracket." fi if [[ "$var" == "value" ]]; then echo "Using double brackets." fi You can check if a script is connected to a terminal to decide whether to enable terminal-specific features, like colored output:if [[ -t 1 ]]; then TERMINAL=true else TERMINAL=false fi Bash supports inline if/then statements using the && and || operators:[[ -n "$MY_VAR" ]] && echo "MY_VAR is not empty" || echo "MY_VAR is empty" Output redirection in Bash allows you to direct the output of commands to files or other outputs. The most common operators are: >: Redirect standard output to a file (overwrites the file). >>: Redirect standard output to a file (appends to the file). 2>: Redirect standard error to a file. &>: Redirect both standard output and standard error to a file. eg.# Overwrite output to a file ls > output.txt # Append output to a file echo "Additional line" >> output.txt # Redirect error messages to a file ls /nonexistent_directory 2> error.log # Redirect both output and errors echo "Test" &> combined.log or, if you're feeling extra fancy, you can do some cool stuff with cat:cat < myfile.txt # look, ma, a comment! do x and y plz and try ${favorite_piza}! EOF You can also output to devices, like /dev/null. As a useful example, to run a find command on your entire (local) system but skip any errors:find / -xdev -type f -name "*xyz*" -ls 2> /dev/null Connectivity checks can be done in a few ways, but the most reliable would be with a pure socket implementation, which has been supported in bash since version 2.04.This method uses timeout with Bash's /dev/tcp feature to check if a port is reachable:timeout 10 bash -c "cat < /dev/null > /dev/tcp/server/port" &>/dev/null This only works in modern Bash versions where timeout is available, however, so it's best to check if timeout is available with a quick if command -v timeout; then beforehand.This method uses /dev/tcp but has no set timeout, meaning it will block for the OS's default time if the connection doesn't succeed:/dev/null In either case, you can get the result with a simple $? on the next line to see the return code. If it's non-zero, you have a problem.Sometimes you need some fancy stuff to make your scripts pretty or make it work well. Here's a couple fun ones.To reliably determine the current script's path in a one-liner:SCRIPT_PATH="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" &> /dev/null && pwd)"
This method works even if the script is being sourced. Thanks to this SO answer.To strip ANSI color codes from a string:echo -e "$1" | sed 's/\x1B\[[0-9]*$\?:;[0-9]$*\?m//g' This is useful when you want to remove color formatting for logging or processing output, or when you don't have a terminal. You can check for all of these by simply doing a if [[ -t 1 ]]; then if/else case.
Largely thanks to the answers in this SO post, but I had to hand-edit the regex for it to work reliably across multiple versions and OSes.Here's a few useful things you can throw in your .bashrc (or .bash_profile)alias grep="grep --color=auto" alias lgrepi="ls -la | grep --color=auto -i" alias rgrepi="find \${PWD} -name '.*' -prune -o -type f -print0 | xargs -r0 -P2 grep --color=auto -HIni" alias grepi="find \${PWD} -maxdepth 1 -name '.*' -prune -o -type f -print0 | xargs -r0 -P2 grep --color=auto -HIni" And a quick summary of the aliases: grephas been updated to add colors to the output when it can lgrepisearches your current directory for filenames matching your query (case-insensitive) rgrepisearches your current directory (and subdirectories) for content matching your query (case-insensitive) grepiis the same asrgrepi, but won't recursively search subdirectories Sometimes you need to copy a bunch of files fast, and with progress bars. rclone is a great way to do that but it doesn't really match the syntax of cp at all. Here's an rcp command that does just that:function is_remote_mount() { dir="$1" grep -E ' nfs[3,4]* | cifs ' /proc/mounts | awk '{print $2}' | while read -r mount_point; do if [[ "$dir" == "$mount_point"* ]]; then return 0 fi done return 1 } function rclone_cp() { args=("$@") last_index=$(( $# - 1 )) dest="${args[$last_index]}" unset 'args[$last_index]' base_flags="--verbose --update --copy-links" # Enable progress if running in an interactive terminal if [ -t 1 ]; then base_flags="$base_flags --progress" fi temp_dir=$(mktemp -d) echo "Created temporary directory ${temp_dir}" trap 'rm -rf -- "$temp_dir"' EXIT for src in "${args[@]}"; do # Convert relative path to absolute path if [[ "$src" != /* ]]; then src="$PWD/$src" fi if [[ -d "$src" ]] && [[ "$src" != */ ]]; then # If source is a directory and does not end with '/', copy the directory itself ln -s "$src" "$temp_dir/" elif [[ -f "$src" ]]; then # If source is a file ln -s "$src" "$temp_dir/" fi done temp_flags="$base_flags" # Determine if source or destination is a remote mount if is_remote_mount "$dest"; then temp_flags="$temp_flags --transfers=16 --checkers=16 --buffer-size=512M" else temp_flags="$temp_flags --transfers=4 --checkers=4 --buffer-size=256M" fi # Determine if source or destination is remote (via rclone) if [[ "$dest" == *":"* ]]; then temp_flags="$temp_flags --transfers=32 --checkers=32 --buffer-size=1G --fast-list" fi rclone copy "$temp_dir" "$dest" $temp_flags } alias rcp='rclone_cp'

Ansible

egg82 — Thu, 12 Jun 2025 00:00:00 GMT

Ansible is a provisioning system developed by Red Hat and is widely used in industry to save a lot of SA time and effort. It makes everyone happy- from the SAs required to build machines to the security office who needs to make sure those machines are in spec, to the CFO in charge of saving money.First, let's ask: what is provisioning, and why do people use it? What problem is it really solving, and what other problems does/can it create?Let's imagine the following scenario: You're in charge of IT for a small mom-and-pop shop selling candy. You build a couple of Windows systems with Excel, Word, etc on them for the three employees on staff and put together a couple Netgear managed switches and the ISP's router for wired and wireless connectivity in the shop.Throughout the first couple of years, occasionally, you're asked to troubleshoot a problem on an employee machine, create and remove accounts, or outright build or rebuild a machine for a new employee. Not a big deal. It keeps you busy on the slow days.One day, the shop expands, opening a second location. You set up a couple more systems, install the usual software, and link the stores with a VPN. All goes smoothly - until the third store opens. Then the fourth. Now, every time a new employee is hired or a system breaks down, you’re manually setting up machines and traveling to each site.A year later, the shop has ten locations. Each store has its quirks: different software needs, custom inventory systems, and varied machine specs. What used to be a quick setup now takes days. You’re driving all over town, USB stick in hand, building machines, installing updates, and troubleshooting everything from user accounts to network issues.Late one night, while you're buried under a pile of checklists and half-configured machines, the owner calls: “We’re opening another store next week. Can you set up five more systems?”You realize it’s unsustainable. There must be a better way.Enter Ansible. Its one and only purpose in life is to run commands on machines. Doesn't sound super fancy, because it's not. But Ansible doesn’t need to be fancy. It’s simple by design - no agents, no complex setup. Just plain, easy-to-read YAML files that let you describe what you want done. Need something more advanced? You can use a Python-like language in your YAML that adds flexibility when you need it, without overwhelming you when you don't.To install it, you set up a RHEL machine somewhere - anywhere it can reach your other systems - and sudo dnf install ansible-core. Everything's done from the command-line, and once you get it all laid out the way you like it, provisioning an entire machine goes from hours of running commands and manually installing software to typing a single command, hitting the Enter key, and coming back after a coffee break.Ansible works by connecting to your machines over SSH (or WinRM for Windows) and running tasks defined in playbooks - YAML files that describe the steps needed to configure your systems. You start by creating an inventory, which is just a list of machines you want to manage.Here's a supplementary video from NetworkChuck - I'm not a super-fan of his "clickbaity-ness" or "hollywood hacker-ish" aesthetic but I'd be lying if I said he didn't make some of the best beginner-friendly explainer videos I'd ever seen. This one is no exception.

you need to learn Ansible RIGHT NOW!! (Linux Automation)Start learning Ansible with a lab on Linode with $100 credit: https://bit.ly/nc_linode*Sponsored by Linode➡️Checkout ALL my training at CBT Nuggets: https://...www.youtube.comNote He uses different commands and file locations than in this document. Both the video and this document are valid ways to do all of this. Define your inventory A list of machines you want Ansible to manage. Write a playbook A playbook contains tasks like installing software or setting up configurations. Run the playbook Ansible connects to the machines and applies the necessary tasks. The inventory file is where you define the machines that Ansible will manage. In this example, we have three groups of machines: webservers, workstations, and a database server. Each group contains the IP addresses or hostnames of the machines, allowing Ansible to apply different configurations depending on the role of the server.[webservers] 192.168.1.10 # websrv1.candy.shop 192.168.1.11 # websrv2.candy.shop [workstations] 192.168.2.52 # wks1.candy.shop 192.168.2.53 # wks2.candy.shop 192.168.2.54 # wks3.candy.shop [databases] 192.168.3.5 # db1.candy.shop The playbook is where the real magic happens. Here, we define tasks for each group of machines. In this example, we’ll: Install and configure Apache on Linux web servers. Install common software like Google Chrome and 7-Zip on Windows workstations. Install and configure MariaDB on a Linux database server. Ansible will connect to the machines in each group and run the tasks defined for that group. Let’s look at how this playbook is structured:--- - hosts: webservers tasks: - name: Install Apache yum: name: httpd state: present - name: Start and enable Apache service: name: httpd state: started enabled: true - hosts: workstations tasks: - name: Install Google Chrome on Windows workstations win_chocolatey: name: googlechrome state: present - name: Install 7-Zip on Windows workstations win_chocolatey: name: 7zip state: present - name: Set PowerShell execution policy to unrestricted win_shell: "Set-ExecutionPolicy Unrestricted -Force" args: executable: powershell - hosts: databases tasks: - name: Install MariaDB yum: name: mariadb-server state: present - name: Start and enable MariaDB service: name: mariadb state: started enabled: true Under the hood, those tasks - even win_chocolatey and service - jut run commands on the remote system. It's just that Ansible abstracted away all those ugly "check, then install, then verify" commands behind a single win_chocolatey task set.When you've saved your playbook and inventory files to a directory on your new "Ansible provisioning server", you can run this in that directory to start provisioning all of your systems:ansible-playbook -i inventory.ini playbook.yml And, finally, go get some coffee! ☕For larger projects or environments with many systems, you’ll likely need more than one playbook. While it’s tempting to cram everything into one massive YAML file, it’s much better to break things up. One playbook per "logical task" helps keep things organized and manageable.For example, instead of having a single playbook for everything: One playbook for setting up web servers. Another for configuring workstations. And maybe a separate playbook for managing your database servers. This way, you can reuse tasks and target specific groups of machines without sifting through a gigantic playbook for every small change.If your system is complex enough, you may even organize your playbooks into "common/core items", "specific items for that system type", and "specific items for X software on that system" - it all depends on how you want to organize things. Since Ansible is all files-based, you can have multiple directories for other sites, projects, or testing. Ansible doesn't care, it just runs the tasks you give it in the order you tell it to.As you grow more comfortable with Ansible, there are several features you can use to optimize and customize your automation even more.Roles are a way to break down tasks into reusable components that can be shared across multiple playbooks. Think of roles as modular pieces that can be plugged into different projects.For example, you could have a powershell role that installs and configures your PS environment, a 7zip role that handles 7-zip setup, and a chrome role for installing and hardening Chrome. Each role has its own directory structure for tasks, variables, handlers, and files, which keeps things clean and easy to manage.To use a role, you would structure your playbook like this:--- - hosts: workstations tasks: - name: Include Powershell roll include_role: name: powershell - name: Include 7zip roll include_role: name: 7zip - name: Include Chrome roll include_role: name: chrome
You would then create new directories, powershell, 7zip, and chrome and place your tasks, templates, variables, etc into specific subdirectories of that role. It keeps everything nice and organized!Roles make it easy to keep your playbooks modular and allow you to build on top of them without starting from scratch each time.You can use tags to run specific parts of a playbook instead of running (or re-running) the whole thing. This is especially useful when you only want to run a subset of tasks, such as software updates, without rerunning everything else in the playbook.Here’s an example:--- - hosts: all tasks: - name: Install Apache yum: name: httpd state: present tags: apache - name: Install MySQL yum: name: mariadb-server state: present tags: database You can then run only the tasks with a specific tag like this:ansible-playbook -i inventory.ini playbook.yml --tags "apache" Or skip tags like this:ansible-playbook -i inventory.ini playbook.yml --skip-tags "apache,database" This would effectively run nothing, since you're telling Ansible to skip both of the tasks in that playbook.Instead of hardcoding paths, usernames, IP addresses, etc, you can define variables and use them throughout your playbook. You can even define different variables for different groups or systems.Example:--- - hosts: all vars: apache_port: 8080 tasks: - name: Configure Apache to listen on custom port lineinfile: path: /etc/httpd/conf/httpd.conf regexp: '^Listen' line: "Listen {{ apache_port }}" Variables can be stored in your inventory file, defined directly in the playbook, or loaded from external files.Ansible Vault allows you to encrypt sensitive data like passwords or API keys. You can then reference these secrets in your playbooks (as variables) without exposing them in plain text.To create an encrypted file:ansible-vault create secrets.yml Choose a password that you'll remember, since you'll be using it every time you run any playbooks that require this secrets file.Inside this file, you can define variables like:db_password: myS3cr3tP@ssw0rd admin_password: abcXYZ123! Then, in your playbook, reference the secrets file as a variables file:--- - hosts: databases vars_files: - secrets.yml tasks: - name: Set up the database mysql_user: name: admin password: "{{ db_password }}" And, finally, run the playbook with:ansible-playbook -i inventory.ini playbook.yml --ask-vault-pass This will prompt you for the password you set for the secrets vault earlier, so hopefully you still remember it.Vaults are a great way to do encryption-at-rest, and many other Dev*Ops projects support handling Ansible Vaults (secrets files).
Templates allow you to dynamically generate configuration files or scripts using Jinja2 templating. Instead of hardcoding values into configuration files, you can use variables within templates, which Ansible will substitute when the playbook runs.Here’s an example of a simple Apache configuration template:apache.conf.j2: ServerName {{ server_name }} DocumentRoot {{ document_root }} You can then reference this template in your playbook and pass in its variables:--- - hosts: webserver vars: apache_port: 8080 server_name: www.example.com document_root: /var/www/html tasks: - name: Deploy Apache config from template template: src: apache.conf.j2 dest: /etc/httpd/conf.d/apache.conf Templates are especially useful for generating configuration files that change across environments or systems.Sometimes, playbooks need to call other playbooks to keep things organized and manageable. Instead of having one massive playbook, you can structure your automation by calling smaller, more specific playbooks from a "master" playbook.Here’s how:--- - hosts: localhost tasks: - name: Run common setup tasks include_tasks: common_setup.yml - name: Run webserver playbook include_tasks: webserver.yml - name: Run database playbook include_tasks: database.yml

Raft-style Elections: Dynamic Membership

egg82 — Sun, 10 Aug 2025 00:00:00 GMT

I'm not usually one for sitting down and reading a PhD thesis. Usually.The Raft consensus algorithm solves a common problem in computer science in a reasonably elegant, simple way. Let's say you have the following situation:You have several systems tasked with (and capable of) running the same job periodically. You need to ensure the following: That the job runs, at minimum, once per day. That multiple systems don't try to run it at the same time. That another system is able to pick up where the first left off in case of a failure. You may assume this could be done with locks, but the final point of "picking up where the first left off in case of failure" complicates that solution due to the need for state management. This is a scenario that Raft would shine in, among many others.
Perhaps somewhat surprisingly, the official Raft homepage contains a really, really good interactive explanation. The Secret Lives of Data is another page which does an excellent job of showcasing the major points of the algorithm in a nice powerpoint-like presentation. I would generally recommend the powerpoint first since it tells you what you're seeing in the interactive exhibit.I really do recommend checking those out. They do a better job of explaining how it works than I could in a single blog post. It'll be required reading for understanding the modifications made later in this post.Let's take a look at our previous scenario of running jobs on a schedule and add one more constraint to it: At any point, a system running this job could be added or removed. One of the biggest issues with implementations of the Raft protocol is that their expectation is that all members are pre-defined. Before you even start the quorum you need to know where and how many Raft members you have. This is a problem for systems that dynamically add or remove systems based on factors such as load. You'd have to re-create the entire consensus every time you add or remove a member - at best. At worst, you're just stuck.A project I did a few years ago required such a system. I created a Raft-adjacent implementation which only handled leadership election, but did so with the added parameter that a member could be added or removed at any time. Without further ado, here's how I did it.With dynamic membership, you're guaranteed to run into a situation in which there's an even number of members in your quorum. Raft handles this by requiring N / 2 + 1 members to elect a leader in the event of an even number of members, which is simple enough, but we need to re-calculate this when systems are added.For an odd number of nodes, we use ceil(N / 2) instead. Both of these can be simplified to a single floor(N / 2) + 1 calculation.For example, let's say we have a leader elected with three nodes. We then add three more nodes which know nothing about the current leadership election. In this instance, we want to set our current leader to a candidate and have it re-issue a new election cycle. This will mean that, for a short time, there will be no leader, but performing a full election starting with the current leader will virtually guarantee that member is re-elected and that all of the new members are up to speed.At some point, you will likely have enough nodes that adding a few won't affect an election at all. There's also no need to re-calculate on membership removal because a current election cycle is as stable as the leader node. If the leader is lost, a new election cycle will only use the remaining members anyway.The implementation I needed to write had no real use-case or need for a replication log. Raft is actually a story of two parts: Leadership elections, for finding a leader node. Replication logs, for allowing the leader to tell the other members how things are. Not requiring this meant that heartbeats were smaller and on-disk persistence wasn't required. Vote tallies were simply stored in-memory on each member because they could easily be removed and re-added if needed anyway.When a new member joined an existing cluster, it assumed no current state. It would do the normal thing of waiting until a timeout to perform a membership election, etc. If any member received a heartbeat of a higher election cycle than it already had, it would immediately set the current election cycle and leader. Heartbeats from leaders below our current election cycle can be safely ignored.Similarly, if an existing leader or member node got a vote request from a (often new) member with an election cycle lower than it already had, the existing member would deny the vote and send back information on the current cycle and leader.And, finally, if a new member joins during an election we can assume everyone is voting for the same leader (as is most often the case) and cast our vote accordingly. If this new member can see current votes being cast it can also set itself up accordingly and assume it is to follow a new leader.These ensured that new members joining the quorum would quickly and reasonably safely get up to speed on the current state of things.There's one final issue that can come up with dynamic membership: When a new quorum first starts, it will likely only have one member to start with but may very quickly get a second.The temptation is to have each node set itself as the leader on initial startup but this will mean that, if the current election cycle is 1, we'll end up in a split-brain where each node simply assumed it was the leader and we'll have to wait for one of them to time out before a "real" election takes place.We found that, sometimes, it was nice to be able to force a leader in a few relatively rare cases such as removing an existing leader (and knowing we were going to do it) without wanting to wait for a new election cycle.In these cases, "proposing" a new leader is as simple as setting a new election cycle and proposing one of the members as the new leader for that cycle (and then becoming a candidate or follower on the node that set the new cycle). Assuming everything is reasonably stable already, no new election cycles would be going on anyway so that would effectively be guaranteed leadership for the chosen member.That's really about it. Raft is a robust and simple algorithm and adding to it is equally simple, with a few "gotchas" here and there and some optimizations that can be done while sacrificing a tiny bit of reliability.While there weren't any fun pictures in this one, I hope you enjoyed a somewhat-technical write-up regardless.

Pentesting Fundamentals

egg82 — Thu, 12 Jun 2025 00:00:00 GMT

This isn't going to be a comprehensive guide on "how to hack" because, frankly, I don't know enough to be able to teach everything. Everything is a lot of things. I can, however, tell you what it is and what it isn't, and show you where you can start.Hacking - usually called penetration testing or "pentesting" - is often thought of as, basically, this guy:I'd love to tell you that this is something Hollywood decided hackers looked like back in the '80s but it's not that far from the truth in some respects.
Hacking culture started as phone phreaking (what we call it today) in the '70s with John "Captain Crunch" Draper. The term has expanded considerably since then to include everything from car hacking to lockpicking to social engineering. Hacking really just means making something do what it wasn't originally designed to do.So, Hollywood aside, what can you actually do with hacking? What does it really look like?In order to make something do what it wasn't designed to do, you first need to learn how it works. You have to fully understand what you're about to break so that you can break it in just the right way to do what you want.A blacksmith, for example, needs to learn how metal works. Not just the different kinds of metals, but also the ratios involved with different types of steel and how those work when put to heat or after quenching. They need to know exactly how the metal is going to behave when they work it, what the grain structure will look like when it's cooled, and what kind of cooling and treatment the metal will need after shaping in order to get the desired result.
Much like any trade that creates things, hacking requires an intimate understanding of the thing you're working with (and lots of practice) in order to be successful. The wood didn't fall off the tree into a pile of two-by-fours in exactly the right length for your house frame. Similarly, the Flipper sitting in someone's hand didn't magick anything from thin air in order to open your garage door or change the numbers on the gas station sign.
Hacking seems a lot like magic when you don't have an understanding of the principals behind what you're seeing.Quote "Any sufficiently advanced technology is indistinguishable from magic." Arthur C. Clarke What all this means is that hacking is bound by the same principals as any other thing in this universe. It isn't magic. It's understanding something well enough to be able to do something cool with it. You can hack anything if you have a good enough grasp of how it works. There are no wizards- just people curious about what makes things tick.This is often the question people start with: "How do I hack X?"The answer, ultimately, lies in learning the technical stuff. But, while true, it's not a satisfying answer. That road requires watching hundreds of hours of YouTube, reading many books, and developing a deep understanding of the technology you want to change over the course of years.So, in the interest of saving you some time, I'll boil down the most-asked question here. I'll be making the assumption that you would like to "hack Google" or "hack Twitter" or Discord or whatever platform is today's current. Usually, this is with the intent of stalking someone (no matter how you justify it, it's still stalking) but for the sake of argument we'll provide the following scenario:Your friend has asked you to try to hack their account to see if it's secure. You're the "bad guy" in this instance, trying to gain access to any - or all - of their online accounts.We're back at "understanding things" already, but this is important. You must know your own limitations, as well as limitations imposed by the universe (those pesky laws of physics), limitations in law (hacking is usually illegal just about everywhere), and limitations imposed by companies such as Google and your local internet provider.The laws of physics and nature, I hope, should be obvious. If you have a way around them then you're already a hacker and you've also got a Nobel prize waiting for you.
Human laws are a little more subtle, but they can become very real very quickly. Just ask Kevin Mitnick. While I am not a lawyer and you should absolutely look at any laws you may possibly be about to break, here's my understanding of US laws around this kind of subject:
Hacking ispretty much universally illegal, except for the following: When you own the network that people connect to, you are free to change the network in any way you see fit This doesnotcover trying to access the person's machine if they're on your network If you can see it, you're free to look at it But unless you own the network the machine is communicating on, you arenotfree to change it You also can't connect to password-protected wifi without permission If you have permission from the owner of the device, you can also do whatever, as long as it's covered in thewrittencontract
This leaves some interesting holes where you can legally do things like Evil Twin attacks or snoop on unencrypted traffic (or decrypt the traffic) in your local coffee shop as long as you don't change it. Unless you're the owner of the network, of course.Again, remember, I'm not a lawyer. Don't go to jail for doing stupid things.Finally, there's limits imposed by companies. eg. Instagram/Facebook, your Internet Service Provider (ISP), etc. Almost always these limits are technical in nature. For example, Google definitely will not allow you to keep guessing someone's password on their platform. Not for long, anyway. There's many anti-bot measures such as CAPTCHAs and browser integrity checks to prevent automated software from using their login forms, and it'll definitely lock you out after a few wrong guesses and will probably alert the account owner.Your ISP will likely be oblivious or turn a blind eye until someone like Facebook or Google tells the FBI about your activity, and then it becomes their problem. ISPs really don't like your problems being their problems, and they will absolutely terminate your account with them if the FBI comes asking about you.So, know your limits. What can you safely get away with? If you don't know and still decide to go for it this will very likely come back to haunt you in a really bad way. Some people get lucky and get away with it for a while, but luck will always run out eventually. You need skill, which comes from knowledge, if you want to do this kind of thing and stay in everyone's good books.Quote "Give me six hours to chop down a tree and I will spend the first four sharpening the axe." Abraham Lincoln This, once again, falls under the "hacking is understanding" category. Research is hard, but it's important.Remember that, in this scenario, your friend has essentially asked you to try to hack them. Anything you can get. So, the first question you should be asking is: what can you get? What accounts do they even have? What services do they use? What do they log into? The internet is a big place, and you could spend your entire life trying every combination of username, email, and password on every website that exists and get nowhere.Let's break this down step-by-step. What we need, first, is information about the target. We'll say that your friend's first and last name is Roy Williams, and that you know he's around 35 years old and that he lives somewhere in California. You also know that he has a dog named Spot, which may come in handy later.Next, we need to be able to find any potential email accounts, Facebook profiles, etc associated with the target. Knowing the name, age, and location will help us significantly, here, since we can use the US's public records to our advantage. You can use any online "people search" website to find out more information on your friend, like exactly where he lives, his relatives, and maybe even some online accounts and phone numbers to start.Most people don't know these services exist, and it's expensive and/or time-consuming to constantly remove yourself from them. All of these websites use the same sets of data, so it's mostly just a matter of which one you like the best. Google "people search" and find one you like. They'll all likely require a few dollars for the information, so get your credit card ready as well.We'll say that with his name and approximate age and location you found what looks to be his exact address, several phone numbers that might or might not belong to him (and you may be able to confirm this yourself since he's presumably your friend), a few email addresses that also might or might not belong to him, and his Facebook profile.Now, you can make use of these email addresses. You can search all sorts of websites using them to see if he used them to register anything- additionally, it's a safe assumption that, if it's a gmail email address, he also has a Google account that he's actively using. You can make use of plenty of tools for this kind of broad search. Beware, though, that a lot of these will very much be a "install/run it yourself" type of thing and that very many of them will only work on a Linux system.
The biggest thing to know here is how to effectively use Google and, likely, how to use Linux. There are a lot of projects on Github which help you with this exact problem, but over the years the most popular or updated ones change significantly so if you're just Googling for "how to find accounts for X person" you're going to hit a wall of outdated recommendations really quickly, if you even find anything legitimate at all.Here's the kind of Google searches I would perform to find these kinds of tools: account finder github username search github
Currently, the most common tool is Sherlock but this can (and probably will) change in the future.I would append "github" to the end of the result because otherwise you'll get all sorts of fun websites that either find the accounts that you already found in your people search or are just outright scams or worse. This is a dangerous area for people who aren't familiar with how the internet works.If you don't have a username to search for, you can try another tool or extrapolate and assume that the first part of the email would be a likely username or, more commonly, first letter last name (eg. rwilliams) would be a good one. It's uncommon to be able to get that kind of name, though, so people usually put numbers in their usernames. Normally, that's an important date to them like their birthday. Check their email address again for confirmation or look at their profile that you pulled up on the "people search" website that you picked. Try a few different usernames and see what comes up with the most hits.We'll assume you found a few services that look like they might be valid and some usernames and/or email addresses to try. The problem, now, is that it requires more than a valid email address to log into most of these. Well, most of the time, anyway.The holy grail, here - the thing you're really looking for - is access to the email itself. Let's take a moment to understand why:When you ask Facebook to reset your password, what happens? Where does it send that password reset link to? Same with Google. Same with Amazon, your local library, and pretty much every other service under the sun. They ask for your email address and, when you "forgot" your password and need to reset it, they'll send all the info to your email that you registered with. If you can access someone's email address, you will own everything they care about.This is why most of the popular email providers will have two-factor authentication, although most people won't enable it because it's effort to set up. If you're targeting someone tech or security-minded you'll be in for a rougher time, here, but nobody's perfect.Speaking of, how are we going to get passwords? Well, we have a few options here, and some are sketchier than others. We'll start with the least sketchy and the most effort. When people create passwords, they will usually do something like - in our example, it would be something like Spot89. Often, password fields will require a special character so most people will use ! or a substitution. eg. Spot89! or Sp@t89. There's quite a few passwords to try, so we'll get some help with a program.Again, Google will be your friend: wordlist password generator github
Currently, tools like bopscrk, cracken, and wgen show up. This is where you put in every detail you can think of about the person. Where they work, pets names, birthdays, family names, hobbies, favorite music, favorite colors, etc. You'll get a wordlist where something resembling at least one of their passwords will almost certainly show up. Password re-use is very common (using the same password across different services) but most people will use a couple of different passwords and forget that some of their accounts exist so you're pretty likely to hit at least one.The sketchier but possibly easier method is to look around for people selling leaked and/or cracked password databases. Websites get attacked all the time, and someone else has probably done the hard work of figuring out people's passwords, so you can take advantage of that and buy a couple of data dumps if you know where to look online. This isn't something you should Google, more of a "if you know then you know" kind of thing. You're very likely to fall victim to something nefarious here if you don't know what you're doing, but it is an option.
Now, you can use a service like haveibeenpwned to find password dumps and try out your passwords. Really, you should be trying your generated password lists against an offline database stored on your own computer, because, as I mentioned earlier, most online services don't like you guessing passwords for their users.So, you can use haveibeenpwned to look at the email addresses you found and see if there's any compromised database dumps that you can check. If you've been on the internet for any length of time, you'll be in one of these dumps, so you'll definitely find something. The problem with these dumps is: You have to find the actual dumped data - this can be a little tricky, but most of the time you'll get them via some clever Googling
The passwords will almost certainly behashed, which is a one-way operation. You'll need to guess passwords until you get it right. That's how "password cracking" works The data is usually in a format that you'll need to convert in order to do said password guessing
hashcat is one of the few constants, here. It specializes in taking your password guesses, transforming them (which will be very helpful with things like character substitutions), and trying a lot of guesses per second - mostly using your GPU. Once you have a password dump (and have it converted into a format that hashcat understands) you'll need to learn how to use hashcat and then try it against your friend's password.Assuming you're successful on at least one of the services, you can manually try this password or variations of it that you think are likely on other services. You might get lucky and get in- and, well, then you're in!
Alternatively, you can try social engineering your friend (or, more likely, their friends, family, and co-workers) into giving you access to their accounts. Don't underestimate the power of a well-crafted phishing email or a microphone and camera in someone's face!
Unfortunately, I'm not well-versed in this subject, but the SE Village at DefCon is a great place to start learning. It's very much a people-focused thing, so if you're better at getting people to do or say things than you are at getting computers to do or say things, then this approach might be for you. It's often more effective than the purely-technical approach I detailed above.