Why I don’t identify as sysadmin anymore

I used to be a sysadmin. And I like to think that I was pretty good at my job. These days, my work is much more managerial with a strong emphasis on communication. The command line and I only meet occasionally, and I have to get help whenever I do something more complicated with regular expressions. Still, I apparently have retained enough skills that the Head of Cloud Operations at my current workplace occasionally says things like „oh, you can do that?“ in a positively surprised tone. But I usually do know my limits and what not to touch.

This is the story of when I failed to recognize my limits.

Knowing that I’m not an admin, this webpage resides on a server that is run by a webhosting company. They worry over security patches, uptime, sensible database configuration and so on, because I know that I’ll probably screw this up. I am allowing myself to change database tables, tinker with some of the htaccess settings and… DNS. That is one of those things that I don’t touch a lot, as there is no need for this on a daily basis.

And thus, I completely forgot about the SPF record when changing the MX entries to support my G Suite setup when I switched hosting last year. As a quick recap, let me quote Google what it’s about:

The purpose of an SPF record is to prevent spammers from sending messages with forged From addresses at your domain. Recipients can refer to the SPF record to determine whether a message purporting to be from your domain comes from an authorized mail server.

Quite the important and nifty functionality. I actually knew of it already, but didn’t realize that the new webhoster had this implemented as a standard. So while I successfully switched all the MX entries, I overlooked the SPF setting.

Alas, there are a lot of email servers that happily ignore a faulty SPF setting, even Google kept receiving emails despite the wrong setting. And you don’t really get an error message on the senders part, because no one talks back to (even only suspected) spammers. So nearly everything seemed fine: I was sending and receiving mails on a daily basis and this configuration error wasn’t discovered for nearly half a year. Only when I started to worry that one email recipient didn’t answer at all and a friend who actually is an admin looked into the matter for me.

So, the lesson here is: The Dunning Kruger effect is sometimes closer to home than you think. Whenever you do something only occasionally, maybe talk to someone who does it regularly, to make sure that technology hasn’t advanced past your own experience in the meantime. And if you use G Suite, here's how to set the SPF correctly.

22. März 2017 testing Keine Kommentare zu Why I don’t identify as sysadmin anymore

How I imprinted on (not only computer) RPGs

Back when I was a kid (well, teenager), I had an Atari 800XL and played a great many list of games on it. The one that hooked me most though is a rather obscure one: Alternate Reality (The City) and it’s sequel Alternate Reality (The Dungeon).

For me, that game was eye-opening. The game world felt really alive in a lot of tiny ways, and was in many more way ahead than other games. It used a raycasting engine, on 8‑Bit home computers nonetheless. It was basically an open-world game, where you didn’t wander through a set plot, but had to connect the dots yourself, and figure out who wanted what. That meant that you could suddenly die, because you wandered in the wrong part of the map, or that people you never met knew of you.

This thoroughly spoiled me for most of what followed. I couldn’t fathom why things had to be so static, why I couldn’t just rob this bank or plead with this monster. And the places made sense, even if they were labyrinthine. Also, with monsters like the „Clothes Horse“, it forged my sense of what is appropriate in a game and what isn’t.

Gosh, I miss that game :)

13. April 2016 playing Keine Kommentare zu How I imprinted on (not only computer) RPGs

Subsonic & Caddy — oh my…

Dangnabbit.

Subsonic has the nasty habit of generating certain URLs in a very hardcoded way.

The login screen for example. If you call index.view and are not logged in, it redirects you to $ServerSubsonicThinksItIs/login.view.

That will be localhost, if you want to do the „oh yay, let’s reverse proxy here“ thing. To make matters worse, $ServerSubsonicThinksItIs also contains protocol and port. This is apparently due to the implementation of Subsonics „generate a yourhost​.subsonic​.org“ redirection.

As far as I can see, only the login and settings pages are the ones that are affected by this. They are still accessible if you manually add /login.view to your public address.

Of course, you can tell Subsonic to believe that it is on the servername you chose for it. As long as Subsonic and the reverse proxy server are in agreement of which host, path & port are shown to the browser, everything is fine. With non-encrypted http that is.

Of course you can re-enable https on Subsonic, in order to successfully proxy everything from A to B, but…

…Caddy checks the validity of the SSL certificate of whatever site it reverse-proxies to. And throws a Bad Gateway if it doesn’t like what it sees. This is actually done with a reason, but it doesn’t help me right now.

As far as I can see, I either have to wait for…

  • Subsonic finally learning to generate sane URLs or
  • Caddyserver being able to ignore faulty certificates upstream
20. Dezember 2015 testing Keine Kommentare zu Subsonic & Caddy — oh my…

Subsonic with Caddy

Ok, now that there is Let's Encrypt, I thought that I shouldn’t have to do this certificate hackery all the time when Subsonic updates or the StartSSL certificate expires.

Caddyserver is the first Webserver/Proxy that I found that has most of the process automated and build in, so I’m using this. Getting started with this was pretty straightforward: Download, expand,  create a config file and before you start up, make sure that Caddy can run on port 80 without requiring to be root:

sudo setcap cap_net_bind_service=+ep ./caddy

My Caddyfile eventually ended up looking like this:

server.mydomain.de {
 proxy / localhost:4040 {
 proxy_header Host server.mydomain.de
 proxy_header Scheme https
 }

Well, it turned out to not be quite as easy. First I also had to make Subsonic stop hogging port 443. Those settings are found in /etc/default/subsonic. And while you’re at it, also edit out any certificates you may have added earlier in /usr/bin/subsonic. Oh, and make sure to note down which port non-https Subsonic runs, hopefully not 80, because Caddy needs that! I may have chosen 4040.

Restart Subsonic and make sure that you can reach it on the correct ports.

As stated, Caddy also needs to run on Port 80 for a brief moment. This is so it can do the certificate domain validation process. So edit your /etc/apache2/ports.conf to make sure not to block anything.

Restart apache2.

Now, to find out if everything works, start Caddy. If all is good, you’ll see a glorious all-ok message and can now access the server via https with a valid certificate.

But as you’ve undoubtedly noticed, Caddy isn’t running as a service yet. Thankfully Ubuntu has Upstart. So, add in a new file /etc/init/caddyservice.conf (thank you Mathias):

description "Caddy Server startup script"
author "Mathias Beke"
start on runlevel [2345]
stop on runlevel [016]
setuid runasme
setgid runasme
respawn
respawn limit 10 5
script
 cd /home/runasme/
 exec ./caddy

At this point I want to take a short moment to mention that whoever thinks that the Upstart Cookbook is an easy entry-level document is clearly out of their mind. Seriously, this „Cookbook“ has a whole chapter named „Critique of the System V init System“!

But I digress. If you don’t make the same mistake as me and confuse init with init.d, then you’re golden and now have a Caddy running as a service, forwarding requests to your Subsonic installation.

The only issue I haven’t been able to solve is that Subsonic has some stupid static URL linking going on so I have to open the settings in a new tab. But I can live with that.

19. Dezember 2015 playing, testing Keine Kommentare zu Subsonic with Caddy