Recently, I’ve found myself walking several friends through what is essentially the same basic setup:
- Install Ubuntu server
- Install Docker
- Configure Tailscale
- Configure Dockge
- Set up automatic updates on Ubuntu/Apt and Dockge/Docker
- Self-host a few web apps, some publicly available, some on the Tailnet.
After realizing that this setup is generally pretty good for relative newcomers to self-hosting and is pretty stable (in the sense that it runs for a while and remains up-to-date without much human interference) I decided that I should write a few blog posts about how it works so that other people can set it up for themselves.
As of right now, there’s:
- An introduction (with Ubuntu basics)
- Tailscale setup
- Optional Docker Explainer
- Dockge setup with watchtower for automatic updates
- MicroBin as a first self-hosted webapp
Coming soon:
- Immich
- Backups with Syncthing
- Jellyfin
- Elementary monitoring with Homepage
- Cloudflare Tunnels
Constructive feedback is always appreciated.
EDIT: Forgot to mention that I am planning a backups article
This is very cool, but also very dangerous. Many projects release versions that need some sort of manual intervention to be updated, and automatically updating to new versions on docker can lead to data loss in those situations.
Here’s a recent example from Immich:
https://github.com/immich-app/immich/releases/tag/v1.133.0
It is my humble opinion that teaching newbies to do automatic updates will cause them to lose data and break things, which will probably sour them from ever self hosting again.
Automatic OS updates are fine, and docker update notifications are fine, but automatic docker updates are just too dangerous.
That’s reasonable, however, my personal bias is towards security and I feel like if I don’t push people towards automated updates, they will leave vulnerable, un-updated containers exposed to the web. I think a better approach would be to push for backups with versioning. I forgot to add that I am planning a “backups with Syncthing” article as well, I will take this into consideration, add it to the article, and use it as a way to demonstrate recovery in the event of such an issue.
it’ll still cause downtime, and they’ll probably have a hard time restoring from backup for the first few times it happens, if not for other reason then stress. especially when it updates the wrong moment, or wrong day.
that’s the point. Services shouldn’t be exposed to the web, unless the person really knows what they are doing, took the precautions, and applies updates soon after release.
exposing it to the VPN and to tge LAN should be plenty for most. there’s still a risk, but much lower
Consider warning the reader that it will not be obvious if backups have stopped, or if a sync folder on the backup pc is in an inconsistent state because of it, as errors are only shown on the web interface or third party tools
Yeah I agree with the warnings. One of the things I’m trying to ensure I get across accurately (which will be discussed later in the series) is how to do monitoring. Making sure backups are functioning properly would need to be a part of that.
My experience after 35 years in IT: I’ve had 10x more outages caused by automatic updates than everything else combined.
Also after 35 years of running my own stuff at home, and practically never updating anything, I’ve never had an outage caused by a lack of updates.
Let’s not act like auto updates is without risk. Just look at how often Microsoft has to roll out a fix for something an update broke. Inexperienced users are going to be clueless when an update breaks something.
We should be teaching new people how to manage systems, this includes proper update checks on a cycle, with appropriate validation that everything works afterwards, and the ability to roll back if there’s an issue.
This isn’t an Enterprise where you simply can’t manually manage updates across hundreds or thousands of servers, and tens of thousands of workstations - this is a single admin, small environment.
I do monthly update checks, update where I feel it’s warranted, and verify systems afterwards.
I don’t disagree with any of that, I’m merely making a different value judgement - namely that a breach that could’ve been prevented by automatic updates is worse than an outage caused by the same.
I will however make this choice more explicit in the articles and outline the risks.
with properly limited access the breach is much, much less likely, and an update bringing down an important service at the bad moment does not need to be a thing
This is really the truth. Auto-updating is really bad form when you are getting into server management. The first admin position I had back in the day had the rule that no automatic updates are to run, a manual update can only be run after 1 month of that update being released, and it had to accompanying documentation confirmed before it could be approved. The one time we did not follow that we ended up having to re-image the server in question from backup (as that was the quickest solution to getting it back online).
Immich is still unstable. This shouldn’t happen to a stable project.
What it tells me is that you need a regular backup
This absolutely can happen to stable projects. This has happened with Mastodon many times, and Mastodon has been stable for years.
It also has happened with Nextcloud many times, and again, Nextcloud has been stable for years.
It’s not a stability thing, it’s an automation thing. We as devs can only automate so much. At a certain point, it becomes up to you, as the administrator, to manually change things. Things like infrastructure changes, and database migrations, where the potential downtime if we automate it is something we need to consider.
Your probably right, you can’t catch each bug I guess
I use diun for update notifications. I wish there was something that could send me a notification, and if I gave it an okay or whatever it would apply the update. Maybe with release notes for the latest version so I could quickly judge if I need to do anything besides update.