Hello everybody,
my plan is to switch from Android to GrapheneOS. In this process, I want to get rid of my reliance on my google account as much as possible.
To this end, I’d like to selfhost some “critical” data, e.g.
- contacts
- calendar
- online drive for files (e.g. google drive alternative)
- some basic note-taking app (like google keep)
and so on.
I do some selfhosting already, though it is not that kind of “cannot lose this” data. So I’d like to share my thoughts and ask for your opinions and experience for the process.
More details for what I want
contacts
- have to be syncable to the phone
- if possible, some webinterface to edit / browse
calendar
- has to be syncable to the phone
- webinterface + sync to desktop / phone
- if possible, send invite-links to events to others
drive
- files of my choosing must be offline-available
- ever other file should not use storage on the phone
- if possible, able to share links to download files
- if possible, able to share links to view with online editor (see below)
document editor
- think google sheets / google docs
- if possible, able to share links to view documents online
smartphone photos
- auto-backup camera folder
There may be some things I’m not thinking about right now, but this seems to pretty much be it.
If possible, all of this should be accessible only via vpn.
What I already have
I have a pfSense physical appliance that’s already managing my home network, got an OpenVPN already setup, dynamic DNS working properly for the lack of a static IP, etc.
I own 2 mini-PCs (some Intel NUC, some passive-cooled zotac with an intel with 4c/8t). One of them (zotac) is currently running as my Proxmox Virtual Environment Hypervisor, managing 3 VMs.
I also have a second PC which misses some critical parts, so it is not currently in working condition. I think there’s an AM4 mainboard and 16 or 32GB of DDR4 RAM in there. I could make a NAS or a new hypervisor out of this, but the case (Fractal Design Define 7) is quite big and a full PC is probably worse for energy-efficiency than my 2 mini-PCs and is going to be more expensive.
Not much in terms of storage sadly
- 1x 6TB external USB HDD (used for backups)
- 1x 2TB external USB HDD (used for data)
What I plan to do
The kind of data I’m going to be hosting myself now is very import, so it cannot be lost or corrupted.
But the feature list doesn’t seem to be overly complicated. This seems like something nextcloud could do.
This means, I will probably need to buy
- 2x 4 TB HDD for storage for data RAID
- 2x 8-10 TB HDD for backups
- 2x external RAID case
Then I could connect the data RAID to the already running zotac pc and spin up new VMs for nextcloud and whatever else I might need and start serving my data from home.
The Intel NUC will be used as a Proxmox Backup Server, connected to the backup RAID. Keeping some daily, weekly and monthly backups.
On the phone-side, I’d have the vpn always active. Whenever active, sync of contacts, calendar entries, photos etc. should be possible.
Questions
Is there anything I missed? Did any of you already try something like that? Does anybody here see a potential problem with any of the above?
Can anyone recommend a RAID-1 external enclosure without a fan and some quiet and energy-efficient HDDs?
You must log in or register to comment.
I assume you basically want protection against disasters, but not high uptime.
(E.g. you likely can live with a week of unavailability if after a week you can recover the data.)
The key is about proper backups. For example, my Nextcloud server is running in a datacenter. Every night I replicate the data to a computer running at home. Every week I run a backup to a USB drive that I keep in a third location. Every month I run a backup to a USB drive on the computer I mentioned at home.
So I could lose two locations and still have my data.
There is much written about backup strategies, for example https://en.wikipedia.org/wiki/3-2-1_backup_rule … Just start with your configuration, think what can go wrong and what would happen, and add redundancy until you are OK with the risks.
I use Nextcloud (currently using their AIO docker images) for all if that. Not sure if it checks all the boxes perfectly, but if not it is probably as close as youll find ready-made.
are you using any of the video/audio calls on that setup? how much horsepower does it need?
I have only tested them a bit and it worked fine, but I havent relied on it “in production”, as it were. I use that machine for too many things for it to be any kind of benchmark. Mine is running on an old HP workstation w/ a 7th gen iGPU chip. I am also relying on it for plex and unmanic using QSV for transcoding. It only ever has trouble if I over-tax it with those.
Nextcloud with a 3-2-1 backup strategy is ace. Proxmox can auto-backup, it’s slick.
A few things. I also think nextcloud is the way to go for what you want. I’ve gotten rid of anything Google I can. Except for maps. Man, there just is no substitute especially when mobile.
I always do, but I’m going to suggest Unraid for a NAS. Pay the money and then just enjoy it. I fought with truenas for over a year before I succumbed. You can totally play around with zfs, striped arrays whatever. I do not recommend an external enclosure. I think you’ll come to hate it for lack of ability. I recommend biting the bullet and building a machine or putting your current PC components into a real case with upgradability if possible.
Also, I wouldn’t plan on running Nextcloud in a VM. Nextcloud is pretty beefy and a VM adds complexity that I suggest against. A docker AIO version of nextcloud running on as close to bare metal as you can is probably the best option for performance.
Did unraid implemented a true raid already, or is still periodical sync like snapraid? If not, this kind if solution would be great for the multimedia library that does not change often, but I’d go for the more traditional raid for critical data.
Except for maps. Man, there just is no substitute especially when mobile.
I thought there was an android app for open street maps, but I couldn’t find any on play.google.com either.
I do not recommend an external enclosure […] you’ll come to hate it for lack of ability
I feel kinda the same, but on the other hand, having a full-blown ATX system running in my living room isn’t going to be my first choice. If I can’t manage with the zotac mini PC, I can still take the drives out of the enclosure and put them in a full ATX case. That’s more of a “last resort” though.
A docker AIO version of nextcloud running on as close to bare metal as you can is probably the best option for performance.
I’m not worried about performance all too much. The only thing constantly connected will be my phone, for syncing contacts, calendars and, every now and then, a new photo or two. Sometimes I open the calendar in my browser on my desktop or laptop to add / change an event. I really don’t use it too extensively.
And to aid in CPU and performance of the VM, I can always have a VM with the “host” CPU type, which should forward CPU capabilities and features to the VM.
I think you can get Open Street Maps in the F Droid app store. But, as much as I appreciate OSM, it’s just not the same as Google maps. The speed, accuracy and information doesn’t seem to have an equal.
You definitely can run Nextcloud in a VM. With decent hardware, it will do it. I guess I would say it depends on needs and expectations. My install is not snappy to me. I’ve got what I feel is a very beefy server but still. Just feels a little slow at times. Totally functional. Just has a small amount of lag when doing anything. I’ve read people say they have none at all. But when you’re busy and relying on it, my suggestion is to eke out everything you can for it for a better experience. Not make or break by any means.
If you don’t mind, what’s the problem with truenas?
I don’t think it’s a problem per se, as much as it’s a difference in priorities. But the docker implementation in TrueNAS is more of an afterthought. I think they’ve fixed some issues but checking out their forums, many of the issues I faced seem to still exist. Docker packages corrupting and not being accessible in any way, not updating, just seemingly, not robust. Also, I disliked the file permission structure but that’s more preference I think. I would say TrueNAS is a great NAS just not the best hypervisor and NAS.
As far as the “what you want” stuff goes, Nextcloud can do all of it and I use it for exactly that.
Oh, it’s nice to hear somebody already did that, thank you!
Did you have any hiccups or general problems with nextcloud or calendar/contacts/photos sync? Did you do any specific thing to harden security, other than using
ufw,fail2banand changingsshdconfig?Haven’t had any issues whatsoever.
I’ve done nothing special regarding security and have it exposed to the public internet. I intend on having fail2ban look at its logs but I’ve not yet set that up (entirely out of laziness).
If you want to be very secure I would recommend having it entirely behind a VPN. I personally use tailscale+headscale for my internal only services but like I said I have Nextcloud publicly exposed as I want to be able to access it from potentially any device.
I’ve done nothing special regarding security and have it exposed to the public internet. I intend on having fail2ban look at its logs but I’ve not yet set that up
That sounds kinda dangerous. I remember years ago, when I rented my first vcloud-server, within the first 10 minutes I had bots trying to get in via SSH. I’d be way too paranoid.
I would recommend having it entirely behind a VPN
Yes, that’s my plan. I intend to create a new OpenVPN server on my pfSense with access only to the nextcloud VM. This would also allow me to share the vpn config files with my friends without a password, as the authentication is done by inline-cert vpn config.
Nextcloud does have a problem with the online editor. It frequently bugs out and moves things out of order or just doesn’t feel snappy.
Some time ago there was also saving issues
I only sync my computer and I have nothing to report. It just works. There’s just a small bug when you just create a file on windows and start editing it, the file shows as being edited by Nextcloud for a couple of seconds, but then it works. It’s just on the initial creation.
Are the documents you edit with the online editor files which are visible in the online drive? Does nextcloud use the open document specifications for saving documents (e.g. .odt, .ods)? Can you view these files without opening them in the editor (like the preview in google drive)?
If so, that is acceptable. The document thing is more for completion, I don’t handle documents all too often. And if the online editor is bad or not working but the files are visible and offline-syncable in the drive to some desktop client and they are using the open document format, I can edit them with libreoffice.
Thanks for the heads-up!
There is a somewhat native file editor in nextcloud but for your needs I would recommend setting up integration with either OnlyOffice or Collabora (LibreOffice). I am currently using OnlyOffice and while my utilization isn’t much or often it seems to do what it should without any fuss.
Thanks for the tipp!
I’ll definitely try the native file editor and collabora, just to see how they compare for me. I even found a tutorial by nextcloud on how to integrate collabora (see this post)
-
Are the documents you edit with the online editor files which are visible in the online drive?: Yes. It works like Google drive basically, and yea, I don’t use an external editor or something. I just create or upload a file to the cloud, and edit it there using the built in web editor (you just open the file and it opens the editor)
-
Does nextcloud use the open document specifications for saving documents (e.g. .odt, .ods)?: Yes. I believe they use a modified version of Collabora or something. By default, you use the same extensions you’d use with libre software like collabora or libreoffice. It supports opening documents from word, PowerPoint and excel… but often fucks up the formatting in some parts (much like libreoffice)
-
Can you view these files without opening them in the editor (like the preview in google drive)?: No. You view them only via the editor. It should respect permissions though, so if you share a file with read access only, they won’t be able to edit it in the editor.
You can use any format you want in Nextcloud, it’s just that they might not be supported by their built in editors, but they’ll work fine.
The reason I use the built in editors is that multiple people can work on a single file at the same time
You’re welcome!
Thank you for answering!
Good to know that most things I would need seem to be already working nicely in nextcloud :)
It should respect permissions though, so if you share a file with read access only, they won’t be able to edit it in the editor.
I’ll definitely have to try that before trying to send out links.
Sure! I always try this
A small downside: you can’t seem to restrict people commenting on a file you shared. There’s a built in “discuss about this file” feature but it shouldn’t really be a problem, unless you intend on sharing a file to a looot of people, because afaik you can’t moderate it
I’ll definitely have to try that before trying to send out links.
Yea, I recommend you always test features to see how they work and what they imply (and if they’re bugged, because Nextcloud often updates and sometimes breaks small things)
you can’t seem to restrict people commenting on a file you shared
That’s okay. My circle of friends I’d share files with is not all too big. So everything stays between a few people anyway.
Nextcloud often updates and sometimes breaks small things
Does breaking stuff happen often? I plan to use the docker image nextcloud:stable-fpm in the hopes of bypassing some bugged releases.
-
I’ve done this.
For contacts, calendar, and files, I use OwnCloud, although NextCloud is as good/better. I couldn’t figure out Self-Signing certificates, which is supposed to be pretty easy, but I am kind of a dummy. NextCloud requires it. On my phone, I use DAVx5, and I replaced the GrapheneOS stock apps with Fossify apps as needed - although that is up to personal preference.
For photos, I use Immich, which is hands-down the best option.
NextCloud also has options for document editing, photo backup, and notes, but I can’t testify to those.
Syncthing is an ideal way to seamlessly sync files and folders between devices, but you will end up with the files on both devices. I use it sparingly, and they are phasing out android support. It’s still very useful to migrate large file libraries and act as a stopgap for other services.
There are tons of notes apps, and it largely comes down to preference. I settled on NotallyX, which can import your Keep notes directly (using Google Takeout iirc). It also has the option to store files externally, which means they can be synced for backup. There are also self-hosted web-apps, like Memos, or the slightly more adventurous Blinko.
I got a lot of great input from this community not too long ago.
The one Google feature I am not able to reproduce is Google Messages. If you use texting to any degree, there are some FOSS apps with pretty nice basic features (I’m using Fossify which is nice). However, there are none that have solid group-messaging features, reactions, and other RCS capabilities. It seems to be a technical/logistical/legal hurdle that is presently insurmountable. Lots of people don’t use texting anyway, so it may not be a concern for you.
Edit: And of course, you need a way to tie everything together. I use Tailscale, which is ridiculously easy to self-host.
Thank you for sharing your experience of the process!
On my phone, I use DAVx5
I’m a little confused after looking at the website. What exactly does DAVx5 do? The regular re-sync of contacts, calendar and files itself? Shouldn’t that be done by the contacts app / calendar app on regular intervalls?
with Fossify apps
I just downloaded fossify calendar on my android a few days ago to test it and got to see the other fossify apps :)
syncthing phasing out android support
Oh man, I already use syncthing for ~5 GB of files and I use it on my android too. Seems I’ll be trying syncthing-android-fdroid in the future then.
There are tons of notes apps
There really are a lot! NotallyX looks nice and simple, but memos also looks very interesting. And thank you for the link, I’ll go dive into that tomorrow.
The one Google feature I am not able to reproduce is Google Messages
I do not need RCS-compatible messengers. What I send via SMS is nothing more than pure text, also no group chats. I use signal and element for my “fancy” messaging needs :)
I use Tailscale
I’ll look into it some more over the next days, but on a quick glance, this seems like it is an online service where you need an account? If that’s the case, I’d prefer using my already running OpenVPN server to do the job.
DAVx5 basically acts as the connector between your server and your calendar/contacts/files apps. I would imagine that this could be built into an app, but there are a lot of ways that such apps can sync or operate locally. I’m guessing that it is just a little more specialized than most developers want to get.
Thanks for the Syncthing-Fork tip! For now the official version is working for me, but I’ll have to migrate myself soon.
From my understanding, OpenVPN provides the same secure remote access as Tailscale, by a slightly different method. You should be fine to use what you’ve already set up.
DAVx5 basically acts as the connector between your server and your calendar/contacts/files apps
Thank you for the explanation. I’ll probably be testing a lot of FOSS apps on my current android before I make the switch, so it’s good to know that I have to look out not just for usability, but also connectivity!
3-2-1?
Thank you for your input!
I also thought about the 3-2-1 backup rule, but am unsure if that is overkill.
My VM-backups and file-level-backups are proxmox backup server (pbs) backups. Meaning, to have them offsite, I’d need to rent a dedicated root server on which I am able to install pbs to act as an offsite sync-target. With TB of backups, this is gonna get very costly very fast.
I thought about regularly exporting encrypted calendar and contacts onto some free online storage, hoping I can automate this process.
With what I have layed out in my post, to lose contacts and calendar events, both my intel NUC and the zotac mini-PC have to be corrupted at the same time. Or both RAIDs simultaniously failing both drives. Am I not paranoid enough or is that an acceptable level of failure-safety?
What @AtariDump@lemmy.world said is correct, if it’s critical data, 3-2-1 is necessary. I personally use BuyVM as my offsite as it’s got pretty cheap storage (~$5USD/1TB/month), but if you’ve got family or friends with a decent internet connection, it’s trivial to set up a remote sync job to any offsite Proxmox Backup Server, perhaps on a box stored at their house.
Now, just to throw it out there, my actual ‘critical data’ is way smaller than my total backed up data, including my media library, random ISOs, etc. - it can be worthwhile to determine if you really need to backup everything offsite or if you can sort out some less necessary data, and only upload some data to a remote server. Maybe the answer is yes, and you’ll need to account for that!
I took a look at the BuyVM offer you mentioned since it sounds really good, but am I understanding correctly that to make use of the 1TB storage offer I would have to also order a dedicated VM with them to actually make use of it? (i.e. no mounting from a vps with a different provider)
That’s correct, I also pay for their cheapest VPS, which is about $3, pretty good overall for my purposes!
Now, just to throw it out there, my actual ‘critical data’ is way smaller than my total backed up data
That’s also the case for me. I’d probably count a few GB as critical. Contacts, Calendar, some photos, some documents.
If nextcloud (or some other alternative) has the ability to regularly export these things to an on-disk location, I could definitely backup that to some cheap hetzner server. This will not be a pbs backup, but I can get by with an offsite-backup done by something like restic or rsnapshot
Thank you for your advice!
The offsite rule is mainly in the scenario where your house burns down for example, or if someone steals your stuff. It can happen.
Maybe your electricity will have variations and will fuck up all your devices in a specific location…
Yes, you’re right. As David From Space said in this comment, the real critical data is far less then all of the backed up data.
So I definitely can have an offsite-backup, it just depends on if I can single these things out in nextcloud, possibly via regular export to the filesystem.
Yea if you don’t need much then you can do with exporting not a lot of stuff.
Google is evil but I know that GDrive has pretty low prices on data storage
There are many cold storage services out there with good pricing. If you need a VPS with good storage (to automate sync, etc… idk), I know I would use Interservers, based in the USA, priced at 3$/TB/month (HDD)
But if you only want to sync a small amount then you can do with free services, probably. Don’t forget to encrypt everything when uploading to these services! Don’t want them to be able to see the content of your files.
Or a house fire, or flood, or lightning strike, or theft. Or just plain fat fingering something and deleting it all.
If you really mean life-or-death critical, yeah, 3-2-1 is the starting point.
If you really mean life-or-death critical
No data I own is “life-or-death” critical.
I can ask around for contact info again, same with calendar events I had planned. Some documents can be restored via the original service or by paying a fee to get a new original document, I still have folders full of originals in paper form. Some info can be restored by looking through my bank account or online buying activity. Losing my photos would be really sad, but nothing of that will kill me or destroy my life.
But I definitely can save the most critical stuff (probably a few GB only), if nextcloud (or some alternative) has the ability to regularly export these to an on-disk location. This way, some backup utility like restic or rsnapshot shoud be able to do the job.
Okay so not critical, just mildly inconvenient if lost.
I would just keep one copy in RAID, and for the most important stuff a second copy locally or in the cloud. Yes, RAID is not backup, but a disk failure is probably the most likely failure scenario. Corruption is the second most likely.
I also thought about the 3-2-1 backup rule, but am unsure if that is overkill.
Maybe you shouldn’t be home hosting critical data if you think this is overkill.
You’ve got a point, but now I gotta ask: Where do you store your original paperform documents? You know, the real-life critical things. Maybe I’m wrong, but I feel like most people store these things at home, possibly tucked away in a neat, little, sorted folder, for preservation. Which would be a nightmare for all the same reasons, but seems strangely accepted and widely practiced.
No data I own is life-or-death critical. Losing everything would be really bad, but many things can be restored in alternative ways, except the photos.
Also, I may be able to backup the most important stuff (which would only be a few GB at most) to an offsite server, as long as nextcloud (or an alternative) is able to export contacts, calendar and photos, or I can single these out in some other way. As long as this somehow works, I can rent a cheap hetzner server with a few GB of storage and have that be the backup target for the most critical stuff.
My critical documents that MUST be in print are stored in a fireproof safe. If it doesn’t need to be in print it’s scanned, shredded, and backed up.
Photos are scanned and stored. They’re also backed up.
For document editing, I have had fairly good luck with OnlyOffice, although it is not without its issues. Others also recommend Collabora, which plays well with NextCloud and LibreOffice.
Thank you for the tipps!
A quick google search even reveals a nextcloud tutorial on how to install it . I’ll definitely try that out.
ZFS on the bottom, Nextcloud on top. Which distro is up to you. Mine is to be set up with nix on my next refresh this fall. Already in testing.
Thank you for the tipp!
Though I gotta ask: would ZFS still bring an advantage, considering that the RAID is going to be managed inside the external RAID enclosure, so ZFS would never see the actual disks? Or did I misunderstand how these enclosures work?
Memos is pretty usefull for me. App on fdroid momemos is superb. Syncthig takes care of google drive ish needs. Immich for photos.
Mealie keeps food interesting.
Have not done calendar or contacts yet.
Running a few on a low power pi5 using docker for the most backup pihole runs on baremetal.
1 16tb external and 2 5tb external. Not the best but i dig it.
I need a nuc.
Memos is pretty usefull for me. App on fdroid momemos is superb. Syncthig takes care of google drive ish needs. Immich for photos. Mealie keeps food interesting.
I’m going to have to test a lot of new android apps, I guess. Thanks for the mentions!
Regarding syncthing, according to gedaliyah’s answer here, syncthing will be dropping the android app :(
For whats it’s worth I’ve been running a fork of the syncthing android app for well over a year now and it’s not skipped a beat.
https://github.com/Catfriend1/syncthing-android
Up to personal preference if you trust a fork for this work. But with sufficent backups I’m comfortable with it.
Ah yes forgot to mention in my family we run syncthing fork from fdroid also.
I’ve been slowly, very slowly, migrating away from Synology stuff, but everything you mentioned are my holdouts because they have been rock solid for decades. Even the cheap used products can swing those apps.
I hate to be the bad influence (no, who am I kidding, not really) and suggest more servers, but if you can find something cheap, I’d maybe give it a try.
All of this will be sitting in my living room somewhere, so I’d like to keep the number of devices and the space I need for the setup to a minimum.
I do know Synology has very solid products, but I’d rather do it myself and have full control over the servers. I use Fedora and my VMs all run debian. I also try to deploy as many services as possible with docker, as that makes it very easy to migrate stuff to another machine and test the next version before using it in production, if the need arises.
Totally get it. And if you find a cool solution, let us know.
I will. Thank you for your advice!
I’m a little reassured that at least I am not the only one seeing no problems with this plan :)
