Posted in r/selfhosted by u/dharapvj • 61 points and 12 comments
This is an automated archive made by the Lemmit Bot.

The original was posted on /r/selfhosted by /u/dharapvj on 2024-09-30 04:35:30+00:00.

I would like to show-off my humble self hosted setup.

I went through many iterations (and will go many more, I am sure) to arrive at this one which is largely stable. So thought I will make a longish post about it’s architecture and subtleties. Goal is to show a little and learn a little! So your critical feedback is welcome!

Lets start with a architecture diagram!

Architecture

Architecture!

How is it set up?

  • I have my home server - Asus PN51 SFC where I have Ubuntu installed. I had originally installed proxmox on it but I realized that then using host machine as general purpose machine was not easy. Basically, I felt proxmox to be too opinionated. So I have installed plain vanilla Ubuntu on it.
  • I have 3 1TB SSDs added to this machine along with 64GB of RAM.
  • On this machine, I created couple of VMs using KVM and libvirt technology. One of the machine, I use to host all my services. Initially, I hosted all my services on the physical host machine itself. But one of the days, while trying one of new self-hosted software, I mistyped a command and lost sudo access to my user. Then I had to plug in physical monitor and keyboard to host machine and boot into recovery mode to re-assign sudo group to my default userid. Thus, I decided to not do any “trials” on host machine and decided that a disposable VM is best choice for hosting all my services.
  • Within the VM, I use podman in rootless mode to run all my services. I create a single shared network so and attach all the containers to that network so that they can talk to each other using their DNS name. Recently, I also started using Ubuntu 24.04 as OS for this VM so that I get latest podman (4.9.3) and also better support for quadlet and podlet.
  • All the services, including the nginx-proxy-manager run in rootless mode on this VM. All the services are defined as quadlets (.container and sometimes .kube). This way it is quite easy to drop the VM and recreate new VM with all services quickly.
  • All the persistent storage required for all services are mounted from Ubuntu host into KVM guest and then subsequently, mounted into the podman containers. This again helps me keep my KVM machine to be a complete throwaway machine.
  • nginx-proxy-manager container can forward request to other containers using their hostname as seen in screenshot below.

nginx proxy manager connecting to other containerized processes

  • I also host adguard home DNS in this machine as DNS provider and adblocker on my local home network
  • Now comes a key configuration. All these containers are accessible on their non-privileged ports inside of that VM. They can also be accessed via NPM but even NPM is also running on non-standard port. But I want them to be accessible via port 80, 443 ports and I want DNS to be accessible on port 53 port on home network. Here, we want to use libvirt’s way to forward incoming connection to KVM guest on said ports. I had limited success with their default script. But this other suggested script worked beautifully. Since libvirt is running with elevated privileges, it can bind to port 80, 443 and 53. Thus, now I can access the nginx proxy manager on port 80 and 443 and adguard on port 53 (TCP and UDP) for my Ubuntu host machine in my home network.
  • Now I update my router to use ip of my ubuntu host as DNS provider and all ads are now blocked.
  • I updated my adguardhome configuration to use my hostname *.mydomain.com to point to Ubuntu server machine. This way, all the services - when accessed within my home network - are not routed through internet and are accessed locally.

adguard home making local override for same domain name

Making services accessible on internet

  • My ISP uses CGNAT. That means, the IP address that I see in my router is not the IP address seen by external servers e.g. google. This makes things hard because you do not have your dedicated IP address to which you can simple assign a Domain name on internet.
  • In such cases, cloudflare tunnels come handy and I actually made use of it for some time successfully. But I become increasingly aware that this makes entire setup dependent on Cloudflare. And who wants to trust external and highly competitive company instead of your own amateur ways of doing things, right? :D . Anyways, long story short, I moved on from cloudflare tunnels to my own setup. How? Read on!
  • I have taken a t4g.small machine in AWS - which is offered for free until this Dec end at least. (technically, I now, pay of my public IP address) and I use rathole to create a tunnel between AWS machine where I own the IP (and can assign a valid DNS name to it) and my home server. I run rathole in server mode on this AWS machine. I run rathole in client mode on my Home server ubuntu machine. I also tried frp and it also works quite well but frp’s default binary for gravitron processor has a bug.
  • Now once DNS is pointing to my AWS machine, request will travel from AWS machine --> rathole tunnel --> Ubuntu host machine --> KVM port forwarding --> nginx proxy manager --> respective podman container.
  • When I access things in my home network, request will travel requesting device --> router --> ubuntu host machine --> KVM port forwarding --> nginx proxy manager --> respective podman container.
  • To ensure that everything is up and running, I run uptime kuma and ntfy on my cloud machine. This way, even when my local machine dies / local internet gets cut off - monitoring and notification stack runs externally and can detect and alert me. Earlier, I was running uptime-kuma and ntfy on my local machine itself until I realized the fallacy of this configuration!

Installed services

Most of the services are quite regular. Nothing out of ordinary. Things that are additionally configured are…

  • I use prometheus to monitor all podman containers as well as the node via node-exporter.
  • I do not use *arr stack since I have no torrents and i think torrent sites do not work now in my country.

Hope you liked some bits and pieces of the setup! Feel free to provide your compliments and critique!