New to WIP?
We're a group of makers shipping together. We help each other stay accountable and reach our goals.
Apply for accessBack
Question
Open source PaaS users: I have questions
Update 3: Dokku works great, and also supports k3s as a scheduler for multi-server when you need that level of complexity. Basically all other self-hosted PaaS apps are poorly documented, non-working, or bloated. This is the only one I can recommend from personal experience. Blog here: https://ben-makes-stuff.beehiiv.com/p/selfhost-app-2024-right-way
Update 2: I tried Kamal and it was unbelievably buggy. Deployments succeeded the first time then failed until I manually wiped the host, at which point they'd fail again. I had a valid healthcheck, container and everything was set up properly and I validated the container was up and working, but Kamal did not work as intended. This is not stable software and I can't recommend it to anyone at this time, might be better when Kamal 2 comes out.
Going to give Dokku a try *and* k3s and write up a blog post comparing all of these options when I'm done.
Update: I'm going with Kamal, Dokku has too many features that I don't need: https://wip.co/posts/open-source-paas-users-i-have-questions-23jdt4#comment_28859
Thank you to everyone who responded!
=====
I've been spending some time thinking about my move away from ThE cLoUd as my AWS credits are expiring in less than a year and I will have to start paying out of pocket. AWS has way too many hidden fees and will eat into my profit margins too much, so sticking with them is a no go.
I'll be starting out with: no Kubernetes, no load balancer, no AWS - just a single VM on Hetzner or some other low cost provider with Cloudflare in front of it. I'll vertically scale until the wheels fall off, then add complexity as needed. No SQLite for various reasons, sticking with Postgres.
I've evaluated the following options for deploying apps to this kind of setup:
1. Dokku
2. Coolify
3. Caprover
4. Kamal
5. K3s
So far it seems like Dokku is the best fit because it seems to have the best docs of any of these and supports hosting on a single node out of the box.
Here are my questions:
1. Does anyone have experience with Dokku (or any of the above) in production environments or for deploying your apps? Was it easy to work with? Any gotchas?
2. Have you tried to use the Postgres extension and how did you deal with backups and upgrades? I found docs for both and both seem relatively easy, just looking to confirm that this really is the case.
Thanks in advance!
Update 2: I tried Kamal and it was unbelievably buggy. Deployments succeeded the first time then failed until I manually wiped the host, at which point they'd fail again. I had a valid healthcheck, container and everything was set up properly and I validated the container was up and working, but Kamal did not work as intended. This is not stable software and I can't recommend it to anyone at this time, might be better when Kamal 2 comes out.
Going to give Dokku a try *and* k3s and write up a blog post comparing all of these options when I'm done.
Thank you to everyone who responded!
=====
I've been spending some time thinking about my move away from ThE cLoUd as my AWS credits are expiring in less than a year and I will have to start paying out of pocket. AWS has way too many hidden fees and will eat into my profit margins too much, so sticking with them is a no go.
I'll be starting out with: no Kubernetes, no load balancer, no AWS - just a single VM on Hetzner or some other low cost provider with Cloudflare in front of it. I'll vertically scale until the wheels fall off, then add complexity as needed. No SQLite for various reasons, sticking with Postgres.
I've evaluated the following options for deploying apps to this kind of setup:
1. Dokku
2. Coolify
3. Caprover
4. Kamal
5. K3s
So far it seems like Dokku is the best fit because it seems to have the best docs of any of these and supports hosting on a single node out of the box.
Here are my questions:
1. Does anyone have experience with Dokku (or any of the above) in production environments or for deploying your apps? Was it easy to work with? Any gotchas?
2. Have you tried to use the Postgres extension and how did you deal with backups and upgrades? I found docs for both and both seem relatively easy, just looking to confirm that this really is the case.
Thanks in advance!
👋 Join WIP to participate
cc @marc because I searched before making this post and it appears you spent some time working with Dokku and I remember you tried Kamal for WIP (maybe it's still on Kamal?), would you happen to have any insights?
I tried Dokku quite a while ago and used it for some of my smaller apps for a while. I can't quite remember why I stopped using it. I think there was some limitation with regards to Postgres versions or something and once you want something they don't offer out-of-the-box, things get complicated.
These days I use Render for my largest app that require stability and where I can afford some higher price, and Kamal + Digital Ocean (and their hosted PostgreSQL instances) for some of my smaller apps that are all share the same servers, etc.
Kamal works for me, but took a while to figure out and get running consistently.
They are all different trade-offs. But if costs are important to you, I think Kamal is a decent option. Just prepare having to learn how it works, how Docker works, etc.
Good to know, thanks! Yeah, cost is the most important right now as many products are just getting off the ground.
I'm already familiar with Docker, just need to learn about Kamal. I take it you were able to configure Kamal to host everything on the same VPS for your smaller apps?
Yes, all the (small) websites are hosted on the same server. People say Kamal isn't optimized for that, but it works fine for me.
I do host all databases separately using their managed services. You can include those in your Kamal server if you prefer, but for me I prefer having it managed for me so I have automatic backups, can easily login remotely via TablePlus, etc.
Sorry for going off-topic, but why no SQLite? And what are you hoping to get from using something like Dokku? Is it to just save time with the server administration?
No worries:
For SQLite:
I don't want to spend time converting my schema and data to SQLite format. SQLite doesn't fully implement the SQL standard so certain things that I'm using today are missing. I have tables with array columns for example, and I'd have to spend extra time fixing these tables, adding join tables, etc.
I have customers that need reports generated from the database and shared with them. I have certain reporting tools that require a SQL connection to generate these reports and they don't support SQLite. I'd have to find a new tool that does and self-host it which would then eat up more system resources.
For security reasons, I use separate SQL logins for these reporting situations and SQLite doesn't support multi-user - since it's just a file there's only one set of "user credentials" -- if you can access the file, you can open a sqlite connection.
For Dokku or something like it: yes, it saves time with server administration. Makes it easy to deploy an app - otherwise you need to write some script yourself that logs in via SSH, copies your binaries over, and manages the process on the server (or does something equivalent to pull the latest docker image, kill the old app if any, etc). Also need to install services like Postgres manually and screw around with config files. Doable, just requires extra time and I'd rather spend my time building apps vs. doing too much maintenance.
Option 6: If you don't need a lot of features, a simple watchtower+github action+docker compose setup goes a long way! I've had setups run like that for years without issues now (also on Hetzner)
Interesting, never heard of Watchtower - is it this thing? github.com/containrrr/watchto…
If so I'll give it a look.
Yes! It basically just polls your registry (probably github?) and then updates your docker images. Very easy to use, put both behind caddy and everything in one docker-compose and you have a pretty solid setup imo.
Cool, seems nice and simple - the only thing this is really missing is management for Postgres.
Dokku has some nice tools for dealing with that it seems and also supports Caddy as a reverse proxy, so I'm still leaning in that direction.
What management features do you need, besides backups? I've used Postgres (on hetzner) + Grafana Cloud for monitoring + custom backup script (dumps data every few hours to a few locations) and that works for me, never missed anything else
Automated backups and upgrades for new PG versions
I'm sure I could write a script to do both (would really only want to automate backups), but Dokku already has all of that: github.com/dokku/dokku-postgr…
I will give this a look though - I do get your point about simplifying things and just doing it yourself.
Yes, but Kamal is not really that much harder at all and already solves zero downtime and assets bridging for you...
When I look at what Dokku became after I used it back in the day, I am not impressed. It left its simplicity mindset I think.
As you already know :) I am team Kamal and Kamal 2 will actually solve one of your reasons for Dokku. I too deploy 1 app == 1 VM.
Kamal Handbook has a single server example that can be used today and I'll update it for Kamal 2 when it comes out. There is also a basic pg backup to S3 example.
Josef is too classy to include a link to his book, so I'll do it for him:
kamalmanual.com/handbook/
The purchase price pays itself back easily in the time you save.
Thanks Marc, I name you the official book ambassador :D
Ok, after a good amount of research I've made a decision: Kamal it is. Dokku, Caprover, Coolify, etc are quite powerful but I've realized I don't need 99% of the features offered.
I thought about the absolute core of my use case and it's: deploy and run a web api behind a reverse proxy, deploy and run scheduled jobs, ship logs to grafana, avoid downtime and manual tasks when deploying.
Kamal supports all of that just fine with Traefik as the reverse proxy, Vector as a stateless accessory service for log shipping, and linux's builtin crontab utility for scheduled work.
Because some might find this interesting: while the postgres upgrade tool in Dokku appears nice when you read the docs, upon further investigation it only handles minor upgrades which are trivial anyway (all you need to do is update the image version and restart the container with the same data mounted) - major ones still need to be done manually with a DB dump so it doesn't provide much value.
To that end, I may deploy Postgres myself as an accessory service with Kamal and script out major upgrades as needed, but I don't really need to worry about hosting Postgres myself because I found a managed provider (Aiven) with a free tier (5GB disk, 1 cpu, 1GB RAM) that's more than good enough for what I need right now.
In other news:
Thanks in part to @marc, the newly appointed Chief Marketing Officer of Kamal Handbook Enterprises, Inc: I have pulled the trigger on the patron copy of Josef's book!
Give this man a raise and a c-suite title. He's a straight shooter with upper management written all over him.
(Also, bought this because I already told @strzibny in person that I would buy it when I was getting more serious about moving off AWS and I am a man of my word 😎)
Thanks to everyone who responded, very helpful discussion.
Thanks a lot Ben! Kamal Handbook Enterprises hmmm do I need to do some hiring now and scale it to the moon? 😂 And nice find with Aiven, didn't know about them.
What you probably want is rancher, I've been using it for years on end in Hetzner, I was very critical on them but for what you want it should be perfect. You get a web based interface, and can click on objects which is really useful cause it gets you a nice overview of the whole cluster. Plus you can install stuff helm charts through it and so on. If you need some help I could give you a more detailed direction
Ultimately I'm skipping everything related to K8S, the level of complexity there is not warranted for what I'm trying to do
This is something I'll revisit at a much different stage when I have a lot more scale to worry about, thanks for letting me know about Rancher as I wasn't aware it existed
then just skip images all together....just use a simple vm and deploy your binaries there
That's too simple to the point where it will cause problems and extra maintenance.
To be clear, what I'm looking for is a "middle ground" solution that doesn't involve all of the configuration required to deploy and manage a K8S cluster, but still gives me some of the benefits I might get from that.
I have production experience with K8s - it works great especially at a large scale (my last company was doing hundreds of thousands of QPS), but I'm not a fan for small deployments.
Things I need:
- Reverse proxy (because I'm not going to serve traffic directly from the app, that is a risky proposition given bot/nefarious traffic - dev servers aren't meant to be true web servers for that reason)
- Environment variable and secret injection
- Some way to deploy apps easily and replace old apps, ideally with zero downtime
- Some way to set up postgres
This is exactly what Kamal, Dokku, and similar solutions provide without needing too much extra config - so I'm leaning in the direction of one of those.
maybe github.com/kube-hetzner/terra… would be a good middle ground for you, It should be way easier than a full k8 implementation....
I'll give it a look - thanks
As a fun sidenote, 37signals were talking with Rancher Labs to support their off cloud Kubernetes deployment and found it so expensive that it was one of the reasons leading to Kamal :)
I use Caprover and it's awesome. I am rather a beginner, but didn't have any problems. I run 2 of my websites there marekbrze.com and ziarnisty.pl and also Umami Analytics, Directus CMSes, Kutt Link shortener, n8n. Recently I created autoamtic backup solution using n8n and google drive
Cool, that's really smart to take advantage of the google drive free tier since it's just for backups anyway.
Caprover seems decent, I'll admit that I was just a bit put off because the documentation seemed a bit amateur/wordy, the website and admin ui didn't look as nice, and there were some spelling mistakes that bothered me
I'll give it another look though!
Yup, the docs are not well maintained. I even contributed to them twice after I learned how to do some things. Generally the Slack Community for Caprover is good and they know some stuff that is not in the docs