# boring infrastructure, on purpose: why salix.host runs synapse while i build vitis

· #matrix #self-hosting #vitis

salix@host:~/posts/boring-infrastructure-on-purpose

salix.host runs Synapse. At the same time, I am building Vitis, an experimental Matrix homeserver in Rust.

That may look a little strange. If I am interested enough in a new homeserver to build one, why do I keep using Synapse in production?

The answer is simple: salix.host has real users. Vitis is still where I am allowed to break things.

Those are two very different jobs.

Real users change the rules

When I run software only for myself, a broken update is annoying. I can restore a backup, read the logs, fix the problem, and call the whole thing a useful lesson.

It is different when other people depend on the server.

A bad update can stop someone from signing in. A database problem can make rooms unavailable. A federation bug can leave messages waiting between servers. Even when a room is end-to-end encrypted, the homeserver still holds the encrypted events and the room data needed to keep everything working.

People did not create an account on salix.host to test my latest database idea. They created one because they wanted a Matrix account that works.

That changes how I make choices. The most interesting option is not always the right option. Sometimes the right choice is the one that has already survived years of strange rooms, failed upgrades, broken network links, and tired administrators.

Boring is a compliment

I call the salix.host setup boring on purpose. In infrastructure, boring is usually good.

Boring software has known behaviour. It has documentation. Other people have already found many of the bad edge cases. When it fails, there is a fair chance that someone else has seen the same problem and written down how to recover from it.

Synapse is not perfect. It can use a lot of resources, its database can become large, and upgrades still need care. Bugs do not disappear just because a project is old.

But Synapse has been used in real Matrix deployments for years. It has a large codebase, a long history, and tools that already understand how it works. Its problems are often known problems. That matters more to me than winning a benchmark or using the newest design.

Reliable software is not software that never fails. That software does not exist. Reliable software is software whose failures are easier to predict, notice, and recover from.

For salix.host, that is a feature.

Then why build Vitis?

Because using reliable software does not mean there is nothing left to improve.

Vitis is my very early Matrix homeserver experiment. It is written in Rust and forked from the Conduit family of homeservers. It is not public yet, it has no releases, and it is not serving real users.

That gives me space to explore ideas without pretending they are ready.

One area is compatibility with the Synapse admin API. Many existing Matrix administration tools are built around Synapse. A different homeserver may support the Matrix protocol correctly and still require a completely different set of admin tools. Supporting useful parts of the Synapse API could make that gap smaller.

I am also looking at how old room state is stored and cleaned up. Matrix rooms collect a lot of state over time. This includes things such as membership, permissions, and room settings. The server also needs enough old state to understand which rules applied to earlier events. Cleaning up unused data can save space, but deleting the wrong data can damage room history or make later events impossible to check. The cleanup needs to be careful, not clever.

Cache management is another part of the work. A cache keeps often-used data in memory so the server can answer faster. The hard part is deciding what to keep, what to remove, and how to stop memory use from growing forever.

Then there is the local smoke-test system. The idea is simple: start a fresh server, perform a set of basic actions, and fail quickly when something important stops working. A small test like that cannot prove the server is safe, but it can catch obvious mistakes before they reach a larger test setup.

None of this makes a very exciting launch screenshot. That is probably why I enjoy it.

Matrix is larger than it first looks

A homeserver can look simple from far away. It accepts requests, stores events, and sends data to other servers.

Then federation enters the room.

A federated server does not only deal with its own users and its own database. It also accepts data from servers run by other people. Some of those servers are slow. Some disappear for weeks and return later. Some have old software. Some send events from rooms with years of complicated history.

The homeserver has to check which events are allowed, keep room state consistent, handle events that arrive late, and continue working when another server does something unexpected. It also needs to support clients, logins, devices, media, moderation, account management, database upgrades, and many other ordinary jobs around the protocol.

This is why a homeserver can pass every small test I wrote and still fail in production. My tests cover the problems I know about. Real rooms are very good at finding the problems I did not imagine.

Synapse has already met a lot of those problems. Vitis has not.

Tests are not the same as trust

A good test suite is important, but production trust needs more than tests.

I need to know what happens when an upgrade stops halfway through. I need to know that a backup can really be restored, not only that a backup file exists. I need to know what happens when the disk becomes full, the database slows down, or another homeserver becomes slow or disappears during a request.

A backup that has never been restored is only a hopeful file.

Monitoring matters too. It is not enough for the process to be running. I need to see when federation queues are stuck, when memory use keeps rising, when database work becomes slow, or when an important task has quietly stopped.

Administration also has to be boring. Creating and disabling accounts, handling abuse, checking federation, reading useful logs, and recovering from mistakes should not require me to edit the database by hand at 3am.

These parts are less fun than designing a new storage system. They are also the parts that decide whether a server is safe to run for other people.

What Vitis would need before production

Vitis does not need to become perfect. It does need to earn trust step by step.

Before I would consider using it for real accounts, I would want at least:

  • database upgrades that can run safely and fail clearly;
  • backups that I have restored more than once;
  • good compatibility with common Matrix clients and other homeservers;
  • useful monitoring, logs, and administration tools;
  • long test runs with old rooms, large rooms, and unreliable federation;
  • a clear way to move back to Synapse if something goes wrong.

That last point is important. A new system is much safer when leaving it is part of the plan. Migration should not be a one-way door.

I would also start small. A private test server with throwaway accounts can find many problems without risking anyone’s real account. After that, I can run my own account on it and learn what daily use feels like. Only much later would it make sense to think about moving a public service.

A fast benchmark would not be enough. A clean codebase would not be enough. Even passing every protocol test would not be enough by itself.

The server would need to survive boring days, boring upgrades, boring backups, and boring failures.

One project, two different jobs

Building Vitis is not a vote against Synapse. Running Synapse is not giving up on Vitis.

Synapse runs the service. Vitis is where I learn, test ideas, and find out which ideas were worse than they looked in my head.

The two projects also help each other. Running Synapse shows me what an administrator actually needs. It reminds me that compatibility, recovery, and clear errors matter just as much as speed. Building Vitis helps me understand the protocol and the storage work below the tools I use every day.

Maybe Vitis will become ready for production one day. Maybe it will stay an experiment and teach me useful things. Both results are fine.

For now, salix.host will keep running Synapse. Vitis will keep breaking in a place where breaking is useful.

That is not a contradiction. It is the line between a project and a service.

A project can be interesting. A service has to be there tomorrow morning.

~$