Are there any Discord servers or somewhere in the Matrix to chat about hosting a Lemmy instance? I’ve got Lemmy running, but I think there are several of us in the same boat struggling with federation performance issues and it might be good to have some place to chat real time.
You must log in or # to comment.
deleted by creator
My server is struggling with federation. Pretty much everything I see in the logs with debug turned on is this:
2023-06-20T01:55:28.018419Z WARN Error encountered while processing the incoming HTTP request: lemmy_server::root_span_builder: Header is expiredThis is exactly what I am seeing. I just tried upping federation_worker_count in the postgres database. I saw someone in another thread mention trying that so we’ll see.
The matrix space have multiple rooms, one explicitly related to instance admins
This is perfect. Thank you!
From the docs / troubleshooting:
“Also ensure that the time is accurately set on your server. Activities are signed with a timestamp, and will be discarded if it is off by more than 10 seconds.”
Honestly- a lot of the performance issues aren’t due to OUR servers- but, the upstream servers.
beehaw.org, lemmy.world, for example- I think their servers are completely overloaded, and are having issues keeping up.
I don’t have sync issues for the smaller/other servers at all. Just the big ones.
I have 128G of ram, 32 cores dedicated. I have the federation worker count set at 256. There is NO shortage of resources, and my server sits more or less, idle.
Due to this only really impacting those larger instances- I believe the blame may lie there.
I think it is less about pointing fingers as to who’s at blame, and trying to see if there are things we can do to resolve/alleviate that.
I recall reading somewhere that @Ruud@lemmy.world mentioned before that the server is scaled all the way up to a fairly beefy dedicated server already, perhaps it is soon time to scale this service horizontally across multiple servers. If nothing else, I think a lot of value could be gained by moving the database to a separate server than the UI/backend server as a first step, which shouldn’t take too much effort (other than recurring $ and a bit of admin time) even with the current Lemmy code base/deployment workflow…
Well- I do know- most of the components do scale.
The UI/Frontend, for example, you can run multiple instances easily.
The API/MiddleTier, I don’t know if it supports horizontal scaling though. But, a beefy server can push a TON of traffic.
The database/backend, being postgres, does support some horizontal scaling.
Regarding the app itself, it scales much better if EVERYONE didn’t just flock to lemmy.ml, lemmy.world, and beehaw.org. I think that is one of the huge issues… everyone wanted to join the “big” instance.
