This happened after updating to the newest version, so we were convinced it was broken code but no one else was reporting similar issues.
For debugging, we undid our patches and added tracing to sidekiq and mastodon, which lead us to believe mastodon was filling memory because of elasticsearch failures. We'll reapply the patches shortly.
We've been having a lot of troubles with mastodon recently (filling memory and swap), which we traced back to elasticsearch freaking out about there not being enough space on the storage drive.
We cleared off some old backups/unnecessary data from the disk, so everything should be fine now. Sorry about the recent downtime.
mastodon moderation tools suck. performing moderation actions takes way longer than it should (it loads for a while), and for some reason disabled accounts can still post? I disabled 2 accounts a week ago and they somehow both posted today...
and to top it all off, after I suspended both of the accounts, the posts from one were still available on local timeline until I manually deleted the post.
I switched to fastmail today to try it out, they added labels in beta which was the last thing I needed to switch. gonna stay on the trial for a month but it seems pretty good so far.
the android app felt a bit sluggish but it's whatever. I can use a different mail app if it's a problem, plus I don't use my phone for viewing my mail very often.
and since I couldn't get alsa to work, I have to attach my microphone to the VM to be able to call people from inside which sucks but it's whatever
doom eternal runs good, so all is well
looking glass is working but the quality is ass, not sure why. maybe it's a wayland thing who knows. for now, I'm just using the GPU directly by switching inputs on my monitors, works fine. steam streaming works well too
setting up alsa audio with QEMU was a fucking nightmare, so I opted for scream, which is a windows virtual sound driver that just sends the pcm data over the network. multicast didn't work, so I had to change it to be unicast with my host's IP but it works great with very low latency
you'd expect the scream shit to be a pain in the ass and slow as hell, but I guess there's hardly any latency anyways since it's virtio net so whatever
sway was working pretty well for me but recently it's just been crashing, `dpms on` hasn't been working properly, desktops have been switching between monitors when using `dpms on`, and redshift-wayland has been randomly been turning off on a monitor
firefox still doesn't default to wayland either, extension menus sometimes don't appear, and there are random crashes (like when playing a video fullscreen on mastodon???)
pretty disappointing but I still believe in sway and wayland and I hope it gets better soon. I'll just put up with it
added a "hide bots" option to local and federated timelines on uwu.social web. obviously, it only works if the account is marked as a bot account (check the profile for a bot label)
admin of the uwu.social