We've been having a lot of troubles with mastodon recently (filling memory and swap), which we traced back to elasticsearch freaking out about there not being enough space on the storage drive.
We cleared off some old backups/unnecessary data from the disk, so everything should be fine now. Sorry about the recent downtime.
This happened after updating to the newest version, so we were convinced it was broken code but no one else was reporting similar issues.
For debugging, we undid our patches and added tracing to sidekiq and mastodon, which lead us to believe mastodon was filling memory because of elasticsearch failures. We'll reapply the patches shortly.
@dean What exactly is Elasticsearch used for, fulltext search of toots? If it's more bother than worth, you could consider disabling it entirely.
@hentaiquotes essentially yes. It's not that much of a hassle, it's just picky about running on partitions with not enough space remaining. The fix was easy enough for us, we just had to remove a bunch of old data/backups we didn't need anymore
I don't think we'll run out of space on this server for a long time, and if we do we'll just upgrade it