@koakuma I recently installed ethernet from my PC to the router and from the TV to the switch under my desk
steam link from the TV to my PC is a good time
@koakuma uwu.social has a patch which will show muted accounts in lists
I mute all of the bots I follow and have a bots list
@koakuma local only I think, seems to be a design decision by gargamel himself
uwu.social is back online after our scheduled HDD replacement. There were a few roadbumps here and there, so it took 6.5 hours instead of the 4-5 we planned for.
Hetzner put a 16k hour drive in the server instead of a 0 hour server at first, so we had to wait for them to replace it twice.
The bootloader was on the removed disk, so we weren't able to boot the server until we got KVM and figured out the cause.
Next on our agenda is moving all Mastodon media to a cloud storage provider to reduce strain on the disks in the future and to free up some space. We will probably start copying media in the background shortly. The main copy operation will not require downtime, but when we do the switch from local to B2 it will require some more planned downtime in the future.
Thank you for your understanding in regards to the downtime we've been having recently.
@hentaiquotes Since it was a very long downtime, we owed it to the users to have in-depth status updates about what was going on behind the scenes.
Here's a link to the maintenance page for future reference: https://owo.whats-th.is/5V8RY3p.html
@Hanuwu if you want to donate you can send money to my paypal: https://paypal.me/deansheather
Moving forwards, to avoid a situation like this again we will:
- Keep making backups, and backing up even more things than before (system SSH keys, Mastodon media, etc.)
- Set up alerts for old drives so we can replace them safely
- Replace drives on all servers if they are old
- Never use Minio again for something with as much content as Mastodon, instead using a cloud provider to mitigate drive strain
Next time we have extended unexpected maintenance again, we will endeavour to put a updates page up again.
Aurieh and I learned a lot from this event. Hopefully, it never happens again.
We're sorry about the downtime that occurred yesterday. One of the drives in uwu.social's server failed. We were running in RAID-1, but because the software raid was misconfigured we encountered corrupted filesystems due to high I/O.
All data was recovered successfully. The RAID array has been rebuilt with a new (0 hour) drive. One of our concerns was that the remaining drive in the replica would fail during the rebuild but that didn't happen, which we're grateful for. We will be putting that hard drive out of service within the next month (which will require scheduled downtime).
It seems like the drive died while processing the huge amount of strain imposed on it by Mastodon and Minio during the old media removal process. We will move to Backblaze B2 before running out again (which will require more scheduled downtime).
@koakuma assuming you're using qemu and want a bridge for your VM, this worked for me (on the host):
ip link add name br0 type bridge
ip link set dev br0 type bridge stp_state 1
ip link set dev br0 type bridge forward_delay 500
ip link set br0 up
ip link set dev eth0 up
ip link set eth0 master br0
then in my libvirt xml:
<interface type='bridge'>
<mac address='52:54:00:51:3e:92'/>
<source bridge='br0'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
</interface>
@hentaiquotes @Main_Tomato yeah, just got an alert about full disk not sure why it was delayed. working on cleaning up the server a bit
@kura australia officially in a recession now :(
@koakuma not for a while, mastodon added a feature a release or two ago that lets me turn it off
Recently I made a shitty DDR pad to make up for the fact the local arcade sold their ITG cabinet.
The sensors are made out of tin foil lol. Eventually I'll make a improved version with real sensors (weight sensors).
Haven't made a bar for it yet, using an unused bedside table temporarily (it sucks since it's the wrong height and position)
admin of the uwu.social
cool guy