Server maintenance this weekend (1 Viewer)

pete

chronic procrastinator
Staff member
Since 1999
Joined
Nov 14, 1999
Messages
63,049
Solutions
3
Location
iPanopticon
Website
thumped.com
ucs_Cray1_install[1].jpg

So the server that we're running now is getting a bit old, and it's time for it to be replaced. I've now got a nice new machine with twice the RAM, twice the SSDs and twice the CPU cores for slightly less money (I think - I forgot that I needed to add the cost of a few extra IP addresses, which was mistake number A) and I'd really like to get everything moved over before I get charged another month's fees for the server we're currently on.

So this weekend everything will have to be taken offline so that I can move the virtual machines over to their new home. There's a lot of data involved, in the region of 100 to 150GB depending on whether or not I can either skip transferring some of the VM disk files and recreate them on the new server or even just copy them in advance.)

Thanks to mistake number B (forgetting that the network speeds on these almost-but-not-quite-bargain basement servers is capped at "fast enough for most purposes but not really suited to massive file transfers") it's going to take at least 10-12 hours, depending on:
  1. What time I start at on Saturday
  2. What time I get out of bed at on Sunday
  3. Oh yeah, the SCP transfer needs to actually work first time....
I tested the process with the mastodon server last night and it went pretty well, so as long as nothing goes horribly wrong with that, I'll see you back here at about 11pm Saturday night for goodbyes and (hopefully) sometime before lunchtime on Sunday for congratulations and messages of appreciation.

Thoughts, prayers, and subscription renewals etc. will all be gratefully accepted.
 
Last edited:
Oh i almost forgot - I need to reboot this server (or rather, the server it's running on) because it's not accepting SSH connections, and no SSH=no SCP=No direct file transfers.
 
Urrrgg what a mare. For the benefit of future me -

When I set up the new server it turned out it couldn’t establish an ssh connection with the old server (and viceversa) as the version running on the old server was so out of date. Upgrading it wasn't really an option due to the risk of making shit worse, and i really didn't want to lose the ability to connect from here as well... So, while i was connected i made some changes to the config (added some newer cipher settings and the like) so that the new server could establish a connection and restarted the SSH process. That worked, and I was able to transfer the mastodon server overnight.

In the meantime, the SSH connection to the old server from here dropped and I discovered yesterday morning that i couldn't reconnect... or rather it would reconnect and then immediately drop the connection. I thought maybe a reboot would help (it didn't)… ssh -v still indicated key exchange problems and the like. Not a problem - all i needed to do was undo the cipher changes that I'd made and...… yeah, now I'd no access to do that.

Normally at this point you'd just walk to the actual server and sort it out, but that's not really an option since the server in question is sitting in a datacentre in Roubaix, which I'm sure is lovely at this time of year but not very practical for getting to in a hurry. To get access back I had to put the server into rescue mode, which is the equivalent of booting a broken machine up from a live CD or whatever. Did this, got SSH access back and... where are my files?

Long story short, I had to figure out what filesystem had to be mounted (/dev/sda5 as it happens) to get access to the hypervisor OS files which for some reason are stored in tar files within tar files (took me about an hour to figure that one out), extracted /etc/ssh/ folder, undid the change i'd made the night before, replaced the tar file, almost modified the wrong files a couple of times in the process, then crossed my fingers and rebooted.

I can now SSH from the old to the new server again which means we're still on for 10pm Saturday night for the big move.
 
Though seriously Pete, I don’t understand any of it but appreciate your dedication to this. This is the only site I go to that is any craic.
 
View attachment 16715

So the server that we're running now is getting a bit old, and it's time for it to be replaced. I've now got a nice new machine with twice the RAM, twice the SSDs and twice the CPU cores for slightly less money (I think - I forgot that I needed to add the cost of a few extra IP addresses, which was mistake number A) and I'd really like to get everything moved over before I get charged another month's fees for the server we're currently on.

So this weekend everything will have to be taken offline so that I can move the virtual machines over to their new home. There's a lot of data involved, in the region of 100 to 150GB depending on whether or not I can either skip transferring some of the VM disk files and recreate them on the new server or even just copy them in advance.)

Thanks to mistake number B (forgetting that the network speeds on these almost-but-not-quite-bargain basement servers is capped at "fast enough for most purposes but not really suited to massive file transfers") it's going to take at least 10-12 hours, depending on:
  1. What time I start at on Saturday
  2. What time I get out of bed at on Sunday
  3. Oh yeah, the SCP transfer needs to actually work first time....
I tested the process with the mastodon server last night and it went pretty well, so as long as nothing goes horribly wrong with that, I'll see you back here at about 11pm Saturday night for goodbyes and (hopefully) sometime before lunchtime on Sunday for congratulations and messages of appreciation.

Thoughts, prayers, and subscription renewals etc. will all be gratefully accepted.
I thought you were running in AWS?
 
I thought you were running in AWS?
nope - i use AWS S3 for backup storage and SES for mail delivery, and i use cloudflare R2 for object storage and caching and the like. i'll probably consolidate it all on cloudflare at some point. but the server the site is hosted on is itself is a VM on an ESXi server in OVH
 

Users who are viewing this thread

Activity
So far there's no one here
Old Thread: Hello . There have been no replies in this thread for 365 days.
Content in this thread may no longer be relevant.
Perhaps it would be better to start a new thread instead.

Support thumped.com

Support thumped.com and upgrade your account

Upgrade your account now to disable all ads... If we had any... Which we don't right now.

Upgrade now

Latest threads

Latest Activity

Loading…
Back
Top