Performance (1 Viewer)

[nuke terrorist]

Any word on getting out of hospital Pete? Hope things are improving health-wise

 

[pete]

Any word on getting out of hospital Pete? Hope things are improving health-wise

Nope.

 

[nuke terrorist]

Aww sorry.

 

[pete]

Aww sorry.

It’s grand. Just boring.

 

[Cornu Ammonis]

We should all come in and visit you. You’d love it.

 

[pete]

Fuck off

 

[Cornu Ammonis]

Fuck off

I’d bring grapes. And an RTE Guide.

 

[pete]

NOPE

 

[Cornu Ammonis]

you’ll come and visit me when I’m sick though, right?

 

[pete]

 

[pete]

taking the nuclear option....

 

[pete]

taking the nuclear option....

Didn’t work.

 

[pete]

So a full removal and reinstall of php-fpm hasn’t helped. Looks like what i thought was the cause is just another symptom.

I’ve been digging into log files and noticed a lot of dodgy traffic hitting one of the server IPs so I’ve blocked that in case it’s effectively acting as a DOS attack. I’ve also increased some child process settings in php-from but to be honest that’s more of a sticking plaster than an actual fix.

Sure we’ll see how it goes today.

 

[rettucs]

ok NEW PLAN

i've gone back and turned a load of stuff off. if everything stays stable i'll turn them back on again one by one until it breaks again. a process of enimilation as they say.

how did this go?

 

[pete]

how did this go?

didn't help. that was mostly Cloudflare optimisations but whatever is going on seems to be happening at lower level on the server.

i managed to lock myself out completely earlier while denying access to anyone trying to bypass cloudflare/FQDN and hitting the server IPs directly. oops.

 

[pete]

i don't have a paid Datadog account so there's no log parsing/event correlations available which is a bit of pain as it means i've to manually trawl through multiple log files looking for... something? at east i've some timestamps to start with anyway.

 

[rettucs]

i don't have a paid Datadog account so there's no log parsing/event correlations available which is a bit of pain as it means i've to manually trawl through multiple log files looking for... something?

do you need datadog? what are you using it for? maybe a lightweight out-of-box prometheus/grafana combo would give less jip.

I hate DD.

 

[pete]

Oh i like datadog because it's so simple to get dashboards up and running and i don't have to worry about configuring any software myself. I tried Grafana before but i didn't have the patience to get it working.

 

[rettucs]

fair enuf

 

[rettucs]

So a full removal and reinstall of php-fpm hasn’t helped. Looks like what i thought was the cause is just another symptom.

I’ve been digging into log files and noticed a lot of dodgy traffic hitting one of the server IPs so I’ve blocked that in case it’s effectively acting as a DOS attack. I’ve also increased some child process settings in php-from but to be honest that’s more of a sticking plaster than an actual fix.

Sure we’ll see how it goes today.

what was the source IP, as a matter of interest? or, were there multiple?

just happened to be reading this, this morning

Stack under attack: what we learned about handling DDoS attacks - Stack Overflow

stackoverflow.blog

but, given that resources seem fine on the underlying system, its unlikely to be DDOS. But, still seems unusual