Performance (1 Viewer)

[Cornu Ammonis]

Wonders never cease!

 

[egg_]

@pete I can't edit my profile, but I don't think I have in years anyway, just checking in case this is a new problem

Also I can't remember my pwd and I have a new phone and I think maybe my profile has an old email address. Help!

 

[pete]

@pete I can't edit my profile, but I don't think I have in years anyway, just checking in case this is a new problem

Also I can't remember my pwd and I have a new phone and I think maybe my profile has an old email address. Help!

we shall continue this separately

 

[pete]

Freed up a load of disk space but didn’t help. I’ve now switched the site to php8.1. What could possibly go wrong

 

[pete]

Freed up a load of disk space but didn’t help. I’ve now switched the site to php8.1. What could possibly go wrong

Didn’t help.

 

[pete]

found and fixed a stupid problem with my datadog config; it's not the cause of the issue but now that it's fixed it might help identify what is

 

[rettucs]

that's gone now, and shit hung again. this what htop looked like while hung

View attachment 15717

kill elasticsearch and see how much memory it frees.

 

[pete]

kill elasticsearch and see how much memory it frees.

Ah I need elasticsearch for indexing the ~1.7m posts here.

Anyway I HAVE A LEAD. Now that I’ve fixed datadog I’ve a lot more visibility into what’s going on.

 

[rettucs]

Ah I need elasticsearch for indexing the ~1.7m posts here.

Anyway I HAVE A LEAD. Now that I’ve fixed datadog I’ve a lot more visibility into what’s going on.

we can call it a planned outage and go and have a coffee and be prick-free for a few minutes

 

[pete]

Well well well

 

[pete]

So considering the Apache server is behind TWO PROXIES designed to reduce load (cloudflare & Nginx), what do you think might cause datadog to report such a high volume of connections in the middle of the night (ok 7am)?

The answer’s in the question, I think.

 

[rettucs]

So considering the Apache server is behind TWO PROXIES designed to reduce load (cloudflare & Nginx), what do you think might cause datadog to report such a high volume of connections in the middle of the night (ok 7am)?

The answer’s in the question, I think.

the russians. definitely the russians

 

[magicbastarder]

clearly the traffic is pete himself.

 

[dudley]

Is it me???

 

[pete]

My working theory was that for a very long time, the Apache integration in my datadog agent has been misconfigured. It runs a probe against Apache every X seconds to pull performance stats, and these have been failing with a 406 error for i don’t know how long.

That’s what i fixed this morning - I had to replace a few lines of Apache directives, did the job. As soon as I did that I started getting much more granular data in the datadog dashboard…. aaaannd the number of open connections fell off immediately.

It thought that these open connections that were failing were being left open, and are what was causing Apache to fail, but since it just happened again while I was writing this post that ain’t it.

The search continues.

 

[pete]

 

[dudley]

 

[therealjohnny]

Did it go down earlier, around 10:00? Thumped never loads for me when I'm out and about. It's been that way for a long time.

 

[pete]

Did it go down earlier, around 10:00? Thumped never loads for me when I'm out and about. It's been that way for a long time.

Maybe!

 

[pete]

ok NEW PLAN

i've gone back and turned a load of stuff off. if everything stays stable i'll turn them back on again one by one until it breaks again. a process of enimilation as they say.