Performance (1 Viewer)

pete

chronic procrastinator
Staff member
Since 1999
Thread starter
Joined
Nov 14, 1999
Messages
59,114
Solutions
3
Location
iPanopticon
Website
thumped.com
@pete I can't edit my profile, but I don't think I have in years anyway, just checking in case this is a new problem

Also I can't remember my pwd and I have a new phone and I think maybe my profile has an old email address. Help!
we shall continue this separately
 

pete

chronic procrastinator
Staff member
Since 1999
Thread starter
Joined
Nov 14, 1999
Messages
59,114
Solutions
3
Location
iPanopticon
Website
thumped.com
Freed up a load of disk space but didn’t help. I’ve now switched the site to php8.1. What could possibly go wrong
 

pete

chronic procrastinator
Staff member
Since 1999
Thread starter
Joined
Nov 14, 1999
Messages
59,114
Solutions
3
Location
iPanopticon
Website
thumped.com
found and fixed a stupid problem with my datadog config; it's not the cause of the issue but now that it's fixed it might help identify what is
 

pete

chronic procrastinator
Staff member
Since 1999
Thread starter
Joined
Nov 14, 1999
Messages
59,114
Solutions
3
Location
iPanopticon
Website
thumped.com
kill elasticsearch and see how much memory it frees.
Ah I need elasticsearch for indexing the ~1.7m posts here.

Anyway I HAVE A LEAD. Now that I’ve fixed datadog I’ve a lot more visibility into what’s going on.
 

pete

chronic procrastinator
Staff member
Since 1999
Thread starter
Joined
Nov 14, 1999
Messages
59,114
Solutions
3
Location
iPanopticon
Website
thumped.com
So considering the Apache server is behind TWO PROXIES designed to reduce load (cloudflare & Nginx), what do you think might cause datadog to report such a high volume of connections in the middle of the night (ok 7am)?

The answer’s in the question, I think.
 

pete

chronic procrastinator
Staff member
Since 1999
Thread starter
Joined
Nov 14, 1999
Messages
59,114
Solutions
3
Location
iPanopticon
Website
thumped.com
My working theory was that for a very long time, the Apache integration in my datadog agent has been misconfigured. It runs a probe against Apache every X seconds to pull performance stats, and these have been failing with a 406 error for i don’t know how long.

That’s what i fixed this morning - I had to replace a few lines of Apache directives, did the job. As soon as I did that I started getting much more granular data in the datadog dashboard…. aaaannd the number of open connections fell off immediately.

It thought that these open connections that were failing were being left open, and are what was causing Apache to fail, but since it just happened again while I was writing this post that ain’t it.

The search continues.
 

therealjohnny

Well-Known Member
Supporter
Joined
Nov 1, 2002
Messages
35,334
Solutions
3
Did it go down earlier, around 10:00? Thumped never loads for me when I'm out and about. It's been that way for a long time.
 

pete

chronic procrastinator
Staff member
Since 1999
Thread starter
Joined
Nov 14, 1999
Messages
59,114
Solutions
3
Location
iPanopticon
Website
thumped.com
ok NEW PLAN

i've gone back and turned a load of stuff off. if everything stays stable i'll turn them back on again one by one until it breaks again. a process of enimilation as they say.
 

Users who are viewing this thread

21 Day Calendar

Lightning Bolt
Whelans
25 Wexford St, Portobello, Dublin 2, D02 H527, Ireland
Panikatax, Chimers (Australia), Fierce Shook, Carpark Sci-fi
Bello Bar
1 Portobello Harbour, Saint Kevin's, Dublin, Ireland
Kurt Vile & The Violators
Vicar Street
Vicar St, Saint Catherine's, Dublin, Ireland

Latest Activity

Loading…

Support thumped.com

Support thumped.com and upgrade your account

Upgrade your account now to disable all ads... If we had any... Which we don't right now.

Upgrade now

Latest posts

Trending Threads

Latest threads

Top