Performance (1 Viewer)

pete

chronic procrastinator
Staff member
Since 1999
Thread starter
Joined
Nov 14, 1999
Messages
58,970
Solutions
3
Location
iPanopticon
Website
thumped.com
@pete I can't edit my profile, but I don't think I have in years anyway, just checking in case this is a new problem

Also I can't remember my pwd and I have a new phone and I think maybe my profile has an old email address. Help!
we shall continue this separately
 

pete

chronic procrastinator
Staff member
Since 1999
Thread starter
Joined
Nov 14, 1999
Messages
58,970
Solutions
3
Location
iPanopticon
Website
thumped.com
Freed up a load of disk space but didn’t help. I’ve now switched the site to php8.1. What could possibly go wrong
 

pete

chronic procrastinator
Staff member
Since 1999
Thread starter
Joined
Nov 14, 1999
Messages
58,970
Solutions
3
Location
iPanopticon
Website
thumped.com
found and fixed a stupid problem with my datadog config; it's not the cause of the issue but now that it's fixed it might help identify what is
 

pete

chronic procrastinator
Staff member
Since 1999
Thread starter
Joined
Nov 14, 1999
Messages
58,970
Solutions
3
Location
iPanopticon
Website
thumped.com
kill elasticsearch and see how much memory it frees.
Ah I need elasticsearch for indexing the ~1.7m posts here.

Anyway I HAVE A LEAD. Now that I’ve fixed datadog I’ve a lot more visibility into what’s going on.
 

pete

chronic procrastinator
Staff member
Since 1999
Thread starter
Joined
Nov 14, 1999
Messages
58,970
Solutions
3
Location
iPanopticon
Website
thumped.com
So considering the Apache server is behind TWO PROXIES designed to reduce load (cloudflare & Nginx), what do you think might cause datadog to report such a high volume of connections in the middle of the night (ok 7am)?

The answer’s in the question, I think.
 

pete

chronic procrastinator
Staff member
Since 1999
Thread starter
Joined
Nov 14, 1999
Messages
58,970
Solutions
3
Location
iPanopticon
Website
thumped.com
My working theory was that for a very long time, the Apache integration in my datadog agent has been misconfigured. It runs a probe against Apache every X seconds to pull performance stats, and these have been failing with a 406 error for i don’t know how long.

That’s what i fixed this morning - I had to replace a few lines of Apache directives, did the job. As soon as I did that I started getting much more granular data in the datadog dashboard…. aaaannd the number of open connections fell off immediately.

It thought that these open connections that were failing were being left open, and are what was causing Apache to fail, but since it just happened again while I was writing this post that ain’t it.

The search continues.
 

therealjohnny

Well-Known Member
Supporter
Joined
Nov 1, 2002
Messages
35,147
Solutions
3
Did it go down earlier, around 10:00? Thumped never loads for me when I'm out and about. It's been that way for a long time.
 

pete

chronic procrastinator
Staff member
Since 1999
Thread starter
Joined
Nov 14, 1999
Messages
58,970
Solutions
3
Location
iPanopticon
Website
thumped.com
ok NEW PLAN

i've gone back and turned a load of stuff off. if everything stays stable i'll turn them back on again one by one until it breaks again. a process of enimilation as they say.
 

Users who are viewing this thread

21 Day Calendar

Big Brave (Southern Lord) + Percolator
Bello Bar
1 Portobello Harbour, Saint Kevin's, Dublin, Ireland
Lori Goldston - Droichead Arts Centre
Droichead Arts Centre
Stockwell Ln, Downtown Drogheda, Drogheda, Co. Louth, Ireland
Pixies
Iveagh Gardens
Stable Ln, Saint Kevin's, Dublin, Ireland
La Luz
The Grand Social
35 Liffey St. Lower, North City, Dublin, D01 C3N0, Ireland
Arab Strap
Whelans
25 Wexford St, Portobello, Dublin 2, D02 H527, Ireland
Arab Strap
Whelan;s
25 Wexford St, Portobello, Dublin 2, D02 H527, Ireland

Latest Activity

Loading…

Support thumped.com

Support thumped.com and upgrade your account

Upgrade your account now to disable all ads... If we had any... Which we don't right now.

Upgrade now

Latest posts

Trending Threads

Latest threads

Top