Multi-Gaming Community
It is currently 28 Mar 2024, 23:08

All times are UTC+01:00




Post new topic  Reply to topic  [ 16 posts ] 
Author Message
nl 
 Post subject: Server troubleshooting
PostPosted: 17 Aug 2011, 19:49 
Offline
Community slut (13473)
User avatar
Before I start I shall say sorry to banana for breaking things already 2 days after he left

As you undoubtedly have noticed, our TF2 servers, like many others, are still experimenting crashing issues.

These crashing issues roughly started to occur after the "pew pew / starwars" update introducing the laser weapons. The servers have suffered several different bugs causing them to crash. So far VALVe has found 5 bugs causing crashes and fixed them. However, there is still one, and perhaps 2 bugs unidentified, and thus not fixed.

The crashes:

One crash occurs mid game and we suspect it still has to do with the laser weapons as we've had several reports from people that the crash occurred exactly at the moment someone would change from a class to soldier with the laser weapons equipped. However, the last time we've seen this crash was the 15th at 09:53CET, a day later an update was released. We suspect the bug might have involved the steam backend as we had crashes with the exact timestamp, on different machines. Nothing was said either about fixing a bug in the TF2 server software with the last update. There is no way telling without official word from VALVe if this fixed, but currently it looks like it.

The second crash always occurs right after a map change when all the clients are connecting again to join the game. The server doesn't really crash, the process simply freezes up and all clients receive a time-out disconnect message. On the console, as far as we can currently see, no error messages are displayed. The last line is simply cut off (always a client connecting). This crash is the one that is currently still bugging us and you as player.

To try and help VALVe out in locating the issue we're going to run a series of test today and tomorrow to try and gather some extra information.

The first test is running tonight. Ditch specially wrote some software for us to monitor our servers. Rather then checking if the process is still alive, we actually query the gameserver to see if it responds. If it fails to respond over a period of time, the process is killed and restarted. This feature ensures our TF2 servers will always be available, also in case of freeze ups instead of actual crashes. We will shut this script down tonight so we can examine the state of the servers when they freeze up. This will have as result that servers after a crash won't directly be available again.

The second test will be ran tomorrow. This test will be much more impacting, as we will disable all the custom mods and plugins(metamod source/sourcemod) on the majority of the servers. This will mean that tomorrow during the entire day stuff such as SpA-points, calladmin, stats, custom maprotation etc. won't be working at all. This is required that we can proof with a 100% certainty the crashes are not caused by any of the plugins running.


Top
   
de 
PostPosted: 18 Aug 2011, 15:33 
Offline
is Wonder Woman (5950)
User avatar
It will be interesting to see the results. Good luck :5:

_________________
Classy is when a woman has everything to flaunt, but chooses not to show it!


Top
   
nl 
PostPosted: 19 Aug 2011, 09:00 
Offline
Community slut (13473)
User avatar
To my displeasure the servers have been running rock solid stable. This means the root cause of the freezecrashes are somewhere in Sourcemod(base), or one of our plugins (spapoints, dynamic map rotations, etc.).

During the next few day's we'll be doing a lot more troubleshooting to find the cause.

Plan of attack:
- Upgrade sourcemod, monitor servers

In case of further crashing:
- Remove all custom plugins from Sourcemod, only run stock plugins, if server still crashes, we'll need sourcemod to find/fix the issue
If the server doesn't crash:
- Bring back our custom plugins one by one till the servers become unstable again

Once we've located the exact source of the issue we can start debugging that specific add-on.


Top
   
gb 
PostPosted: 19 Aug 2011, 11:02 
Offline
Has no REAL life! (8841)
User avatar
Shout if you need a hand or anything mate.

_________________
-"You've really worked out your banter, haven't you?"
-"No, not really. This is a different thing, it's spontaneous and it's called wit."


Top
   
nl 
PostPosted: 19 Aug 2011, 11:17 
Offline
Community slut (13473)
User avatar
[SpA]ProtectMyBalls wrote:
Shout if you need a hand or anything mate.
Please fix it :mrgreen:


Top
   
gb 
PostPosted: 19 Aug 2011, 11:28 
Offline
Has no REAL life! (8841)
User avatar
Done. Have you tried an unexpected manual reboot? :ugly:

_________________
-"You've really worked out your banter, haven't you?"
-"No, not really. This is a different thing, it's spontaneous and it's called wit."


Top
   
de 
PostPosted: 19 Aug 2011, 11:36 
Offline
is Wonder Woman (5950)
User avatar
Haha yeah, good luck with that Balls...... :shock:

I'll give you a hand Saint, when I'm on top of the Mountain, I'll give you a birds eye view :5:

Oh und gute idee, mach nichts kaputt :ugly:

_________________
Classy is when a woman has everything to flaunt, but chooses not to show it!


Top
   
nl 
PostPosted: 19 Aug 2011, 11:45 
Offline
Community slut (13473)
User avatar
[SpA]annarack wrote:
Haha yeah, good luck with that Balls...... :shock:

I'll give you a hand Saint, when I'm on top of the Mountain, I'll give you a birds eye view :5:

Oh und gute idee, mach nichts kaputt :ugly:
Der ganzen welt ist KAPUT, JA!


Top
   
nl 
PostPosted: 19 Aug 2011, 11:47 
Offline
Community slut (13473)
User avatar
I'll be rebooting servers through out the day to get the new sourcemod version going. The servers on MrWhite will remain running modless as verification cases. (Servers sometimes crashed in sync on both machines, so it's interesting to see that *if* servers on MrBlonde crash with mods, it also crashes on the modless servers on MrWhite)


Top
   
gb 
PostPosted: 19 Aug 2011, 11:48 
Offline
Has no REAL life! (8841)
User avatar
was a joke anna :roll:

_________________
-"You've really worked out your banter, haven't you?"
-"No, not really. This is a different thing, it's spontaneous and it's called wit."


Top
   
de 
PostPosted: 19 Aug 2011, 11:48 
Offline
is Wonder Woman (5950)
User avatar
Die ganze Welt ist voller Idioten!

_________________
Classy is when a woman has everything to flaunt, but chooses not to show it!


Top
   
nl 
PostPosted: 19 Aug 2011, 11:52 
Offline
Community slut (13473)
User avatar
[SpA]annarack wrote:
Die ganze Welt ist voller Idioten!
das ist richtig! verdammt noch mal.


Top
   
gb 
PostPosted: 19 Aug 2011, 11:56 
Offline
CAST MASTER (481)
User avatar
If you require a lab rat, give me a shout, i'm sure i could break more things for you :D

_________________
Gone, but somehow not forgotten.


Top
   
gb 
PostPosted: 19 Aug 2011, 11:56 
Offline
Has no REAL life! (8841)
User avatar
[SpA]annarack wrote:
Die ganze Welt ist voller Idioten!
Ain't that the truth.

_________________
-"You've really worked out your banter, haven't you?"
-"No, not really. This is a different thing, it's spontaneous and it's called wit."


Top
   
nl 
PostPosted: 22 Aug 2011, 23:36 
Offline
Community slut (13473)
User avatar
Servers are more stable now. I have seen 2 or so crashes the last couple of days.

Not sure if the change to Sourcemod, or the TF2 update coming in between fixed it (more or less).


Top
   
de 
PostPosted: 24 Aug 2011, 16:25 
Offline
Has no REAL life! (4162)
User avatar
Da ist man mal ein paar Tage nicht da... ... ... :shock:

_________________
SaintK: I'm completely lost :mrgreen:


Top
   
Display posts from previous:  Sort by  
Post new topic  Reply to topic  [ 16 posts ] 

All times are UTC+01:00


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB® Forum Software © phpBB Limited