Multi-Gaming Community
It is currently 20 Jun 2025, 19:39

Unanswered topics | Active topics


» »

All times are UTC+02:00



Post new topic  This topic is locked, you cannot edit posts or make further replies.  Page 1 of 2
  [ 52 posts ]  Go to page 1 2 Next
  Print view Previous topic | Next topic 
Author Message
[SpA]SaintK
 Post subject: Network / system issues
PostPosted: 05 Nov 2007, 17:04 
Offline
Community slut (13474)
User avatar
This topic will be used to list networking issues we experianced;

5-11-2007
Little hick up in connectivity to our servers.
Quote:
Because of a human error all 10GE customers failed over to the backup network. This caused traffic to cease between 10GE customers and the rest of the network from 16:06:23 to 16:08:37. Traffic between 10GE customers and traffic between 1GE, 100MB and 10MB customers should not have been affected.


Last edited by [SpA]SaintK on 22 Nov 2007, 16:16, edited 1 time in total.

Top

Loading...

 
   
[SpA]SaintK
 Post subject:
PostPosted: 07 Nov 2007, 22:11 
Offline
Community slut (13474)
User avatar
7-11-2007
Unplanned downtime both servers due to a human error.
Downtime webserver: est 3 hours
Downtime gameserver: 10 minutes
Top

Loading...

 
   
[SpA]SaintK
 Post subject:
PostPosted: 09 Nov 2007, 20:28 
Offline
Community slut (13474)
User avatar
9-11-2007
Maintanance on the servers from 20:00 gmt +1. Last attempt on fixing the kernel. If this won't work, then windows will be installed tomorrow. We can't tell yet how long the downtime will be. This depends on the to come or not to come success.

#update 1
TF2 servers are back up, work on the kernel will continue tomorrow morning around 6.

#update 2
Great news! Both kernels are now compiled and working fine! We have 2 compiled kernels, 1 1000Hz and 1 tickless. We are now running tests to see which one performs best. First numbers are looking good, server now runs at 500frames per seconds, over 24 frames per seconds before!

#update 3
The tickless kernel seems to be giving the best performances. Ingame pings have dropped about 10-20ms for local connections, and 20-40ms for non-NL connections. Hurray!
Top

Loading...

 
   
[SpA]SaintK
 Post subject:
PostPosted: 22 Nov 2007, 16:17 
Offline
Community slut (13474)
User avatar
22-11-2007

Both TF2 servers crashed. I didn't realize that adjusting the system clock would crash the game server processes... Upside, 'thetime' command ingame returns the correct time again :mrgreen:

Servers downtime est. 2 minutes.
Top

Loading...

 
   
[SpA]SaintK
 Post subject:
PostPosted: 24 Nov 2007, 01:29 
Offline
Community slut (13474)
User avatar
24-11-2007 1:27

We seem to have some networking issues at the moment. Lots of people from outside NL have issues reaching the server.

#update 1:45
It looks like the problem cleared itself. People seem to be able to connect to everything again.
Top

Loading...

 
   
[SpA]SaintK
 Post subject:
PostPosted: 29 Dec 2007, 19:56 
Offline
Community slut (13474)
User avatar
29-11-2007 18:51 - 19: 01

There was a small network issue which was resolved after 10 minutes.
Top

Loading...

 
   
[SpA]SaintK
 Post subject:
PostPosted: 15 Jan 2008, 11:01 
Offline
Community slut (13474)
User avatar
15-01-2008 - 11.00 - 11.20

We are currently experiancing networking issues on both the Dutch servers.
On both machines we currently have 30 to 40 percent packetloss.

We have informed our ISP about the issues and the problem will be solved as soon as possible.

#update

Probleem seems to have solved itself. Packetloss is no longer noticable.
Top

Loading...

 
   
[SpA]SaintK
PostPosted: 06 Feb 2008, 19:18 
Offline
Community slut (13474)
User avatar
06-02-08 start: 19:05 end: 19:16

All systems down, reason unknown.
Top

Loading...

 
   
[SpA]Bucky
PostPosted: 07 Feb 2008, 11:17 
Offline
SpA Fookah (4459)
User avatar
sunday 2 February 2008, from 11:00 GMT+0 on till the early afternoon

Network issues at the datacentre made the server unreachable and reachable for amounts of time.

Problem fixed itself, reason unknown

_________________
M.A.S.K. , is the mighty power that can save the day
Top

Loading...

 
   
[SpA]SaintK
PostPosted: 08 Feb 2008, 13:50 
Offline
Community slut (13474)
User avatar
08-02-08 start: 13:45 end: 13:50

It appears that our provider’s core network is being targeted by DDoS attacks which cause the BGP to reroute the traffic from the core routers that have been hit. This has been noticed as small but highly annoying downtimes.

We apologize for the issues.
Top

Loading...

 
   
[SpA]Bucky
PostPosted: 29 Mar 2008, 17:08 
Offline
SpA Fookah (4459)
User avatar
29-3-08
From 12:30 to 17:00
Everything down, reason unknown

_________________
M.A.S.K. , is the mighty power that can save the day
Top

Loading...

 
   
[SpA]SaintK
PostPosted: 29 Mar 2008, 20:21 
Offline
Community slut (13474)
User avatar
Update on outage from today;

A datacenter switch died and caused our network connectivity to drop. The defective switch has been replaced and all services are restored to normal.
Top

Loading...

 
   
[SpA]SaintK
PostPosted: 07 Apr 2008, 15:05 
Offline
Community slut (13474)
User avatar
07-04-08 start: 15:00 end: 15:03

The Amsterdam Internet Exchange (world largest exchange) went down several times for a coupel of seconds. The situation now looks cleared but could still come back.
Top

Loading...

 
   
[SpA]SaintK
PostPosted: 09 Apr 2008, 10:48 
Offline
Community slut (13474)
User avatar
Last night our specialattack domain name was abused during a spamrun. This means all @specialattack.net email owners can get high amounts of bounce mails reporting the other side's mail adress could not be reached. Unfortunatly this is something we cannot prevent and has to die out.

#edit

We'll be implementing a fix later this week to prevent this from happening in the future. With an SPF record you can limit sending from @specialattack.net to the mail servers IP only. This means we're going to have to open the SMTP server for @specialattack.net owners to send their mail trough, rather then sending it trough their own ISP's SMTP server. All users will be informed on this change as soon as it went live.
Top

Loading...

 
   
[SpA]SaintK
PostPosted: 20 Apr 2008, 14:12 
Offline
Community slut (13474)
User avatar
I accidently restarted the webserver, sorry!
Top

Loading...

 
   
[SpA]SaintK
PostPosted: 04 May 2008, 01:25 
Offline
Community slut (13474)
User avatar
We're currently experiancing some network issues on the Dutch servers. We've contacted our datacenter host to look into the problem. One of their corerouters shows a 100% packetloss during the timeouts.

#update
Quote:
Expected Downtime: Unknown
Location: Easynet
Affected IP ranges: All
Reason: dDOS

*******************************

Dear Customer,

At the moment we experience some troubles in (part of) our network.

We are doing our utmost to solve this issue a.s.a.p.
As soon as we have more information available, we will publish it here.

Please be patient while we are working on this problem.

Regards,
LeaseWeb BV

********************************
Update 20:36

We had a ddos attack on one of our customer. This caused bgp sessions to flap. To resolve this we have null routed the target ip. The bgp sessions are stable again. We will keep monitoring the situation.
Top

Loading...

 
   
[SpA]SaintK
PostPosted: 06 May 2008, 15:33 
Offline
Community slut (13474)
User avatar
We seem to have some small issues again with packetloss on the Dutch servers. Its not too much packetloss, but every now and then you can notice a freeze-up ingame.

We've contacted Leaseweb to look at the issue.
Top

Loading...

 
   
[SpA]SaintK
PostPosted: 12 May 2008, 02:13 
Offline
Community slut (13474)
User avatar
Server 4 is down, reason unknown.

#edit

It went up again during the night, unknown what was wrong...
Top

Loading...

 
   
[SpA]SaintK
PostPosted: 14 May 2008, 00:56 
Offline
Community slut (13474)
User avatar
Easynet DDoS
Time: Tue 13 May 2008 22:20

Location: Easynet
Reason: dDOS
*******************************

Dear Customer,

At the moment we experience some troubles in (part of) our network.

We are doing our utmost to solve this issue a.s.a.p.
As soon as we have more information available, we will publish it here.

Please be patient while we are working on this problem.

Regards,
LeaseWeb BV

[update 22:30]
Again the DDoS has been filtered.
We have found certain control-plane ratelimiters are not working as expected. this is possibly caused by the currently running IOS version. Tomorrow morning 01:00 CEST we will perform an emergency IOS upgrade.
Top

Loading...

 
   
[SpA]SaintK
PostPosted: 23 May 2008, 00:28 
Offline
Community slut (13474)
User avatar
MrWhite (web/game server) has been offline for a period of 2 hours. Cause of the system crash is unknown and will be researched.
Top

Loading...

 
   
[SpA]Bucky
PostPosted: 30 May 2008, 01:24 
Offline
SpA Fookah (4459)
User avatar
this morning, at 11:41, sa-info seemed to log off, servers weren't reachable, but sbnc was still connected.

This evening around 11:20, server seemed unreachable, didn't reply to ping.

Server 2 seemed also offline, but not 1

reasons unknown

Serv 3 config will be under review

_________________
M.A.S.K. , is the mighty power that can save the day
Top

Loading...

 
   
[SpA]Bucky
PostPosted: 07 Jun 2008, 02:11 
Offline
SpA Fookah (4459)
User avatar
around 12, server was unreachable ,had to reboot the server.

_________________
M.A.S.K. , is the mighty power that can save the day
Top

Loading...

 
   
[SpA]SaintK
PostPosted: 13 Jun 2008, 23:55 
Offline
Community slut (13474)
User avatar
We're currently experiancing routing issues with traffic origionating from Austria (and possibly country's surrounding it). A interconnect between the local networks in Austria and the local networks in The Netherlands seems to be experiancing issues resulting in a 500ms latancy on the servers.

After reporting these issue's to Leaseweb, they've informed us that they'll re-route the traffic to fix the problem.
Top

Loading...

 
   
[SpA]SaintK
PostPosted: 17 Jun 2008, 12:30 
Offline
Community slut (13474)
User avatar
Server 4 is down for unknown reasons (Machine is not ours, we're trying to retrieve some information on why its down)
Top

Loading...

 
   
[SpA]SaintK
PostPosted: 19 Jun 2008, 16:08 
Offline
Community slut (13474)
User avatar
Ams-IX is currently experiancing BGP routing issues. The result of this is that the majority of the Netherlands, and possibly parts of europe have no traffic routing, which can cause the servers to be unavailable.

#update
Routing seems to be normal again, outage took aprox 15 minutes.
Top

Loading...

 
   
[SpA]SaintK
PostPosted: 03 Jul 2008, 03:36 
Offline
Community slut (13474)
User avatar
MrWhite crashed about 15 minutes ago.
Top

Loading...

 
   
[SpA]SaintK
PostPosted: 03 Jul 2008, 20:57 
Offline
Community slut (13474)
User avatar
Quote:
Expected Downtime: Unknown
Location: EasyNet
Reason: dDOS
*******************************

Dear Customer,

At the moment we experience some troubles on location EasyNet, we are receiving an large DDoS.

We are doing our utmost to solve this issue a.s.a.p.
As soon as we have more information available, we will publish it here.

Please be patient while we are working on this problem.

Regards,
LeaseWeb BV

[update 19:45]
We have managed to take appropriate measures against the DDoS attack.
Traffic levels have restored to normal values.

[update 20:40]
Regrettably the DDoS has changed, our measurea are not effective anymore.
Traffic levels have dropped again.
We are working on a solution.

[update 21:13]
We have been unable to find a effective method of deflecting the DDoS. We are still working on the issue.

[update 22:50]
The bulk of the DDoS traffic has stopped.
However we are still experiencing unstable routing.

[update 23:49]
After the DDoS traffic subsided our routers stil experienced instability. This was caused by a customers network equipment that sent an ARP broadcast storm to our routers.
We are working together with the customer to find the cause of this.


Top

Loading...

 
   
[SpA]SaintK
PostPosted: 04 Jul 2008, 15:32 
Offline
Community slut (13474)
User avatar
updated last nights outage report
Top

Loading...

 
   
[SpA]SaintK
PostPosted: 08 Jul 2008, 14:38 
Offline
Community slut (13474)
User avatar
Quote:
EasyNet Network Outage [Update]
Time: Tue 8 Jul 2008 13:00

Expected Downtime: Unknown
Location: EasyNet
Affected IP ranges: All
Reason: Under investigation
*******************************

Dear Customer,

At the moment we experience some troubles in (part of) our network at location EasyNet.

We are doing our utmost to solve this issue a.s.a.p.
As soon as we have more information available, we will publish it here.

Please be patient while we are working on this problem.

Regards,
LeaseWeb BV

[UPDATE 13:36] Our engineers are on-site, we are experiencing hardware problems with our primary router. Our backup router did not take over, we will investigate this later. We are currently swapping the hardware of our primary router.


Top

Loading...

 
   
[SpA]Bucky
PostPosted: 08 Jul 2008, 17:55 
Offline
SpA Fookah (4459)
User avatar
Quote:
[UPDATE 15:30]
We are still seeing unexplainable high CPU load.
A software upgrade will be carried out in the next 15 minutes to ensure the problems are not caused by the current version.
The routers will reload once.

[UPDATE 16:00]
The software update did not resolve the problems.
We will swap the supervisor of the proimary router again, this time with a brand new one.

[UPDATE 16:51]
Swapping the supervisor did not have the desired effect.
By disabling an amount of BGP sessions we have brought the router to a more stable situation. Traffic should return to acceptable metrics. We will continue to troubleshoot the router, but plan customer imapcting steps for later tonight.

_________________
M.A.S.K. , is the mighty power that can save the day
Top

Loading...

 
   
Display posts from previous:  Sort by  
Post new topic  This topic is locked, you cannot edit posts or make further replies.  Page 1 of 2
  [ 52 posts ]  Go to page 1 2 Next

» »

All times are UTC+02:00

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB® Forum Software © phpBB Limited