22-11-2007
Both TF2 servers crashed. I didn't realize that adjusting the system clock would crash the game server processes... Upside, 'thetime' command ingame returns the correct time again
Servers downtime est. 2 minutes.
| SpecialAttack.net https://forum.specialattack.net/ |
|
| Network / system issues https://forum.specialattack.net/viewtopic.php?t=1600 |
Page 1 of 2 |
| Author: | [SpA]SaintK [ 05 Nov 2007, 17:04 ] |
| Post subject: | Network / system issues |
This topic will be used to list networking issues we experianced; 5-11-2007 Little hick up in connectivity to our servers. Quote:
Because of a human error all 10GE customers failed over to the backup network. This caused traffic to cease between 10GE customers and the rest of the network from 16:06:23 to 16:08:37. Traffic between 10GE customers and traffic between 1GE, 100MB and 10MB customers should not have been affected.
|
|
| Author: | [SpA]SaintK [ 07 Nov 2007, 22:11 ] |
| Post subject: | |
7-11-2007 Unplanned downtime both servers due to a human error. Downtime webserver: est 3 hours Downtime gameserver: 10 minutes |
|
| Author: | [SpA]SaintK [ 09 Nov 2007, 20:28 ] |
| Post subject: | |
9-11-2007 Maintanance on the servers from 20:00 gmt +1. Last attempt on fixing the kernel. If this won't work, then windows will be installed tomorrow. We can't tell yet how long the downtime will be. This depends on the to come or not to come success. #update 1 TF2 servers are back up, work on the kernel will continue tomorrow morning around 6. #update 2 Great news! Both kernels are now compiled and working fine! We have 2 compiled kernels, 1 1000Hz and 1 tickless. We are now running tests to see which one performs best. First numbers are looking good, server now runs at 500frames per seconds, over 24 frames per seconds before! #update 3 The tickless kernel seems to be giving the best performances. Ingame pings have dropped about 10-20ms for local connections, and 20-40ms for non-NL connections. Hurray! |
|
| Author: | [SpA]SaintK [ 22 Nov 2007, 16:17 ] |
| Post subject: | |
22-11-2007 Both TF2 servers crashed. I didn't realize that adjusting the system clock would crash the game server processes... Upside, 'thetime' command ingame returns the correct time again Servers downtime est. 2 minutes. |
|
| Author: | [SpA]SaintK [ 24 Nov 2007, 01:29 ] |
| Post subject: | |
24-11-2007 1:27 We seem to have some networking issues at the moment. Lots of people from outside NL have issues reaching the server. #update 1:45 It looks like the problem cleared itself. People seem to be able to connect to everything again. |
|
| Author: | [SpA]SaintK [ 29 Dec 2007, 19:56 ] |
| Post subject: | |
29-11-2007 18:51 - 19: 01 There was a small network issue which was resolved after 10 minutes. |
|
| Author: | [SpA]SaintK [ 15 Jan 2008, 11:01 ] |
| Post subject: | |
15-01-2008 - 11.00 - 11.20 We are currently experiancing networking issues on both the Dutch servers. On both machines we currently have 30 to 40 percent packetloss. We have informed our ISP about the issues and the problem will be solved as soon as possible. #update Probleem seems to have solved itself. Packetloss is no longer noticable. |
|
| Author: | [SpA]SaintK [ 06 Feb 2008, 19:18 ] |
| Post subject: | Re: Network / system issues |
06-02-08 start: 19:05 end: 19:16 All systems down, reason unknown. |
|
| Author: | [SpA]Bucky [ 07 Feb 2008, 11:17 ] |
| Post subject: | Re: Network / system issues |
sunday 2 February 2008, from 11:00 GMT+0 on till the early afternoon Network issues at the datacentre made the server unreachable and reachable for amounts of time. Problem fixed itself, reason unknown |
|
| Author: | [SpA]SaintK [ 08 Feb 2008, 13:50 ] |
| Post subject: | Re: Network / system issues |
08-02-08 start: 13:45 end: 13:50 It appears that our provider’s core network is being targeted by DDoS attacks which cause the BGP to reroute the traffic from the core routers that have been hit. This has been noticed as small but highly annoying downtimes. We apologize for the issues. |
|
| Author: | [SpA]Bucky [ 29 Mar 2008, 17:08 ] |
| Post subject: | Re: Network / system issues |
29-3-08 From 12:30 to 17:00 Everything down, reason unknown |
|
| Author: | [SpA]SaintK [ 29 Mar 2008, 20:21 ] |
| Post subject: | Re: Network / system issues |
Update on outage from today; A datacenter switch died and caused our network connectivity to drop. The defective switch has been replaced and all services are restored to normal. |
|
| Author: | [SpA]SaintK [ 07 Apr 2008, 15:05 ] |
| Post subject: | Re: Network / system issues |
07-04-08 start: 15:00 end: 15:03 The Amsterdam Internet Exchange (world largest exchange) went down several times for a coupel of seconds. The situation now looks cleared but could still come back. |
|
| Author: | [SpA]SaintK [ 09 Apr 2008, 10:48 ] |
| Post subject: | Re: Network / system issues |
Last night our specialattack domain name was abused during a spamrun. This means all @specialattack.net email owners can get high amounts of bounce mails reporting the other side's mail adress could not be reached. Unfortunatly this is something we cannot prevent and has to die out. #edit We'll be implementing a fix later this week to prevent this from happening in the future. With an SPF record you can limit sending from @specialattack.net to the mail servers IP only. This means we're going to have to open the SMTP server for @specialattack.net owners to send their mail trough, rather then sending it trough their own ISP's SMTP server. All users will be informed on this change as soon as it went live. |
|
| Author: | [SpA]SaintK [ 20 Apr 2008, 14:12 ] |
| Post subject: | Re: Network / system issues |
I accidently restarted the webserver, sorry! |
|
| Author: | [SpA]SaintK [ 04 May 2008, 01:25 ] |
| Post subject: | Re: Network / system issues |
We're currently experiancing some network issues on the Dutch servers. We've contacted our datacenter host to look into the problem. One of their corerouters shows a 100% packetloss during the timeouts. #update Quote: Expected Downtime: Unknown
Location: Easynet Affected IP ranges: All Reason: dDOS ******************************* Dear Customer, At the moment we experience some troubles in (part of) our network. We are doing our utmost to solve this issue a.s.a.p. As soon as we have more information available, we will publish it here. Please be patient while we are working on this problem. Regards, LeaseWeb BV ******************************** Update 20:36 We had a ddos attack on one of our customer. This caused bgp sessions to flap. To resolve this we have null routed the target ip. The bgp sessions are stable again. We will keep monitoring the situation. |
|
| Author: | [SpA]SaintK [ 06 May 2008, 15:33 ] |
| Post subject: | Re: Network / system issues |
We seem to have some small issues again with packetloss on the Dutch servers. Its not too much packetloss, but every now and then you can notice a freeze-up ingame. We've contacted Leaseweb to look at the issue. |
|
| Author: | [SpA]SaintK [ 12 May 2008, 02:13 ] |
| Post subject: | Re: Network / system issues |
Server 4 is down, reason unknown. #edit It went up again during the night, unknown what was wrong... |
|
| Author: | [SpA]SaintK [ 14 May 2008, 00:56 ] |
| Post subject: | Re: Network / system issues |
Easynet DDoS Time: Tue 13 May 2008 22:20 Location: Easynet Reason: dDOS ******************************* Dear Customer, At the moment we experience some troubles in (part of) our network. We are doing our utmost to solve this issue a.s.a.p. As soon as we have more information available, we will publish it here. Please be patient while we are working on this problem. Regards, LeaseWeb BV [update 22:30] Again the DDoS has been filtered. We have found certain control-plane ratelimiters are not working as expected. this is possibly caused by the currently running IOS version. Tomorrow morning 01:00 CEST we will perform an emergency IOS upgrade. |
|
| Author: | [SpA]SaintK [ 23 May 2008, 00:28 ] |
| Post subject: | Re: Network / system issues |
MrWhite (web/game server) has been offline for a period of 2 hours. Cause of the system crash is unknown and will be researched. |
|
| Author: | [SpA]Bucky [ 30 May 2008, 01:24 ] |
| Post subject: | Re: Network / system issues |
this morning, at 11:41, sa-info seemed to log off, servers weren't reachable, but sbnc was still connected. This evening around 11:20, server seemed unreachable, didn't reply to ping. Server 2 seemed also offline, but not 1 reasons unknown Serv 3 config will be under review |
|
| Author: | [SpA]Bucky [ 07 Jun 2008, 02:11 ] |
| Post subject: | Re: Network / system issues |
around 12, server was unreachable ,had to reboot the server. |
|
| Author: | [SpA]SaintK [ 13 Jun 2008, 23:55 ] |
| Post subject: | Re: Network / system issues |
We're currently experiancing routing issues with traffic origionating from Austria (and possibly country's surrounding it). A interconnect between the local networks in Austria and the local networks in The Netherlands seems to be experiancing issues resulting in a 500ms latancy on the servers. After reporting these issue's to Leaseweb, they've informed us that they'll re-route the traffic to fix the problem. |
|
| Author: | [SpA]SaintK [ 17 Jun 2008, 12:30 ] |
| Post subject: | Re: Network / system issues |
Server 4 is down for unknown reasons (Machine is not ours, we're trying to retrieve some information on why its down) |
|
| Author: | [SpA]SaintK [ 19 Jun 2008, 16:08 ] |
| Post subject: | Re: Network / system issues |
Ams-IX is currently experiancing BGP routing issues. The result of this is that the majority of the Netherlands, and possibly parts of europe have no traffic routing, which can cause the servers to be unavailable. #update Routing seems to be normal again, outage took aprox 15 minutes. |
|
| Author: | [SpA]SaintK [ 03 Jul 2008, 03:36 ] |
| Post subject: | Re: Network / system issues |
MrWhite crashed about 15 minutes ago. |
|
| Author: | [SpA]SaintK [ 03 Jul 2008, 20:57 ] |
| Post subject: | Re: Network / system issues |
Quote: Expected Downtime: Unknown
Location: EasyNet Reason: dDOS ******************************* Dear Customer, At the moment we experience some troubles on location EasyNet, we are receiving an large DDoS. We are doing our utmost to solve this issue a.s.a.p. As soon as we have more information available, we will publish it here. Please be patient while we are working on this problem. Regards, LeaseWeb BV [update 19:45] We have managed to take appropriate measures against the DDoS attack. Traffic levels have restored to normal values. [update 20:40] Regrettably the DDoS has changed, our measurea are not effective anymore. Traffic levels have dropped again. We are working on a solution. [update 21:13] We have been unable to find a effective method of deflecting the DDoS. We are still working on the issue. [update 22:50] The bulk of the DDoS traffic has stopped. However we are still experiencing unstable routing. [update 23:49] After the DDoS traffic subsided our routers stil experienced instability. This was caused by a customers network equipment that sent an ARP broadcast storm to our routers. We are working together with the customer to find the cause of this. |
|
| Author: | [SpA]SaintK [ 04 Jul 2008, 15:32 ] |
| Post subject: | Re: Network / system issues |
updated last nights outage report |
|
| Author: | [SpA]SaintK [ 08 Jul 2008, 14:38 ] |
| Post subject: | Re: Network / system issues |
Quote: EasyNet Network Outage [Update]
Time: Tue 8 Jul 2008 13:00 Expected Downtime: Unknown Location: EasyNet Affected IP ranges: All Reason: Under investigation ******************************* Dear Customer, At the moment we experience some troubles in (part of) our network at location EasyNet. We are doing our utmost to solve this issue a.s.a.p. As soon as we have more information available, we will publish it here. Please be patient while we are working on this problem. Regards, LeaseWeb BV [UPDATE 13:36] Our engineers are on-site, we are experiencing hardware problems with our primary router. Our backup router did not take over, we will investigate this later. We are currently swapping the hardware of our primary router. |
|
| Author: | [SpA]Bucky [ 08 Jul 2008, 17:55 ] |
| Post subject: | Re: Network / system issues |
Quote: [UPDATE 15:30]
We are still seeing unexplainable high CPU load. A software upgrade will be carried out in the next 15 minutes to ensure the problems are not caused by the current version. The routers will reload once. [UPDATE 16:00] The software update did not resolve the problems. We will swap the supervisor of the proimary router again, this time with a brand new one. [UPDATE 16:51] Swapping the supervisor did not have the desired effect. By disabling an amount of BGP sessions we have brought the router to a more stable situation. Traffic should return to acceptable metrics. We will continue to troubleshoot the router, but plan customer imapcting steps for later tonight. |
|
| Page 1 of 2 | All times are UTC+02:00 |
|
Powered by phpBB® Forum Software © phpBB Limited |
|