SpecialAttack.net
https://forum.specialattack.net/

Network / system issues
https://forum.specialattack.net/viewtopic.php?t=1600
Page 2 of 2

Author:  [SpA]SaintK [ 09 Jul 2008, 01:42 ]
Post subject:  Re: Network / system issues

Our USA server just went down. Reason unknown. We have contacted the host to inform them of the problems.

Author:  [SpA]SaintK [ 09 Jul 2008, 19:59 ]
Post subject:  Re: Network / system issues

As we all know Murphy's law, our USA server just went down again. I've reported the issue again to our USA host which will look into this incident more propperly now as this is the 3rd downtime in less then 48 hours.

Author:  [SpA]SaintK [ 09 Jul 2008, 21:42 ]
Post subject:  Re: Network / system issues

Quote:
Ticket CKI-58194 has been updated by Saint K. [customer].

Hi,
The server is down again. So it might be another issue (4th time now).
In case of any maintenance (planned) I would like to be informed prior to when its being executed.
Regards,
Quote:
Ticket CKI-58194 has been updated by Sean McGuire [staff].

Your server is now up and running. There was some maintenance that was being done in the datacenter. There should be no more interceptions.
Thank You,
Sean McGuire
Quote:
Hi,
Server just went down again, and earlier today again for 15 minutes (I informed devicenull on this).
Please research this issue, as our USA location is worthless at the moment.
Cheers,

Author:  [SpA]SaintK [ 10 Jul 2008, 00:50 ]
Post subject:  Re: Network / system issues

Quote:
Dear Customer,

We do apologize for the inconvenience. This was still a portion of the maintenance being performed on the network in NJ/NY. We did get another ambiguous email from the NOC staff about it but it was too short of a notice to let clients know. If you have any questions, please let us know.

Thanks,

Author:  [SpA]SaintK [ 10 Jul 2008, 14:19 ]
Post subject:  Re: Network / system issues

MrWhite just died for a while. We expect a faulty PSU and have ordered a new one. We'll be going to the datacenter as soon as the new hardware arrived.

Author:  [SpA]SaintK [ 15 Jul 2008, 16:20 ]
Post subject:  Re: Network / system issues

Quote:
Dear Customer,

We have discovered several network connectivity problems between our network and AMS-IX peers networks.

Some customers note they are unable to establish IPSEC VPN sessions.
Others notice lower up/download speeds.

The problems started at around 03:00 CEST.
At the same time AMS-IX did a topology swap due to a MLX software upgrade.

We have notified AMS-IX NOC.
AMS-IX NOC thinks these problems are caused by a CAM issue on their foundry switches.


Regards,
LeaseWeb BV

Author:  [SpA]Bucky [ 24 Jul 2008, 06:23 ]
Post subject:  Re: Network / system issues

Mrwhite was down tonight/this morning - new psu and case has been ordered, it's currently being shipped

Author:  [SpA]SaintK [ 04 Aug 2008, 18:01 ]
Post subject:  Re: Network / system issues

Quote:
Time: Mon 4 Aug 2008 00:00
Location: Easynet
Reason: Under investigation *******************************

Dear Customer,

At the moment we experience some troubles in (part of) our network.

Servers in easynet are not reachable from certain destinations.

We are doing our utmost to solve this issue a.s.a.p.
As soon as we have more information available, we will publish it here.

Please be patient while we are working on this problem.

Regards,
LeaseWeb BV

Author:  [SpA]SaintK [ 07 Aug 2008, 11:46 ]
Post subject:  Re: Network / system issues

Servers 4&5
Quote:
Start window: Thu 14 Aug 2008 21:00
End window: Fri 15 Aug 2008 06:45

Dear customer,

One of our darkfiber providers (KPN) has scheduled a maintenance window from 14 August 21:00 till 15 August 06:45 at EvoSwitch. All traffic will be routed via our other darkfiber providers so the impact to our customers will be minimal. Short periods of packetloss / higher latency can occur during convergence.

Kind regards,

LeaseWeb

Timezone is CEST

Author:  [SpA]SaintK [ 12 Aug 2008, 10:32 ]
Post subject:  Re: Network / system issues

Quote:
Dear Customer,

At the moment we experience some troubles in (part of) our network.

We are doing our utmost to solve this issue a.s.a.p.
As soon as we have more information available, we will publish it here.

Please be patient while we are working on this problem.

Regards,
LeaseWeb BV

[update 01:19]

We have found and solved the problem.
Ans AMS-IX peer incorrectly sent us too many routes but had no capacity to route the traffic.
The session has been shut and the peer contacted.

Author:  [SpA]SaintK [ 12 Aug 2008, 19:39 ]
Post subject:  Re: Network / system issues

For some reason the traffic targeted at MrBlonde (servers 1 and 2) is being routed over New York. We have notified Leaseweb of the issue.

#update

Routing issue's should be fixed. Cogent made a fk up with the routing.

Author:  [SpA]SaintK [ 13 Aug 2008, 23:02 ]
Post subject:  Re: Network / system issues

Quote:
Start window: Wed 20 Aug 2008 00:01
End window: Wed 20 Aug 2008 06:00

Dear customer, We have scheduled this maintenance for two reasons. The first one is to test the redundant setup. During the outage on 08-07-2008 our backup router failed to take over when the primary failed with a hardware failure.
The last few weeks we have tested the backup router and have found the problem of the failure(further explained in the rfo). We will test the backup router in controlled steps so impact to customers is minimal. If at any time we run in to problems we will fall back to the primary router. During the swap customers can experience packetloss and loss of connection for a few minutes.
Customers located in suite 1D will also be swapped physical to the backup uplink we have supplied in the racks. When all customers are swapped over to the backup router we will turn off the primary router. If everything checks out we will start we the second phase of the maintenance. The second phase is to replace some structured cabling. We already replaced all the second uplinks in suite 1D to our customers and our shared racks. At this step we are running on the second uplinks and replace the primary uplinks. All cables are already in place and only needs to be connected. After we have completed this step we will swap back to the primary router with the new uplinks. We have chosen for a large time frame so we can do this maintenance in small steps and verify every step. During this time frame customers should only be affected for a short time during the swap to the backup router and back to the primary router. During the maintenance customers can check noc.leaseweb.com for updates. At anytime during the maintenance the can also email or call us for an update. Kind regards, LeaseWeb - NOC
Unfortunatly this maintanance will take place right at the moment of the TF2 Heavy update.

Author:  [SpA]SaintK [ 11 Sep 2008, 12:57 ]
Post subject:  Re: Network / system issues

The UK server is currently down. We're investigating the problem.

Author:  [SpA]SaintK [ 24 Sep 2008, 15:02 ]
Post subject:  Re: Network / system issues

Leaseweb is currently experiancing massive DDoS attacks on their network. Our location seems to be pretty much allright, but it does lag out from time to time. The location of 9mm's server (custom maps / arena 1) is currently unreachable.

Author:  [SpA]SaintK [ 27 Sep 2008, 20:48 ]
Post subject:  Re: Network / system issues

We're currently experiancing packetloss issue's in the UK network (server 3). We're investigating the problem.

#update
Network seems stable again.

Author:  [SpA]SaintK [ 03 Oct 2008, 08:46 ]
Post subject:  Re: Network / system issues

We had a nasty harddisk crash yesterday on our main web/database server MrWhite. After maintanance in the datacenter we've managed to restore and bring back the server.

We'll be ordering new harddisks to setup a RAID1+0 config in the server to ensure in case a HDD fails the server won't directly go down, aswell as a bit more security that all the data will stay save.

Author:  [SpA]SaintK [ 03 Oct 2008, 18:27 ]
Post subject:  Re: Network / system issues

Yesterday's crash has wrecked the MySQL database. The SpA points and Psychostats database are containing errors and we're currently unable to fix them. As result of this, both the stats and SpA points are currently offline. No data has been lost, we just have to clean up the mess.

Author:  [SpA]SaintK [ 04 Oct 2008, 13:51 ]
Post subject:  Re: Network / system issues

Because of the crash we're running some check and repair actions on the database. This causes the websites to slow down.

Author:  [SpA]SaintK [ 04 Oct 2008, 17:10 ]
Post subject:  Re: Network / system issues

Blackhawk managed to repair Psychostats, which is recording now again and rewarding SpA points for play time. Currently SpA points is only not working on servers 1 and 2, we expect to have a solution shortly.

Author:  [SpA]SaintK [ 04 Oct 2008, 17:35 ]
Post subject:  Re: Network / system issues

SpA points is working again!

Author:  [SpA]SaintK [ 10 Oct 2008, 13:03 ]
Post subject:  Re: Network / system issues

Quote:
Start window: Fri 10 Oct 2008 02:00
End window: Fri 10 Oct 2008 02:30
Dear Customer,
During this maintenance window we will upgrade our connection to the AMS-IX at Telecity location from 20 Gbps to 40 Gbps. The upgrade will be a seamless upgrade, no service interruption is expected. If for some reason problems do arise we will notify you via this website.

Kind Regards,
LeaseWeb NOC

Author:  [SpA]SaintK [ 14 Oct 2008, 10:53 ]
Post subject:  Re: Network / system issues

A part of the Amsterdam Internet Exchange is currently down. This could cause people to be unable to reach our servers.

Page 2 of 2 All times are UTC+02:00

Powered by phpBB® Forum Software © phpBB Limited