CP6 Web Server issue Resolved
Priority - Critical Affecting Server - CP6

20th September 2020

9:30PM UK Time:

 

We are aware of an issue with websites down on CP6 Web Server.

We are investigating the issue



10:05pm

We have identified the issue as a fault apache module that is stopping the web server functioning.

We have escalated this issue to cPanel and awaiting feedback from them.

 



10:45pm

This issue is now resolved by rebooting the server and then running this fix by cpanel: https://support.cpanel.net/hc/en-us/articles/360052252934-Apache-rebuild-failures-due-to-mod-negotiation 

Carina - Migration to new Enterprise Cloud Server Resolved
Priority - Medium Affecting Server - CP3

Start: 27/10/2019 20:00
End: 27/10/2019 23:59
UK Time  (check UK Time here)

Migration status will be updated here during migration period.

Jupiter-FL DDoS Attack Resolved
Priority - Critical Affecting Server - CP-US101

The shared IP for Jupiter-FL is currently under a large DDoS attack resulting in services going offline.

We're currently working with the datacenter to mitigate this attack as quickly as possible.

Update: The attack has been mitigated and filtering rules updated and the server is back online.

Venus-FL - DDoS Attack Resolved
Priority - Critical

The server is currently under a large DDoS attack which has taken all services offline.

We're working to mitigate the attack as quickly as we can to restore service. Please bear with us.

Jupiter-FL - DDoS Attack Resolved
Priority - Critical Affecting Server - CP-US101

The server is currently under a large DDoS attack which has taken all services offline.

We're working to mitigate the attack as quickly as we can to restore service. Please bear with us.

Update: As per the recent email we have changed the IPs for this server to use a cloud DDoS scrubbing service. All services should be back online once this DNS information has propagated in around an hour.

Mars-FL - DDoS Attack Resolved
Priority - Critical

The server is currently under a large DDoS attack which has taken all services offline.

We're working to mitigate the attack as quickly as we can to restore service. Please bear with us.

Update: The attacks are continuing and are against several IPs on this server. We're working on a solution as quickly as we can but it's impossible to provide an ETA on when the server will be back online currently.

Update: Datacenter techs are currently working on setting up a DDoS mitigation solution for us with higher capacity. This will require server configuration changes and further details will be emailed to all customers shortly.

Update: The server and shared IP are now on a much higher level of protection however this doesn't extend to existing dedicated IPs. This does mean the main server and shared IPs have changed however any customers using our DNS cluster won't need to make any changes. An email with more information is being sent to all customers shortly.


Update: Issues with SSL certificates have now been resolved and existing certificates are now working correctly.
We also had an issue where some clients could not access their sites, after much discusion with the DDoS scrubbing provider we have adjusted the MTU from 1500 to 1400 which allows Path MTU Discovery to work correctly and customers that were previously having issues should now be able to access their sites correctly.

Hercules - DDoS Attack Resolved
Priority - Critical

The server is currently under a large DDoS attack which has taken all services offline.

We're working to mitigate the attack as quickly as we can to restore service. Please bear with us.

Update: The attacks are continuing and are against several IPs on this server. We're working on a solution as quickly as we can but it's impossible to provide an ETA on when the server will be back online currently.

Update: Datacenter techs are currently working on setting up a DDoS mitigation solution for us with higher capacity. This will require server configuration changes and further details will be emailed to all customers shortly.

Update: The server and shared IP are now on a much higher level of protection however this doesn't extend to existing dedicated IPs. This does mean the main server and shared IPs have changed however any customers using our DNS cluster won't need to make any changes. An email with more information is being sent to all customers shortly.


Update: Issues with SSL certificates have now been resolved and existing certificates are now working correctly.
We also had an issue where some clients could not access their sites, after much discusion with the DDoS scrubbing provider we have adjusted the MTU from 1500 to 1400 which allows Path MTU Discovery to work correctly and customers that were previously having issues should now be able to access their sites correctly.

Gemini - DDoS Attack Resolved
Priority - Critical

The server is currently under a large DDoS attack which has taken all services offline.

We're working to mitigate the attack as quickly as we can to restore service. Please bear with us.

Update: The attacks are continuing and are against several IPs on this server. We're working on a solution as quickly as we can but it's impossible to provide an ETA on when the server will be back online currently.

Update: Datacenter techs are currently working on setting up a DDoS mitigation solution for us with higher capacity. This will require server configuration changes and further details will be emailed to all customers shortly.

Update: The server and shared IP are now on a much higher level of protection however this doesn't extend to existing dedicated IPs. This does mean the main server and shared IPs have changed however any customers using our DNS cluster won't need to make any changes. An email with more information is being sent to all customers shortly.

Update: We are aware of the SSL issues on this server since the change and are working as quickly as we can to resolve the issue.

Update: The SSL issues have been tracked down to an nginx issue on the scrubbing cloud. Upstream techs are currently working on this issue.

Update: Issues with SSL certificates have now been resolved and existing certificates are now working correctly.
We also had an issue where some clients could not access their sites, after much discusion with the DDoS scrubbing provider we have adjusted the MTU from 1500 to 1400 which allows Path MTU Discovery to work correctly and customers that were previously having issues should now be able to access their sites correctly.

Gemini DDoS Attack Resolved
Priority - Critical

Gemini is currently under a large DDoS attack.

We are working to identify the target and mitigate the attack however this will take some time to complete. We'll keep this post updated

Please bear with us.

Update: Our upstream provider have told us the level of protection can't be raised without switching all IPs on the server to different IPs. Obviously this isn't something we can currently do without DNS changes for customers however we have identified the target of the attack and are currently working to remove them from the network. 
The attack is being closely monitored and once traffic drops to managable levels the server will be back online. As this is dependant on the attacker and how long they've paid for the DDoS we're unable to provide an ETA on this.

Gemini Reboot Resolved
Priority - Low

Gemini is scheduled for reboot on Friday 15th September to resolve an issue with the server not seeing the correct number of CPUs. 

Expected downtime is around 20 minutes.