Active Incident

Updated a few seconds ago

Incident Status

Operational

Components

OpenStack

Locations

STO1 - Stockholm South, STO2 - Stockholm West, STO3 - Stockholm North



April 24, 2024 10:20 CEST
April 24, 2024 08:20 UTC
[Investigating] We are aware of issues with newly created instances experiencing network issues and are currently investigating it closer. In some cases, a possible workaround might be to simply restart the newly created instances that are experiencing issues. If this does not resolve your issues then feel free to reach out to us with your instance ID at https://support.elastx.se

April 24, 2024 14:02 CEST
April 24, 2024 12:02 UTC
[Investigating] We are currently working together with our supplier to locate the cause of the issue.

April 24, 2024 15:03 CEST
April 24, 2024 13:03 UTC
[Investigating] Investigation is underway together with our supplier. The next status update will be posted at around 17:00 CEST

April 24, 2024 16:59 CEST
April 24, 2024 14:59 UTC
[Investigating] The investigation together with the supplier is ongoing. The next update will be around 20:00 CEST.

April 24, 2024 20:37 CEST
April 24, 2024 18:37 UTC
[Investigating] The investigation together with the supplier is ongoing, we are now waiting for a plan on how to proceed. The next update will be around 23:00 CEST.

April 24, 2024 22:21 CEST
April 24, 2024 20:21 UTC
[Investigating] We are still waiting for a plan on how to proceed from our provider. Since the impact is quite low ,no more actions will be taken today and the next update will be around 10:00 tomorrow.

April 25, 2024 11:17 CEST
April 25, 2024 09:17 UTC
[Investigating] No news, we are still waiting for an update from our vendor. Next update will be at 14:00.

April 25, 2024 16:50 CEST
April 25, 2024 14:50 UTC
[Investigating] Our supplier support is still processing the given information about the issue and we are waiting for their feedback. Next update will be at 21:00.

April 25, 2024 21:09 CEST
April 25, 2024 19:09 UTC
[Investigating] We have received feedback from the supplier and are planning for actions later tonight. As part of these actions we will upgrade STO1 and STO2 network infrastructure during an emergency maintenance window starting 22:00.

April 26, 2024 04:54 CEST
April 26, 2024 02:54 UTC
[Investigating] We have have not carried out the announced emergency maintenance fully as our tests have shown that the issue remains. Hence we are not going to risk degrading service further. Our supplier is notified and we will resume in the morning or as soon as we get more feedback. Next update around 10:00.

April 26, 2024 10:02 CEST
April 26, 2024 08:02 UTC
[Investigating] We are awaiting further analysis from our supplier. Next update will be around 13:00 CEST.

April 26, 2024 13:06 CEST
April 26, 2024 11:06 UTC
[Investigating] We are awaiting more information from our supplier. The next update will be before 17:00 CEST.

April 26, 2024 16:26 CEST
April 26, 2024 14:26 UTC
[Investigating] Investigation together with our supplier is ongoing. The next update will be on Monday unless the situation changes over the weekend.

April 28, 2024 21:04 CEST
April 28, 2024 19:04 UTC
[Investigating] We upgraded the remaining switches in sto1 today. This did not resolve the issue but it will be easier to find the root cause. There are still delays in creating new networks in sto2 and sto3 but all other tasks should be operational. Currently it is not possible to perform any network changes in sto1. All existing network configurations should be operational.

April 29, 2024 13:55 CEST
April 29, 2024 11:55 UTC
[Investigating] Investigation together with supplier is still ongoing. Next update around 16:30 CEST.

April 29, 2024 16:27 CEST
April 29, 2024 14:27 UTC
[Investigating] We are still investigating together with our supplier. Next update will be around 10:00 tomorrow unless the situation changes.

April 30, 2024 09:07 CEST
April 30, 2024 07:07 UTC
[Identified] We have had some success in resolving the issue. Still service is degraded, however it seems like new resources can be created in all AZs (with some delays) now. We are waiting for a patch from the supplier which we will test and roll out. Please contact us if you want to create new instances and experience problems. Next update 13.00 today.

April 30, 2024 13:08 CEST
April 30, 2024 11:08 UTC
[Identified] No news. The next update will be around 16:30 today.

April 30, 2024 17:04 CEST
April 30, 2024 15:04 UTC
[Identified] The supplier is still working on fix for this problem. The current status is: - New networks will take some time before they are deployed and fully operational. - New instances on existing networks will for the most time work instantly but in some cases it can be a delay before it is connected to the network. - All other network operations like security group, floating ip and router changes should be instant. The next update will be around 13:30 tomorrow.

May 1, 2024 13:14 CEST
May 1, 2024 11:14 UTC
[Identified] We are still waiting for a fix. The status of the network is the same. The next update will be around 18:00.

May 1, 2024 19:18 CEST
May 1, 2024 17:18 UTC
[Identified] The vendor is still working on a fix. The next update will be around 10:00 tomorrow.

May 2, 2024 10:07 CEST
May 2, 2024 08:07 UTC
[Identified] The vendor developer team is working on a fix and we expect an update during the night. The next update will be around 10:00 tomorrow.

May 3, 2024 13:32 CEST
May 3, 2024 11:32 UTC
[Identified] We have been informed that we will not get a fix this week, at the earliest on Monday. The next update will be around 10:00 Monday.

May 6, 2024 13:16 CEST
May 6, 2024 11:16 UTC
[Identified] We have been informed that we get a fix this week, at the earliest on Tuesday. The service seems stable but slow when creating new resources. Please contact us (support@elastx.se) if you experience issues. The next update will be around 10:00 Tuesday.

May 7, 2024 10:37 CEST
May 7, 2024 08:37 UTC
[Identified] We have been informed that we get a fix this week, at the earliest on Tuesday. The service is stable but slow when creating new resources. Please contact us (support@elastx.se) if you experience issues. The next update will be around 10:00 Wednesday.

May 8, 2024 13:43 CEST
May 8, 2024 11:43 UTC
[Identified] We have received a fix and are implementing this. We will work on implementing the fix today and monitor the situation over the weekend. The service is stable but perhaps slow when creating new resources. Please contact us (support@elastx.se) if you experience issues.

May 13, 2024 21:29 CEST
May 13, 2024 19:29 UTC
[Identified] Using the provided fix we can continue to remediate the issue. However its proven to be a tedious task and will continue overnight. Our supplier is looking for options to speed up the process. If you have questions, please contact us at support@elastx.se. Next update will be provided tomorrow around 10.00 am.

May 14, 2024 11:10 CEST
May 14, 2024 09:10 UTC
[Identified] Status has not changed. Cleaning is ongoing. Our supplier is working on a quicker solution. Please contact us at support@elastx.se if you have questions. Next update tomorrow around 10am or sooner if we progress.

May 15, 2024 14:25 CEST
May 15, 2024 12:25 UTC
[Identified] Cleaning is still active. The supplier has escalated this as far as possible to get a quicker solution in place. Again, the platform operates normally except for delays when creating new objects in Openstack. Contact us on support@elastx.se if you have questions. Next update upon any progress or tomorrow morning.

May 16, 2024 11:23 CEST
May 16, 2024 09:23 UTC
[Monitoring] Together with our supplier we have now completed major part of the cleaning to improve performance when creating new networks. We can now see that new networks are created without any unexpected delays. We will continue to monitor the performance and work together with the supplier to ensure that we can avoid degraded performance in the future.

Scheduled Maintenance

Schedule

May 22, 2024 09:00 - 17:00 CEST
May 22, 2024 07:00 - 15:00 UTC

Components

OpenStack

Locations

STO1 - Stockholm South, STO2 - Stockholm West, STO3 - Stockholm North

Description

We are upgrading the HSM cluster that is the backend storage for Barbican in Openstack. This service is used to store secrets, mainly legacy encrypted volume keys, and SSL certificate keys when using Octavia L7 proxies. We do not expect any disturbance but there is an increased risk during this service window. There should be no impact to any already provisioned resources.

1

Active Incidents

0

Active Maintenances

1

Upcoming Maintenances

Cloud Connect




Operational

Cloud VPN




Operational

DBaaS




Operational

Mail Relay




Operational

Kubernetes




Operational

OpenStack




Operational

Virtuozzo PaaS




Operational

External Services

History

No incidents in the last 7 days