Postmortem -
Read details
Nov 8, 04:49 PST
Resolved -
Service Update: Issue Resolved
We’re pleased to inform you that the issue has been resolved, and services are now fully online.
A small number of sites may still be missing; we will reach out directly to the respective owners to arrange restoration. Rest assured, the data is intact, but it requires additional time to be fully restored from the failed hardware.
Thank you for your patience and support during this process.
Nov 8, 00:27 PST
Monitoring -
Service Restoration in Progress
The restoration process has begun, and websites are starting to come back online as we speak. Our team is actively working to ensure that all services are fully operational as soon as possible.
ETA: Sites will continue coming online over the next few hours as restoration progresses.
We appreciate your patience and understanding during this process, and we’ll provide further updates here as needed.
Thank you for your continued support.
Nov 7, 22:45 PST
Update -
We have successfully mounted new hardware and are currently configuring the system and uploading backup data from 2024-11-06. Our team is working to restore access, and we anticipate that services should be back online within a few hours.
ETA: Approximately 3 hours
Data Recovery in Progress:
We are also continuing data recovery from the previous hardware. Once this process is complete, all systems and data will be fully up to date.
ETA: Approximately 5 hours
We appreciate your patience and understanding during this restoration process. Thank you for your continued support.
Next Update: Approximately 3 hours
Nov 7, 19:28 PST
Update -
We are currently in the process of extracting data from hardware that has encountered a failure.
Our team is working diligently to restore all affected services as quickly as possible.
Estimated Data Transfer Time: Approximately 6 to 7 hours to transfer, assuming ideal conditions.
We appreciate your patience and understanding as we work through this recovery process.
Further updates will be provided here as we make progress.
Thank you for your continued support.
Nov 7, 17:38 PST
Identified -
Service Outage Update: US.CA Server
We are currently addressing an unexpected outage on our US.CA server caused by a series of hardware issues, including a RAID failure, motherboard malfunction, and power supply failure due to an unknown cause. Our team is fully engaged in resolving this complex situation to restore services as swiftly as possible.
What We’re Doing
To expedite recovery, we are setting up a new machine and have begun the data restoration process. The initial restoration will use recent backups from November 3-4, aiming to get core services up and running quickly. Additional data will be restored gradually over the next few days to ensure complete access.
What to Expect
We anticipate the first phase of restoration to be completed within a few hours. Full data recovery will take longer, but we’ll continue providing updates throughout the day as progress is made.
Thank you for your patience and understanding as we work diligently to resolve these issues. Further updates will follow—please reach out if you have any questions.
Nov 7, 16:08 PST