This weekend, Kickserv was unavailable for an extended period of time. Part of this outage was planned, and part of it was unanticipated. We understand how much you rely on Kickserv to manage your business. This weekend, we failed to maintain the level of service you expect. We are very sorry for this, and would like to share with you what took place and the steps we’re taking to ensure you're able to access Kickserv.
Successful Planned Maintenance
At midnight GMT on Feb 14, we began our scheduled maintenance by bringing down our app and database servers. After testing the database work we were performing last week, we estimated a downtime of roughly 3 hours. The database maintenance was completed successfully within this 3-hour period, and we began to bring our servers back online at 02:02 GMT.
By 02:40, all of our servers were running, but we began to notice severely slower response times as our database indexes were being rebuilt. Normally, this index rebuilding can happen concurrently as users are accessing the application, but higher-than-anticipated traffic on our servers prevented the indexes from being properly rebuilt.
As a result, we made the decision to limit access to the application while the indexes were being rebuilt. This took until approximately 08:16 GMT, when the servers were brought back up.
We consider a lengthy downtime like this unacceptable and always strive to provide the best service we can. We are very proud of our track record. Our average monthly uptime for 2015 was 99.974%.
That said, we need to do a better job in the future. Fortunately, this was a one-time, extensive database migration that will not need to happen again, and the new database implementation will allow us to develop features that were not possible before and provide improved speed and reliability as our servers stabilize over the next few days. We will continue to investigate improved methods of maintenance to limit any application downtime in the future.
Finally, we were made aware that some of our users were not notified of the planned maintenance. It is our responsibility to make sure you're informed of any lapses in service, and we have outlined a plan to ensure ALL of our users are notified in the future.
We also urge you to subscribe to status updates at our status page (http://status.kickserv.com). You will be notified by email of any outages or planned maintenance periods in the future.
Wrapping Things Up
We understand how important Kickserv is to enable your business to operate and succeed. The entire team at Kickserv would like to apologize for this unexpected outage. We will continue to investigate this incident and strive to improve our response in the future.