What happened with GCal-synced accounts today. We are truly sorry for the problems. Why the cloud can be trusted.

Yesterday we quietly introduced some speed improvements to the Google Calendar sync in Nozbe and were very happy about our first tests… and when we put this all into production everything seemed fine. But it wasn’t.

There was a hidden bug in our code that started messing up with the tasks synced with Google and duplicating some of them and generally messing things up.

Only accounts that synced with Gcal were affected and we’re truly sorry for this. We managed to fix the issues and revert the damage.

When we realized what was happening we had to first disable the Gcal sync and quickly start debugging the problem. Then we processed to reverting the damage by bringing back the tasks some users lost and deleting the duplicates. I believe we’ve managed to undo the problem. Luckily our system was designed to handle problems like this.

The beauty of the cloud and our triple-backup technology

When problems like this occurred (which happens extremely rarely) we saw our users questioning the cloud computing and trusting our cloud-based system with their information.

However, in moments like these, our cloud-based system is the exact reason why there is nothing to worry about. Let me explain.

Your data is safe at all times. Even if our system encounters a bug or malfunctions. We use triple-backup system to make sure your data is never lost.

Here’s how we do it:

1) We backup the entire Nozbe database every 2-3 hours to two remote geo-redundant locations.
2) We backup the entire Nozbe system with all the databases every single day to two geo-redundant locations.
3) When you insert your data to Nozbe (actions, projects, contexts, notes, files…) they are being duplicated, even when removed, to make sure you (or any of our scripts) cannot accidentally remove your data.

(additionally our enterprise-grade servers have redundant hard disks arrays and all the other usual preventive measures)

That’s why it took us a little while but we could revert the damage and quickly track which actions got lost or duplicated… and bring all back to normal.

You trust us with your data so we treat it with extreme care no matter what

That’s why we’re making sure we have our backup technology in place to protect your data at all times. Your trust is all we care about. And in moments like these the cloud computing and a properly designed cloud system can show its strengths.

Recent failures of Flickr, Tumblr or Google Mail showed that some big players don’t do sufficient backups to keep the cloud safe. We are a small player and maybe that’s why we care so much more about your data than the big guys.

Again, we’re truly sorry that some of you were not as productive today as you’re used to

Although only a part of our users syncs with Google Calendar and only a fraction of these were affected, we wanted to make sure you get this message and know we care about you all and we can’t express how sorry we are that today some of you were not as productive as you’re used to.

We haven’t brought back the Google Calendar sync just yet (we will be bringing it back tomorrow) as we have to perform some additional tests to make sure it’s working properly. The problem is entirely ours, Google is not to blame. We messed up. We are sorry.

If there are still some issues with your account or we have overlooked something, please do contact our customer support

Thank you for your support!

Although it was a tough day for us and we have lost some customers today, I’m happy that our backup technologies worked as they should have and many of you have expressed support and understanding… and patience. We applaud you for that and thank you. Things fail, problems do happen but it’s our job to keep the system running and care about your data no matter what. Thanks for your trust again. Hope that in spite of the system glitches today, we’ve proven we are trust-worthy. Thanks.

- Michael Sliwinski, founder of Nozbe