Friday, January 30, 2009
Tuesday, January 27, 2009
Friday, January 23, 2009
Saturday, January 17, 2009
Update: thanks to a very helpful user who was willing to allow us to use his account for debugging, we found an additional problem with Google contact sync last night. Users with larger numbers of contacts should benefit from the fix which is now live on the production service.
Update: if you have no contact sync (or indeed no calendar sync) please send a message to firstname.lastname@example.org with subject 'I have no contacts (or calendar)' and include your Nuevasync username and the number of contacts in your list if you know that. There may be some account-specific problem and we'd like to investigate some affected accounts. Thanks.
Friday, January 16, 2009
The heavy traffic over the last few days is due to two things. First the transition to multiple calendars generates more work for the servers. Second, we have three times as many new users signing up every day than we did last week.
The status on the transition to multiple calendars is as follows: about half our users have had the feature enabled. Roughly 75% of their devices have made the transition. When we're sure that most of the eligible devices have transitioned and the current performance and capacity issues have been addressed, the remaining users will then be multical-enabled.
The web page that reports sync status and the elapsed time since the last successful sync will display incorrect information at present. This is because the recording of the status and error data has been temporarily disabled to reduce disk traffic (and improve overall reliability).
Thursday, January 15, 2009
The worst aspect of this incident was that Apple devices interacted in an unfortunate way with the degraded service: if the device wasn't able to sync a change made on its end it could trigger an out of sync condition in the service. This is handled on the iPhone by deleting all the existing data, then re-fetching new data from the service. Normally this would happen in a few seconds but yesterday the service wasn't always responding to the re-fetch and as a result the device ended up empty until it was able to successfully re-sync.
For us this is a worse situation than if the service were completely down because in that case you'd still have all your data on the phone. Therefore we modified the service code last night to allow us to globally block these out of sync triggers on a temporary basis. Hence we can guarantee that nobody will have their contacts vanish all of a sudden. The downside to this is that nothing will sync for the affected devices either. This safety mode was enabled last night. It will be turned off once we're sure service is stable this morning (things look good at present).
All the devices that got into the half-synced state where contacts and calendar events were 'vanished' should have picked up sync again and fetched new data.
We do have plenty of servers and network bandwidth available.
Update : people have asked 'what can I do' to get syncing again. The answer is : nothing. Sync should pick up again naturally. The safety mode has been turned off now, so unless there are further service load problems everyone should see normal service return soon.
Wednesday, January 14, 2009
Tuesday, January 13, 2009
In planning the roll out for multiple calendars there was some concern that the extra load on our servers from processing the transition for large numbers of devices all at once might bring down the entire service. Transitioning a device from the old to the new calendar content involves re-fetching all calendar events and quite a bit of churn in the persistent store, so more CPU and disk and network resources are used than for normal syncing. This led to the adoption of a progressive roll out strategy.
In addition, in order to reduce the risk of widespread service instability, we decided to initially deploy first in a mode where 'old' users did not get the new features but newly signed up users did. The idea was that new users never had working sync and therefore if something unfortunate happened they would be much less upset than existing users who had had working sync beforehand.
The plan was to watch for any reports of badness from new users for a while, then if none were seen, begin to enable multiple calendar support for existing users.
What actually happened was that there were no reports of new problems but there was an unrelated server stability issue over the weekend. Investigating that issue delayed the beginning of the 'old user' roll out, and in the meantime someone leaked the existence of multiple calendars for new users in a blog comment.
This prompted the 'official' announcement which of course led to frustration for those users who had not yet been 'enabled'. It was necessary to proceed with caution: enabling 100 users first then watching server load for a while, followed by 1000 users and so on. As of tonight roughly 40,000 users have had multiple calendar support enabled and we expect to enable the remainder tomorrow, assuming no server overload issues show up.
Monday, January 12, 2009
NuevaSync has several new features for the New Year:
Support for multiple, separate calendars on Apple iPhone and iPod touch devices is here! Now you can tell which appointment belongs in which calendar, add appointments to the calendar of your choosing, and filter your display to show only events from a specific calendar. What is most important of all, your calendars each get their own color. We are currently phasing this feature in: if you don't see it on your phone yet, you will very shortly.
You can now select exactly which of your calendars you want -- or do not want -- to sync. This feature is available for all users and device-types: just click on the setup link next to the calendar section after logging in to the NuevaSync site.
Read-only calendars can now be selected for sync. This is great for common calendars, like holidays, and for shared calendars from school or work. You can make changes and remove entries from a read-only calendar on your phone, but those changes will not be applied to the remote version. On an iPhone or iPod, read-only calendars are marked with an "*" after the name.