The servers weren't quite able to keep up with their workload during the busy part of the day Thursday but are operating normally now. As load rises on Friday morning it is possible that service will degrade again. We are working on two fronts to address our overload problems: 1) deploying additional server capacity and 2) making a performance optimization in the software. One or both of these measures will be introduced on Friday to address our increased load.
The heavy traffic over the last few days is due to two things. First the transition to multiple calendars generates more work for the servers. Second, we have three times as many new users signing up every day than we did last week.
The status on the transition to multiple calendars is as follows: about half our users have had the feature enabled. Roughly 75% of their devices have made the transition. When we're sure that most of the eligible devices have transitioned and the current performance and capacity issues have been addressed, the remaining users will then be multical-enabled.
The web page that reports sync status and the elapsed time since the last successful sync will display incorrect information at present. This is because the recording of the status and error data has been temporarily disabled to reduce disk traffic (and improve overall reliability).
Friday, January 16, 2009
Thursday, January 15, 2009
iPhone Calendar Color Selection
The mystery of the iPhone calendar coloring is...well...still a mystery to us. At present the situation is this : the iPhone picks the colors. We don't yet know how to send it color information. If we learn how to do that (assuming it's possible) then we will allow users to pick the colors. Google does allow users to select colors, and assuming a stable calendar/color assignment on the device, it is usually possible to modify your Google colors so they match. However, sometimes the iPhone randomly changes the colors. There is also a strange black on yellow color that can show up, and in extreme cases all the calendars get that color. We suspect that it has some deep meaning (perhaps related to the 18th century practice of flying a yellow flag on a ship to indicate the presence of fever?). The yellow plague and the random color reassignment can usually be fixed by clicking the link 'Folder list on the device does not match? Reset it. ' on the calendar setup page.
Chinese, Japanese Calendar Names
Several users reported that in the calendar selection page, if the calendar name is a multibyte string, it was displayed as ???? characters. This bug has been fixed and those calendars should be readable now.
Service Status and Vanishing Contacts
After a marathon debugging session we found and fixed two performance-related bugs in the service last night. One had led to more disk I/O being done when a device performed a re-sync with contacts enabled. The data we store was being re-written when it hadn't really changed. We had many devices re-syncing yesterday when they converted to multiple calendars. This led to disk saturation and hence service slowness. Slow service leads to increased load because some devices time out and trigger yet more re-syncing. The second problem was a memory leak that became worse with the increased load, eventually leading to the service bogging down completely which triggers an automated re-start. Both these bugs were fixed late last night.
The worst aspect of this incident was that Apple devices interacted in an unfortunate way with the degraded service: if the device wasn't able to sync a change made on its end it could trigger an out of sync condition in the service. This is handled on the iPhone by deleting all the existing data, then re-fetching new data from the service. Normally this would happen in a few seconds but yesterday the service wasn't always responding to the re-fetch and as a result the device ended up empty until it was able to successfully re-sync.
For us this is a worse situation than if the service were completely down because in that case you'd still have all your data on the phone. Therefore we modified the service code last night to allow us to globally block these out of sync triggers on a temporary basis. Hence we can guarantee that nobody will have their contacts vanish all of a sudden. The downside to this is that nothing will sync for the affected devices either. This safety mode was enabled last night. It will be turned off once we're sure service is stable this morning (things look good at present).
All the devices that got into the half-synced state where contacts and calendar events were 'vanished' should have picked up sync again and fetched new data.
We do have plenty of servers and network bandwidth available.
Update : people have asked 'what can I do' to get syncing again. The answer is : nothing. Sync should pick up again naturally. The safety mode has been turned off now, so unless there are further service load problems everyone should see normal service return soon.
The worst aspect of this incident was that Apple devices interacted in an unfortunate way with the degraded service: if the device wasn't able to sync a change made on its end it could trigger an out of sync condition in the service. This is handled on the iPhone by deleting all the existing data, then re-fetching new data from the service. Normally this would happen in a few seconds but yesterday the service wasn't always responding to the re-fetch and as a result the device ended up empty until it was able to successfully re-sync.
For us this is a worse situation than if the service were completely down because in that case you'd still have all your data on the phone. Therefore we modified the service code last night to allow us to globally block these out of sync triggers on a temporary basis. Hence we can guarantee that nobody will have their contacts vanish all of a sudden. The downside to this is that nothing will sync for the affected devices either. This safety mode was enabled last night. It will be turned off once we're sure service is stable this morning (things look good at present).
All the devices that got into the half-synced state where contacts and calendar events were 'vanished' should have picked up sync again and fetched new data.
We do have plenty of servers and network bandwidth available.
Update : people have asked 'what can I do' to get syncing again. The answer is : nothing. Sync should pick up again naturally. The safety mode has been turned off now, so unless there are further service load problems everyone should see normal service return soon.
Wednesday, January 14, 2009
Server Stability Problems Today
It turns out we were a bit over-ambitious with the number of users enabled for multiple calendar sync. The conversion process went very smoothly over night, but when the load rose this morning (load peaks around 9am pacific time every day) the service hit an overload state. There could be unreliable service and short outages today as we deal with the situation. It's unlikely the problems will last beyond today. Apologies for the poor service today, we expect to have things back to normal very soon.
Tuesday, January 13, 2009
Colored Calendar Phase-In
Some details on how iPhone multiple calendar support is being enabled for our users:
In planning the roll out for multiple calendars there was some concern that the extra load on our servers from processing the transition for large numbers of devices all at once might bring down the entire service. Transitioning a device from the old to the new calendar content involves re-fetching all calendar events and quite a bit of churn in the persistent store, so more CPU and disk and network resources are used than for normal syncing. This led to the adoption of a progressive roll out strategy.
In addition, in order to reduce the risk of widespread service instability, we decided to initially deploy first in a mode where 'old' users did not get the new features but newly signed up users did. The idea was that new users never had working sync and therefore if something unfortunate happened they would be much less upset than existing users who had had working sync beforehand.
The plan was to watch for any reports of badness from new users for a while, then if none were seen, begin to enable multiple calendar support for existing users.
What actually happened was that there were no reports of new problems but there was an unrelated server stability issue over the weekend. Investigating that issue delayed the beginning of the 'old user' roll out, and in the meantime someone leaked the existence of multiple calendars for new users in a blog comment.
This prompted the 'official' announcement which of course led to frustration for those users who had not yet been 'enabled'. It was necessary to proceed with caution: enabling 100 users first then watching server load for a while, followed by 1000 users and so on. As of tonight roughly 40,000 users have had multiple calendar support enabled and we expect to enable the remainder tomorrow, assuming no server overload issues show up.
In planning the roll out for multiple calendars there was some concern that the extra load on our servers from processing the transition for large numbers of devices all at once might bring down the entire service. Transitioning a device from the old to the new calendar content involves re-fetching all calendar events and quite a bit of churn in the persistent store, so more CPU and disk and network resources are used than for normal syncing. This led to the adoption of a progressive roll out strategy.
In addition, in order to reduce the risk of widespread service instability, we decided to initially deploy first in a mode where 'old' users did not get the new features but newly signed up users did. The idea was that new users never had working sync and therefore if something unfortunate happened they would be much less upset than existing users who had had working sync beforehand.
The plan was to watch for any reports of badness from new users for a while, then if none were seen, begin to enable multiple calendar support for existing users.
What actually happened was that there were no reports of new problems but there was an unrelated server stability issue over the weekend. Investigating that issue delayed the beginning of the 'old user' roll out, and in the meantime someone leaked the existence of multiple calendars for new users in a blog comment.
This prompted the 'official' announcement which of course led to frustration for those users who had not yet been 'enabled'. It was necessary to proceed with caution: enabling 100 users first then watching server load for a while, followed by 1000 users and so on. As of tonight roughly 40,000 users have had multiple calendar support enabled and we expect to enable the remainder tomorrow, assuming no server overload issues show up.
Monday, January 12, 2009
New Features!
NuevaSync has several new features for the New Year:
Support for multiple, separate calendars on Apple iPhone and iPod touch devices is here! Now you can tell which appointment belongs in which calendar, add appointments to the calendar of your choosing, and filter your display to show only events from a specific calendar. What is most important of all, your calendars each get their own color. We are currently phasing this feature in: if you don't see it on your phone yet, you will very shortly. 
You can now select exactly which of your calendars you want -- or do not want -- to sync. This feature is available for all users and device-types: just click on the setup link next to the calendar section after logging in to the NuevaSync site.
Read-only calendars can now be selected for sync. This is great for common calendars, like holidays, and for shared calendars from school or work. You can make changes and remove entries from a read-only calendar on your phone, but those changes will not be applied to the remote version. On an iPhone or iPod, read-only calendars are marked with an "*" after the name. 
Subscribe to:
Posts (Atom)