New Service Code Deployed

Since 08:00 UTC today we have new code running that provides significantly improved capacity and reliability. So far everything looks to be operating as expected.

Nuevasync Review at TechCrunch

Jeff Widman at TechCrunchIT has posted an interview with David.  Particularly if you are curious how Nuevasync came to be, make sure to check it out.

Support News

We've recruited a new member of staff who will work 100% on user support. It's been challenging to keep up with the support workload recently, particularly at times when there were service reliability issues because those tend to lead to large numbers of new tickets being raised. Users should experience more consistent response to support issues and our engineering staff will be able to focus entirely on fixing the problems reported.

If You Have no Contacts...

Thanks to everyone that reported they had no contacts. A pattern is emerging. Almost all the users reporting lack of contacts have the same problem, which in fact isn't a new problem nor is it related to the recent reliability issues. This blog article has the details. If you have a lack of contacts, or indeed any problem, please log into our web site and check your account status first (if you see a red light that indicates a problem that needs to be fixed), then your sync status. These pages 'understand' a large number of known support issues and will often display information on how to fix the problem you have.

Update: thanks to a very helpful user who was willing to allow us to use his account for debugging, we found an additional problem with Google contact sync last night. Users with larger numbers of contacts should benefit from the fix which is now live on the production service.

Contact Sync Status

There are quite a few blog comments here suggesting that contact sync just isn't working at present. As far as we are aware contact sync is working normally. We can sync contacts on test devices, add contacts on each end and everything seems ok. There are situations in which the iPhone will delete all its contacts and then re-fetch the contact list from our service. If for some reason it can't contact the service at that point then the device has an empty contact list. It's possible that this event has been happening more often than normal, or that there's some reason why the re-fetch isn't working. We'd expect devices to catch up again and retain their contact list.

Update: if you have no contact sync (or indeed no calendar sync) please send a message to support@nuevasync.com with subject 'I have no contacts (or calendar)' and include your Nuevasync username and the number of contacts in your list if you know that. There may be some account-specific problem and we'd like to investigate some affected accounts. Thanks.

iPhone/iPod Default Calendar

We're not yet up to date on the web site with all the tips and tricks for the iPhone multiple calendar feature. Here's one useful thing: for reasons not yet clear the iPhone sometimes picks a 'default calendar' seemingly at random. This can be annoying because when new events are created on the device they are assigned to the default calendar unless another one is explicitly selected. We intend the default to be your Google main calendar but this doesn't seem to always be the case. Until we get this resolved, it will be necessary to change the default calendar after the initial sync. This can be done in the settings page, under 'Mail, Contacts, Calendar...', scroll down 'till you see 'Default Calendar'.

iPhone Multiple Calendar Support Fully Deployed

All existing users now have multiple calendar support enabled. The next time your device syncs it should pick up colored calendars. No user intervention is required. Sync status logging is still turned off so the sync status page will not display correct information yet. It might say that your sync has an error when in fact it succeeded.

Service Status

The servers weren't quite able to keep up with their workload during the busy part of the day Thursday but are operating normally now. As load rises on Friday morning it is possible that service will degrade again. We are working on two fronts to address our overload problems: 1) deploying additional server capacity and 2) making a performance optimization in the software. One or both of these measures will be introduced on Friday to address our increased load.
The heavy traffic over the last few days is due to two things. First the transition to multiple calendars generates more work for the servers. Second, we have three times as many new users signing up every day than we did last week.
The status on the transition to multiple calendars is as follows: about half our users have had the feature enabled. Roughly 75% of their devices have made the transition. When we're sure that most of the eligible devices have transitioned and the current performance and capacity issues have been addressed, the remaining users will then be multical-enabled.
The web page that reports sync status and the elapsed time since the last successful sync will display incorrect information at present. This is because the recording of the status and error data has been temporarily disabled to reduce disk traffic (and improve overall reliability).

iPhone Calendar Color Selection

The mystery of the iPhone calendar coloring is...well...still a mystery to us. At present the situation is this : the iPhone picks the colors. We don't yet know how to send it color information. If we learn how to do that (assuming it's possible) then we will allow users to pick the colors. Google does allow users to select colors, and assuming a stable calendar/color assignment on the device, it is usually possible to modify your Google colors so they match. However, sometimes the iPhone randomly changes the colors. There is also a strange black on yellow color that can show up, and in extreme cases all the calendars get that color. We suspect that it has some deep meaning (perhaps related to the 18th century practice of flying a yellow flag on a ship to indicate the presence of fever?). The yellow plague and the random color reassignment can usually be fixed by clicking the link 'Folder list on the device does not match? Reset it. ' on the calendar setup page.

Chinese, Japanese Calendar Names

Several users reported that in the calendar selection page, if the calendar name is a multibyte string, it was displayed as ???? characters. This bug has been fixed and those calendars should be readable now.

Service Status and Vanishing Contacts

After a marathon debugging session we found and fixed two performance-related bugs in the service last night. One had led to more disk I/O being done when a device performed a re-sync with contacts enabled. The data we store was being re-written when it hadn't really changed. We had many devices re-syncing yesterday when they converted to multiple calendars. This led to disk saturation and hence service slowness. Slow service leads to increased load because some devices time out and trigger yet more re-syncing. The second problem was a memory leak that became worse with the increased load, eventually leading to the service bogging down completely which triggers an automated re-start. Both these bugs were fixed late last night.
The worst aspect of this incident was that Apple devices interacted in an unfortunate way with the degraded service: if the device wasn't able to sync a change made on its end it could trigger an out of sync condition in the service. This is handled on the iPhone by deleting all the existing data, then re-fetching new data from the service. Normally this would happen in a few seconds but yesterday the service wasn't always responding to the re-fetch and as a result the device ended up empty until it was able to successfully re-sync.
For us this is a worse situation than if the service were completely down because in that case you'd still have all your data on the phone. Therefore we modified the service code last night to allow us to globally block these out of sync triggers on a temporary basis. Hence we can guarantee that nobody will have their contacts vanish all of a sudden. The downside to this is that nothing will sync for the affected devices either. This safety mode was enabled last night. It will be turned off once we're sure service is stable this morning (things look good at present).
All the devices that got into the half-synced state where contacts and calendar events were 'vanished' should have picked up sync again and fetched new data.
We do have plenty of servers and network bandwidth available.

Update : people have asked 'what can I do' to get syncing again. The answer is : nothing. Sync should pick up again naturally. The safety mode has been turned off now, so unless there are further service load problems everyone should see normal service return soon.

Server Stability Problems Today

It turns out we were a bit over-ambitious with the number of users enabled for multiple calendar sync. The conversion process went very smoothly over night, but when the load rose this morning (load peaks around 9am pacific time every day) the service hit an overload state. There could be unreliable service and short outages today as we deal with the situation. It's unlikely the problems will last beyond today. Apologies for the poor service today, we expect to have things back to normal very soon.

Colored Calendar Phase-In

Some details on how iPhone multiple calendar support is being enabled for our users:
In planning the roll out for multiple calendars there was some concern that the extra load on our servers from processing the transition for large numbers of devices all at once might bring down the entire service. Transitioning a device from the old to the new calendar content involves re-fetching all calendar events and quite a bit of churn in the persistent store, so more CPU and disk and network resources are used than for normal syncing. This led to the adoption of a progressive roll out strategy.
In addition, in order to reduce the risk of widespread service instability, we decided to initially deploy first in a mode where 'old' users did not get the new features but newly signed up users did. The idea was that new users never had working sync and therefore if something unfortunate happened they would be much less upset than existing users who had had working sync beforehand.
The plan was to watch for any reports of badness from new users for a while, then if none were seen, begin to enable multiple calendar support for existing users.
What actually happened was that there were no reports of new problems but there was an unrelated server stability issue over the weekend. Investigating that issue delayed the beginning of the 'old user' roll out, and in the meantime someone leaked the existence of multiple calendars for new users in a blog comment.
This prompted the 'official' announcement which of course led to frustration for those users who had not yet been 'enabled'. It was necessary to proceed with caution: enabling 100 users first then watching server load for a while, followed by 1000 users and so on. As of tonight roughly 40,000 users have had multiple calendar support enabled and we expect to enable the remainder tomorrow, assuming no server overload issues show up.

New Features!

NuevaSync has several new features for the New Year:

Support for multiple, separate calendars on Apple iPhone and iPod touch devices is here!  Now you can tell which appointment belongs in which calendar, add appointments to the calendar of your choosing, and filter your display to show only events from a specific calendar.  What is most important of all, your calendars each get their own color.  We are currently phasing this feature in: if you don't see it on your phone yet, you will very shortly.

 

You can now select exactly which of your calendars you want -- or do not want -- to sync.  This feature is available for all users and device-types: just click on the setup link next to the calendar section after logging in to the NuevaSync site.

 

Read-only calendars can now be selected for sync.  This is great for common calendars, like holidays, and for shared calendars from school or work.  You can make changes and remove entries from a read-only calendar on your phone, but those changes will not be applied to the remote version.  On an iPhone or iPod, read-only calendars are marked with an "*" after the name.