Thursday, January 15, 2009

Service Status and Vanishing Contacts

After a marathon debugging session we found and fixed two performance-related bugs in the service last night. One had led to more disk I/O being done when a device performed a re-sync with contacts enabled. The data we store was being re-written when it hadn't really changed. We had many devices re-syncing yesterday when they converted to multiple calendars. This led to disk saturation and hence service slowness. Slow service leads to increased load because some devices time out and trigger yet more re-syncing. The second problem was a memory leak that became worse with the increased load, eventually leading to the service bogging down completely which triggers an automated re-start. Both these bugs were fixed late last night.
The worst aspect of this incident was that Apple devices interacted in an unfortunate way with the degraded service: if the device wasn't able to sync a change made on its end it could trigger an out of sync condition in the service. This is handled on the iPhone by deleting all the existing data, then re-fetching new data from the service. Normally this would happen in a few seconds but yesterday the service wasn't always responding to the re-fetch and as a result the device ended up empty until it was able to successfully re-sync.
For us this is a worse situation than if the service were completely down because in that case you'd still have all your data on the phone. Therefore we modified the service code last night to allow us to globally block these out of sync triggers on a temporary basis. Hence we can guarantee that nobody will have their contacts vanish all of a sudden. The downside to this is that nothing will sync for the affected devices either. This safety mode was enabled last night. It will be turned off once we're sure service is stable this morning (things look good at present).
All the devices that got into the half-synced state where contacts and calendar events were 'vanished' should have picked up sync again and fetched new data.
We do have plenty of servers and network bandwidth available.

Update : people have asked 'what can I do' to get syncing again. The answer is : nothing. Sync should pick up again naturally. The safety mode has been turned off now, so unless there are further service load problems everyone should see normal service return soon.

83 comments:

Spencer E Holtaway said...

Wow well handled, thanks guys.

Any updates on lack of coloured calendars when all calendars are showing on iPhone? Probably the least of your worries right now, but just in case!

Thanks again for the service,

Spence

Valley Buddha said...

"All the devices that got into the half-synced state where contacts and calendar events were 'vanished' should have picked up sync again and fetched new data."

By the sound of this, I should have my contacts back. I've tried both wifi and OTA and nothing yet. I will live but hopefully this is just for a few more hours.

Edward said...

Thanks for the hard work and the service. Glad to hear you have enough servers, but count me amongst those who'd be happy to donate to the cause.

Question--for those of us who ended up with blank contacts/calendars--should it automatically re-sync or should we manually re-sync? My contacts came back automatically but my boss is still missing his.

SomeDutchGuy said...

Indeed; good work on fixing the problems. Its understandable that a service has a few problems occasionally, and as a programmer I've seen more than my fair share of them. Its how you handle them when they arise that matters, and you guys are phenomenal.

I'd just like to report that all the calendars show up on my iTouch as separate calendars but they all share the same color. Any Ideas?

Fabian Inthamoussu said...

You are the best guys!!!

GREAT SERVICE, AND FOR FREE!!!

Keep the good work!

I can't wait to see all these features + email via IMAP gateway, ando for free !!!

Again, you are the best!!!

. said...

Thanks for keeping us up to date. Your great communication and service!

Anonymous said...

Great work! I love this service. A minor thing that happens when the iphone looses everything is that the assigned ringtones dissappear. Does anyone know of a way to store the ringtones in google contacts? I know this is really minor, but if there is a way to do that I would love to find out.

David Boreham said...

@Spencer: a side effect of the safety mode we have turned on since last night is that no phones will convert to multiple calendars until we turn it off. We'll do that this morning and the remaining unconverted iPhones should convert after that.

George Harper said...

Everything came back for me this morning. I agree with the other posts -- the communication was terrific. I think this is a great service -- keeps me from having to be tethered to my mac for calendar updates, which is a godsend. Happy to contribute (I guess I'll wait for the premium service).

A few things I've noticed (not complaints, just items for you to notice for the future):

1. Before this rollout, the web configuration for google calendars could see all calendars I had access to -- now I only see my personal and work ones (it'd be great to sync our google calendar for company holidays -- as well as a couple co-workers).

2. The two calendars are given the same color: yellow

3. I'm not sure which calendar the PERSONAL one refers to -- possibly my gmail one, but not sure how it sees that one

4. Curious under what conditions notifications/alerts are synchronized and provided on the iPhone (I have popup configured on Google calendar 15 minutes before, but there's no Alert on my iPhone

Spencer E Holtaway said...

@david - that's fantastic news. Hope it all runs smoothly for you guys!

Spencer

Josh B. said...

Thanks for all of your work. This is fantastic.

I have noticed that everything seems to be back and working except that my Read-only Calendar does not show up on my iPhone.

I wonder if this is a localized problem on my account/set up/iPhone or if it is system-wide.

Thanks,

Josh

Unknown said...

You guys rock. I love using the nuevasync service!

Earl A Hudson said...

Great work. This is the best service I have on my iPhone. And you did an outstanding job moving to color and getting things back up and running after problems.

jmvidal said...

Still no contacts for me. I tried turning contacts on/of but still nothing.

Anonymous said...

I'm havin some problems since this morning with an iPhone 3G.
Firstly I'd (google) calendarS (6) working ok, 1 color for each of them. Just when I started to making some changes on colour and adding 2 more read-only calendars, started to have problems.
- All my contacts (gmail) where lost, now a 75% are synced (but impossible to sync all)
- All my events where lost, each calendar did appear but all were yellow, now, like my contacts, I've only a pieze of all events and are yellow all.

What can I do?
I've tried reconfigue nuevasync, reconfigure exchange account and turning off-on syncronization on the iPhone.

David Castro said...

lost all my contacts around 10:30 a.m. Eastern time, tried resyncing and all the other trouble shooting, nothing is working so far. Maybe a gremlin crept back into the system?

stirman said...

Still no contacts for me either... looking forward to an update! Thanks for the communication, it's much appreciated.

samhnadubhadh said...

I'm looking forward to the service working, but I'm still having major problems.

After yesterday, I woke up this morning and everything seemed to be working. All of my calendars were different colors and it seemed like they had all synced. I tried adding an event on my iPhone, but it didn't end up on my google calendar. So I tried manually resyncing, but that just erased everything and now I'm back to where I was yesterday.

Help?

Unknown said...

Sorry, but my iPhone calendar (having come back last night (GMT)) has emptied again and is now completely blank.

Good luck!

Jim Warwick

Steven said...

Hey guys mine resolved itself yesterday, although it was slow getting pushed updates, felt like I was fetching more than it was pushing. Today everything seems fine until my contacts have vanished again! Calendar entries are still up and look good but no contacts. Just thought I'd let you know that maybe the code fix to stop this from happening may not be working as expected.

Unknown said...

My phone still has no calendar or contact data. This is very disappointing since I will now have to sync to my mac so I can use my iphone today. I really love this service, so I guess I'll check back periodically to see if this ever starts working again.

Unknown said...

my phone has still no calendar data.. it's not working at all and the default calendar in the iphone settings is always wrong.

Jason (the other one...) said...

I hate to sound like a whiner but...

I'm one of the "older" users and evidently last in line to be able to use advertised (now on the front page of the website) new features in trade for stability.

Regardless, I've had calendar and contact data just fine for the past few days even with these server stability issues. Now that you say things are fixed I have no contact data as of this morning.

Steve said...

Contacts and Calendar on my iPhone are empty. About an hour ago some were there... now they are all gone.

Unknown said...

Still having issues... earlier today, all my calendar items disappeared from my iPhone.

I created a second calendar in my GMail calendar, and my iPhone picked up the new calendar when it sync'ed (and turned them both black/yellow). Syncing takes ages and I'm still not getting any calendar events at the end of it.

Toph said...

Love the service... would be willing to pay for it (and I'm usually not that interested in pay services).

Yesterday my contacts went missing. They are still missing. My calendar is still there, but it is the old one - with only one color.

I believe I am one of the older users... I'm happy to help you debug if needed (I'm in IT). My user name at NS is tofferr.

While I was writing this post I did get my contacts back. My calendar is still one color.

Josh B. said...

UPDATE:

Well, since my Read-Only Cal wasn't showing, I couldn't leave well enough alone and re-synced. Unfortunately, that first got rid of everything (I expected that) but then, nothing came back. After another re-sync no data showed but my calendars appears (in the window that shows them as a list) but with yellow dots. Re-synced again, now they're colorful again, but still no data. Same for contacts.

Thanks for your hard work to get this back up and running. I know you're probably working incredibly hard to do-so. Good luck. I'm getting impatient, but that's my issue, not on you guys.

-Josh B.

Paul said...

Whoops, my contacts just disappeared. I had 'em an hour ago. Methinks there still be problems.

Anonymous said...

I had to reformat the iphone today, and no contacts/calendars are syncing.

Jason Seriff said...

All of my contacts have disappeared both from the phone and from gmail. Is this data cached anywhere in the nuevasync software to be re synced to both, or is all of my data irrevocably lost at this point?

Thomas said...

I am all about getting this solved. And when we have problems, we throw lots of money at it.

Let's all do this... Send these guys some money via PayPal (www.paypal.com) at: info@nuevasync.com

Just send them something. We're throwing $50 at them to get this started.

Let's go guys!! You're doing a great job and we are eager to help you get this done.

Jas said...

Hi,

I'm waiting patiently but have now been without calendar or contacts for 48 hours. Latest post gives me the impression it should all be back up and running. I'm happy to keep waiting but wanted to check I'm not missing something?

Thanks guys.

Chandrashekar R. Gaajula said...

Thanks for working out the glitches.

However, only a few of my contacts have showed up on my iPhone. There is still a plenty that is missing.

What can I do to get all my contacts.

Michael said...

Great information, but perhaps the all-clear signal was given too soon. At 12:29PM 1/15/2009, I have no contacts. I don't remember checking contacts earlier today, or yesterday, so I can't say when it started.

Keep up the good work! You'll figure this out.

Anonymous said...

All of my contacts were back by 7am CST this morning. By noon they were all gone again. And they are still MIA.

Unknown said...

Thanks for the updates on the issues, but just an FYI --- calendars were syncing this morning and now all my data is gone (including contacts).

This is extremely frustrating, good luck on getting this issue resolved.

Unknown said...

Wow --- now they are back. Wild times, thanks for the hard work!

eaa511 said...

Yeah, I actually lost my contacts and calendar this morning around 10am EST. They are still missing...Overall the service has been great. I've been using it for a couple of months and it has been seamless til now. Looking forward to seeing it come back soon!

Greahm said...

After a re-Push some are back? And, I now have even more contacts that have lost the Name (NO NAME), oh joy!

terence said...

I am experiencing the out-of-sync/calendar vanishing issue at approximately 10:30am PST.

Also got a duplicate entry in google calendar on the comp for some reason

James said...

I don't have any contacts or calendar items currently. 11:45am Pacific on 1/15.

Steve said...

At 2:54 EST, my contacts have come and gone a couple times today. Currently, they are gone. And when they were back, there were a few that were missing.

Unknown said...

I think Neuvasync needs to post another update. Contrary to this posting from early this morning that there shouldn't be any more issues, my contacts have again disappeared.

Steve said...

At 3:20PM EST all of my contacts are back now.

Paul said...

My contacts have disappeared and haven't returned yet. It's been several hours now.

Anonymous said...

Likewise at 3:43 est my contacts and calendar are empty. Just to let you know. Do appreciate the seRvice and your prompt responses.

Unknown said...

Nothing. No contacts, no calendar. I gave up and started a different account, thinking a clean start would be better.

Still nothing -- it's been an hour......

Is MobileMe an option?

dShefman said...

No calendar data yet. It is 2:50. Thanks for the update... Good luck getting up.

Anonymous said...

Still no contacts for me. All day. Very sad news. :(

Any further updates?

James said...

I am still having problems, have tried re syncing the whole thing, but it takes about 10 attempts to complete a sync and every time it does it is only pulling back about 137 out of 237 contacts and they are random ones which are missing so it is not missing the last of them, but random ones!

Anyone having this issue?

Unknown said...

Still no contacts for me.

Unknown said...

and now i have my calendar back.. but some important calender items are not there... one entry ok, other one is not on my iphone.

what we can do? a reset? NO USE THE LAST TIMES!

Anonymous said...

No contacts. No calendar items.

ETA?

Ross Grady said...

Mysteriously, I now have exactly 50 contacts (out of a total of 278). Hmm.

I'm irritated, but NOT at Nuevasync. The *proper* way to handle device sync is to get the delta & then select one of howevermany methods to apply it. In the event of sync failure, rolling back to the last known good state would seem to be a no-brainer. Unless you're Apple, that is. Why yes, if my contacts are out of sync with the server, I *would* prefer you just DELETE THEM ENTIRELY until the problem is resolved. That's SO MUCH MORE USEFUL. Yeesh. Thanks a lot, Apple.

Hey Nuevasync, you need to turn on your "safety condition" again, until you get this stuff straightened out.

Pete said...

Well done. I like it when information is shared.

Looks like the gremlins are still at work. I got my contacts back, but now gone again. 5:15pm ET 15.Jan

Adam said...

Everything is up and running for me! Love the multiple calendars!
Thanks so much for the service!

mike77p said...

Still no contacts as well for me.

High Level Guy said...

My calendar kept flipping between updated and empty. Now it seems back in business. No contacts yet.

Agree w Ross, consider switching on the safety condition until everything is stable for at least a whole day or two.

Unknown said...

Thank you for sharing information- in fact this rapid disclosure of information is the only reason I didn't just jump ship on beta testing!

As an update about 2pm PST (Jan 15) contacts did a 'Houdini' and haven't showed up yet. Calendar seems to be intact... the colored dots preceding the calendar names changed a couple times though.

Perhaps when Nuevasync goes to a fee model they will remember the infinite patience of those tenacious beta-testers that stuck with them through thick and thin? ;)

Cheers,

Bill B said...

As of 551pm EST I still don't have Contacts or Calendar - I got them back briefly this morning, but they disappeared again by lunch time. ;-<

Unknown said...

It's 4:55PM CST and my contacts have been down for roughly 2 hours.

Stephan said...

The performance problems are still there. Calendar extremely slow to sync. Completely unable to sync contacts. Had my friend switch to iTunes Contacts sync and cease using Nuevasync altogether.

8thWarrior said...

New user just signed up today and completely unaware of stability issues. Was directed to Nuevasync as a solution and followed the instructions but no joy. Troubleshooted for 3 hours before I accidentally came across this blog. Delighted to find out it is not me and looking forward to getting Google Calendar working on my iPhone soon. Currently have occasional events from (4) calendars showing up.

Antoine said...

To Brian: Have you tried turning off the sync from within settings and then turning it back on. That's how I got my data back. I have copied the info below from NuevaSync's troubleshooting page.

Apple

Tap "Settings"
Tap "Mail, Contacts, Calendars"
Tap the NuevaSync account, for example "person@gmail.com"
Tap the off button for the service you wish to resync, for example "Calendar."
Wait for a moment as the device will say "Turning off
Tap the on button for the service to turn it back on.
You're done!

Josh B. said...

Everything is back and working for me, even calendar colors (granted they are not synced to my Google Cal colors, but hey).

Contacts look good too. Tried a few test entries on the iPhone and they sync correctly with Google.

Nice. Thank you.

Unknown said...

Still no contacts yet, even after turning sync on and off several times over a several hour period.

Steve said...

I'm back baby!! All my contacts and calendar!

Michael said...

As others have said before, problems will happen, and it's how they are handled that separates the professionals from the amateurs. You guys have done a great job keeping everyone informed! Reading that it is a limitation in the iPhone causing the "missing contacts syndrome" made it much easier to bear. I was definitely surprised when it happened yesterday! I had my contacts disappear briefly again this afternoon, but they came back a little while ago.

It is reassuring that everyone posting here seems to understand that this is beta software and it's a work in progress. Personally, I am willing to accept the risk and inconvenience of the occasional failure because the upside is so good!

Unknown said...

Not a single contact at January 16, 0:43 GMT.

Would be great to get a new update, since the latest one on the blog ("everyone should see normal service return soon") seems to be outdated...

Thanks a lot guys!

Unknown said...

All my contacts and calendar items returned and looked perfect for about 20 minutes... now they're gone again. 4:55 pt.

Unknown said...

As of about 4:45pm (PST) it looks like everything is back to normal... for the most part (phew!)

Just a guy said...

I just looked at my phone and my contacts have vanished.

Anonymous said...

My contacts have all dissappeared again. Maybe this is no change, but they were there all day! Anyone else having this problem?

Unknown said...

My contacts are gone as well! I do appreciate all the communication and the service is great!

Andy said...

My calendar is gone right now as well (9:04 PM CST)

jasonod said...

I lost my contacts again around mid-day when I most needed them. I had a number of calls to make and didn't have the phone numbers. I will try not to b!tch b/c this is a great and (still) free service. But it is frustrating. I still don't have them back :-(

Greahm said...

Unfortunately, Nuevasync is struggling with some problems the past day or so. However, as you all have commented, it is a great (free) service. As I think Tim suggested earlier throw them a few bucks via PayPal (info@nuevasync.com) to help them out and show them that you appreciate the normally great service.

Anonymous said...

Still no luck with contacts. I'm syncing with Windows Mobile 6.1. I've tried everything I can think of.

It will hit the server, pull down two contacts and act like that's it.

:(

I have all my contacts backed up to a vcard. Does anyone know an alternate method for loading contacts to a Treo800W using a vcard? (I don't use outlook so I can't sync with that).

Once this is solved, I'll definitely be throwing some cash via the paypal account.

Anonymous said...

My calendar is fine now.
My contacts are all recovered, but for some reason, when i start contacts app (on iphone), never stop syncronizing...

samhnadubhadh said...

Oh my god working fine for me the first time. The service is the godsend I've been waiting for

Unknown said...

My contacts have been gone for over 2 days. Any updates on status?

PGMAW said...

I've lost my contacts - not sure when this happened, but just noticed it today after also noticing the colour-coded multi-calendar implementation. Is there anything I can do to get my contacts re-synced?

A big Thank You for this incredible service. It has been working flawlessly for me for months.

Greahm said...

If you are still missing or not syncing see previous post by Antoine:
Tap "Settings"
Tap "Mail, Contacts, Calendars"
Tap the NuevaSync account, for example "person@gmail.com"
Tap the off button for the service you wish to resync, for example "Calendar."
Wait for a moment as the device will say "Turning off
Tap the on button for the service to turn it back on.
You're done!