Tuesday, September 23, 2008

Fixing Duplicates

Though not common, over time there have been users who have reported entries duplicated either on their device or at Google/Plaxo.  As we've made continued improvements to squash duplicates, we've kept in touch with those users.  But, we want to broaden access to that information and make sure all our users know first, where duplicates can come from, and second, how to deal with them.

By far the most common cause actually isn't a NuevaSync problem at all, but a bug in the Apple synchronization client.  The precise details are rather convoluted, but the quick version is that if an error ever happened when syncing, it was possible for the device to get confused and start adding entries it already had.

In response to that, we’ve written and deployed code that actively intercepts duplicates before they get committed.  It isn't possible to intercept 100% of duplicates this way, but the results so far have been excellent.  And since the code is in our server, it applies equally to devices of all types and versions: iPod, iPhone, Windows Mobile, Nokia, etc.

Apple apparently noticed the problem as well, and another major step came with the release of the 2.1 firmware.  When we began testing the 2.1 beta releases a few weeks ago, we noticed an improvement in the devices' behavior; they did not create duplicates in the same circumstances as the 2.0 devices.  This is a great enhancement for Apple users, and though we are still a little reserved about 2.1 (cf. some of our earlier posts), we are very pleased with this particular fix.

While those changes (and several other, smaller ones we've made) work to prevent duplicates, what do you do if you already have them?  To answer that we have created some experimental tools to "dedupe" your account.  The tools for Google Contacts and Calendar have been available for some time, but today marks the release for Plaxo.  All these tools are specifically targeted at removing duplicates which might have been created through us.  If your service offers its own tool, you may want to use that instead.  Plaxo, for example, offers an advanced duplicate merging and removal tool which is well worth a look.  Our tools are experimental and by their nature designed to delete entries.  Make a backup first and exercise caution when using them!

They can be accessed at:

Detection and removal can take a few minutes, so be patient when it runs.  If a lot of entries are removed, it is probably a good idea to resync your device afterward (cf. https://www.nuevasync.com/PublicSite/user/troubleshooting.htm), but that is entirely up to you.

By raw numbers, duplicates have never been common, but nevertheless they have been one of--and perhaps the--most serious of issues to us. With these new measures, we’re hoping most users will never have a problem.  If you do, e-mail us at support@nuevasync.com and we'll do everything we can to get it fixed and you back syncing.

As always, many thanks to our users!


Jason said...

I've been having loads of duplicates but I'm pretty sure it's not Nuevasync's fault. Along with Nuevasync I'm using Lotus EasySync to sync my work Lotus Notes calendar and I'm sure it's the culprit.

Anyhoo, the deduping tool saved me lots of time. Thanks a bunch for letting us use it!

CWB said...

So is the freezing issue resolved? Id really like to use the service again..

Darrell said...

+1 on cwb's comment--I'd love to start using the service and have my biggest problem be duplicate entries. The freezing issue is so bad on my device I can't even consider using this service until it is fixed. I'd be happy to pay something (i.e. purchase a non-beta version) if necessary. If nothing further will be done, let us know so we can look elsewhere for solutions.

CWB said...

I totally agree. I would gladly pay for the service when the freezing issue gets resolved.

David Boreham said...

Folks, the freezing issue is not yet completely resolved, however we've made changes to the service that have reduced the probability it will happen. We have tens of thousands of 2.1 firmware devices using the service but only a few users reporting freezing now. So it could be worthwhile giving it a try again. We'll update the blog when we have a complete freezing resolution (which may ultimately depend on Apple releasing an update).

Paddy said...

Duplicates and Freezing, resetting the phone an losing all previous phone settings prompt me to ask if anyone knows of an alternative service or software, free or paid that is known to be reliable.

kevin said...

love you guys for fixing that

Hendrikn said...
This comment has been removed by the author.
Hendrikn said...

I just wanted to say that my Nokia Symbian S60 device with Nokia Mail for Exchange is working brilliantly!

Since the server-fallout a few weeks ago, i had no troubles whatsoever!

vestini said...

Last week my iPhone calendar froze up, two days after I had upgraded to Apple firmware 2.1.

So I performed David's suggested solution: change password on NuevaSync server and re-enter account on iPhone. Bingo! Everything has been working perfectly since.

That includes adding and deleting events on my iPhone, on the Google Calendar, or on the Sunbird calendar on my PC. Any change to one of the calendars syncs perfectly with the others within a minute or two. It's GREAT!

ThomThom said...

I use Zindus to sync my contacts from Thunderbird w/ my Google Apps for Domains email acct and it seems to work pretty good.

To deal with duplicates (since Google Contacts freaks out on dups) - I use this: https://addons.mozilla.org/en-US/thunderbird/addon/2505

To get my Google Apps contacts into my iPhone, I l o v e Nuevasync so far. I am also testing out myFUNAMBOL ...

As far as duplicates - the TB de-duplicator add-on that I use goes deep looking for duplicate email addresses, full names and phone numbers - not just exact dups - AND - rather than just deleting an exact dup, they give you an interface by which to compare a dup pair, select the one you want to keep, also allowing you to cut and paste info from one contact to the other before you eliminate the dup.

If you decide to keep both contacts of the suspected dup pair, you just skip it and go to the next one. One of the smartest approaches to de-dupping contacts I've ever seen.

Bartje in India! said...

I still experience the freezing. Few days ago, I re-synced my phone, but now after a few days it again freezes. Now because I changed a date of a appointment.

Anonymous said...

I'm not getting freezing, but slow load and scrolling for contacts.
Google searching shows rumours this is from corrupted/duplicated contacts, so this dedupe tool sounded like it was worth a spin.

But it told me this when I tried it:
"Duplicate Remover - Google
Unable to check for duplicates! Duplicate removal might not be supported for Google yet."

Is the dedupe tool supposed to work for users syncing to/from google?

Anonymous said...

Just removed account from iphone, changed p/w here on nuevasync, and then re-added account. Now showing tons of dupes on iphone only. GContacts seem fine. Hmmm.

Jim Clark said...

This utility doesn't work. All sorts of errors occur when it's run.

Leena said...

I removed NuevaSync account from my E71. Got fed up with duplicates on my Google Calendar. It didn't just duplicate the events, there were litterally n occurrences of the same event on my Google Cal. Very annoying!