Pages

Friday, March 17, 2017

*Monitoring Google Contacts for changes - are you losing contacts?


NOTE: This page has moved to https://datamakes.com/2017/03/17/monitoring-google-contacts-for-changes-are-you-losing-contacts/

Have you ever thought you are losing contacts stored in Google?  That wonderful moment when you are trying to dial your phone - and the person you want is not in your address book.  Then you think about it and realize that you definitely have had them in there before..........how frustrating.

I believe that your contact list is probably one of the most important personal information you keep in your phone.  Contacts last over time and you don't always notice when they disappear until it is too late.

To help combat this problem, I have written a set of bash scripts which run on Linux to help you recognize a problem and provide you a way to correct it.

This solution is specifically for backing up Google Contacts - but the concepts would work for any contact storage engine where you can get vcards.

To make this work, you will need to get vdirsyncer installed properly.  There are very complete instructions at the vdirsyncer web-site at https://vdirsyncer.pimutils.org/en/stable/installation.html  

Please pay specific attention to the "Google" section in https://vdirsyncer.pimutils.org/en/stable/config.html  Specifically you will need to create an API key (client_id and client_secret) and install an additional python module to access Google.  All of this is fully documented so you should be able to follow those instructions.

Once you have vdirsyncer installed, let's speed things up and jump right to the configuration.  Here is a my vdirsyncer config file:
[general]
status_path = "<path>/status"

[storage googlecontacts]
type = "google_contacts"
token_file = "<path>/google.token"
client_id = "<client_id from the Google API console>"
client_secret = "<client_secret from the Google API console>"
read_only = "true"

[storage vcf]
type = "filesystem"
path = "<path>/contacts"
fileext = ".vcf"

[pair google]
a = "googlecontacts"
b = "vcf"
collections = ["from a"]
conflict_resolution = "a wins"
Now for a discussion of the important points in this config file:
  • You must substitute everything in < > with proper values
  • vdirsyncer seems to really require all of the quotation marks (") above - leave them in
  • The file <path>/google.token provides access to your Google account via an OAuth token - protect this file
  • The read_only parameter in the googlecontacts storage configuration means that no changes from your local PC will ever appear on Google.  NOTE: There should never be changes on your local system, unless something goes horribly wrong.....this is just a safety measure
  • The "a wins" also specifies that Google Contacts is the authoritative source of information
Once you create this configuration file, you will need to perform a one-time only step of running
vdirsyncer -c <config file> discover
This step will either automatically start a browser for you to authenticate with Google - or if a browser can not be started - a URL will be provided that you must browse to.  Once you authenticate, you will be given a long complex string which you will paste into your vdirsyncer window.  This is used to generate your OAuth token, which is stored in the token file specified above.

At this point in time, you probably want to run
vdirsyncer -c <config file> sync
just to make sure everything is working.  If everything goes well, you will end up with a bunch of vcard (.vcf) files in your <path>/contacts directory.

Now that everything is working, let's automate this.  The script below will notify you when:
  • A contact is deleted - the deleted contact is stored for your review
  • A contact is added
  • A contact is changed - both then old and new version are stored and dated for your review
 #!/bin/bash

BASEDIR=<path>
TODAYDIR=$BASEDIR/contacts/default
YESTERDAYDIR=$BASEDIR/yesterday
CONFIG=$BASEDIR/<config file>.conf
CHANGEDIR=$BASEDIR/changes
YESTERDAY=`date +%Y-%m-%d -d "yesterday"`
TODAY=`date +%Y-%m-%d`
CHANGED=0

/usr/local/bin/vdirsyncer -c $CONFIG sync | egrep -v "Syncing "

#Search for deletions
for vcf in `ls $YESTERDAYDIR/*.vcf`
do
   card=`echo $vcf | xargs -n 1 basename`
   NAME=`cat $YESTERDAYDIR/$card | egrep ^FN: | cut -f2 -d: | sed -e 's/ /_/g'`
   if [ ! -f "$TODAYDIR/$card" ]; then
      echo DELETED: $NAME \($card\)
      mv $YESTERDAYDIR/$card $CHANGEDIR/$NAME.$card.DELETED.$TODAY
      CHANGED=1
   fi
done

#Search for additions
for vcf in `ls $TODAYDIR/*.vcf`
do
   card=`echo $vcf | xargs -n 1 basename`
   if [ ! -f "$YESTERDAYDIR/$card" ]; then
      NAME=`cat $TODAYDIR/$card | egrep ^FN: | cut -f2 -d: | sed -e 's/ /_/g'`
      echo ADDED: $NAME \($card\)
      CHANGED=1
   fi
done

#Search for changes
for vcf in `ls $TODAYDIR/*.vcf`
do
   card=`echo $vcf | xargs -n 1 basename`
   if [ -f $YESTERDAYDIR/$card ]; then
      if [ `stat --printf="%s" $TODAYDIR/$card` -ne `stat --printf="%s" $YESTERDAYDIR/$card` ]; then
         NAME=`cat $YESTERDAYDIR/$card | egrep ^FN: | cut -f2 -d: | sed -e 's/ /_/g'`
         echo CHANGED: $NAME \($card\)
         cp $TODAYDIR/$card $CHANGEDIR/$NAME.$card.CHANGE.$TODAY
         cp $YESTERDAYDIR/$card $CHANGEDIR/$NAME.$card.CHANGE.$YESTERDAY
         CHANGED=1
      fi
   fi
done

# Copy all of todays entries into yesterdays directory - for comparison tomorrow
cp $TODAYDIR/*.vcf $YESTERDAYDIR

if [ "$CHANGED" == "1" ]; then
   <any code you want to specifically execute to notify you of a change - remember cron will automatically E-Mail you a log of this session, if any changes were found.  This is for any additional notification options.  For example, I use an API to send myself a text message>
fi
You should only have to change the items in red, make sure all of the directories exist, and add this script to your crontab.

Now you will automatically be notified when your contacts change.  If you see any unexpected changes, you will have all of the necessary information to restore the missing data.

Rest easy knowing that your contacts are safe.