Friday, November 23, 2007

Automated gmail backup via IMAP

This is how I setup automated gmail backup using IMAP via mbsync. Parts are MacOS X specific.
  1. Enable IMAP in your gmail account.
  2. Install mbsync, if using MacPorts do: sudo port install isync
  3. Create a new directory ~/Backup/gmail
  4. Save this certificate as ~/Backup/gmail/gmail.pem. This is the gmail IMAP SSL certificate, retrieved via
    openssl s_client -connect imap.gmail.com:993 -showcerts
    -----BEGIN CERTIFICATE-----
    MIIDYzCCAsygAwIBAgIQcdBJTwL0s64EVDDexAG1jTANBgkqhkiG9w0BAQUFADCB
    zjELMAkGA1UEBhMCWkExFTATBgNVBAgTDFdlc3Rlcm4gQ2FwZTESMBAGA1UEBxMJ
    Q2FwZSBUb3duMR0wGwYDVQQKExRUaGF3dGUgQ29uc3VsdGluZyBjYzEoMCYGA1UE
    CxMfQ2VydGlmaWNhdGlvbiBTZXJ2aWNlcyBEaXZpc2lvbjEhMB8GA1UEAxMYVGhh
    d3RlIFByZW1pdW0gU2VydmVyIENBMSgwJgYJKoZIhvcNAQkBFhlwcmVtaXVtLXNl
    cnZlckB0aGF3dGUuY29tMB4XDTA3MDUxMTE1NTUzMFoXDTA4MDUxMDE1NTUzMFow
    aDELMAkGA1UEBhMCVVMxEzARBgNVBAgTCkNhbGlmb3JuaWExFjAUBgNVBAcTDU1v
    dW50YWluIFZpZXcxEzARBgNVBAoTCkdvb2dsZSBJbmMxFzAVBgNVBAMTDmltYXAu
    Z21haWwuY29tMIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDp8NJaYAUMfsA4
    uW1y5wXh6sE31Uh/s0dLgeTu4NbzA36Ru3bmpB4zxCNUgnNT73OfhhtTitx8BPoB
    zdY4Tgwc1asVXSw0h2iNoj6/KIiCv4r5FFqWMQxdHZh3n6/VJnNzCPnD62fJy9D8
    j9jIfU6NGM91+zgsEexU7JuHB+y2jwIDAQABo4GmMIGjMB0GA1UdJQQWMBQGCCsG
    AQUFBwMBBggrBgEFBQcDAjBABgNVHR8EOTA3MDWgM6Axhi9odHRwOi8vY3JsLnRo
    YXd0ZS5jb20vVGhhd3RlUHJlbWl1bVNlcnZlckNBLmNybDAyBggrBgEFBQcBAQQm
    MCQwIgYIKwYBBQUHMAGGFmh0dHA6Ly9vY3NwLnRoYXd0ZS5jb20wDAYDVR0TAQH/
    BAIwADANBgkqhkiG9w0BAQUFAAOBgQBIuR0Dr4wURb1CjxMVjWA9/lPZl2f2Kr++
    naPcrIw+gJMLwU88OCfs7XqOHQ/n9dRnQ+mXcrmJKHVQAh0d038JKOaglVBn6LdX
    nJovtY8DyeYPXMGHdIwxPj7H583HQRGqkDF1usr29X3JZxcpPi3ICk+lRYoSHBvH
    /MXIPo3WJA==
    -----END CERTIFICATE-----
    
  5. Save this certificate as ~/Backup/gmail/thawte.pem. This is the CA for the gmail certificate, downloaded from http://www.thawte.com/roots/.
    -----BEGIN CERTIFICATE-----
    MIIDJzCCApCgAwIBAgIBATANBgkqhkiG9w0BAQQFADCBzjELMAkGA1UEBhMCWkEx
    FTATBgNVBAgTDFdlc3Rlcm4gQ2FwZTESMBAGA1UEBxMJQ2FwZSBUb3duMR0wGwYD
    VQQKExRUaGF3dGUgQ29uc3VsdGluZyBjYzEoMCYGA1UECxMfQ2VydGlmaWNhdGlv
    biBTZXJ2aWNlcyBEaXZpc2lvbjEhMB8GA1UEAxMYVGhhd3RlIFByZW1pdW0gU2Vy
    dmVyIENBMSgwJgYJKoZIhvcNAQkBFhlwcmVtaXVtLXNlcnZlckB0aGF3dGUuY29t
    MB4XDTk2MDgwMTAwMDAwMFoXDTIwMTIzMTIzNTk1OVowgc4xCzAJBgNVBAYTAlpB
    MRUwEwYDVQQIEwxXZXN0ZXJuIENhcGUxEjAQBgNVBAcTCUNhcGUgVG93bjEdMBsG
    A1UEChMUVGhhd3RlIENvbnN1bHRpbmcgY2MxKDAmBgNVBAsTH0NlcnRpZmljYXRp
    b24gU2VydmljZXMgRGl2aXNpb24xITAfBgNVBAMTGFRoYXd0ZSBQcmVtaXVtIFNl
    cnZlciBDQTEoMCYGCSqGSIb3DQEJARYZcHJlbWl1bS1zZXJ2ZXJAdGhhd3RlLmNv
    bTCBnzANBgkqhkiG9w0BAQEFAAOBjQAwgYkCgYEA0jY2aovXwlue2oFBYo847kkE
    VdbQ7xwblRZH7xhINTpS9CtqBo87L+pW46+GjZ4X9560ZXUCTe/LCaIhUdib0GfQ
    ug2SBhRz1JPLlyoAnFxODLz6FVL88kRu2hFKbgifLy3j+ao6hnO2RlNYyIkFvYMR
    uHM/qgeN9EJN50CdHDcCAwEAAaMTMBEwDwYDVR0TAQH/BAUwAwEB/zANBgkqhkiG
    9w0BAQQFAAOBgQAmSCwWwlj66BZ0DKqqX1Q/8tfJeGBeXm43YyJ3Nn6yF8Q0ufUI
    hfzJATj/Tb7yFkJD57taRvvBxhEf8UqwKEbJw8RCfbz6q1lu1bdRiBHjpIUZa4JM
    pAwSremkrj/xw0llmozFyD4lt5SZu5IycQfwhl7tUCemDaYj+bvLpgcUQg==
    -----END CERTIFICATE-----
    
  6. Save this config file in ~/Backup/gmail/mbsync-config. Replace LOGIN with your gmail username. This sets up a one-way channel from your gmail "All Mail" store to a local maildir. So the local maildir will be kept in sync with all gmail mail.
    MaildirStore local
    Path ~/Backup/gmail/maildir/
    
    IMAPStore gmail
    Host imap.gmail.com
    User LOGIN@gmail.com
    UseIMAPS yes
    CertificateFile ~/Backup/gmail/gmail.pem
    CertificateFile ~/Backup/gmail/thawte.pem
    
    Channel backup
    Master ":gmail:[Gmail]/All Mail"
    Slave :local:gmail
    Create Slave
    Expunge Slave
    Sync Pull
    
  7. Save your gmail password in Keychain. The simplest way to do this is to login to your gmail account and have Safari remember the password.
  8. Give security permission to access this password. Run this command (replace LOGIN with your gmail username) and when prompted click "Always Allow" to allow security access.
    security find-internet-password -g  -a LOGIN -s www.google.com
  9. Save this script as ~/Backup/gmail/backup-gmail and make it executable chmod +x backup-gmail. Replace LOGIN with your gmail username. This script uses security to retrieve your gmail password from Keychain - to avoid storing it in plain text in the config file. mbsync uses getpass to read the password directly from the TTY, so this won't work when you run backup-gmail directly from Terminal. It will work when run via launchd (see below).
    #!/bin/bash
    security find-internet-password -g  -a LOGIN -s www.google.com 2>&1 |\
        sed -n -e '1s/password: "\(.*\)"/\1/;1p' |\
        /opt/local/bin/mbsync --config ~/Backup/gmail/mbsync-config backup 2>&1 > ~/Backup/gmail/mbsync.log
    
  10. Save this launchd plist as ~/Backup/gmail/com.rectalogic.gmail.backup.plist and then symlink it into your LaunchAgents directory
    ln -sf ~/Backup/gmail/com.rectalogic.gmail.backup.plist ~/Library/LaunchAgents
    This places the backup script under launchd control and will run it every 24 hours. You can run it manually to test via launchctl start com.rectalogic.gmail.backup. Monitor ~/Backup/gmail/mbsync.log for errors.
    <?xml version="1.0" encoding="UTF-8"?>
    <!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
    <plist version="1.0">
    <dict>
     <key>Label</key>
      <string>com.rectalogic.gmail.backup</string>
     <key>OnDemand</key>
      <true/>
     <key>Program</key>
      <string>~/Backup/gmail/backup-gmail</string>
     <key>StartInterval</key>
      <integer>86400</integer>
    </dict>
    </plist>
    

9 comments:

Brandon said...

Whoa, awesome post. I can't wait to give this a whirl later tonight.

brian doll said...

First, thanks very much for this! I've been really meaning to backup my gmail account and now it's working great!

Some small adjustments I needed to make in order to get this to work:
- create the ~/Library/LaunchAgents directory (otherwise the symlink will be to the directory name, not within it)

- in com.rectalogic.gmail.backup.plist, change ~/Backup/gmail/backup-gmail to /Users/USERNAME/Backup/gmail/backup-gmail

- create the ~/Backup/gmail/maildir directory

dexter said...

interesting .. how about google apps ?

I get this error.

Channel backup
Selecting slave gmail... 0 messages, 0 recent
Error: UIDVALIDITY of slave changed

Remko Tronçon said...

Thanks for pointing me to mbsync. I had to apply a few tweaks to mbsync to get it to synchronize my labels (recursively) as well, but it's all working great for me now.

Anonymous said...

Useful, thanks!

Regarding "Error: UIDVALIDITY of slave changed"

I got that same issue; but only on the second account I setup. First I tried it with a throw away account.

I resolved the problem. Having discovered that somebody, certainly not me, had created a ~/.mbsyncrc file I deleted that and my real account worked out.

I susupect that even if you always invoke it with a --config it still creates and reads the .mbsyncrc file.

diafygi said...

I've written a python script for getmail4 that pretty much does the same thing. It's called ImapGrab, and it can be used for any IMAP server. I also added a special option for Gmail. Enjoy!
http://daylightpirates.org/index.php/Programs/ImapGrab

Anonymous said...

Check out this tool: Gmail Keeper, you see how easy it is to backup your emails and labels automatically.

Streeter said...

I thought these instructions were quite helpful. I tweaked them a bit to get an mbsync setup working on my home Ubuntu install. I also have a patch to support recursive label traversal. The steps I went through are here.

Anonymous said...

There are five basic functions that we all need to perform in order to manage and safeguard our email- backup, restore, archive, transfer and organize the data. Beyond Inbox can help you perform all of these in a very efficient and easy way. http://www.beyondinbox.com/documentation/mail-backup--email-backup-for-disaster-and-recovery.html