Life, Football, Technology and Vespas…

Migration Status

Well into the migration now. A successful run of 25000 accounts was done last night and now I have just kicked off another 50000. Running at a rate of 5000 an hour, this should be done by the morning.

We have had some challenges leading up to the migration – as you would have with this magnitude, here are some of them and how we solved them below:

We never had enough time to wait for developers to modify the whole provisioning process. At present they write directly to an ldap directory to which qmail-ldap refers to. All of the necessary provisioning such as password changes, quota changes, forwarding, drop boxes etc, have all been channeled through LDAP. To keep this as simple as possible with the least amount of impact (for now) we implemented a sync replication process to take the changes made on the qmail-ldap master ldap server and translate those changes into zmprov commands. So far this is working well for us as an interim solution until the development team can develop a provisioning application that uses the SOAP API.

Another challenge was the time to migrate the accounts. How we overcame that was to synchronize the accounts on qmail-ldap (from the ldap server) and using the sync replication process, align the Zimbra setup to be identical in terms of accounts, quotas, passwords etc… So in effect, the two systems were running side by side with identical account data. This process took about 2 weeks or so. The second part of the process would be to start moving account data and delivering mail to the Zimbra servers and not have any downtime.

We investigated imapsync and found that it would be far too slow. It is also buggy and during our testing – left a lot to be desired. Imapsync was also the preferred method of migration advised by Zimbra’s professional services. We looked at a functionality within Zimbra called ‘External Data Sources’ and with that we found our solution.

The idea was that using qmail-ldap’s clustering capabilities and referencing http://wiki.zimbra.com/index.php?title=Split_Domain#Configuring_Zimbra_as_the_Secondary_System, we would be able to query the account in the qmail-ldap ldap servers to identify all the necessary account information. Then by adding an additional attribute – mailHost, to the qmail-ldap ldap server, with a value of  the Zimbra mailstore, the qmail-ldap server would (using qmqpd) send the mail to the zimbra mailstore. This meant that all new mail for that account would from then on be delivered to the Zimbra servers. The next thing was to get the old data from the qmail-ldap servers.

Most of our users use pop3 so we have very little issues with folder structures and such so all we needed to do was create an external data source on the zimbra account pointing back to the pop3 server on the qmail-ldap side. One problem – we only had the md5 encrypted password and had no way of getting the cleartext pass. It turns out that qmail-pop3d cannot / will not do clear text comparison if there is an {md5} in the password. Fail. We over come this by implementing courier-pop3d. This server allows plaintext comparison and therefore we could run it on a different port and point the external data source to that server on the new port using cleartext with the md5 hash as the password entry. Works like a charm!

Now if anyone was observant enough they may be wondering… how the hell can you make 5000 zmprov commands an hour on a single thread. The truth is we could run more, but are limited by our existing ldap servers replicating!  Zmprov is retardedly slow starting up, so the idea was to start it once and then run all the commands we need and then quit zmprov when finished – restarting it if it dies (which it does… a lot!)

Here is a piece of code (Thanks Mr Milne) that enable us to do such a thing:

use IPC::Run qw ( harness start run finish timer);

my ($zmprovh,$zmprovin,$zmprovout,$zmproverr,$commands,$user);
my @zmprovcmd = qw (/opt/zimbra/bin/zmprov);

$zmprovh = harness (\@zmprovcmd, \$zmprovin, \$zmprovout, \$zmproverr, (my $t = timer 5) );

** WHILE ACCOUNTS….

zmprov(do what you want);

$zmprovin .= “quit\n”;
$zmprovh->finish or warn “Finishing zmprov returned $?”;
exit;

sub zmprov {
my ($commands) = @_;
my $stamp = &ts();
$zmprovcommandsrun =  $zmprovcommandsrun + 1;
#debug(2,”Trying to run via zmprov ($zmprovcommandsrun): $commands”);
$t->start(5);
#$t->reset;
$zmprovin = $commands . “\n”;
$zmprovout = ”;
$zmproverr = ”;
#$zmprovh->pump until $zmprovout =~ /^prov> \Z/m;
eval { $zmprovh->pump;1} or warn “$@” until $t->is_expired || $zmprovout =~ /^prov> \z/mgc;
if ($t->is_expired) {
#debug(1,”zmprov timed out or died, restarting it”);
$zmprovin = “quit\n”;
$zmprovh->finish;
$zmprovh->kill_kill(grace=>5);
$t->start;
$zmprovh->pump until $t->is_expired || $zmprovout =~ /^prov> \z/mgc;
}
if ( $zmproverr ne “”) {
spilog(“ERROR zmprov output: $zmproverr”);
}
}

Uber quick… If anyone is interested in the full script, I will try to get it put on the zimbra wiki. The script makes all the necessary changes to qmail-ldap ldap server and adds the external data source to pop account every few hours. Bingo! A couple of hours after running the script on an account, the mail is now being delivered to the Zimbra servers and the mail on the existing qmail-ldap servers has been popped and delivered into the inbox (FolderId 2 btw).

So – I will update tomorrow and see where we are with the 50,000 accounts.

Advertisements

4 responses

  1. hoang

    Hi! the topic is very enjoyable. Can You detail and how to migration qmail-ldap with zimbra opensource! Ah ha! thank You!

    I’m from VietNamese

    May 26, 2010 at 9:47 am

  2. Thanks for the info.

    we are migrating a similar number of users from mirapoint

    so you have been very helpful

    Thanks again

    November 23, 2011 at 2:06 pm

    • bonoboslr

      I used to manage a mirapoint system to! Let me know how you get on or if you need any advice.

      November 23, 2011 at 2:22 pm

  3. Thank you for your information share make me work easier 🙂 , our account very little if compare with you scale. I use courier-IMAP if you have an issue to suggest it’s welcome.

    thank you
    #me

    August 22, 2012 at 3:34 am

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s