How To Migrate A Massive WordPress Site

A client of mine called me and told me their WordPress website was growing too big for the server it was on, even though it was a VPS, that the WordPress site was going offline periodically.  They asked me if I could help them migrate this gigantic site.

When I refer to gigantic and WordPress in the same sentence, to me that simply means that instead of megabytes of data in the database they are dealing with gigabytes of data in the database.  And instead of thousands of files, there are let’s say hundreds of thousands of files.

Small Biz Now Has Enterprise Level Issues

Luckily I have dealt with both the small and large servers over the years.  The real question has always been, can WordPress handle this high volume of DB and File issues.  Well, it can, but the issues have to be thought out and a problem like this turns from dealing with minor WordPress hick-ups to corporate-level Unix issues.

Why They Had To Migrate

Their server was just not able to handle what they were doing in terms of server speed, hard drive space, etc.  And this was causing the website to be unmanageable.  So, it was time to migrate. However, this was not your run of the mill WordPress migration.  It had all kinds of potential problems associated with it, like thousands of uploaded files, hundreds of thousands of records in some tables.

Why My Normal Tools Won’t Work

For instance, the database was too big to download using PHPMyAdmin.  Not that this is a big deal, it’s just if you have been a WordPress person for the last 5 to 10 years like me, you may not have all the tools in the tool chest to deal with this. I typically use a desktop ftp program to do this stuff, but not for this size. Using local ftp clients means downloading to your Mac or PC. It is not recommended at this level. For instance, if you are not on a fast enough download client, you are screwed…

The Tools

The reason I am writing this blog entry is so I can grab all those tools again in the following order in order to do this kind of migration again.

Here Are The Steps I took:

  1. Sized out the proper new server for the client. (My preference is StormOnDemand by Liquid Web, WHM/Cpanel cloud)
  2. Had the client buy the new server and handed over to me the login credentials.
  3. Received all the login credentials I need from the original server.
  4. Created a new website in WHM/Cpanel, generating a new IP.
  5. Created a temporary domain name for the new site and pointed the DNS to the IP.
  6. Created a Database Name, Database User in CPanel with the same name as the old DB Name and User.
  7. From the Shell command line (I use putty.exe) I used the tar command to zip up all the files beneath public_html.
  8. From the Shell command line I ran a mysqldump of the old database, creating a command line .sql file.
  9. From the Shell command line I ran FTP as new account name (not root) on new server and Put the tar file (like files.tar.gz) onto the new server.  (it was in the multigigs)
  10. From the Shell command line I ran FTP and Put the .sql file onto the new server. (it was over a gig)
  11. On the new server from the shell I unzipped the tar file.  Remember to position this right above where you want it to land.
  12. On the new server I switched over to root user (su root) to be able to import mysql.
  13. On the new server from the shell I ran a mysql import command which updated the database.
  14. Finally, while in test mode, there were two lines wp_options which need to point to the right domain.
  15. Make sure all the previous plugins are activated.
  16. We tested the new site with the temp domain name.
  17. Once we were ready, we switch the old IP at the DNS record to the new IP.
  18. Final thing is to rename the site name from the temp domain name to the actual domain name.
  19. Everything went relative smoothly and the site migrated.

That is all the actions I took, now here are the commands I used specifically:

he tar command to zip up the files to a .gz from right above the public_html directory:

The tar command to unzip the files from a .gz right above the new public_html directory:

The FTP commands I used to connect from Linux server to Linux Server:

The mysql command I used to dump the entire mysql database at the command line on the old server:

The mysql command I used import the entire mysql database at the command line on the new server:

Conclusion

Hopefully if you are reading this, you will use this to reduce the amount of time I took looking around the web for the right commands to do this. You may have to look up those other websites, because each of these commands have dozens of options. For instance, if you need to move more than one file at a time, using FTP, you would have to use mput and not put. So, keep up the good work. I hopefully will use this guide for myself next time.

Enterprise WordPress – Things You Have to Change – Robots.txt, wp-config.php, etc.

After all the hullabaloo in getting WordPress ready and implemented for the enterprise, with a DEV, QA and PROD environment, there are a lot of small gotchas that are still sitting around. For those of you trying out WordPress for the Enterprise level, these are the little gotcha’s to worry about, and Robots.txt is one of them:

  • Robots.txt
    Robots.txt needs to be different in each level of DEV, QA and PROD. If you are using SVN this can be a challenge. If you don’t know what SVN or CVS are, then don’t worry about this one. Basically as you copy the site from DEV to QA to PROD, you end up with the same robots.txt. This can’t happen from a technical perspective, since you really want robots.txt to block the search engine bots on DEV and QA, and not on PROD.
  • wp-config.php
    wp-config.php, the settings file for WordPress, needs to be modified if you are spanning the code across DEV, QA and PROD. Simple answer for this, use the old PHP switch statement and change the site based on server name with case statements. The variable to use for this is $_SERVER[‘SERVER_NAME’]. A good example is below:

    // ** MySQL settings – You can get this info from your web host ** //
    switch( $_SERVER[‘SERVER_NAME’] )
    {
    case “dev.strategicpoints.com”:
    define(‘DB_NAME’, ‘dbname’);
    define(‘DB_USER’, ‘dbuser’);
    define(‘DB_PASSWORD’, ‘dbpassword’);
    define(‘DB_HOST’, ‘localhost or SERVER NAME’);
    break;

    case “qa.strategicpoints.com”:
    define(‘DB_NAME’, ‘dbname’);
    define(‘DB_USER’, ‘dbuser’);
    define(‘DB_PASSWORD’, ‘dbpassword’);
    define(‘DB_HOST’, ‘localhost or SERVER NAME’);
    break;

    case “www.strategicpoints.com”:
    case “strategicpoints.com”:
    define(‘DB_NAME’, ‘dbname’);
    define(‘DB_USER’, ‘dbuser’);
    define(‘DB_PASSWORD’, ‘dbpassword’);
    define(‘DB_HOST’, ‘localhost or SERVER NAME’);
    break;
    }

  • Server Name in Mysql
    It is important to make 2 changes in each DB when migrating MYSQL between DEV, QA and PROD. These are in the wp_options table, and if you have been through this drill before they are pretty explanatory. The option names you have to change are “siteurl” and “home” to the full path of the site such as http://www.strategicpoints.com.

To solve some of these incompatibilities between the DEV, QA and PROD there are plugins and other things that need to be put in place in order to keep these versions in sync. This is a major challenge. As I solve each one of these issue or more issues, I am going to update this blog post.

WordPress In The Enterprise – Article 4, Upgrading WordPress!

This article is the 4th of a series of articles about WordPress In The Enterprise. This article is a work in progress, I am going to add to it as I go about this process, since I am tackling an issue that right now has no easy answer in the enterprise, and that is how to upgrade enterprise driven WordPress implementations. Why upgrade? Well, every chicken little on the WordPress blog/web has said if you don’t you may have a security issue. This is the kind of corporate implementation that includes a development server, a QA server and a load balanced production server.

If you want to start at the beginning and read them in a sequence use the following links below:

WordPress For The Enterprise – Article 1 – Setting The Mood
WordPress For The Enterprise – Article 2 – Issues And Plugins
WordPress For The Enterprise – Article 3 – Implementation Problems

Ok, now that you have read up on the first series of articles, let’s get to the heart of the problem with upgrading from an enterprise perspective. First off, the real issue is that WordPress is not an enterprise product, just yet. This does not mean it won’t be in the future, so like a WordPress plugin that does not yet exist, we have an issue that just needs to be solved. The big issue is testing a new WordPress Version on a development server and then somehow easily upgrading the QA and production servers.

Upgrade Method
If you use the WordPress upgrade method right out of the box, you are somehow violating a holy grail of upgrading from development to QA to production. Or are you? The current WordPress recommended method of upgrading is to load the new set of version files like from 2.8 to 2.84 (not overwriting wp-content’s themes & plugins directories, or anything else unique) and just letting the new version, like 2.8.4 do its magic. By magic, I mean it sometimes, but not always makes modifications to MySQL table structures. If that were not an issue, this would kind of be a mute point, because then we would just copy the development site over and be done with it. If you need the most recent version of WordPress, you can get it by clicking here.

Testing Plugins
Let’s take a little diversion here. Obviously one of the most important issue in the enterprise solution that I am trying to figure out, is testing plugins. Apparently you have to update certain plugins with the most recent version of the plugin to make sure it is compatible with an upgrade. This is a bit taxing on plugin makers who have to keep up with major revisions. The task at hand is to test plugins like seo-all-in-one-pack, google analytics, redirection, and others for compatibilities. If a very important plugin can not be upgraded and remains incompatible with the new Version, well, let’s just say you will be going to the backup.

The Backup
An important step in the upgrade process is the backup. I am not talking about the system backup you get from your server folks when the server crashes dead as a doornail, I am talking about backing up MySQL fully before you do the upgrade. Not explicitly required for the non-enterprise version, but for enterprise it is a must. Simply back it up to a .sql file and keep it handy, even in the development site before an upgrade. Good practice and that is what enterprise WordPress is all about.

The Upgrade Process
Once you have gone through the standard copy of the files over the development, you need to surf wp-admin area of WordPress. Immediately it will come up and there is a button that says UPGRADE to version x.x.x. You click it and hopefully it all goes well. Simple enough and that dev version is upgraded.

The QA Process
Tedious as it is, you need to go through and not just test out the content, permalinks and categories, but you need to take a look carefully at the plugins from both a site perspective on the outside and in the admin and tools areas as well to make sure they are functional on the inside. Once this is complete and tested, it is time to push it.

Pushing It
By pushing it, I mean pushing files up to the QA server, along with a mysql copy. This may be more of an SA task, but up till now it has been in the hands of the person managing the WordPress version. If you had left it purely up to SAs, it probably would not have happened right. All that has to happen at this point in the process is complete duplication of 2 things: files and Mysql DB. The duplication on the QA and production server will complete the process…hopefully.

Until somebody writes the ultimate publishing plugin for WordPress Enterprise, this will be a bit of black hole. Trust me somebody out there is going to fill this void. We have been hard at work on our own internal WordPress publishing plugin, but I am sure there will be open source versions sometime soon on the web.

Dan

Next Article WordPress In The Enterprise – Article 5 – Post Mortem