How To Migrate A Massive WordPress Site

A client of mine called me and told me their WordPress website was growing too big for the server it was on, even though it was a VPS, that the WordPress site was going offline periodically.  They asked me if I could help them migrate this gigantic site.

When I refer to gigantic and WordPress in the same sentence, to me that simply means that instead of megabytes of data in the database they are dealing with gigabytes of data in the database.  And instead of thousands of files, there are let’s say hundreds of thousands of files.

Small Biz Now Has Enterprise Level Issues

Luckily I have dealt with both the small and large servers over the years.  The real question has always been, can WordPress handle this high volume of DB and File issues.  Well, it can, but the issues have to be thought out and a problem like this turns from dealing with minor WordPress hick-ups to corporate-level Unix issues.

Why They Had To Migrate

Their server was just not able to handle what they were doing in terms of server speed, hard drive space, etc.  And this was causing the website to be unmanageable.  So, it was time to migrate. However, this was not your run of the mill WordPress migration.  It had all kinds of potential problems associated with it, like thousands of uploaded files, hundreds of thousands of records in some tables.

Why My Normal Tools Won’t Work

For instance, the database was too big to download using PHPMyAdmin.  Not that this is a big deal, it’s just if you have been a WordPress person for the last 5 to 10 years like me, you may not have all the tools in the tool chest to deal with this. I typically use a desktop ftp program to do this stuff, but not for this size. Using local ftp clients means downloading to your Mac or PC. It is not recommended at this level. For instance, if you are not on a fast enough download client, you are screwed…

The Tools

The reason I am writing this blog entry is so I can grab all those tools again in the following order in order to do this kind of migration again.

Here Are The Steps I took:

  1. Sized out the proper new server for the client. (My preference is StormOnDemand by Liquid Web, WHM/Cpanel cloud)
  2. Had the client buy the new server and handed over to me the login credentials.
  3. Received all the login credentials I need from the original server.
  4. Created a new website in WHM/Cpanel, generating a new IP.
  5. Created a temporary domain name for the new site and pointed the DNS to the IP.
  6. Created a Database Name, Database User in CPanel with the same name as the old DB Name and User.
  7. From the Shell command line (I use putty.exe) I used the tar command to zip up all the files beneath public_html.
  8. From the Shell command line I ran a mysqldump of the old database, creating a command line .sql file.
  9. From the Shell command line I ran FTP as new account name (not root) on new server and Put the tar file (like files.tar.gz) onto the new server.  (it was in the multigigs)
  10. From the Shell command line I ran FTP and Put the .sql file onto the new server. (it was over a gig)
  11. On the new server from the shell I unzipped the tar file.  Remember to position this right above where you want it to land.
  12. On the new server I switched over to root user (su root) to be able to import mysql.
  13. On the new server from the shell I ran a mysql import command which updated the database.
  14. Finally, while in test mode, there were two lines wp_options which need to point to the right domain.
  15. Make sure all the previous plugins are activated.
  16. We tested the new site with the temp domain name.
  17. Once we were ready, we switch the old IP at the DNS record to the new IP.
  18. Final thing is to rename the site name from the temp domain name to the actual domain name.
  19. Everything went relative smoothly and the site migrated.

That is all the actions I took, now here are the commands I used specifically:

he tar command to zip up the files to a .gz from right above the public_html directory:

The tar command to unzip the files from a .gz right above the new public_html directory:

The FTP commands I used to connect from Linux server to Linux Server:

The mysql command I used to dump the entire mysql database at the command line on the old server:

The mysql command I used import the entire mysql database at the command line on the new server:

Conclusion

Hopefully if you are reading this, you will use this to reduce the amount of time I took looking around the web for the right commands to do this. You may have to look up those other websites, because each of these commands have dozens of options. For instance, if you need to move more than one file at a time, using FTP, you would have to use mput and not put. So, keep up the good work. I hopefully will use this guide for myself next time.

WordPress In The Enterprise – Article 5, Post Mortem

This article is article 5 in a series of articles about implementing WordPress in the Enterprise. This article is about some of the aftermath of our big conversion of enterprise sites to WordPress. In some ways it never ends, the tweaking and upgrading and improving of WordPress sites. This is both good and bad. Good in that we don’t have to keep such large web development staffs around and we are keeping up to date with current SEO, security and other WordPress driven stuff. Bad, in that it never ends…

Here are the original articles if you want to start at the beginning

WordPress For The Enterprise – Article 1 – Setting The Mood
WordPress For The Enterprise – Article 2 – Issues And Plugins
WordPress For The Enterprise – Article 3 – Implementation Problems
WordPress For The Enterprise – Article 4 – Upgrading

Some of what I am going to discuss in this article has to do with small details which occurred after the implementation and continue to happen. These small details are important because they are some of those gotchas after a big migration. They are just listed here in random order:

301 Redirects On Additional Domains

Let’s say you own about 10 other domains that you have pointed to the same web site as your base site. This would be like having http://www.strategicpoints.com and then having http://www.internetanalysistools.com and then pointing internetanalysistools.com to strategicpoints.com. It just happens that on additional domains, WordPress should take care of this for you. Just set the DNS to point to the same server and make sure the apache conf (in Linux is set) and you are good to go. The second domain should hit the site and switch over to the real site. This also works the same way for www.strategicpoints.com and strategicpoints.com. Set it to one in wordpress admin and it will always redirect to the other. This is important for the search engines to have one domain for the site to avoid duplication and getting crawled incorrectly on the secondary domains.

Nofollow Meta Tags Left On (kind of like your headlights left on)

Just happened that prior to our migration, we set the dev and QA sites to not be indexed by the search engines. This setting is under Settings/Privacy. It is called Blog Visibility, and the exact wording is “I would like to block search engines, but allow normal visitors”. We had these Dev and QA sites set this way, so that we would not get these sites indexed. However, somehow this template theme was transferred to the components of our enterprise production system which are not WordPress sites. This means that we copied the template as is, and it had the follow Meta tag because of this setting:

<meta name="robots" content="noindex,nofollow" />

In particular we ended up leaving this in production on these non WordPress sites. It was an oversight, and one we immediately fixed. But this just says that even after you do your QA and job and the best you can, there is still a potential problem that you are not aware of in these environments. So you have to be vigilant and look things over periodically.

I am going to keep adding to this article as issues slowly bubble to the surface on the enterprise wordpress implementations we have been through.

Fixing Broken Links After Migrating To WordPress

This article is related to the first three posts about Implementing WordPress in the Enterprise Environment. One of the lonely tasks following the migration of our website from HTML and Cold Fusion based files to PHP and WordPress was to fix several thousand incoming links from Google and other search engines. Without these Redirection fixes, the migration would have basically crushed our traffic.

To solve this problem, there are several places to look, including Google and doing a site:YOURWEBSITENAME.com search and clicking on every link to using Google Analytics. The Google Analytics method is the best, since it automatically tells you what needs to be fixed. This does not mean you should ignore the google search engine and looking at what are big incoming links in advance. Remember once you find a broken link, and you are using the WordPress Redirection Plugin, I highly recommended in a previous blog entry (click here to learn more info), you need to enter the entry into Redirection and fix it. If you are just learning that this did not require an .htaccess entry in Linux or htapi entry in windows, you are learning about a whole new world of improvement…

So back to Google Analytics. First of all you need to have a 404 page, which is where all the pages end up when they find no entry in WordPress, including the redirection plugin. This typically will be a 404.php page in the theme directory you are using. If you are sending people to your site map page, that is fine as well.

There are several ways to get this info in google analytics. I recommend using page title. For instance, on our site, the page title of our 404 page is called “Page Not Found”. Go into Google Analytics and click on “Content” and then on “Content by Title”. If you have just migrated, and the first day has passed, just set the date to the current day, to remove anything that is older than today, and find the “Page Not Found” (or other) title. If you click on this title name itself, another screen will open and reveal all the URLs that arrive at that page. This is your list of incoming broken links, sorted by the most important links at the top of the list.

Now is the detailed nitty gritty work. You need to re-enter each of these links into a url to test them and make sure they are broken. Once you are have found one that is being clicked on from a website out yonger, take it, copy it into the redirection plugin. You do this by Adding a new Redirection, then copying the link into the Source URL. Remember to not use your full path, just the /filedirectoryetc/ path, and then go over to another browser and find the end URL you want to send the visitor to and then copy and paste this path into Target URL and click Add Redirection. The default is a Permanent 301 redirect.

There are a lot of other options in the redirection program like using Regular Expressions. These work fine as well, but make sure you read the instructions carefully, since some redirections are impacted by other redirections…sequentially.

Let me know if you have any comments are questions about this process, because almost all sites come to point of migration, and this blog entry is really about the aftermath of a big migration. Our last WordPress migration did go through, but it had lots of small items that we had to fix along the way. I am going to add another blog entry covering these additional problems we ran into, creating a load balanced WordPress Enterprise solution!