Or: "How to cleanup the mess..."

A couple of days ago a friend that is new to Magento reached out to me asking for some advice on how to update a Magento instance. He had inherited a legacy Magento project from another friend and now the merchant wants him to update the shop to a recent version of Magento.

While there are a couple of posts available on how to update Magento none of them covers how to deal with a really messy project in the first place and how to update that one in a safe way. Let my try to explain my approach.

So, actually this post is not so much about how to update Magento (I admit, the title is a little misleading), but on how to cleanup Magento so that updating becomes easy...

The main goal of the following approach is to follow a process that will allow you to have a working Magento project at (almost) every point in time instead of taking everything apart and then trying hard (and probably failing) to put everything together with the new Magento version.

Frontend Testing

In an ideal world every view and feature is covered by a Selenium test, but I'm sure that's probably not the case with your friend's project. As a shortcut you could write a simple testcase that traverses the most important views and checkout steps and automatically creates screenshots. With Menta that's an easy thing to do, plus using Menta's pdiff feature you can rerun the test after updating and look at a nice interactive report that shows you all differences on a pixel-basis. (A separate blog post on that might follow at some point...)

Perceptual Diff

Hacks

Before we start thinking about how to restructure the project and how to do the update let's have a closer look at what has been done to the Magento core. Last week I've published a project called "Magento Project Mess Detector" (mpmd) which is a plugin for n98-magerun, that will create nice reports for you. Checkout the instructions in how to install n98-magerun and how to install mpmd.

Magento Project Mess Detector

Core hacks

Find out which version (and edition) of Magento your running, go and get a vanilla copy of the source code and extract it somewhere on your development environment (you're not doing all this on production, right? :)

First let's look at all the core file that have been modified by running this command in the Magento root directory:

n98-magerun.phar mpmd:corehacks /tmp/magento-ce-1.8.0.0 /tmp/report_corehacks.html

The generated report will show diffs for every file that has been touched.

Core Hack Report

Before you can proceed with the update you should go through every single one and resolve the problem in a way that does not result in a core hack. Every hack might be there for a reason. If you update Magento ignoring these hacks something will most likely fail.

In case you'll find a "hack" that is an official patch from Magento that was applied to fix a bug please check if the bug fix is included in the Magento version you're about to install. If not you might need to take care of applying that patch to the new Magento sources manually and/or request a new patch for the new version.

You might also see a lot of "core hacks" inside the app/design and skin directories. If this is the case the initial creators probably edited the default theme instead of creating a new one that will fallback to the default theme. In this case you might have to do some serious cleanup or at least copy the whole skin, template and layout files into a new one and postpone the theme refactoring and cleanup.

Code Pool overrides

Besides core hacks there's another way to screw your Magento instance up: Code pool overrides. Magento's autoloader allows you to put any file with the same path and filename into a code pool of higher priority in order to override the original file. The code pools (in order of priority) are:

  • app/code/local
  • app/code/community
  • app/code/core
  • lib

A common pattern that's only slightly better than hacking the core is to copy the whole file into the local code pool and do the hack inside that copy. This way it's at least a little clearer that you're not using the original file anymore, but updating will still result in problems because now your using that copy from the old Magento version.

Again, mpmd helps you to create a report of these overrides by comparing the code pools with each other:

n98-magerun.phar mpmd:codepooloverrides /tmp/report_codepooloverrides.html

Code Pool Override Report

Make sure, you'll check every single override individually and try to find a better solution to the problem (in some rare cases a code pool override is the only way of doing things, but in most cases this feature is being abused out of convenience...)

Project structure

Now that the Magento core is clean and no code pool overrides exists you probably still have a big pile of files in front of you. Sadly the way Magento 1 is designed out of the box will not allow you to easily tell the Magento core and all the modules apart.

The quick solution now could be to simply extract the new Magento core on top of the existing project and rely on the fact that new files will replace older ones or will be created if they didn't exist before. While this is possible I don't recommend doing this since it results in an ever bigger mess. Instead let's try to decompose the project into it's parts and then swap out the Magento core only:

Step 1: Separating Magento core from everything else

With a simple script (might follow as a new mpmd command, stay tuned!) you should traverse every file in your project and if a file is not part of the vanilla Magento core you should move it into a different directory while preserving the relative path. This will most likely be the case for everything in app/code/local, app/code/community, app/design/frontend/.../ and a bunch more files. The only files that should be left in the original place after the script has finished should be (untouched) Magento core files.

Now let's move the other files into a folder called .modman/Project_Everything inside (or even better: next) to your Magento root directory: (in case you placed the .modman folder next to the Magento root directory you need to create a file .modman/.basedir that contains the relative path to the Magento root directory. In this case: htdocs)

.modman
  Project_Everything
    app
      code
        local
          My
            Module...
htdocs
  app
    code
      core
        Mage...

Now we need to create a modman configuration file: .modman/Project_Everything/modman. In case you haven't used the modman tool before check out this tutorial.

The trick in creating the modman configuration is to find the most generic path for every file that's private to this module. That sounds more complex than it is, but in no time you'll learn how to write modman files.

For our current update workflow we need to make sure every file and directory in Project_Everything is being linked to it's original location without overwriting any Magento core directories (so you can't just map the app/code directory inside Project_Everything to Magento's app/code since it would conflict with the existing app/code directory that also contains to core and community code pools)

Step 2: Create modman modules for every Magento module.

The current situation is already a lot better than having everything in one tree, so this following step doesn't necessarily need to be done now, but code be postponed to a post-update cleanup.

In order to refine the files inside .modman/Project_Everything try to isolate every Magento module (There's more than the files in the code pool: Don't forget the files in app/etc/modules/ and the templates/layout/skin/lib/... files.)

Also check these module's original source (e.g. on GitHub). The project developer might have modified them aswell (which might be a good thing or not). In any case you might want to check if there's an updated version and the chance that the original module comes with a modman file is very high. Remove everything that's related to this module from the Project_Everything meta module and place the new one into .modman instead.

Step by step your Project_Everything module should become very empty and in the end you might only have a custom .htaccess and a app/etc/local.xml. I like to manage even these files from within a modman module (Project_Base) so that there's a clear separation between Magento core and everything else

Step 4: Composer

In case you want go one step further you should put the different components of your projects into different repositories (or use the ones that might already exists for some of the modules you're using). I suggest following git repos:

  • Project Skeleton: Contains composer.json, some project specific configuration files, and you might want to bundle modman, n98-magerun.phar and composer.phar with it.
  • Magento core: Create your own repo (especially if you're using EE) or reuse a public mirror like this one.
  • Magento modules: Create or reuse on repo for each module
  • Tools: Pull in the tools you need (e.g. EnvSettingsTool and the deployment scripts)

Put everything together inside the composer.json file in your project skeleton repo:

{
    "name": "my/project",
    "minimum-stability": "dev",
    "require": {
        "aoepeople/composer-installers": "*",
        "aoepeople/envsettingstool": "*",
        "tmp/magento_community": "1.9.0.1",
        "aoepeople/magento-deployscripts": "1.0.3",
        "aoepeople/aoe_scheduler": "*"
    },
    "repositories": [
        {
            "type": "vcs",
            "url": "https://github.com/AOEpeople/composer-installers.git"
        },
        {
            "type": "vcs",
            "url": "https://github.com/AOEpeople/EnvSettingsTool.git"
        },
        {
            "type": "package",
            "package": {
                "name": "tmp/magento_community",
                "type": "magento-source",
                "version": "1.9.0.1",
                "dist": {
                    "url": "https://github.com/OpenMage/magento-mirror/archive/1.9.0.1.zip",
                    "type": "zip"
                }
            }
        },
        {
            "type": "vcs",
            "url": "https://github.com/AOEpeople/magento-deployscripts.git"
        }
    ],
    "config" : {
        "bin-dir": "tools"
    }
}

After running composer.phar install and modman deploy-all you should now have your original project structure back in place.

Step 5: Build and deployment pipeline

You're already running Jenkins and have an integration server? Great! Create a build job for this project and install it on an integration environment. If you don't have any experience with automated builds you might want to check out my presentation on Magento Deployments to get some inspiration :)

Performing the update

Updating now is as simple as modifying the composer.json to point to a different version of the Magento core. Redeploy, test well and you might be done... (besides the fact that after updating there's always something broken, that you'll have to fix manually :)

Conclusion

If you're planning on picking up further development (or making it easier to fix any update related problems) you should probably invest some time to get the project in a better shape at first. While merchants and some project manager tend to think that updating Magento "can't be such a big deal" it mostly depends on how much test coverage you have, what had been done to the Magento core and what's going on in any of the modules. Also, updating a Magento project while onboarding a new development team usually isn't a very good idea. I suggest taking some time first to clean up the project in a similar way like described here and doing the update as soon as the team feels comfortable with the status of the project. Btw, a messy project usually also comes with a messy deployment process. Plan some extra time to have a closer look how code get's deployed to production and be prepared to clean that up aswell...

Comments

This website uses disqus for the commenting functionality. In order to protect your privacy comments are disabled by default.

Enable Comments