An Evaluation Of Using TDS To Improve My Projects Deployment Process
Tue 19 July, 2016 / By Jon D Jones
In today's guide, I'm going to cover the pro's and con's I encountered when integrating TDS into my last project. I was onboard the project after the client had two very bad experiences with their previous partners and they wanted to get an independent analysis of their website. As part of this role, they wanted me to introduce a process that would ensure the website didn't break on a release, that allowed them to work on parallel tasks with other developers and to make a process that would help them move forward as a business.
After doing some initial discovery, the situation I found myself in, in terms of a deployment process, was probably the worst process I've encountered in 11 years of working in this industry. This article is probably a great guide about what you should not do when implementing a deployment strategy for Sitecore.
The quick overview of the deployment process can be seen below. DO NOT COPY THIS PROCESS AT ALL. IT IS SHIT:
- All Config Code Wasn't Checked Into Source Control. Config files lived on each server without back-up
- Manual Changes Required To Deploy Code to Each Servers Config Files
- Each Environment seemed to contain a few assemblies with different versions
- No Rollback Process
The files and the Sitecore website was separated into different projects, which is good. On someone's development machine, someone would do a Visual Studio publish that would copy the pre-compiled classes into the dev/auth/live folder. The process didn't use any type of config transforms so a lot of the config files like the web.config, 401 Re-directs etc.. where not in source control, or backed up anywhere at all. Not having your web.config file in source control creates a lot of hassle when it comes to using Nuget and I wouldn't recommend it personally but some people will agree with excluding it.
To deploy code to a server, a developer would do a Visual Studio publish into a source controlled folder (one folder for each environment). After the files were checked in, a build process was triggered which would copy files onto the server, via a custom deploy app that copied the files to the appropriate folders.
If you had any config changes, you would hope the site didn't break after deployment. After the files copied, you would then log onto that server and manually make any changes using Beyond Compare until the site was back up and running. Say Nuget had added some binding redirects, these had to be manually copied across all web.config on each environment in the web.config.
This process is terrible for a number of reasons. Striving for continuous integration should be a key aim for any big company that relies on its website. For an enterprise level company that makes a lot of money through their website, this wasn't acceptable in terms of risk or disaster recovery. If the live website's config file somehow got deleted, or corrupted, they literally had no roll-back process. The process at worse case would involve me reinstalling Sitecore on the server and then tinkering with files until it worked.
How Do We Fix This Mess?
Instead of this very time consuming, error prone manual process, I came up with a set of goals the business agreed they wanted to move toward. Below lists some of the goals I look for in a deployment process:
- Deploy Between Environments Without Any Manual Intervention
- Able To Set-up New Environments Easily (New Developer, New Server)
- Easily Able To Sync Sitecore Items Automatically
- Be Able To Use Strongly-Typed Models In My Code To Reference Sitecore Items
- Able To Upgrade Easily
- Have The Minimum Number Of Binaries In Source Control
- Make the instance replaceable/Able To Rollback/Re-Install Quickly
- Have All Solution Dependent Files Backed Up
- Able To Unit Test
The goal was to introduce a fully automatic continuous integration process with unit test checks on each enviroment. In terms of deployment, Team Development For Sitecore (TDS) is a widely recommended tool to improve your deployment Sitecore process. After a few meetings explaining the issues to the business, they brought a Sitecore TDS license.
As TDS is renowned for its benefits in deployment, then buying a license seemed like an obvious choice. In today's guide, I'm going to go into more details about some of the difficulties with the current process, what I wanted from TDS and the issues I faced trying to implement TDS and how I found working with TDS in general.
Team Development For Sitecore
As mentioned above, some of the main goals of my process were:
1. To be able to deploy between environments without any manual intervention
2. To be able to easily sync Sitecore items automatically between environments
3. To be able to use strongly-typed models in the codebase to reference Sitecore items
I've written previously in, Installing And Configuring TDS For Sitecore
about the benefits of TDS to improve the deployment process. The main things I wanted to use TDS for was:
1. Moving Sitecore data items between environments
2. Using its ability to integrate with Glass Mapper to generate classes for my Sitecore items
3. Make use of the config transform feature with Sitecore config items
A TDS license costs $750 bucks per developer. As the company had a very small team, a license for TDS was an obvious choice. The license took a few hours to get sent out but we managed to get started on the day of purchase.
Before I get into this in detail, I should say that I personally think that config files in Sitecore, like the web.config (in particular) should be left as close to stock as possible. This will help you greatly when you try to upgrade. If you want to add in application settings, you should create a separate app setting config file and use transform on that file to configure the site how you want. The benefits of having things like web.config as close to stock as possible is that when you upgrade you can simply use the new stock web.config and copy in the extra include files you might need to get your site up and running again.
TDS 4 has the ability to perform config transforms on a code build. TDS can integrate with config transformation tools like SlowCheeta, which out of the box Sitecore can struggle with, so I was keen to see if TDS lived up its promises.
In this client's situation, the web.config wasn't stored in source control. I was told this approach was taken to prevent the publish from potentially overriding settings on the different environments when the code was checked in. By using TDS and it's transformation capabilities, I could create one config files that changed on a per-environment basis in source control to speed up deployment and ensure no human errors were introduced on a build.
After implementing this, this small change alone saved about 30 minutes to an hour on each deployment for each environment. With transforms, I could completely eliminate the issue of having different config files onto different environments. To add in transforms I installed Slow Cheeta, then on each config, in solution explorer right-click on the config file and in the context menu, select 'Add Transformation':
This should create you a config file for each environment configuration you have defined in your project. In my example, I added in the transforms specific to my environment for the dev environment and after setting up TDS to work with my dev build configuration, it worked out the box with no extra configuration required. This is a really cool feature of TDS.
If you want to use config transforms with Sitecore then TDS is definitely a bonus and really easy to use. It works on MSBuild tasks in the background so you don't have to do anything extra to get this to work on the Sitecore transform files. Without TDS I couldn't get this to work.
Sync Data Between Environments
One of the main TDS features I wanted to get working in the solution was the ability to copy Sitecore data between environments as part of the continuous integration process. In the past, I've spent far too much time creating update packages, installing them on different servers, realizing I forgot to include something then repeating the process until everything works as expected.
My experience with Sitecore packages for small releases is that it's error prone (usually by human forgetfulness) and time-consuming. With TDS I was hoping to automate this process as part of CI to improve efficiency. TDS has a brilliant way of syncing Sitecore items, however, I did run into issues implementing it with the environment. One of the big hurdles I had trying to implement this was the way TDS works and the IT policy at the clients.
In the existing deployment process, a Visual Studio build would be made into a folder. The binaries would be checked into source control and then cruise control and a custom deployment app copied the files onto the server.
In order for TDS to sync items automatically, it needs to be installed on the build server and your deployment process needs to perform a source code build on each environment in order for Sitecore items to be synced automatically. This process is definitely how I've set-up other websites in the past and is how a standard continuous integration process would work, in my opinion. This provides a number of benefits, for example, unit tests being able to run and the use of peer review tools to be integrated with the deployment process.
The process the company had been sold into was basically a binary publish onto each environment via source control. I think this approach is bad because a) you can't run unit tests and b) you don't know exactly what's being checked into each environment. All you know is that the binary as a whole has changed.
TDS, however, uses something called the 'Sitecore connector' to sync the data between environments. The connector is used by TDS to interact with Sitecore, e.g. sync and deploy Sitecore items.
The Sitecore connector is triggered by a local Visual Studio build, or through a build server build. When a build on the build server is triggered for example, it can add new Sitecore items added into the authoring or live environments. This process is triggered by special MSBuild instructions performed by the TDS installer.
The TDS installer will first install the Sitecore connector web service during the build, and then afterward use the connector to deploy the items contained inside the TDS project to a target Sitecore instance. This is all configured in the 'Build' tab in your Visual Studio project:
In case you're wondering how this works, it's the settings in this 'Build' properties where the TDS deployment process happens. For each environment (Debug, Test, Auth, Release) you define the website URL and its file location. With this approach, you can tell your Build server to use the settings for the authoring instance when building and deploying
Issues With My Process
The process above definitely meets the desired deployment requirements, however, to throw a spanner in the works, the company's IT department said they were unable to look at changing the deployment app (custom) for three months.
If you find yourself in a similar situation, the other approach is to kick off an item deployment from within Visual Studio. All you need to do is have your selected build configuration configured in TDS , right-click on the Solution and select Deploy Solution. This will cause TDS to build AND deploy items and files to your target instance.
In order to implement this alternative approach, I needed my dev machine to have a network share/access available to each of the servers and then configure TDS to deploy the code. Due to IT, this was also no good, so I was a bit stuck moving forward.
I guess this is one of the main considerations when considering introducing TDS in terms of data sync. For most people, I would assume they would consider TDS at the beginning of a project, not several years in, so having a build server work like a build server isn't an issue.
If you work with data syncing tools in other CMS systems, you don't generally have this requirement to completely change your deployment process. When you use uSync with Umbraco for example, it works by writing any change of data to a folder location, as long as you check that folder location into source control that data will be loaded into the CMS without any other interaction required. When Umbraco loads it then automatically reads in the new items in the folder.
If you look at Episerver and how the page type builder works, this is another very similar approach that could have been used. If TDS worked by creating data as a file that was read in automatically on application start-up rather than application build, it would have meant I could have used it out the box instead of having to look for an alternative approach.
Sync Data Between Environments Review
Granted, Sitecore is more complex than other CMS systems but having to completely change your deployment process to work with a tool is a bit clunky. If you are considering TDS, then I would recommend making sure that you actually have a build server that compiles your work onto each server as an important aspect of getting the most out of TDS. If you do not build your code on the build server you may not be able to use TDS the way you wanted to without a change to the process.
As mentioned above, the deployment process introduced by the last partner was effectively a fancy file copy hooked into source control. In the short term, we wanted to carry on making process towards a more automated solution. This is where TDS package creation came in handy.
In the 'Update Package' tab if you have 'Generate package during the build' enabled AND you have TDS set-up correctly, on a build TDS will create a Sitecore update package for you.
This update package can contain all the file and database changes that have been added into TDS. There is also an option to split this packages into two, one for files and one for the Sitecore database changes.
This package approach solved the problems of sharing data between my environments easily. All that was required was a build for the desired environment, then within the TDS projects bin folder, for each environment, there was a matching folder that contained the update packages:
To deploy my changes all I need to do is a build and then on each environment, run the update package. That's a lot quicker than having to create an update package each time. As I'm additionally using TDS with code generation (mentioned below) the update package ties in with the strongly typed models I use so there's no chance I might forget to sync something across an environment.
Package Deployment Review
Not being able to sync Sitecore data between environments the way I wanted to with the client's deployment process was annoying. Setting up Sitecore to work with deployment packages was really simple. Being able to install packages so easily was a really good alternative to the current approach. I could also check the update packages into source control to make sure I had a rollback process, which was an added bonus.
Code Generation Based On Sitecore Models
Automatic code generation was the last part of my deployment process check-list that I thought TDS could help improve. If you've read my guide, Configuring TDS To Use Code Generation Using Glass Mapper
you can see how easy and quick the process is. The process of setting up TDS with the Glass Mapper and the T4 templates required to generate the code resulted in a few unexpected surprises, but after the main issues were resolved the process took about an hour to get up and running.
Originally, the project used Synthesis as an ORM mapper, if you have never used Synthesis then I recommend reading, How To Map Sitecore Objects into C# Objects using Synthesis
. Synthesis does a pretty good job of mapping Sitecore objects. It does exactly what you would want. Synthesis creates usable classes that map to Sitecore objects. The bit I felt Synthesis fell down was that I wanted my code generation to link in with my Sitecore data deployment. With Synthesis you can generate your models, but from what I understand, there is no way to integrate it so any new template it finds can add into build into a different environment without me manually needing to do something.
As TDSs integrates with Glass Mapper, it makes it really easy to get TDS to sync the items you care about into the TDS project, generate the classes for use in your code with GlassMapper and then hit build and have TDS automatically push those items. As this approach is more in line with my goals, I decided to start phasing out the Synthesis use and turn towards GlassMapper, which went really well.
Code Generation Review
The way TDS integrates with GlassMapper and then deploys those new Sitecore items into different environments is a massive win towards reaching my deployment goals. Synthesis is good at what it does, but its lacks on the integration with TDS which means that even though Synthesis might know about a new Sitecore item, TDS might not. Using GlassMapper that integrates so nicely reduces the errors you might get in deployment.
If you use an ORM mapper in your solution that isn't GlassMapper and you're so far down the road that swapping it might be a problem, then this would probably be another reason you might not want to purchase TDS. Personally, I think spending the time integrating everything is worth the effort as it makes the deployment process super simple.
During the process of trying to integrate TDS with my project, I needed to contact TDS support. Compared to some support vendors (Sitecore being one of them) my experience with TDS support was really quick. I think I got an answer within 30-60 minutes each time I emailed. The advice I was given was relevant and helpful. Having good support was another big plus for using TDS in the project.
As the deployment process was so bad in this situation, there wasn't really any possibility that TDS could make things worse. Even though we didn't have the build server and source control set-up the way we ideally wanted to at the start of the deployment change improvements, TDS still managed to help resolve a number of issues.
A month after I implemented the changes in the guide I could definitely see a vast improvement to my efficiency. I definitely spent considerably less time doing deployments and messing around with manual changes.
As a person who charges a day rate for my services, over a 6 month period, the costs of a TDS license would easily repay the amount of time it would take me to manually do some of the things I've been able to automate. A lot of the work that was manual and took a long time was now done at a click of a button. I've spawned off several new developer machines in less than 5 minutes a pop.
I did find the data sync issue not working the way I wanted annoying, but it was good that TDS also provided an alternative approach I could use until we fixed the build server.