Five key considerations when engaging a DCM partner
Introduction
It has been said by many that for most end user organisations Data Centre migration is a once in a generation event. Consequently, many organisations planning a DCM choose an experienced third-party supplier to work with them to achieve their goals. Most end user organisations are, quite rightly, focused on ensuring that they select the best partner for their needs who has the appropriate experience and highly skilled resources to execute the DCM. What sometimes gets neglected is what they, the end user, will need to do to make the partnership successful. Regrettably candidate partners don’t always spell this out in detail as in a tender situation stressing what will be expected of the customer may be perceived as somewhat negative. Nonetheless the customer needs to understand what they are likely to need to do to make the project a success.
Engagement is key
There needs to be continual and in-depth engagement between the Customer, The Supplier and the Customer’s key third party support partners. Contentious issues need to be put on the table ASAP and worked through to resolution. As an example, on many occasions I have worked on DCMs for organisations where application support has been contracted out to one or more third parties. Now I know the temptation is to try to get them to support the DCM for free and there is probably no harm in trying that. However, in many cases these organisations will push back and want new commercial terms to support the DCM. These things must not be allowed to drag on for many weeks or months. So, whilst the Customer may try to include it in the existing scope, they should be working up an alternative strategy in the background. Too many large DCMs lurch from blocker to blocker. These issues will always arise but engagement, anticipation and flexibility are what’s needed to keep the DCM moving forward.
You need an in-house DCM team even if you are using a third party supplier
You need to stand up a team to work with your supplier and I don’t just mean a Project Manager or two. Some of this team will need to be dedicated to their roles 100% of the time for the life of the project. Many organisations find this hard to take and have embarked on a DCM project expecting to have their people carry out their BAU roles and maybe give 5% to 10% of their time to DCM project related activities. Most of the time that simply will not work. I have seen organisations realise this and go out to the supplier or the freelance market to temporarily back fill the in-house SMEs that are key to making the DCM work. This is a good strategy, but you have to start it early. The backfill resources need to be onboarded and then trained to handle the normal BAU activity before the in-house SME’s become swamped with DCM work. You probably need to on-board the back-fill staff two to three months prior to the DCM ramping up in earnest. Yes, this is yet more cost but if you don’t do this the project timescales will extend and almost certainly result in a much bigger cost. By the way, doing this the other way round and on-boarding temporary staff to form your in-house DCM team almost never works. In these situation you will have a new team who knows very little about your application landscape facing off to the third party supplier who, whilst they may bring a wealth of DCM experience to the project, also know very little about your applications and infrastructure at the outset. Oh and don’t forget your internal third party relationships. If support outsource organisations are embedded with your teams and they support and develop many of your applications, they will need to be involved too. Also you will probably need a commercial discussion with them because they may, reasonably, argue that supporting a DCM is outside the scope of their current contract.
You need to allow the supplier to carry out detailed discovery
It may seem surprising but I have lost count of the number of times DCM projects have suffered significant delays due to the discovery activities being substantially impeded. Discovery mainly happens in the early stages of a DCM project. Its smooth running can be hampered by the fact that the relationship and trust between the Customer and the third party supplier is in its infancy. In my experience the game can often run like this.
Supplier: We have some preferred discover Tools/Scripts we would like to deploy
Customer: You don’t need to do that we have an XYZ CMDB that will give you everything you need (The supplier may not have actually specified everything they need as they were hoping to use their preferred discovery tooling and packaged configuration that they have used on many projects)
Supplier: Spends weeks or sometimes months evaluating the data from the XYZ CMDB and trying to validate it against actual running systems. Often the supplier will not have access to live systems for the purposes of cross validation. Unsurprisingly they often come up short against what they feel they need and by now they have produced a detailed specification of discovery data elements that are required. They present this to the Customer, repeating the request to be allowed to run their preferred tools
Customer: Says that to deploy tools or scripts to all servers will require Infosec evaluation and approval. As Infosec are also involved in BAU work this may take weeks or months before permission to deploy the tools can be granted. As a compromise the Customer says that if very basic scripts or even lists of standard OS command can be provided then they can have their support teams review them and if approved run them on each server.
Supplier: Directs their technical team to produce information in-line with the minimalistic approach suggested by the Customer
Customer: Eventually approves the minimalistic requests. These are run on each server with little or no automation. After a number of weeks the output is provided, in per server form, to the supplier
Supplier: The supplier team spend several weeks consolidating the individual server output and uploading it into whatever discovery repository they are using. After much processing they supply consolidated discovery information back to the customer for validation and approval
Customer: In depth validation in most cases is not a practical reality. In earlier steps we have already learned that the XYZ CMDB is not a suitable source for cross comparison of all discovery elements. Whilst some attributes could be cross validated the number of exceptions would be very large. Customer therefore signs off discovery as being correct.
This sort of scenario, which in my experience is surprisingly common, means that a DCM project can be in a situation where they have taken many months doing discovery and only end up having entry level quality data. To maximise the chance of success and minimise delays in the DCM project the Customer and Supplier both need to bite the bullet and have the tough conversations up-front before anything like my example saga above unfolds. Why doesn’t that happen a lot of the time? Well as I said Discovery takes place early in the life of a DCM project. The supplier’s commercial management team will be reluctant to have difficult conversations with the customer in the early stages of the relationship. Both sides need to get over it somehow. This stuff adds significant cost and many months to the project.
You need to be flexible with regard to Change Management
DCMs can be particularly problematic for an organisation’s Service & Change Management functions. When business services come on-line they will usually have been subject to rigorous accept into service processes. Once they go live the Service Management function acts as their guardian by ensuring any changes to the service are subject to appropriate testing and other quality assurance measures. The rigour that typically accompanies services changes in BAU is often impractical when you are moving tens or hundreds of servers in a relatively short space of time. It is likely that most BAU change management processes would be overwhelmed if a CR were submitted for every server being migrated or worse still for every step in the process of migration. However, that is what many end user organisations try to do. Consider using tactics such as a “Blanket” Change request for a batch of servers being moved rather than one per server. Yes it’s less detailed but the business, particularly Incident & Service management, will be on high alert anyway because you are in the middle of a DCM. The downside of that, by the way, is that pretty much any disruption of service is initially blamed on the DCM, a price the DCM team unfortunately have to accept. In any non-trivial DCM there will be failures. The goal is to try to minimise their occurrence and their impact.
You probably can’t do it all using BAU processes
In some respects, this is a generalisation of the previous point. It’s not every day that a business decides to move 500+ servers to a new Data Centre. It is highly probable that your existing risk evaluation processes have not been drawn up with that eventuality in mind. DCMs by their very nature increase operational risk for their duration. You could attempt to nail that down by moving systems in very small groups or even on an individual basis. Usually that turns out to be unacceptable as in many cases it could expand the project timeline to years. Exactly where you draw the line in terms of risk acceptance criteria for a DCM will be up to you, but it is unlikely to be where the line is today. One example of this is trying to stick to the BAU process for Firewall rule changes. Quite a few organisations have SLAs around Firewall rule changes that allow say 28 days or more for validate and implement rules. Trying to stick with this for a DCM project that will generate hundreds or even thousands of Firewall rule changes can be a very severe bottleneck. Of course resourcing may be an issue. I refer back to my earlier point about the customer needing to stand up a dedicated team. If there are going to be a large number of network changes you may need to allocate a network person 100% to the DCM. Another example is trying to migrate servers within BAU service outage windows. If you are lucky you will be able to do that for some services, but almost certainly not all. You will need to have extended outages on some services to get them migrated. In general, the lower the appetite for outage the more complex the migration will be.
Conclusion
DCMs are hard and that’s a fact. By the end of a DCM project the Customer and the third-party supplier will probably have a good working relationship. Processes will have matured and for the most part will be working well. However, it is also likely the project will be well over budget and have extended well beyond the envisaged timescales. It is not easy to get on top of some of the key factors I have outline in this article but if you can, you stand a much better chance of bringing the DCM in on time and within budget.