Within this guide, you will get access to a number of questions that you will need to consider when picking an experimentation platform. In order to successfully scale up the culture of experimentation with your company, you need to pick a platform that will grow with you. The questions listed in this article will help you to do just that. This article has been split into two parts, the first section are questions you should consider and ideally have answers to before talking to vendors, and the second set of questions that you might want to consider adding to an RFP process.

Type of experiment

  • Where do you want to run experiments (website/mobile/etc...)?
  • Do you want the tool to also provide personalisation capabilities? If so, what type of segments would you want to use to target customers?
  • Does your company need to deal with busy sales periods?
  • Do you want to do client-side experimentation, server-side or both?
  • What types of things do you want to test, UI changes, or more fundamental changes to how the site works?
  • How many tests would you like to run per month?
  • How are you planning to project manage your experimentation program?

Price

  • How many monthly active users do you have?
  • How many people will need access to the tool?
  • Do you have a budget?
  • Is the budget approved?
  • Will you need help putting together a business plan?

Project

  • What is the main driver behind the business?
  • What would the first year of a successful implementation look like?
  • Are you working towards a timeline?
  • What websites do you want to experiment on (Urls)?
  • Do you have the technical resources to work on the project?
  • What is the main driver behind moving to another system? Why would you not stick with your current platform?
  • If something requires a developer change do you do that in-house or agency

Data And Security

  • Do you need to talk to legal/compliance and do you have any rules you need to adhere to?
  • Does the stats engine work in real-time?
  • How is the sample size calculated by your system?

Site Performance

  • What is an acceptable level of page performance?
  • Do you have any performance requirements?

Support and Help

  • What level of support do you need? Do you need a lot of hand-holding, or, are you independent?

RFP Questions

After considering these questions, you will be in a great position to go and talk to use vendors. To start learning about the capabilities each platform provides, the remaining questions will help you get the information you might need from a vendor:

Type of experiment

  • Does your software come with a visual editor?
  • Does your software MVT testing?
  • Do you support multi-arm bandit testing?
  • Do you offer split URL testing?
  • Does the tool support multi-page testing?
  • Can I preview my changes easily?
  • Do you offer A/B/x testing?
  • Does the tool support targeting/segmentation?
  • Does the tool easily track clicks and page views?
  • Can we experiment on logged-in and anonymous users?
  • Can a group of users be excluded from any experiment?
  • Is it possible to run omnichannel experiments with consistent variation assignment
  • Can we customize the traffic allocation?
  • Can we import custom segments into the software?
  • Is it possible to prevent overlapping experiments?
  • Can we schedule experiments?
  • Do you support tests with dynamic traffic allocation?
  • Can mutually exclusive tests be run on your platform?
  • Does the tool support re-usable templates?
  • Does the tool offer AI-based personalization?
  • Does the tool offer 1:1 personalization?
  • Can I build custom widgets within the tool?

Price

  • Is there a cost for leaving an experiment set to 100% once completed?
  • Do you charge for non-production environments?
  • Is there an overage charge if traffic is higher than expected?

Data And Security

  • What type of goals can be defined for an experiment?
  • Can additional metrics be set up?
  • Does the tool provide a real-time view of test performance?
  • Does the tool provides a confidence indication?
  • Does your system support different user/group permissions?
  • Are you ISO 27001 accredited?
  • Describe the statistical approach used by the tool?
  • Do you provide access to raw experiment data?
  • Are all transmissions made by the tool encrypted?
  • Is access to sensitive information granted on a need-to-know basis?
  • Where are your data centres located?
  • Do you have a documented security incident response plan?

Technical

  • Does your tool work with single-page applications?
  • Will the tool impact my website's page load time?
  • How do you mitigate the flickering effect?
  • Can I integrate your tool with my current analytics platform?
  • Can I integrate your tool within my customer data platform?
  • Can I integrate your tool within my data lake?
  • Can I export my data from the tool easily?
  • Which SDKs do you support?
  • Does the tool offer feature flagging?
  • Does the tool offer staged rollouts?
  • Do you provide documentation for your SDKs and APIs?
  • Does the tool have a REST API?

Support and Help

  • Do you offer training?
  • Can you provide developer support?
  • What does your professional services model look like?
  • Where is your support team located?
  • Do you provide training videos?
  • What if s your NPS tech support score?

Happy Coding 🤘