Recommendations are a product decision with an algorithm attached, and the order matters. A suggestion layer placed over a search problem hides it from the team while the customer still runs into it.
This guide covers how to check the navigation first, what signals you actually have, which rules must never be broken, and why a hand-picked list tested against a control group answers the question cheaply.
Key takeaways
- Check navigation first: A recommendation layer over a search problem hides it from the team while the customer still meets it.
- Write the never-suggest rules: Stock, compatibility, recent purchases and restrictions constrain every approach and are cheapest to agree now.
- Try a curated list: Twenty hand-picked products against a control group answers the question for the price of a month.
- Hold back a control group: Without one, any uplift gets attributed to the feature and nobody can separate it from the season.
Start with the business outcome
Success is customers finding something they would not otherwise have found, visible in the order rather than in the click. Write down which decision the customer is struggling with, because that is what the recommendation has to help with.
Check the navigation first. If search returns poor results or the categories are wrong, a recommendation layer hides the problem from the team while the customer still experiences it, and the uplift you measure will be against an artificially low baseline.
Four areas to define before choosing features
1. User Need
Describe the choice a customer is struggling with and check whether a better category page would already solve it. A recommendation layer placed on top of a navigation problem hides it from the team while the customer still experiences it.
2. Signals
List what you can observe today, how many customers you observe it for, and how quickly it goes stale. A shop with ten thousand products and a thousand monthly buyers does not have enough signal, and no algorithm fixes a data shortage.
3. Rules
Write down what must never be suggested: out of stock items, incompatible combinations, something bought last week, anything with a regulatory restriction. These constraints apply to every approach and are cheaper to agree now.
4. Feedback
Decide how a bad suggestion is noticed and whether a customer has any way to say so. Without that loop the first embarrassing recommendation is found by a customer on social media rather than by the team.
Decide what to measure
Measure decision usefulness, data quality, adoption, exception handling, review effort, and downstream outcome. Monitor failure patterns over time because a successful pilot does not guarantee reliable operation under new conditions.
Manage the most likely risks
| Decision area | Risk to make visible | Practical safeguard |
|---|---|---|
| User Need | The real problem is navigation | Fix search and categories first and measure what that alone changes. |
| Signals | There is not enough signal | Count how many customers you observe and how quickly the behaviour goes stale. |
| Rules | Something inappropriate gets suggested | Write the never-suggest rules before any approach is chosen. |
| Feedback | Uplift cannot be attributed | Hold back a control group from the first day rather than adding one later. |
Build a roadmap around evidence
1. Check how people browse today
Look at search terms that returned nothing and where customers give up.
2. Write the rules
List what must never be suggested, including stock, compatibility and anything with a restriction.
3. Test a curated list
Show twenty hand-picked products to half your customers and leave the other half alone.
4. Compare order value
If a curated list moves nothing, stop; if it does, that is the baseline an algorithm must beat.
For the data quality that decides whether any signal is usable, see Data Quality: Dimensions, Examples, and an Improvement Plan.
Plan adoption and operating ownership
Merchandising teams hold the rules that protect margin, and those rules are rarely written anywhere a developer would find them. Get them into the room early and have them add their constraints to the never-suggest list. Decide who reviews what the system is recommending each week, because the first embarrassing suggestion is otherwise found by a customer rather than by you.
Rehearse launch as an operating change
Plan a day-in-the-life rehearsal for a recommendation engine. Use representative accounts, realistic data, the actual staff roles, and one common exception. Move through User Need, Signals, Rules, and Feedback while observers record confusion, missing access, unclear ownership, and support questions.
This approach treats adoption as part of the product. It gives the business a practical route to define the customer decision, available signals, business rules, feedback, control, and evaluation while protecting customers and staff during the period when the workflow is still becoming familiar.
A 30-day validation plan: Recommendation engine development
Days 1 to 5: check whether browsing is the problem. Look at how customers currently find products and where they give up. If search returns poor results or the categories are wrong, fix that first. A recommendation layer on top of a navigation problem hides it from the team while the customer still experiences it. Check the search terms that returned nothing, because those are customers telling you in their own words what the catalogue is missing.
Days 6 to 15: write the rules that must never be broken. List what should never be suggested: out of stock items, incompatible combinations, something the customer bought last week, anything with a regulatory restriction. These rules constrain every approach and they are cheaper to agree now than to retrofit after the first embarrassing suggestion. Ask the merchandising team to add theirs, since the rules that protect margin are rarely written down anywhere a developer would find them.
Days 16 to 30: test a hand-built list against a control group. Pick the twenty products a good salesperson would suggest, show them to half your customers, and leave the other half alone. Compare order value. If a curated list moves nothing, an algorithm will not either, and you have the answer for the price of a month. Keep the groups the same size and run them over the same weeks, as a comparison split across a promotion period will mislead in whichever direction the promotion ran.
A practical worksheet: Recommendation engine development
Recommendations are a product decision with an algorithm attached, and the order matters. Complete these rows before anyone compares approaches, because the first two usually settle the question.
| Decision area | What to record |
|---|---|
| User Need | The choice a customer is struggling with, and whether a better category page would already solve it |
| Signals | What you can observe today, how many customers you observe it for, and how quickly it goes stale |
| Rules | What must never be suggested, such as out of stock items, prohibited combinations, or something just purchased |
| Feedback | How a bad suggestion is noticed, and whether a customer has any way to say so |
| Measurement | The effect on the whole order rather than on clicks, tested against customers who saw no recommendations at all |
Hold back a control group from the start. Without one, any uplift gets attributed to the recommendations, and the team will be defending an investment with a number that cannot distinguish the feature from the season.
Related guides
- The operational half of this is in Data Quality: Dimensions, Examples, and an Improvement Plan.
- If your team is already past the planning stage, read BI Dashboard: Examples, KPIs, and Design Principles.
If Recommendation Engine Planning: Product Value Before Algorithms leaves you needing the data joined up before anything is built on it, Discuss Your Data Product.
Ali Boran Gazel