Vendor Management · Reading a Claim
The Five Questions to Ask Any Vendor Claim
The industry sells your board numbers all day long. The board’s job is not to distrust every one of them. It is to know, in about ninety seconds, which ones have earned the trust and which are wearing it.
The Bottom Line
Every performance claim that reaches a board — from a product representative, a contractor, a consultant, a management company, or a software demonstration — passes through the same five questions. They require no statistics and about ninety seconds. Most claims survive them. The ones that do not tend to fail in one of three recognizable ways, and learning to see those three failures is most of the skill.
Three Ways a Claim Fails
1. Selection bias — who is in the sample
Start with a claim you have heard: the communities a management company manages show property values above the market.
Suppose that is completely true. It still may say nothing about the management, because of one question the claim does not raise: which communities hire professional management in the first place?
Larger, wealthier, more amenity-rich associations are both more likely to retain a professional manager and more likely to have high property values, for reasons that have nothing to do with the manager. The claim takes a pre-existing sorting and dresses it as an effect. The manager did not lift those values. The kind of community that was already valuable is the kind of community that hires the manager.
Once you see this, you see it everywhere in industry marketing. The tell is a comparison between a self-selected group and everyone else, presented as if membership in the group were random.
The correcting question: who is in this sample, and how did they get in? If the members selected themselves — by being wealthy enough, disciplined enough, or motivated enough to opt in — then the outcome may be measuring the thing that got them in, not the thing being sold.
2. Correlation presented as causation
Now a subtler one, because it sounds like plain common sense: associations with reserve studies have fewer special assessments.
Almost certainly true. Almost certainly not what it appears to be.
The board that commissions a professional reserve study is, by that very act, revealing itself to be the disciplined, forward-looking, funding-inclined kind of board — the kind that was going to fund its reserves and avoid special assessments whether or not the study existed. The study did not cause the outcome. A third thing, the board’s underlying discipline, caused both the study and the funded reserve.
Hand that same board no study and it still outperforms the community down the road that has neither a study nor the temperament to want one.
This is the most abused move in vendor and even consultant marketing. Two things move together; the seller implies the first causes the second; a hidden third variable is quietly driving both.
The correcting question: what else is true of the group that bought this, and could that other thing explain the result?
And an important caveat, because rejecting a reason is not rejecting a product. The reserve study is absolutely worth having, for reasons that have nothing to do with this particular claim. It is just not worth having for the reason the claim gives — and a board that buys on the strength of a causal story that is not true has learned nothing about how to evaluate the next claim.
3. The number with no denominator
Back to the roof coating: it extends roof life by forty percent.
Forty percent measured how? Against what? Over what period, on how many roofs, in what climate, compared to which untreated control?
A percentage is a ratio, and a ratio with an unstated denominator is not data. It is decoration.
“Forty percent longer” could rest on a manufacturer’s accelerated-aging laboratory test of a handful of samples, with no field roofs, no independent control, and no association anywhere near the study. It could also rest on solid, independent, long-run field evidence. The number alone cannot tell you which, and the number alone is all you were given.
This is the discipline behind the phrase statistical significance, stripped of its math: a result means something only if the sample is large enough, measured honestly, and compared against a genuine control, so that the effect is unlikely to be noise or wishful selection.
You do not need to compute anything. You need to ask for the denominator and the control, and to notice when they are missing.
The Five Questions
Here they are, in the order they are useful.
- Compared to what? Every performance claim needs a control. Lasts longer, saves more, performs better — than what, measured how? No comparison, no claim.
- Who is in the sample, and who selected them? If the group chose itself, the result may be measuring the choosing rather than the product.
- What is the denominator? A percentage or an improvement figure with no base, no sample size, and no duration is decoration, not evidence.
- Who paid for the number, and who benefits if you believe it? A study funded by the seller is not disqualified. It is not independent, and it is weighed accordingly.
- Does the mechanism make sense, or only the correlation? Ask the seller to explain why the effect happens. A real effect has a cause you can follow. A spurious one has only a chart.
The Change of Subject Is Itself the Finding
This is the most practically useful sentence in the article.
A vendor who has the denominator and the control will produce them, usually with enthusiasm, because having them is a competitive advantage and they know it. A vendor who does not have them will change the subject — to a reference list, to a warranty, to how long they have been in business, to a story about another community.
None of those are answers to the question you asked. And the redirect tells you what the substantiation is, without anyone having to say so out loud.
Write down which of the five questions did not get answered. That note is worth more at the next meeting than the brochure.
The Same Skepticism, Pointed at a Return
The most dangerous version of the unexamined number is not a coating. It is a contractor pitching a project on its rate of return: this investment pays back at sixteen percent.
It is the same move in a more sophisticated suit — a single impressive figure resting on assumptions the seller chose and did not show.
The five questions work on a return exactly as they work on a coating. Compared to what? On whose assumptions? Who benefits if you believe it?
And one addition that applies specifically to a financial model: the assumptions belong to the board. The discount rate, the escalation rate, the utility rate, the occupancy assumption, the service life — those are the association’s inputs, not the vendor’s. Ask for the model, re-run it on the board’s own numbers, and keep both versions. The gap between them is frequently the whole story.
A board that has learned to ask a roofer for his denominator will know, without being told twice, to ask a contractor for his.
If the Decision Rests on a Savings Claim, Decide How You Will Check
Here is the step almost nobody takes, and it costs nothing.
Where an award rests on projected savings, agree before the award on how the savings will be measured: against what baseline, over what period, and by whom.
A savings claim with no agreed measurement method cannot be verified afterward, and so it will not be. Two years later the invoice is still arriving, nobody can say whether the savings materialized, and the honest answer to “did that work?” is that the association never set itself up to find out.
Ten minutes at the front end. Ask for it in writing.
Scale the Scrutiny, and Do Not Become the Board Nobody Wants to Bid
Full written substantiation is appropriate where the award is material, where the claim is central to the decision, or where the commitment is long. Routine, low-value, competitively bid purchases do not need it, and a board that treats every proposal as suspect will lose good vendors to boards that do not.
Set the threshold as policy, in a quiet month, so it is not being decided under pressure with a representative sitting at the table.
And note one signal: a claim that cannot survive the delay required to substantiate it has told you something. Where a genuine deadline exists, proceed if the board judges it right — and record that the claim was accepted without substantiation and why. That record costs one sentence and is worth a great deal at the postaudit.
The Quiet Second Effect
A board that runs these questions out loud does two things at once.
It protects the community from the claim that does not survive them. And, more quietly, it changes how it is sold to — because a vendor learns very quickly which boards can be closed with a statistic and which cannot, and the pitches that arrive afterward are different pitches.
That second effect compounds over years and never appears in any minutes.
What a Board Should Do Next
- Adopt the five questions as the standing evaluation for material claims, in writing.
- Set the substantiation threshold as policy rather than case by case.
- Request substantiation in writing before award, not after.
- Re-run any vendor return model on the association’s own assumptions and keep both versions.
- Establish the savings measurement plan before the award.
- Use the phrase “not substantiated” in records and communications. It is what the board actually knows, and it says nothing about the vendor’s honesty.
- Flag every claim-based award for a postaudit, so the claim gets tested against what happened.
Related CIC-SC Resources
- Governance Standard OPS-005 — Vendor Claim Substantiation
- Governance Standard OPS-004 — Capital Project Postaudit
- Governance Standard OPS-001 — Vendor Procurement and Contract Standard
- Authorizing Expenditures and Spending Limits
References & Sources
- Knight, Ian. Association Financial Strategy (Fundamentals of Association Management series, Book 4), Ch. 4 — How to read a vendor’s claim.
- Knight, Ian. Association Financial Strategy, Ch. 9 — The return on association spending; Ch. 10 — The purpose and design of association contracting.
- Common Interest Community Standards Council, Governance Standard OPS-005 — Vendor Claim Substantiation.
CICSC provides educational resources and governance standards. CICSC does not provide legal, accounting, tax, engineering, insurance, or reserve study services. Boards should consult qualified professionals for matters requiring professional judgment.