Lessons Learned

I joined EBRI with a passion for the insights that quality research can provide, and a modest concern about the dangers that inaccurate, sloppy, and/or poorly constructed methodologies and the flawed conclusions and recommendations they support can wreak on policy decisions.  While my tenure here has only served to increase my passion for the former, on (too) many occasions I have been struck not only by the breadth of assumptions made in employee benefit research, but just how difficult – though not impossible – it is for a non-researcher to discern those particulars.

We have over the past couple of years devoted some of this space to highlighting some of the most egregious instances, but as I close this chapter of my professional career, I wanted to share with you a “top 10” list of things I have learned in my search to find reliable, objective, actionable research:

1. There are always assumptions in research; find out what they are. Garbage in, garbage out, after all (the harder you have to look, the more suspicious you should be).
2. Just because research validates your sense of reality doesn’t make it “right.” But just because it invalidates your sense of reality doesn’t necessarily make it right, either.
3. Take the conclusions of sponsored research with a grain of salt.
4. Self-reported data can tell you what the individual thinks they have, but not necessarily what they actually have.
5. Sample size matters. A lot.
6. “Averages” (e.g., balances/income/savings) don’t generally tell you much.
7. There’s a certain irony that those who propose massive changes in plan design, policy, or tax treatment, frequently assume no behavioral changes in response.
8. When it comes to research findings, “directionally accurate” is an oxymoron.
9. In assessing conclusions or recommendations, it’s important to know the difference between partisan, bipartisan and nonpartisan.
10. In an employment-based benefits system, the ability to accurately gauge employee response to benefits change is dependent on the reaction of the employers who provide access to those benefits.

One of the things that I’ve always loved about the field of employee benefits was that there was always something new to learn, and with each position along the way I have gained a new and fresh perspective.  I’ll always treasure my time here as a member of the EBRI team, the opportunity I’ve had to contribute to this body of work – and I’m looking forward to continuing to draw on EBRI research for insights and analysis in my new position, as I have for most of my professional career.

That said, I’ll close by commending to your attention one of my favorite “lessons” – a quote attributed to Mark Twain, and one worth keeping in mind along with the 10 “lessons” above:

“It’s easier to fool people than to convince them they have been fooled.” 

Here’s to not being fooled.

- Nevin E. Adams, JD