January 18, 2008
Behavior, Data and Experiences
Posted by Kevin | Print This Article
In a comment here, Scott Briscoe writes:
“I advocate taking a variety of inputs–data, expert opinion, nonexpert opinion, advice from whatever sources seem salient–and mixing it all together with your own experience and thought to make a decision. It’s more risky, because if it fails you don’t have the crutch to say ‘but the data showed…’ or ‘but the committee recommended…’ — you own the decision. You live by it. You try to make it work, you adjust based on environment and input and then you try to make that work, etc.”
I don’t think Scott and I are necessarily disagreeing except that we are ascribing different weights to different things. Scott seems to think (and I may be misreading) that all things are equal; I think that “experience and thought” only come into play when one has solid data around which to wrap them. Behavioral data, in my opinion, is the most important foundation for any organizational decision-making.
I’ve been focusing on online behavior in these posts because, well, I love it. It’s real-time! It’s a treasure trove of information. But it’s not the only behavioral data that counts. You should know a lot about your members’ behavior, including what programs they participate in, what products they purchase, what types of questions they ask, what meetings they attend, what training they register for, etc.
An example: In our organization, we represent small businesses. They range from 1 employee to more than 1,000 (actually, not many of those). A few months ago, for the first time, we compiled mounds of data in an effort to determine which of our programs and content areas appeal to which size segments, because there is a real difference between a company with, say, 3 employees, and a company with 30 employees, and a real difference between a company with 30 employees and a company with 125 employees.
We first broke our membership down into six “size ranges” and then figured out which percentage of our membership came from each range. Then we took everything we do that is measurable in terms of participation — from participation in online areas, to participation in partner programs, to purchases of specific products, to participation in different types of education, to types of questions asked (we have an online q&a submission system that allows us to track different categories of inquiry), to lots of other things. Then we broke the participants in each area down by size type.
And then we brought it all together and compared to overall membership percentages. The idea was to gauge which programs/services have the most appeal to differing segments based on their participation. If a particular size range was OVER-represented in a particular program relative to their percentage of our overall membership, then we learned something. If a particular size range was UNDER-represented in a particular program, then we learned something. And if a particular program saw participation fall along roughly the same percentages as our overall membership, then we learned something.
I’m simplifying our process here and am limited in what I can share. Once we got all this together, we had much more than a pretty table with lots of nice colors. We clearly knew which areas the very small companies (which comprise a large part of our target market base) were willing/able/interested in participating. We clearly knew which areas pulled mostly larger companies, who are a smaller segment of our market base but clearly have different interests/time/resources. And we immediately knew exactly what our “sweet spot” was in terms of a specific size business that provides both maximum participation and resources for the association.
I’m sure you can imagine how such data can be used, both in creating new products and in marketing both membership and products. There are certainly other ways one could do similar analyses for different types of organizations based on different structuring criteria. Some of what we learned from this process was exactly what we expected to learn (based on, yes, our own “experience”), but some of it was a complete surprise.
In any event, once you have access to solid behavioral data that you can keep updated on a regular basis, THAT’s when you take “experiences and thoughts” and begin figuring out how to apply what you’ve learned.
A simplistic example:
If you know a particular type of member likes to travel to events while another type of member never signs up for any event, you have some choices: figure out how to give the non-attending member type some other kind of educational service that doesn’t involve events, figure out how to give the attending member type more events they can take advantage of, or both (or neither, if you have some other agenda). Or, even better, you can use the demographics of your attending member types to go after similar people who aren’t attending because now that you know that people just like them are attending, you can target your materials much more closely. (The wrong solution, in my opinion, would be to decide that what the non-attending member type needs is better marketing. They’re not going to attend. You can kill yourself trying to “market” to them, or create new products/services they might like better — the latter is easier, though for some reason many people think it’s harder.)
Based on various data, you could even decide that a particular member type is so non-engaged that they may not be worth pursuing at all.
But I heartily agree that experiences and thoughts play a major role in deciding what your association is “going to do” once you actually know how your members react in real life to various things. If your association is going to be truly innovative (and I argue that it should) then you should be launching things that your customers would not necessarily think that they need if you asked them (which is why asking them is not necessarily a good use of your time). But I would argue with Scott that basing such risks merely on your own gut (or that of your senior staff or a committee member) is dangerous territory. How many times have associations had to respond to one influential member who complained about a particular program which led to a complete re-evaluation of said program without any attempt to determine if other people had the same opinion? How many “pet projects” have associations launched for various leaders (both staff and volunteer) that ultimately went nowhere?
Gut is good. I trust my instincts, too. But I’ve learned over the years (often the hard way) that the best instincts are well-informed instincts.
I do love me some data. Secret confession: when studying for my MPA, the two social research and statistics classes I took were my favorite. It’s true.
But (and you knew there would be a but) data is too often misused and overused. I think I agree with one of the things you are saying, and that is that association execs don’t collect enough or examine enough data — or the data they do examine isn’t relevant.
But it’s also often the case that data is used as some trump card when making decisions. Data never showed Sony that people wanted a Walkman. One of the problems with data is at best it’s going to give you a slice of the middle of the bell curve. Developing things for the middle of the bell curve is pretty much by definition mediocre. Why not aspire for the left of the curve, the early adopters? Data can help, I don’t deny that, but intuition–whether you call it experience, luck, whatever–is what drives developments of products and services that resonate with the left of the curve, which are products and services that evolve to serving the middle of the curve.
Scott, that’s my point exactly. Data never showed Sony that people want a Walkman. But it certainly showed Sony that people were walking/jogging/etc. As I said, it’s pointless to ask members what they want. But unless you know how they actually behave (as opposed to how YOU behave, or how your committee members behave) you will never give them what they need.
Comment update: On further reflection, I want to make sure you know just how much I agree with you that data is often used as a crutch. Data is data — it is objective. It is what you do with it that matters, and this is where “experiences and thought” (and, in some cases, let’s face it, just plain smarts) come into play. If something doesn’t work and someone says “well, this is what the data showed” then all they are saying is that what they did in response to the data was wrong or misguided or executed poorly. The data (assuming you gathered it correctly) is blameless. It’s what you do with it that determines whether your organization is a success, is mediocre, or crashes and burns. But a far, far, far more likely “crutch” or excuse for associations is NOT “that’s what the data showed” but rather “that’s what our plan said to do” or “that’s what the Board told me to do.”