How to Measure Anything
I’s often wondering how can people make decisions in work and life, as valuable information is always limited, even though seems like there are so many. Does Big Data can resolve data measurement requests of us? What about the intangibles, which is very important, in business world? I believe the author did a good job to answer this. Applied Information Economics (AIE) is good practice in decision making processes of business, IT security, and etc.
Applied Information Economics (AIE) Process
Phase 0: Project Preparation
Initial research. Interviews, secondary research, and prior reports are studied so the AIE analyst can get up to speed on the nature of the problem.
Expert identification. Four or five experts who provide estimates is typical, but I've included as many as 20 (not recommended).
Workshop planning. Four to six half-day workshops are scheduled with the identified experts.
Phase 1: Decision Modeling
Decision problem definition. In the first workshop, the experts identify what specific problem they are trying to analyze. For example, are they deciding whether to proceed with a particular investment, or is the dilemma just about how to modify the investment? If the decision is an investment, project, commitment, or other initiative, we need to have a meeting with decision makers to develop an investment boundary for the organization.
Decision model detail. By the second workshop, using an Excel spreadsheet, we list all of the factors that matter in the decision being analyzed and show how they add up. If it is a decision to approve a particular major project, we need to list all of the benefits and costs, add them into a cash flow, and compute an ROI (as in any simple business case).
Initial calibrated estimates. In the remaining workshops, we calibrate the experts and fill in the values for the variables in the decision model. These values are not fixed points (unless values are known exactly). They are the calibrated expert estimates. All quantities are expressed as 90% confidence interval (CI) or other probability distributions.
Phase 2: Optimal Measurements
Value of information analysis (VIA). At this point, we run a VIA on every variable in the model. This tells us the information values and thresholds for every uncertain variable in the decision. A macro I wrote for Excel does this very quickly and accurately, but the methods discussed earlier in the book are a good estimate.
Preliminary measurement method designs. From the VIA, we realize that most of the variables have sufficient certainty and require no further measurement beyond the initial calibrated estimate. Usually only a couple of variables have a high information value (and often they are somewhat of a surprise). Based on this information, we choose measurement methods that, while being significantly less than the Expected Value of Perfect Information (EVPI), should reduce uncertainty. The VIA also shows us the threshold of the measurement—that is, where it begins to make a difference to the decision. The measurement method is focused on reducing uncertainty about that relevant threshold.
Measurements methods. Decomposition, random sampling, subjective-Bayesian, controlled experiments, Lens Models (and so on) or some combination thereof are all possible measurement methods used to reduce the uncertainty on the variables identified in the previous step.
Updated decision model. We use the findings from the measurements to change the values in the decision model. Decomposed variables are shown explicitly in their decision model (e.g., an uncertain cost component may be decomposed into smaller components, and each of its 90% CIs is shown).
Final value of information analysis. VIAs and measurements (the previous four steps) may go through more than one iteration. As long as the VIA shows a significant information value that is much greater than the cost of a measurement, measurement will continue. Usually, however, one or two iterations is all that is needed before the VIA indicates that no further measurements are economically justified.
Phase 3: Decision Optimization and the Final Recommendation
Completed risk/return analysis. A final Monte Carlo simulation shows the probabilities of possible outcomes. If the decision is about some major investment, project, commitment, or other initiative (it's usually one of them), compare the risk and return to the investment boundary for the organization.
Identified metrics procedures. There are often residual VIAs (variables with some information value that were not practical or economical to measure completely but would become obvious later on). Often these are variables about project progress or external factors about the business or economy. These are values that need to be tracked because knowing them can cause midcourse corrections. Procedures need to be put in place to measure them continually.
Decision optimization. The real decision is rarely a simple “yes/no” approval process. Even if it were, there are multiple ways to improve a decision. Now that a detailed model of risk and return has been developed, risk mitigation strategies can be devised and the investment can be modified to increase return by using what-if analysis.
Final report and presentation. The final report includes an overview of the decision model, VIA results, the measurements used, the position on the investment boundary, and any proposed ongoing metrics or analysis for the future, follow-on decisions.