Measuring Return on Investment

Those responsible for managing information and knowledge resources continue to find it challenging to demonstrate the value these resources bring to an organization. Outsell Inc., an information industry research and advisory firm, projects 2011 revenues for companies in the information industry will reach $374 billion.(1) With organizations investing large sums of money in acquiring information and building knowledge sharing platforms and systems, it is understandable that managers expect accountability for such expenditures. But can we realistically specify bottom line financial benefits that result directly from access to information? It is important to acknowledge that calculating the financial benefits, or your return on investment (ROI), is more of an art than a science. How can we precisely measure return on investments in information resources, knowledge management systems, or learning programs when the end use or application of such information may be several steps removed from the time the actual information is initially obtained? Some would argue that it does not pay to attempt to link balance sheet benefits to such investments. ROI is, however, an important business metric for senior managers, and knowledge of this formula, as well as other techniques for measuring value, is advantageous in negotiating for budgets and project funding. The keys to establishing credibility for the metrics you adopt are 1) knowing what metrics will resonate within your organization and 2) demonstrating how the data you present supports the organizations business goals.

Typical Investments
All organizations today deal with a broad range of information and knowledge products. When thinking about assessing value and impact to the business, it is practical to focus on specific products, understand in detail how and by whom these products are used, monitor use over a period of time, identify decisions made as a result of having access to these products, and then assess the benefit. It is also necessary to have a thorough understanding of costs. You may choose to look only at direct costs, excluding such costs as labor, IT support, and real estate and if you do, it is important to clarify that when stating the assumptions on which value and benefit calculations are made. As knowledge professionals embarking on this activity, you will want to consider proprietary internal knowledge resources in addition to products and information services that are licensed or purchased. Internal proprietary resources can include: Custom taxonomies Databasese.g., those created for product literature, competitor data, research reports, etc. Employee blogs Expertise directories Newsletters Licensed or purchased resources can include: Databases E-book collections E-journal subscriptions E-journal back files Market research studies Newsfeeds Syndicated research Potential scenarios that could signal a need to measure return on investment may be: Confirmation that a particular resource brings value to users Resources that you know are having a positive impact on users are in need of higher visibility Funding for broader dissemination/access is being questioned Funding for renewal is being questioned Additional headcount is being requested to create or manage a particular service

Meaningful Metrics
After you determine the product or service for which you will assess return on investment (ROI) and document the associated costs, it is necessary to focus on the return part of the equation. In a manufacturing environment, it is relatively simple to calculate the financial benefit of a new piece of equipment when production increases by a precise number of tangible units. Sale of the additional units over time justifies the investment and provides a suitable return for the business. In the knowledge economy, determining ROI is not a straightforward calculation since many perceived benefits are intangible.

How do you decide what to measure? Sometimes the best answers for how to arrive at meaningful metrics start with the best questions: What are the organizations critical business goals? How do senior executives measure the organizations performance and success? What kind of return is likely to be most valued in this organization? How do executives talk about success? The knowledge managers ability to arrive at meaningful metrics requires thorough understanding and support for the goals of the parent organization. It is also helpful to be in tune with the culture of the organization and to know something about the management style of the person to whom you will be presenting the business case. Revenuewhether it is new revenue or money savedis always understood but proofs of financial benefits might be a stretch. Some managers prefer qualitative metricsstories that illustrate a win or real business benefit and have an emotional impactinstead of spreadsheets filled with assumptions and calculations. Managers have different perceptions of value or return depending on their business goals and belief systems and those perceptions are subjective. To accommodate managers who focus on quantitative data and outcomes, hard metrics that can potentially be derived from access to high-quality information and knowledge resources include: Time saved Money saved, costs reduced Reduced time-to-market New revenue generated Revenue increased Risk avoided Soft metrics include: More informed decision-making Improved response to customers, competition, market conditions Employee productivity Employee satisfaction/retention Enhanced ideation Prevention of duplication of efforts Some organizations will place greater value on metrics and stories that illustrate how information and knowledge resources contribute to performance goals, such as: Safety Reduction in errors (e.g., medical errors) Increased membership

Collecting Data & Feedback for Estimating ROI

Consumers of subscription-based information resources typically view such resources as necessary tools for their work. The same is true of internally-generated knowledge assets. Knowledge workers are not accustomed to connecting their work back to specific articles, reports or stories since business decisions are made with a myriad of inputs, blended with ones education, experience and conversations with others. It is incumbent on those managing these resources to systematically ask users to think about and articulate how access to targeted information resources, organized to facilitate finding precise and accurate answers, increases productivity and leads to solving business problems. Such input can be collected through: User/client surveys Interviews and focus groups Feedback forms embedded in delivery systems Collecting anecdotes and testimonials This data along with web analytics and vendor usage data are essential for estimating ROI. The methodology you choose will depend on tools available to you and your personal management style. The bottom line is that ROI cannot be estimated without user data and feedback.

Approaches to Measuring Benefits

In theory, ROI calculations involve objective cost information and measurable benefits. With information and knowledge services, estimating benefits is often based on subjective valuations, such as useful, helpful, contributed to, and I learned something new. The subjective valuations help paint an overall picture, but knowledge professionals need to continue efforts to build strong and objective financial ROI cases. Consider the following techniques for presenting financial benefits as a result of access to quality content. Return on investment: an accounting and management concept used to determine financial benefit or value received in comparison to capital invested. Businesses use ROI projections to decide where to make investments and later, to analyze the success of investment decisions after the fact. It is calculated by dividing the anticipated (or actual) return amount by the cost of the investment for a single period or averaged over multiple time periods. ROI is expressed as a percentage and is determined with this formula: ROI = Return (or benefit received) Cost of the investment x 100 Cost of the investment Cost-Benefit Analysis (CBA): a related metric for determining the impact or consequence of an investment. CBA requires documenting the cost of a product, service or activity and comparing the costs with a list of all benefits to be achieved. This method also requires assigning value to hard-to-quantify intangible benefits. Story-telling and anecdotes: simple narrative descriptions of results such as how business was won or risks avoided because of having the right information at the right time may have greater impact and may be perceived as more credible than multiple spreadsheets and models, depending on preferences of the listener.

Presenting Estimates of Value

When presenting ROI or cost-benefit analysis data, it is critical that your methodology for collecting data and analyzing value is deemed credible. As a general rule, the following practices add to credibility and understanding. Be rigorous about calculating the total investment (costs) for the product/service being analyzed Make sure the user sample size is large enough to be representative or statistically significant Always describe the source of the data used for the calculations State assumptions about users and uses Analyze facts and environmental realities, i.e., the business context of the assessment Be conservative in estimating savings or new (or incremental) revenues Present data in charts or graphs for ease of understanding Over time, you will be able to build models based on the data collected. For example, feedback collected over a three-year period indicates that a specific type of information product is benefiting the sales organization through 1) generating promising new leads, 2) minimizing time wasted with poor quality leads, and 3) resulting in larger sales due to increased knowledge of the customer business.

Challenges & Opportunities

In practice, ROI measurement for information and knowledge resources is an elusive target. We are dealing with the fact that content from these resources is mixed with other inputs in the decision-making process so there is usually only a fuzzy correlation with business results. Many factors contribute to performance and productivity and the intangible nature of information and knowledge makes it challenging to assess that contribution. While we have made progress in understanding that contribution, we need to be persistent in gathering data that will enable us to build stronger business cases to show the value and impact of information and knowledge resources. Another challenge lies in the diversity of user/client needs. In order to measure value to the user and the organization, it is critical to know business drivers for the work group or department being evaluated. Not understanding the business needs of users and not aligning our offering of information and knowledge resources to meet those needs makes it unlikely that we will achieve a positive ROI. On the upside, estimating ROI deepens our understanding of how we are managing information and knowledge resources to help various stakeholder groups achieve their business goals. This exercise opens up new opportunities for conversation and collaboration with peers and management and gives us leverage when negotiating for resources and funding. Looking back at which resources deliver the greatest benefits and those that failed to deliver the anticipated return sharpens our strategic focus on how to shape the portfolio of resources and services offered in the future. 1). Successful Information Companies Look for Revenue in All the Right Places, According to Outsells 2011 Outlook. Press release. January 12, 2011. http://www.outsellinc.com/press/press_releases/report_outlook_2011

Checklist for Creating a Return-on-Investment Business Case

Establishing a defensible Return on Investment (ROI) case or Cost-Benefit Analysis requires careful collection of cost data, information on business use of the product or service being evaluated and value received. Following the methodology described here ensures that you include essential components.
Decide which product or service will be evaluated for ROI Note why this particular product or service is selected for evaluating ROI Review current organizational goals and determine what goals are supported by the product or service you are evaluating Gather all cost information related to the product or service for specific work groups or business units. If using only invoiced costs (i.e., no costs related to staffing or IT support), make sure this is noted in your assumptions section of the final report. If determining ROI for a product or service developed by knowledge professionals, calculate staff time and costs for development and maintenance, production costs and delivery costs. Collect usage data (by department or work group) as context Determine how to collect user feedbacke.g., surveys or interviews, and prepare the survey instrument or questionnaire to elicit user input on applications and value attributes related to time, money, knowledge of the business or competitors or marketsdepending on organizational goals and priorities Analyze survey or interview responses and classify benefits to users Choose appropriate method (formula such as ROI or anecdotes and testimonials) for linking costs to benefits Consult with internal departments, e.g., HR, to estimate benefits or savings on average labor costs or Finance for savings related to reduced time to market Prepare a report or presentation for management that includes: Description of product/service evaluated Costs related to that product/service Information on users and level of use Information on business applications Benefits to users (from surveys and interviews) Estimate of financial benefit relative to cost (or anecdotes of specific wins) List of assumptions used in the analysis Track investments and benefits over a period of years in order to create models for productivity, quality or profitability