Vous êtes sur la page 1sur 20

Gartner Reprint

9/25/18, 4*51 PM

Licensed for Distribution

Licensed for Distribution

How to Operationalize Machine Learning and Data Science Projects

Published 3 July 2018 - ID G00333499 - 21 min read By Analysts Erick Brethenoux, Shubhangi Vashisth, Jim Hare

The democratization of machine learning platforms is proliferating analytical assets and models. The challenge now is to deploy and operationalize at scale. Data and analytics leaders must establish operational tactics and strategies to secure and systematically monetize data science efforts.


Key Challenges

Organizations often start their machine learning efforts as science projects, without rst understanding the real business issues and challenges faced by their business teams.

Many organizations struggle when it comes to systematically productizing machine learning results, as the process of deploying analytical insights into production is still seen as an art.

Although it is critical to deliver technically sound advanced analytic models, data scientists and data engineers often don't have the right skills to successfully operationalize models.

Many organizations struggle when it comes to systematically productizing machine learning results, as the production process is either overlooked or left solely to the DevOps team.

Even if successfully productized, machine learning efforts still face signicant maintainability and scalability challenges.

Gartner Reprint


9/25/18, 4*51 PM

To systematically leverage their data science and machine learning efforts while strengthening their analytics strategy, data and analytics leaders should:

Enhance the business value of analytical deployments while prioritizing use cases by establishing close, ongoing dialogue with, and explicit buy-in from, their business counterparts.

Ensure the integrity (technical and economic), transparency and sustainability of deployed analytical models by establishing a systematic operationalization process.

Maximize operationalization success by securing the help of domain experts, process engineers, IT professionals and business practitioners, in addition to existing data science talent.

Minimize the heavy technical debt and complex maintenance procedures of deployed machine learning models by monitoring and revalidating their business value on an ongoing basis.

Strategic Planning Assumptions

By 2020, 50% of organizations will lack sufcient articial intelligence and data literacy skills to achieve business value.

By 2021, 80% of data science labs that have failed to establish measurable operationalization practices will be outsourced.


According to the Gartner Data Science Team survey, 1 (#dv_1_the_gartner) conducted at the end of 2017, even within organizations beneting from the expertise of mature data science teams, less than half of data science projects end up being fully deployed. Another survey (Data and Analytics Trends) conducted in 2017 points to the fact that this phenomenon is not unique to data science, and that deploying AI projects into business processes or applications remains the principal barrier to delivering business value (see Figures 1 and 2).

Gartner Reprint

9/25/18, 4*51 PM

Figure 1. More Than Half of the Data Science Projects Are Not Fully Deployed

Gartner Reprint 9/25/18, 4*51 PM Figure 1. More Than Half of the Data Science Projects Are

Source: Gartner (July 2018)

Gartner Reprint

9/25/18, 4*51 PM

Figure 2. The Main Barrier to Delivering Business Value Is the Lack of Successful Productizing Projects

Gartner Reprint 9/25/18, 4*51 PM Figure 2. The Main Barrier to Delivering Business Value Is the

Source: Gartner (July 2018)

Machine learning provides a signicant business (that is, monetary) impact for organizations when analytical assets are deployed within business processes — that is, when analytical assets are consumed by people or processes. An example would be when a call center operator makes a recommendation in real time to a customer (a recommendation promoted by an analytical propensity model), or a machine accepts or declines millions of credit card transactions every

Gartner Reprint

9/25/18, 4*51 PM

minute (acceptance calculated in microseconds by an analytical fraud detection model). Embedding those models inside appropriate business processes delivers that value. For those organizations that have fully deployed and implemented their data science efforts, the return on investment has been signicant. 2 (#dv_2_ve_ways)

The focus of data science teams has traditionally been on developing analytical assets (see Figure 3), while dealing with the operationalization of these assets has been an afterthought. Many reasons underline that oversight:

Lack of a process-centric understanding from the data science team

Unwillingness to deal with the operational aspect of advanced analytics, along with a misunderstanding of their practical business impact

Lack of a formal operationalization methodology

A decit of formal communication channels between the data science team, the IT operational team and the line-of-business stakeholders

Machine learning operationalization is a critical step in aligning analytics investments with strategic business objectives — the "last mile" toward business value.

The focus of data science teams has traditionally been on developing analytical assets (see Figure 3), while dealing with the operationalization of these assets has been an afterthought.

Making sure that analytical models (that have supported concrete decisions) can be legally traced, monitored and secured requires discipline and management capabilities. Deploying analytical assets within operational processes in a repeatable, manageable, secure and traceable manner requires more than a set of APIs and a cloud service; a model that has been scored (executed) has not necessarily been managed.

The current analytical open-source movement is producing a wide range of analytical assets that will eventually have to be consumed — but current open-source deployment techniques provide "dissemination means" not "managed production means." From that perspective, containerization

Gartner Reprint

9/25/18, 4*51 PM

techniques are efcient and effective in pushing models into production, but they do not relieve the organization from adopting a formal operationalization process.

In the last decade where the "industrial" operationalization of analytics has started to boost the competitiveness of early adopters, we have learned, often the hard way, that publishing a model is not enough. That "publish moment" is followed by many steps that make the analytical model operationalization cycle as important as the analytical model development cycle.

But establishing best practices beyond the analytical asset development process requires much more than laying out a few operationalization steps; it starts with understanding the organization's business priorities, and then systematically applying a model life cycle discipline.

Figure 3. A Battle-Tested Analytical Model Development Process Cycle

Gartner Reprint 9/25/18, 4*51 PM techniques are ef fi cient and effective in pushing models into

Source: Gartner (July 2018)


Establish a Close, Ongoing Dialogue With Business Counterparts

Gartner Reprint

9/25/18, 4*51 PM

No data science project should be started without considering its nal business impact. 2 (#dv_2_ve_ways) Line-of-business stakeholders should be able to clearly articulate what are the tangible business benets they are expecting from the analytical asset to be developed. What is the business problem that the business is trying to tackle? Who is the primary consumer of the model? What is the business process that will host that model? How will the impact of implementing the model be measured? All while leaving room for unanticipated outcomes and avenues for further exploration.

The Gartner Business Value Model can be used to establish the appropriate set of metrics. 3 (#dv_3_the_gartner) The Gartner Business Value Model is a structured framework and denition of nonaccounting metrics that can be applied generically to help organizations identify how their business activities will impact nancial performance. Ultimately, "hard core" business metrics should be derived as, in practice, they are the most effective: for example, "reducing customer churn by 2%," "reducing incoming calls to the call center by 7%," "improving cross-selling on item X and item Y by 15%."

In addition to clearly identifying the business benets of the data science project, the exercise has another goal. That second goal is to bridge the gap between the data science team, the business stakeholders and the IT engineering practitioners and to better align those three entities through an effective communication process. Their alignment will be critical during the operationalization phase and will guarantee the sustainability of the production efforts.

Finally, model explainability is a concept that is gaining in importance as more organizations productize machine learning models. Model transparency is critical to sound model operationalization. It is essential in order to determine what went wrong when insights are starting to deviate, to be able to explain why and how an insight was derived, and simply to secure the trust of decision makers.


Identify the team that will support each individual data science project — a team including data science, business and IT stakeholders (specically IT engineering practitioners).

Collectively develop candidate use cases, while identifying the quantiable business outcomes from each of these use cases. Agree on measurable business impacts before gathering the rst set of data.

Gartner Reprint

9/25/18, 4*51 PM

Prioritize the selected data science projects based on their technical feasibility and measurable economic viability — that is, on their ability to be properly measured while in production.

Establish a Systematic Operationalization Process

The model development cycle (illustrated in Figure 3) is a variation on the Cross-Industry Standard Process for Data Mining (CRISP-DM) methodology that has been used by data science practitioners for two decades. That methodology, simple and powerful, imposes a development discipline that promotes the integrity and robustness of the resulting analytical models. The development cycle has always been the glamorous part of the full analytical process, and developers, especially coders (now fully enabled through robust sets of development tools), have thrived on that rst cycle.

The model operationalization cycle, the "industrialization" of analytics, on the other end is less glamorous, but it is where business value is actually delivered. Neglecting the operationalization cycle has often prevented organizations from fully realizing the business value of their analytical efforts, while generating distrustful attitudes toward analytics.

Operationalization Cycle Functionality

That operationalization cycle has not been as formalized as the analytical model development cycle; but successful organizations have adopted various technologies to provide the foundation to manage operational decision services in a reliable, repeatable, scalable and secured way. That foundation should provide:

Catalog. A "centralized" way to store and secure analytical assets to make it easier for analysts to collaborate and to allow them to reuse or exchange models or other assets as needed (it could be a secured community or a collaboration space).

Governance. Protocols to ensure adherence to all internal and external procedures and regulations, not just for compliance reasons but, as an increasing amount of data gets aggregated, to address potential privacy issues.

Capabilities. Automated versioning, ne-grained traceable model scoring and change management capabilities (including champion/challenging features) to closely test, monitor and audit analytical asset life cycles from a technical as well as a business performance

Gartner Reprint

9/25/18, 4*51 PM

perspective (through key performance indicators [KPIs]).

Coherence. Establish simple protocols to provide functional bridges between the development and operationalization cycles; enhance cooperation and consultations between development and operationalization teams; provide efcient "translation" services between data science and lines of business; improve the transparency of deployed analytical assets.

Operationalization Cycle Process

Once a model has been published out of the development cycle, its introduction into the operationalization cycle takes place in two main phases: the "release" phase and the "activation" phase, depicted in Figures 5 and 6. The goal of the release phase is to set the published model free inside a guarded perimeter to verify that the assumptions made while developing the model still hold true in the real world of the actual business process. Once the model has been tested in the real world and is performing as intended, it is ready for prime time in the activation phase. The goal of the activation phase is to activate that model within existing business processes across the organization at the endpoints identied (and validated) in the release phase of the production process.

Depending on their project size and complexity, not all organizations undergo those two phases. It is also possible that some organizations (especially while implementing real-time model operationalizations) might integrate the release phase as part of their development cycle; but a large majority of analytically mature organizations productize models through those two phases.

It is also important to note at this point that models in production are often part of an ensemble of models working together to provide insights within a particular process. Take, for example, a telco process determining a client's propensity to churn. One model computes the propensity to switch from one carrier to another, another model tests the desire of this prospect for a certain device, and another combines those propensities to nally feed a model that will deliver a propensity score.

Release Phase: Testing Models in Business Conditions

Release phase: The goal is to set the published model free inside a guarded perimeter to verify that the assumptions made while developing the model still hold true in the real world of the actual business process. We have identied six steps in this process (see Figure 4):

Gartner Reprint

9/25/18, 4*51 PM

Figure 4. Release Phase of the Operationalization Cycle

Gartner Reprint 9/25/18, 4*51 PM Figure 4. Release Phase of the Operationalization Cycle Source: Gartner (July

Source: Gartner (July 2018)

Model release. The model is promoted to the release step, ready to be undertaken by the operationalization team and labeled as a "candidate" model (that is, development-vetted, but not yet fully production-ready).

Endpoint identication. Validation of the decision points where the model will be delivering its insight. Those analytics endpoints could be within an existing application, within a business process, as part of an ensemble of models, as an input to another decision modeling mechanism (like a business rule).

Parameter testing. Target business processes might be subject to technical constraints where the velocity, shape, volume and quality of the input data might not exactly align with the data in sandboxes used to develop the models. This step aims at testing that alignment.

Integration testing. Providing that the expected data matches the development assumptions, integration assumptions (that is, REST APIs, microservices call, code integration) also have to be tested to ensure proper performance.


Gartner Reprint

9/25/18, 4*51 PM

Instantiation validation. As models in production are often part of model ensembles, even slight variations in those elemental models (such as a propensity to buy models instantiated across multiple states or regions in the same country) can produce radically different results.

KPI validation. Model performance should not only be measured against technical parameters (such as precision) but also against the agreed-upon business KPIs (measurable business outcomes) set forth as early as the "business understanding" step in the development process (as indicated earlier).

Depending on the outcome of the KPI validation step, three paths are possible:

The model fails due to insufcient data or ill-calibrated assumptions about that data — in this case the process can proceed back to the "new data acquisition" step in the development process.

The model fails to deliver the appropriate business outcome, and in this case it might be wise to re-evaluate the original business assumptions through the "business understanding" step back at the beginning of the development cycle.

The model delivers as promised and is ready to move to phase two of the operationalization cycle: the activation phase.

Activation Phase: Operating Models in Real Business Conditions

Activation phase: Now that the model has been tested in the real world and is performing as intended, it is ready for prime time. The goal is to activate that model within existing business processes across the organization at the endpoints identied (and validated) in phase one of the production process: the operationalization process. We have identied seven steps in this process (see Figure 5):

Gartner Reprint

9/25/18, 4*51 PM

Figure 5. Activation Phase of the Operationalization Cycle

Gartner Reprint 9/25/18, 4*51 PM Figure 5. Activation Phase of the Operationalization Cycle Source: Gartner (July

Source: Gartner (July 2018)

Management and governance. As mentioned in the functionality factors above, once a model is ready for activation, it should be cataloged, documented, versioned (including the original set of variables that has been used to develop it). That model should also be submitted to the governance rules adopted by the data science team and the "production committee" (and in particular application product managers). Management rules should also be established to control the exchanges between the production and the development teams (such as when to retune models, expose challenger models and levels of model explanations).

Model activation. This step is the hand-off of the operationalization-process-validated models to the production team as activated models. At this stage the models are "production ready,"

Gartner Reprint

9/25/18, 4*51 PM

fully documented and compliant with the governance rules.

Model deployment. Depending on how those models will be executed (on-premises or in the cloud, or both, leveraging streaming infrastructures and parallel processing mechanisms like Spark), measures need to be taken to guarantee the smooth processing of the transactions leveraging the model (or the ensemble the activated model is part of).

Application integration. Insights are delivered through decision endpoints, the large majority of which are part of existing applications. Extra coding is sometimes necessary to properly embed a model, or its input, within an application, a business process or another set of insights, to enhance a business outcome. This is where the model will nally deliver its business value.

Production audit procedures. Model telemetry, along with various performance metrics in terms of accuracy, response time, input data variations and infrastructure performance instrumentation, including possible implementation of software agents, have to be implemented to gather the necessary data to monitor models in production. Various instruments, such A/B testing methods, multivariate testing, multiarmed bandits (MABs) and shadow models, can be implemented to judiciously appreciate the performance of models.

Model tracking behavior. Alerts and monitoring methods have to be established to track the performance of models in production. Performance thresholds and notication mechanisms are implemented in this step to systematically ag any divergence or suspicious behavior.

KPI validation. Extended from Phase one, and fed by the two previous steps, the KPI validation step consistently measures the business contribution of the models (or ensemble models) in production. The idea is to get as precisely as possible the business value that can be attributed to the model. To determine that value, data from the "production audit procedures" step should prove invaluable.

Monitor, Re-evaluate, Tune and Manage Models on an Ongoing Basis

From this point, the models remain in the operationalization cycle as long as they are performant, meet attribution thresholds and deliver business value derived from the models in production. As long as models are performing, they keep their place in the production cycle. Otherwise, management rules are in place in the "management and governance" step to re-evaluate those

Gartner Reprint

9/25/18, 4*51 PM

models in the light of changing circumstances captured in the three previous operationalization steps. Those models are then sent back into the development cycle process.

The combined processes are illustrated in Figure 6.

Figure 6. Combined Model Development and Operationalization Cycles

Gartner Reprint 9/25/18, 4*51 PM models in the light of changing circumstances captured in the three

Source: Gartner (July 2018)


Machine learning systems carry a signicant technical debt 4 (#dv_4_d_sculley) that has to be addressed upfront. To confront the problem early, data and analytics leaders should:


Gartner Reprint

9/25/18, 4*51 PM

Start as early as the development process to address the KPIs that will govern the success of analytical models in production.

Establish a well-instrumented and disciplined model operationalization process to scrupulously track the behavior and performance of machine learning models deployed across business processes.

Heed the management and governance step in the operationalization process as its rules and procedures will eventually dene the success or failure of your deployed analytical assets.

Secure the Help of Nonanalytical Personnel

If properly executed, the identication of measurable data science use cases has brought together a team that will secure a successful project deployment.

In addition to traditional data science characters, including (at minimum) data scientists, data engineers, software engineers and business experts, 5 (#dv_5_stafng_data) putting a decision service into an application in production (that is, releasing and activating analytical assets) requires additional skills. These include system administrators, source systems experts, application developers and process engineers (see the "Data Science Operationalization Skills" section below).

Data Science Development Core Skills

Data scientists:

Critical key staff members that can extract various knowledge from data, have overview of the end-to-end process and can solve data science problems. Data scientists monitor technical models' performance, and challenge and retune models as necessary.

Data engineers:

Make the appropriate data accessible and available for data scientists and, as such, can be instrumental in big productivity gains. Data engineers ensure the alignment between operational data and model training/validation data.

Software engineers:

Needed sporadically when custom coding is required (for special visualization, data integration or

Gartner Reprint

9/25/18, 4*51 PM

deployment of certain results, for example). Software engineers should closely cooperate with product managers for analytical asset integration.

Business experts:

Individuals who understand the business domain really well. This can sometimes be the business leaders, or sometimes a range of key specialists. Business experts should also bridge the gap between the data science team and business executives.

Data Science Operationalization Skills

Source system experts:

Those who have intimate knowledge of the data ows and sources at the business application level, and of the data necessary for the development and proper running of models.

System architects:

Have a clear picture of the infrastructure supporting the processes, systems and applications that will integrate machine learning models.

System administrators:

Harmonize and tune IT operations to optimize analytic assets performance at runtime.

Application developers:

Have a deep understanding of the business applications' inner workings and end users' interactions with those business solutions.

Process engineers:

Understand the complexity and middleware relationships of the various business processes leveraging analytical assets.

Beyond the nine roles mentioned so far, a tenth is starting to appear within successful data science teams: the business translator. 6 (#dv_6_the_age) The business translator could be a business-savvy data scientist, an analytically minded business person or a process engineer (process modelers or business analysts focused on process design) mindful of business optimization opportunities derived from analytical assets. Business translators usually emerge,

Gartner Reprint

9/25/18, 4*51 PM

rather than being nominated; they often become the common denominators (and conductors), not just between the various members of the development and operationalization teams but also to the executive stakeholders.


Recruit operationalization core talents (from the IT department) within the data science project as early as the business understanding phase.

Spot and quickly promote business translators who will become the trusted gureheads of the data science extended team.

Provide the data science development team with a clear understanding of the business and production system implications (and potential limits) of their efforts.

Establish a "production" committee, including the head of the data science development team, a designated data science operationalization leader, the business translator, and the application product manager to oversee the full data science cycle from model inception to model retirement.

Monitor and Constantly Revalidate the Business Value Delivered by Machine Learning Models in Production

The KPI validation step of the operationalization process needs to be constantly revalidated while the model is in production. Beyond the organization's changes of business strategy and tactics, market conditions can shift, and customer behaviors or competitive pressure can evolve, impacting the business performance of models. While all of those factors can contribute to models' degradation, others to watch for include: concept drift (where seasonality, decision impacts, local data particularities, dynamic data transitions, and so on can induce a change in class distribution); bias reinforcements (where decisions are increasingly subjective through models' self-fullling recommendations); feedback loops (where models inuence their own behavior as they evolve over time 7 (#dv_7_six_pitfalls) ).

KPIs should also be subjected to a higher level of scrutiny, not just within the business process context of the active model, but across the business processes that are impacted by the decisions inuenced by models (or collection of models). This will avoid the local optimum problem where a

Gartner Reprint

9/25/18, 4*51 PM

few decisions are optimized for local decisions, eventually leading to global issues. By auditing the development, renement and usage of models and other assets throughout their life cycle, the formalization of the operationalization cycle provides a bigger picture advantage: more consistent organizational decisions.


Secure the commitment of the business stakeholders at the business understanding stage for the continuous validation of the business KPIs while the model is in production.

Track the models' (or model ensembles') degradation signals by constantly monitoring the business indicators inuenced by productized models.

Continuously measure and harmonize the overall business impact and collective contribution, within or across processes, of integrated model collections.


  • 1 The Gartner Data Science Team Survey (January 2018) shows that a majority of data science projects are not deployed (see Figure 1). The survey also shows that almost 60% of data science teams are now measured on business KPIs — that is, on the real business impact of their models in production. Results presented are based on a Gartner study conducted on data science teams. The research was conducted during November and December 2017, with 302 respondents in the U.S., U.K., Germany and France. Participating organizations were screened to have at least three years of data science capabilities. The respondents were screened to have both technical as well as strategic involvement in data science initiatives.

2 "Five Ways Data Science and Machine Learning Deliver Business Impacts"

3 "The Gartner Business Value Model: A Framework for Measuring Business Performance"

Advances in Neural Information Processing Systems 28 — Proceedings (NIPS 2015)

Gartner Reprint

5 "Stafng Data Science Teams"

9/25/18, 4*51 PM

7 "Six Pitfalls to Avoid When Executing Data Science and Machine-Learning Projects"

Note 1 Analytical Assets

Analytical assets include machine learning (predictive) and mathematical models and precalculated propensities (such as predictive scores).

By Erick Brethenoux, Shubhangi Vashisth, Jim Hare

Gartner Reprint

9/25/18, 4*51 PM

© 2018 Gartner, Inc. and/or its afliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. and its afliates. This publication may not be reproduced or distributed in any form without Gartner's prior written permission. It consists of the opinions of Gartner's research organization, which should not be construed as statements of fact. While the information contained in this publication has been obtained from sources believed to be reliable, Gartner disclaims all warranties as to the accuracy, completeness or adequacy of such information. Although Gartner research may address legal and nancial issues, Gartner does not provide legal or investment advice and its research should not be construed or used as such. Your access and use of this publication are governed by Gartner’s Usage Policy. Gartner prides itself on its reputation for independence and objectivity. Its research is produced independently by its research organization without input or inuence from any third party. For further information, see "Guiding Principles on Independence and Objectivity."

Gartner Reprint 9/25/18, 4*51 PM © 2018 Gartner, Inc. and/or its af fi liates. All rightsGartner’s Usage Policy . Gartner prides itself on its reputation for independence and objectivity. Its research is produced independently by its research organization without input or in fl uence from any third party. For further information, see " Guiding Principles on Independence and Objectivity ." About Careers Newsroom Policies Site Index IT Glossary Gartner Blog Network Contact Send Feedback © 2018 Gartner, Inc. and/or its Affiliates. All Rights Reserved. https://www.gartner.com/doc/reprints?id=1-5EX8T0M&ct=180906&st=sb Page 20 of 20 " id="pdf-obj-19-54" src="pdf-obj-19-54.jpg">

© 2018 Gartner, Inc. and/or its Affiliates. All Rights Reserved.