Académique Documents
Professionnel Documents
Culture Documents
Engineering
Microsoft IT
Published: 2016
Copyright Information
The information contained in this document represents the current view of Microsoft Corporation on the issues discussed as of the date of publication.
Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft
cannot guarantee the accuracy of any information presented after the date of publication.
This white paper is for informational purposes only. Microsoft makes no warranties, express or implied, in this document.
© 2016 Microsoft Corporation. All rights reserved.
1
Contents
1 Acknowledgements ............................................................................................................... 5
2 Forward ................................................................................................................................... 6
3 Introduction............................................................................................................................ 8
5 Our approach........................................................................................................................ 13
2
5.2.3 Runtime detection and prevention ............................................................................................ 20
8 Conclusion ............................................................................................................................ 41
3
9.1.1 SDL ......................................................................................................................................................... 43
4
1 Acknowledgements
Authors
Anmol Malhotra
Talhah Mir
Contributors
Aaron Clark
Glenn Leifheit
Jonathan Griggs
Manish Prabhu
Shoham Dasgupta
Reviewers
Andrew Marshall
Brijesh Desai
Bruce Jenkins
Dave Christiansen
Karen Luecking
Michael Howard
Ralph Hood
Rob Polly
5
2 Forward
2.1 Bret Arsenault
The pace at which business is moving today requires that
technology be more agile, to keep up with the rapidly
evolving needs of companies and organizations around the
world. Technology companies need to ensure that security is
keeping pace with the speed of software, and address the
security gaps created by moving to agile workflows. While
security has always been a primary focus for us at Microsoft,
today’s threat landscape demands that we adapt the way we
address security as a business. We work constantly to ensure
that security is top-of-mind for everyone at the company. It’s
clear that to build a strong security posture, we must engage
everyone from our engineering teams all the way through to
Corporate Vice President
our senior leadership.
and
Chief Information Facing new pressures, modern engineering teams are leading
Security Officer the transformation to agile development and are delivering
Microsoft what customers need, as they need it. With an agile
methodology, Microsoft IT provides the flexibility and speed
with which solutions are released in as short a time as
operationally feasible. To properly land the value of these
accelerated development cycles, companies need to ensure
that they have the right security processes and automated
tools in-place to address new risk exposure that is created by
a high-speed development environment. Importantly,
leadership must also make sure we are creating a security
culture and driving the right behavior with engineers -
enabling them to succeed, while delivering the best possible
products to our customers.
6
2.2 Sue Barsamian
The rapid growth of the app economy and the increasing
pressure to innovate has put the software developer in the
driver’s seat in modern IT. Developers are now deeply involved
in every part of the software development lifecycle as the
boundaries between software and hardware continue to blur
and infrastructure moves to the cloud. Developers are now
responsible for driving innovation and keeping up with the
increasing need for a faster time-to-market. This notion has
challenged the traditional development lifecycle, pushing for
more agile processes and greater collaboration across
development, QA, security, and operations. Securing the
software development process has never been easy, but in the
Senior Vice President and midst of such seismic shifts in software development,
General Manager, application security is more challenging than ever.
Security Products
In this faster-paced new development lifecycle, security
Hewlett Packard Enterprise organizations must adapt to becoming a natural part of the
development process or they risk getting in the way. Even
worse, they could be left behind as applications become more
complex and more vulnerable than ever. The Microsoft ISRM
team has taken a unique and aggressive approach to this
challenge by partnering with development organizations to
build security into the process while staying true to the
discipline of the acclaimed Microsoft Security Development
Lifecycle (SDL). By teaming with us in HPE Security Fortify,
Microsoft has enabled effective, unobtrusive application
security automation at scale that provably secures their
applications and saves time and money during development.
We are excited to share the experience and lessons learned by
bringing together the world’s largest software company and
the leading application security solution. Together we have
built a world class Application Security program that could
provide a model for helping you secure the applications that
run your business.
7
3 Introduction
3.1 Setting the scope
Microsoft’s ISRM organization, which is part of Microsoft IT, has a mission to ensure that all of
the company's information and services are protected, secured, and available for appropriate
use through innovation and a robust risk management framework. Microsoft is committed to
building and implementing best-in-class security programs and processes and is constantly
working to reduce exposure to cybersecurity risks.
ISRM supports Microsoft’s overall security mission by providing key security services that help to
protect Microsoft’s corporate systems, services, data, and users. The service lines through which
we deliver these services include risk management, threat and vulnerability management,
identity and access management, security and incident management, and security monitoring.
Across Microsoft IT and throughout the company, the ISRM team is continuously evolving the
security strategy and taking actions to protect key assets and the data for our organization. One
primary focus for the team is to protect line-of-business (LOB) applications for Microsoft IT.
ISRM drives the SDL for IT applications.
For more detailed resources related to the SDL model, including books and websites, see
Appendix A.
The SDL defines the standards and best practices for providing security and privacy for new and
existing LOB applications currently under development or being planned for development. IT
LOB applications are a set of applications that are vital to running an enterprise organization
including accounting, legal, finance, human resources, payroll, supply chain management, and
resource planning applications, among others.
8
maintain development infrastructure that support continuous integration/continuous delivery
(CI/CD).
Just as the SDL is agnostic to any specific development methodology, practice, or tool, the
concepts in this showcase whitepaper apply to this modern engineering world, broadly
speaking. Our goal is to empower modern engineers with a set of tools, guidance, and
processes to empower them to write, deliver, and maintain more secure applications and
services.
With an agile methodology, Microsoft IT provides the flexibility and speed with which solutions
are released in as short a time as operationally feasible. Agile teams are receiving faster
customer feedback though an iterative design and feature approach, and mature agile teams
often release every day or even multiple times a day. While this is great for business
enablement, this poses a huge challenge for security in terms of how to effectively and
efficiently drive security and privacy in these CI/CD scenarios. For example, consider a security
process that takes two weeks to complete sign off on a release. This model plainly fails when
applied to an agile application which may take, for example, a single week to ideate, create, and
be ready for release. Additionally, the more traditional security approach – to review every
application release – worked well when release cycles spanned months, but this approach is
highly inefficient against modern engineering practices where schedules are much more
condensed.
Given the ubiquity of customer data and critical data, security and privacy are of utmost
importance to consumers. For example, would you feel comfortable using a banking application
on your mobile phone if security and privacy aspects were overlooked by the engineers?
Security can be friction, but it can’t be completely ignored either.
9
So, this is our challenge:
“How can we make security low friction (efficient) while maintaining its effectiveness
in this new world of modern engineering?”
This challenge demands that the security culture and approach are modernized and adapted for
shorter release cycles and sprints. Security teams must support decentralized security processes,
but they must also drive greater automation and move beyond point-in-time assessment
practices. Under these modern engineering challenges, they need to adopt a solution that can
scale and that can provide continuous assurance.
We will also discuss some of the key trends we see in application security that have started to
redefine not just how we look at application security but how application security processes
such as the SDL are completely re-scoped. For example, with development roles merging with
operations, application security processes also need to evolve to effectively secure the Ops in
DevOps. Finally, we’ll close by sharing some of the lessons we learned in this journey of driving
security into modern engineering practices.
We don't want to single out DevOps, or say that the concepts discussed in this whitepaper work
only with DevOps, because it can be difficult to identify a common definition of DevOps. For the
10
purpose of this discussion, the important aspect of the DevOps movement is to recognize
certain fundamental principles that define for us what we consider “Modern Engineering.” For
more information, see Appendix A.
The principle focus is around people and culture. Development roles and operational roles have
merged, and the expectation from a service or software engineer is not just to develop and test
code, but also to be able to deploy and operate the code effectively. Engineers have full control
over the runtime environment so they can build with predictability. This avoids the "throw it over
the wall" mentality that can occur when development hands off to operations.
Speeding up the pace of innovation by shortening the release cycles using agile
methodology, and maintaining control over the entire technology stack (from code
through to the infrastructure and operational practices.)
Enabling faster feedback loops from customers which can result in application features
and bug fixes.
Considering these principles as key components of modern engineering, anything that gets in
the way of this process is friction. More traditional approaches to software security assurance,
that rely on gates, are seen as friction in modern engineering practices. Here are a few examples:
11
4.2.1.1 Manual security assessments
Typical white box code or black box security assessments last anywhere from few days to few
weeks depending on the size and complexity of the application. With application development
sprint cycles shrinking to days, just engaging security teams and scheduling code reviews is a
challenge, let alone reviewing anything on time. This time-consuming process is
counterproductive for the needs of business and engineering teams because it is clearly a speed
bump for fast-paced release cycles.
12
4.3.2 Intelligent automation
It's natural for any security team to look at some of the challenges articulated thus far and
assume that automation is the solution. It’s correct that automation is a big part of any solution.
However, automation is very easy to get wrong and very hard to get right. A common response
is for security teams to find a wide range of automation tools and “throw them over the wall” at
engineers. In our experience, this doesn't always work for two primary reasons:
It’s important to understand that the solution to the challenge of maintaining CI/CD and security
compliance is not just automation. We needed intelligent automation that can yield the kind of
information that engineers can act on to drive positive impact.
5 Our approach
Our approach to the challenges is itself an agile one in which we are evolving the solution
through incremental feedback and updates. We are on this journey to evolve and to enhance
how we integrate security with modern engineering and how we enable our engineering teams
to succeed. The following are the four core tenets around which we are building our solution:
13
5.1 Knowledge management
Having a sound knowledge management process is one of the key aspects of any effective
security solution. Underneath any SDL process are the requirements, which we call Technical
Control Procedures (TCPs). Defining clear technical requirements for engineering teams is the
foundation of an SDL program, and over the years our Control Assessment Library and
Methodology (CALM) board has continuously refined our requirements.
14
FIGURE 3 AREAS OF COVERAGE FOR TCPS
TCPs are built around positive instead of negative attributes and are focused towards
engineering teams. We identify what the known positive controls are that a system should
implement to be secure. In our taxonomy of a TCP, we define (among other attributes):
Why to implement Align to company security standard(s) and policy for compliance
measurements.
The following are few example categories from the TCP library and their underlying procedures:
15
TABLE 1 TCP CATEGORIES
TCPs are the backbone of everything we drive with the engineering teams. The following are
areas of the SDL to which TCPs are integral:
16
FIGURE 4 ALIGNMENT TO COMPANY POLICY
We started piloting a guidance factory approach in 2015. In this pilot, an analyst posts quick
guidance in response to questions from the engineering team to a SharePoint site. We
17
leveraged that site as a key input to our TCP maintenance process and continue to evolve them
over time. With this agile approach, engineers receive up-to-date guidance that delves deeper
than our currently published TCPs in a timely fashion, and in the long term we leverage the
guidance produced into the TCPs.
5.2 Automation
To enable teams for CI/CD and continuous assurance for security, automation must be a focal
point in the solution. In support of this, our focus is to develop low-friction security services. We
have identified the following three classes of services as key building blocks for our solution to
deploy and maintain:
18
consistent in terms of their rules and can cover millions of lines of code in a short period of time.
This type of security service can provide the scale and code coverage that we needed. In fact,
combining static analysis with human code review is one of the best ways to ensure all aspects
of the application have been vetted. By leaving the common and tactical issues to the
automated tool, we found that our most experienced security SMEs can focus their time on
other important aspects such as design issues and business logic issues in higher-risk
applications.
The following are security principles that dynamic analysis addresses and reasons why we think
it is an essential part of the overall security automation strategy:
5.2.2.1 Hygiene
Dynamic security analysis helps engineers to test an application during runtime. As they test
their application workflow, dynamic security analysis is another tool or service in their arsenal to
leverage and verify that the application has fixed security defects that only surface when you run
the application.
19
5.2.2.2 Coverage
Dynamic security analysis can uncover different types of issues that may be non-discoverable to
static analysis or can help validate issues discovered during static analysis. For example, the
impact of environment and server-level configuration issues are easier to detect by running
dynamic security scans. Potential information leaks that can occur as a result of interaction with
the application can also be easier to discover through dynamic analysis versus static analysis.
These issues can have severe impact and may be overlooked without effective dynamic
scanning.
Runtime detection and protection for application – a technology that we think is still maturing in
the industry – provides continuous assurance during application runtime. This is a nascent space,
but it is a must-have service capability for the future and will become a crucial part of the overall
application security strategy.
We have identified the following risk areas that will benefit from a runtime detection service:
20
5.2.3.3 Legacy applications
Runtime detection can be an incredibly powerful tool to protect legacy applications which may
no longer have any engineering support and are “sunset” applications. Such applications tend to
use older technology stack which were less secure and easy targets for attacks. There could be
cases where some applications are so old that they might not have gone through any software
assurance processes whatsoever.
5.3 Implementation
5.3.1 Static analysis
There is no such thing as perfect security, but the pursuit of thoroughness is important because
one vulnerability is all it takes to compromise an application. In ISRM, we use HPE Fortify SCA to
conduct automated static security analysis of our applications. After evaluating many static
analysis tools, we chose Fortify SCA because of its thoroughness, its ability to facilitate
collaboration, and most importantly, its coverage against our TCPs.
Fortify SCA is a static application security solution. It traverses your code, identifies potential
security issues, and compiles a list for you to review. A human determines which issues are
serious flaws that must be fixed and which issues are false positives or low priority and can be
deprioritized or ignored.
Fortify offers two desktop IDEs to view and aggregate issues. Our engineers use the Visual
Studio plugin to view Fortify SCA results inside their development environment, while most of
our analysts use the Fortify proprietary IDE, Audit Workbench. Everyone accesses scan results
from the same Fortify Software Security Center (SSC) server, which ensures that all comments
and reviews are preserved from one scan to the next. Build engineers will configure regular
scans which will automatically upload results to the Fortify SSC server, ensuring everyone has
recent scans to work from.
We started the journey with Fortify in 2014, and we use the following three capabilities with
Fortify for our needs in ISRM:
Build integration
This is our most preferred method. We’ve integrated Fortify SCA static analysis with the
application build processes. Based on the schedule and release cadence, static analysis
21
runs as well. All the results from the scan are then automatically uploaded to the
Software Security Center (SSC) server.
Self-hosted scans
Self-hosted scans are one-time scans. This provides a scanning method for those
applications that either do not have a build environment for integration or that are one-
time applications. Examples are marketing campaign applications that focus on a specific
event or applications with a shelf life of less than six months, for example. These
applications leverage self-hosted scanning to perform Fortify SCA static security
scanning. Results are then uploaded to the SSC server. Engineers are encouraged to
utilize this feature so they can see scan results within their IDE before their code is
uploaded to build servers.
Static security analysis on-demand
On-demand static security analysis provides engineering teams with white glove
treatment. A central security team scans the code on behalf of the engineering teams
and helps the team triage the issues. This is similar to HPE’s Fortify on Demand service,
but it is run within the organization by our central security team. We developed this
capability primarily to handle venture integration scenarios. For example, this capability
comes in handy when your company inherits LOB applications as a result of an
acquisition and the engineering teams managing those applications have little or no
exposure to static analysis. You will need to hand hold them to get them started and
slowly move to self-hosted scans and finally to build integration.
22
5.3.3.1 Build track
The build track is where the heavy lifting occurs. We work with the build owner to ensure that
Fortify SCA is installed on their build environment and is configured and scheduled to run based
on their release cadence. We then run the first scan and hand off to the development track.
23
5.3.5 Shortfalls and opportunities
We faced several challenges when we began to deploy Fortify SCA in our large enterprise space.
The following are some of the shortfalls, including ways that Fortify SCA is helping us address or
will be addressing these shortfalls in the future.
Achieving better horizontal scaling is still an opportunity. However, the performance of the
Fortify SSC servers has improved greatly over the past several releases and made this scaling
less necessary.
While using Fortify SCA as our analysis tool, we found an opportunity to openly communicate to
the Fortify team about our needs as an enterprise organization along with our feedback. In
doing so, we’ve found tremendous value in moving beyond a customer/vendor relationship. The
partnership we’ve built with the Fortify team in our journey has yielded important advances in
our ability to secure our code. We believe our relationship with the Fortify team, and
continuously sharing information, has uncovered valuable new insights that not only helped us,
but helped Fortify as well.
24
allows our engineering teams to remain in compliance with our internal security policies without
slowing them down. This integration of Fortify within Microsoft Visual Studio Team Services
highlights one of the most beneficial achievements to come out of our relationship with the
Fortify team.
The goal with VSTS integration was not merely the basic use of Fortify in VSTS like you would
use any source repository system. This was about building a solution that allows minimal input
from the user and that would enable the teams to use Fortify with less friction in a deployment
phase. We wanted to make the Fortify scan feel more like doing a hosted build in VSTS.
Another significant aspect is the ability to enable Fortify scanning through a virtual machine,
which is preconfigured and ready for teams to use directly in Microsoft Azure. Teams can pick
the Fortify build machine out of the Azure store and it will have Fortify ready to go. Scanning
machines will spin up on demand instead of requiring dedicated machines that are only used
periodically for scans. This on demand service improves the efficiency of both the engineering
and security teams.
25
5.3.7 Dynamic analysis
In ISRM we use HPE Fortify WebInspect to conduct dynamic web application security testing. It
complements static code analysis performed through Fortify SCA, and both use a common
Fortify SSC server for management and reporting.
Our approach for deploying dynamic security analysis was pragmatic. We rolled out WebInspect
initially as an SDL requirement to help teams discover issues during application testing. We were
very aware of the tool fatigue issue our engineers raised, so we focused on static security
analysis first and then moved to dynamic analysis once we achieved the sufficient effective
adoption of static analysis.
In the first year, we encouraged all applications (with web interfaces) to run WebInspect by self-
scanning and uploading the results for compliance. Multiple application teams used this model,
and some lessons we learned included the following:
Low quality scans The quality of scans submitted to Fortify SSC were low. Either they
were incomplete scans or misconfigured scans. We realized that engineers must be
trained in the tool to run effective scans. For example, running a basic crawl without
configuring macros can yield incomplete or duplicative scans of the same resource.
Resources for scans Applications teams complained that there was a lack of dedicated
servers from which to conduct self-scans, particularly for larger applications and services.
These lessons paved the way to make changes in our dynamic security scanning strategy. We
decided to onboard higher risk applications to a dynamic security scanning service. This service
uses a security team to engage with the engineering team once and provide that team with
automated, scheduled macro-based scans. Because the scans are led and configured by a
26
security team, we are better assured of the quality of the scans and their results. We are also
evaluating use of APIs to further streamline scans scheduling and kick off process.
The right side of the diagram illustrates continuous dynamic scanning for high-risk
applications. These scans are run as periodic scheduled scans on the WebInspect
Enterprise sensors controlled from the WebInspect Enterprise server.
The left side illustrates dynamic scanning for all other applications. These scans are run
by engineering teams on their own infrastructure and are uploaded to the WebInspect
Enterprise server.
27
5.3.9.1 WAF
A WAF works by sitting in front of an organization’s web application stack. It analyzes all
incoming traffic for attack patterns and blocks any suspected malicious requests from reaching
the web application itself.
However, typical WAFs suffer from a number of limitations, including – but not limited to – the
following:
Utilizes signature-based pattern matching and blacklist filtering, which can be bypassed
as attacks become more sophisticated.
Lacks insight into application logic, data, and event flows because it does not have
application context.
Has high false positive rate from list of detected malicious activity.
Has relatively limited vulnerability category coverage.
Requires extensive coverage and maintenance for firewall rules to stay in sync with
application changes.
Even with these limitations, WAFs can provide a level of protection that justify their cost
(including operational maintenance) and can be an effective control in many scenarios.
5.3.9.2 RASP
RASP is a fairly new security technology. It is hooked into an application or application runtime
environment and can be capable of controlling application execution to prevent real-time
attacks as well as detection.
We have been evaluating a RASP technology from HPE Fortify called Application Defender to
provide runtime detection and protection. Our evaluation for RASP has been focused around
the following key features:
Enterprise management How does the technology scale for an enterprise deployment?
Performance and scalability How does the technology impact the application
performance?
Agent installation and maintenance How easy is it to install and manage agent
updates as new rules for attacks roll out? Does it require server restart?
Platform support Does the technology works seamlessly for on premise applications
and cloud applications, particularly Platform as a Service (PaaS)?
Ruleset quality What is the current ruleset coverage when mapped to our TCPs, and
what is the update frequency of the ruleset?
It’s clear that the deployment and evaluation of any tool that sits in a production environment
needs to be carefully vetted from all aspects listed above. We have been working closely with
the Fortify team to vet solution details and to work on bridging the gaps to make sure this
technology is ready for our needs.
The following are two key areas we see that require enhancements:
28
Azure PaaS support
Applications running in Infrastructure as a Service (IaaS) and PaaS (Web Roles)
environments can be supported, but the deployment automation is yet to be
determined.
App Defender also has challenges with uniquely identifying the Azure PaaS hosts
which prohibits effective monitoring.
Seamless agent upgrades
An update to the agent installed on the host application can require IIS and
application server restart, depending on the application type. This can be a significant
operational burden for engineering teams.
We plan to test the on-premise solution for App Defender. We are testing it against different
scenarios and applications to identify what it would take to deploy this technology in the least
disruptive way for the engineering teams. We intend to deploy an end-to-end scenario where
App Defender is configured in “Detection Mode Only” at first, and then integrate the feed into a
security information and event management (SIEM) technology such as HPE Security ArcSight.
Our goal is to directly enable our engineering teams with the detection capabilities that can be
incorporated into their telemetry systems to not only gauge the operational state of their
application but also the security posture.
A solution like App Defender can provide very advanced application monitoring capability. It is a
key component of enabling continuous assurance, and also helps share the responsibility of
security with the engineering teams which enables our modern engineers to gain the
appropriate security insight into their applications.
To address this shift, we’ve made it a priority to develop what we call the “Secure DevOps Kit for
Azure” which contains a set of tools, extensions, code snippets, and other automations.
Most specifically, the Secure DevOps Kit for Azure contains the following components:
29
A package of scripts and programs that ensure more secure provisioning, configuration,
and administration of an Azure subscription.
A set of extensions, plug-ins, templates, and PowerShell modules that empower a
developer to create, check-in, and deploy code with greater security from the beginning.
A tool that can capture snapshots of “secure” state for a subscription, for the application
resources, watch for drift, and that can enable operational security compliance.
A package that would grant the visibility to enterprise IT teams about the shape and
form of all of the above along with extensions that can provide application layer events
and alerts for individual service line teams.
In the spirit of agile development, this kit is a work in progress. We are hoping to refine many of
the ideas as we iterate and enhance it over sprints with regular feedback from our engineering
community.
It’s easy to come up with metrics such as number of bugs or number of fixes to measure
application quality (security or otherwise). However, that can be a rather antiquated model that
30
worked better with traditional engineering practices. In the modern engineering model, where
the pace of innovation and continuous assurance are the core drivers, we focused on mean-time
based metrics instead such as mean-time-to-triage (MTTT) and mean-time-to-fix (MTTF).
These metrics are driven by mean-time and measure how fast the engineers respond to security
issues by triaging them first and then by fixing them in their release cycles. Enabling our
engineers with automation is critical, but we also need to measure how fast they are leveraging
the automation to drive continuous assurance.
The four stages of competence model (developed by psychologist Noel Birch in the 1970s) is a
helpful illustration of the stages that an individual goes through to learn a new skill. We’ve
applied it to the process that an engineer progresses through as they learn how to achieve
security hygiene with the AppSec Health Index.
31
When learning a new skill, individuals or teams progress through each phase from “unconscious
incompetence,” (an unskilled state) to “unconscious competence” (a state of expertise).
We think this illustration of the progression of skills is a great concept and is a basis for what we
wanted to achieve when creating a health index for our AppSec program.
The AppSec Health Index measures the impact of our SDL in maintaining a secure posture of an
application portfolio. It is a qualitative metric that is used to indicate the application security
health of a team or organization. The index is derived from four underlying quantitative metrics:
32
TABLE 2 APPSEC HEALTH I NDEX QUALITATIVE METRICS
It’s important to note that the targets for each metric should be based on an organization’s
maturity and needs. For example, in some organizations, having SDL compliance for as little as
50% of the portfolio may be a reasonable target. In other organization, aiming for 100%
compliance may seem realistic. Determining the right target is not trivial and can often take
many iterations to converge. However, it’s important to stress that the above numbers are
examples only and should not be treated as targets for any company that wants to achieve the
most ideal application security posture.
If we apply the four stages of competence to our AppSec program, and factor in the AppSec
Health index metrics, the progression of skill might look like the following:
33
FIGURE 11 STAGES PROGRESSION
Finally, to calculate the single qualitative AppSec Health Index, we analyze the four quantitative
metrics to determine the conditions that set its value. One such example is to define the overall
AppSec Health Index into three tiers of Green (on target), Yellow (close to target) and Red (off
target) and set the value based on this:
34
5.4.1.1 Driving competition
The concept of using metrics to drive competition or gamification is not new in security.
Microsoft engineering teams love to compete to be at the top and to be the best among their
peers. We approached this by publishing the AppSec Health Index quarterly on a scorecard for
all engineering teams in Microsoft IT. This drove a sense of competition and encouraged teams
to mimic the best practices of the teams who are on the top of the scorecard.
35
FIGURE 13 SAMPLE - FORTIFY TOP 10 VULNERABILITIES
We also found “Bug density data” valuable to determine an engineering team’s security maturity
and to enable the conversations with the teams to figure out corrective actions.
Oftentimes, security teams undermine the importance of end-to-end user experience while
creating and deploying security solutions. Although as an industry, we’ve made significant
36
progress, we think the security industry as a whole has been behind on this – just look at the
plethora of security solutions in the industry that treat user experience as an afterthought!
Security is critical, but it doesn’t mean we have an excuse to ignore user experience. We focused
on simplicity and user experience when thinking about our end-to-end solution. Some of the
concepts that helped orient our work on this included:
User experience is a journey and we are still working toward achieving the optimal end-to-end
user experience for our process. If engineers are treated like customers and if they are engaged
during tool evaluations, process redesign, and overall user experience discussions, we believe
this can yield very powerful results.
37
FIGURE 15 FORTIFY P LUGIN FOR VISUAL S TUDIO
We went a step further by also making the questionnaire intelligent. When the engineering team
opts to run static and dynamic security tools, certain control questions (that validate the
implementation of TCPs) are automatically hidden, which makes the questionnaire shorter and
smarter, and therefore, enhances the user experience.
38
an ongoing basis. During the last process enhancement we did in ISRM, we were able to reduce
106 varying technical requirements (TCPs) to 73 requirements.
There is much more work we need to do as we integrate security more closely to engineering
systems and processes. One of the big initiatives we are driving is evaluating how we can use
Visual Studio Team Systems (VSTS) work items to drive security activities more effectively and
efficiently.
7 Lessons learned
We have learned so much from our engineering teams and from our partnerships – and we have
a lot more yet to learn. The following are some of the more important lessons we’ve learned so
far.
39
but also can identify gaps to share with your tool vendor to make the tool work even better for
your engineers. Engineers need to love the tools even more than security does.
Avoid an overly architected, rigid process. In fact, the nimbler the process the better. For
example, have at least your knowledge base well defined so you know exactly what you are
evaluating and how it will help you streamline your process in driving software security
assurance.
40
This is our aspiration because there are security issues, such as poor authorization controls or
design issues, that tools have a very hard time detecting. Nevertheless, it’s an aspiration worth
driving towards – automating the detection of all those security issues that can be automated.
The easiest approach we found was to invite your business stakeholders (the engineering teams)
into the discussion. This discussion should include not just the processes and technology, but
the development of metrics such as mean-time-to-triage and mean-time-to-fix to measure
success. If the business can align to your metrics and agree that driving to defined targets will
help create an impact, you’ve got a natural win-win situation.
8 Conclusion
The modern engineering world presents ongoing challenges for us as we adapt to a DevOps
model while also navigating the changing security landscape. Our approach has been to enable
our engineers with security solutions that reduce friction as much as possible. We want to
41
improve and enhance the engineering experience for our engineers with respect to software
security assurance. We feel very comfortable with our iterative approve on our solution because
this is very much the agile way – to jump in and iterate on the solution over time. We’ve done
the heavy lifting already of creating the building blocks for our foundational components of
knowledge management, automation, metrics, and user experience. Making these strategic
shifts was the difficult part, and now we can build on that momentum.
In addition to the challenges we talked about, we also see big opportunities for advancing
application security in this age of modern engineering. There has never been a better time to
push security automation and develop integrated security services for engineering teams as they
think about operating in a DevOps environment and push for CI/CD scenarios. Similar to how
development, test, and operation roles have merged to shape today’s modern engineer, we feel
that a software security assurance program can yield much better results than before if the
processes are baked seamlessly into the engineering process. Security teams should leverage
this momentum of automation to further expand the scope and coverage in their organization
in an effective and efficient manner.
42
9 Appendix A: Resources
9.1.1 SDL
Security Development Lifecycle
https://www.microsoft.com/sdl
The Security Development Lifecycle: SDL: A Process for Developing Demonstrably More Secure
Software (Developer Best Practices)
https://www.amazon.com/Security-Development-Lifecycle-Developing-
Demonstrably/dp/0735622140/ref=sr_1_1?ie=UTF8&qid=1473087879&sr=8-
1&keywords=security+development+lifecycle
43