Copyright

2016 Splunk Inc.

CenturyLink
Thom Woldahl
Lead Engineer


 About CenturyLink
Founded in 1930
Leading provider of Internet, TV, Voice
A global communicaIons, hosIng, cloud
and IT services company
250,000-route-mile US ber network
More than 55 global data centers
Headquartered in Monroe, Louisiana
S&P 500, Fortune 500

2
 About Me
Lead engineer
Based in Phoenix, AZ covering the Mountain West area
(Colorado, New Mexico, Arizona)
First IPTV engineer in the eld for CenturyLink on PRISM product
Previously worked in network planning and engineering
As a Regional OrganizaIon, we support our local eld
operaIons team
Install, maintenance, repair video and high-speed Internet
products as well as POTS (plain old telephone)
Were measured on installaIon and troubleshooIng
success rates
Keep costs down/customers happy

3
 Splunk at CenturyLink
Splunk posiIoned to collect machine data from enIre CenturyLink
network
Data sources:
Syslog from IP/MPLS backbone (routers/ switches), servers,
appliances, gateways, video delivery subsystems, SNMP, call
center call logs
Database logs, applicaIon (transacIonal) logs
AcIve, passive and transacIonal measurements throughout the
network: data pulled into Splunk sodware using network probes
and scripted input into database
Indexing up to 600 GB data per day
Splunk users: 750 regular (2000+ unique)
Teams using Splunk: NOC, Development, Architecture, Planning,
OperaIons, Engineering, Service Provisioning, eCommerce team,
Order Processing(IVR), MarkeIng, Product, IT, etc.
Splunk Apps: Deployment Monitor, Windows App, Splunk on Splunk,
Splunk DB Connect, our own customized Splunk Apps
4
 Before Splunk: inaccessible data, too many customer visits
MulIple visits of on-site customer support
chronic customers (3+ trouble calls in a measured period)
I would only be called in as techs or supervisors would become frustrated
with ineecIve repairs or unable to idenIfy a specic issue Technicians were
Endured laborious, manual processes developing
I only had access to test data, if I was on site during troubleshooIng rela9onships with
Technicians were challenged to capture the data, move it to a USB drive, customers because it
and send it to me in an email for review was not uncommon
Time consuming, clunky process (and a ton of windshield Ime driving to for them to be at
customers homes) their houses mul9ple
We knew the data could tell us criIcal informaIon if we could view 9mes for repair.
the test results

5
 Before Splunk:
I started with post it notes with test readings wriken down To obtain data:
on them Technicians manually upload results
Began to use a spreadsheet with data entered by hand from the test device to a USB
Created a Macro that would import test results into the Plug into a laptop
spreadsheet that techs would email surf to a web address
Massive amount of data needed automaIon upload the data les
Original AutomaIon was pulng the test data into a php IdenIfy the Job InformaIon
based web site
Technician who performed the work
We had the data show red or Green for pass or fail. But
there was no way to report the test failures

6
 Choosing Splunk
With the previous web site, I was not able to make reports or changes without
asking the original developer to please do it for me
He made choices of what data to parse and which akribute to display
Of the 127 test akributes, only 27 were able to view
There was not a way for me to work with results easily.. With Splunk, we
I learned about Splunk at a CenturyLink DSL conference in Denver were able to
A colleague made a key introducIon, which led to Splunk
gure out what
drives trouble,
We were able to put a Splunk connecIon on the exisIng database to capture
new test results as well as ingest ALL the previous 300K + test uploads into address the
Splunk. standards and
I immediately began working with the data, all the while dreaming of what could adjust to support
be accomplished as far as the meaning of the test results. performance
It was like going from a closet (which hung data in a relaIve orientaIon)
To a sock Drawer where I could dig around and look for the missing sock

7
 Splunk Benets
We began looking at akributes of customer
calls & dispatch Ickets & and comparing to
uploaded test data
We built dashboards for all of the 127
dierent akributes to show the median
results and the upper and lower thresholds.
We could search on any akribute or any Splunk integrates
into every por9on
distance we wanted
of what we do.
Then we could look up customer call in Were able to bring
records and trouble Ickets which were in siloed databases
separate data silos and dierent
informa9on all
together.

8
 Splunk Benets
Splunk was the soluIon for managing our data and providing an opportunity to make informed
decisions
Splunk has streamlined and automated our operaIons test data collecIon and analysis
We can connect siloed data bases that contain related types of informaIon
For instance, the technician can now save the test data on his handheld test unit and submit direct to our Splunk
dashboards.
The dashboards know who, where, and when parIcular task is worked, no need for them to ll in the info to
submit data.
Splunk integrates into every discipline of what we do
With the connected data bases and dierent types of informaIon- we get a picture of what is happening with a
customer service. We can begin to validate performance not just test results
All the data comes together in Splunk as one view-a single view to see mulIple data sets
A single log on not mul9ple disconnected web pages
With Splunk, our data bases can communicate with each other to give us a beker view
Splunk gives us useful visuals graphical representaIons not just logs and numbers

9
 Splunk Benets

Visibility / ReporIng /
Log / Data Management
Dashboards

Service Management /
Performance Tracking
Eciency
 Splunk for Log/Data Management
A Splunk forwarder now sends data directly from the meter
to our indexers
Eliminates manual input by the technician
We know who did what and where
We combine all log data into Splunk Technicians no
Machine data sources longer have to put
Databases in what jobs
Modem informaIon theyre working
Dispatch reports on. We already
Customer call in data know.
Our processes are much more streamlined
Independent data bases communicate with each other
Which gives much improved view, which in turn provides beker customer
experience

12
 Users/Partners
NaIonal OperaIons teams
Analysts
DocumenIng EvaluaIng/UpdaIng/Maintaining Process and
Standards
Regional Field OperaIons teams,
Engineers
Analysts
Create compliance reporIng that goes to Local OperaIons teams
Local OperaIons Teams
VP/GMs, Directors, Area Managers, Plant Supervisors
Technicians
Growing by the thousands
We expect 5,000+ users by the end of the year

13
 Splunk for Visibility/ReporIng/Dashboards
Daily, weekly and monthly reports
Automated emails from Splunk to management
Compliance reports, who uploaded what test data for a service
Inventory control and tracking of Set top Boxes and Gateway/Modems, what was
installed and where
Was the process followed during customer turn up Splunk has
Technicians are now accountable to the tests performed for service installaIon become our
Process Repeatability and monitoring periscope. Some of
Make sure technicians are following the right steps the dashboards we
We Test with the same methods have are quite
Run the same test scripts and upload results (based on company standards to incredible, to tell
support performance)
The test equipment is monitored for Firmware Revision you the truth.
We can trace the test equipment to User and maintain the Inventory of Capitol
equipment
The Ime to complete a task
Other groups are asking for our help to unlock their informaIon
14
 We make reporIng available on a local, regional and naIonal level using the same data source from
within Splunk.
We react to the same unltered informaIon

15
Side Benet: We are able to view Revision of Firmware on a technicians
test set, as well as the locaIon and capability of test equipment

16
17
 Splunk for Performance Tracking
We use Splunk to track key performance metrics:
Set top box performance
Modem performance
Copper performance (not just tesIng any longer)
We track problem solving variables as well
Technicians, modems, copper, or a customer issue (whats on the LAN)
With Splunk, we can narrow it down & pinpoint the issue
Seeing into all the moving parts of complex situaIons is like the 6 sides of a
Rubiks cube we can see the actual sources of problems and how they are
related to each other

18
 Splunk as an Mobile Device Interface

19
 40M circuit that the customer is
experiencing ~7.5 M download

An example of
mashing data
from mulIple
sources to create
an opportunity for
evaluaIon

20
 poor performing loop An acceptable test

Missing
bandwidth
Full use of
bandwidth

looks good
Bounce indicates Bridge tap

21
 Splunk Unique tool: The Promise Chart

If the blue line does not touch the colored bar,

the loop can NOT support that provisioned speed

Blue line represent test data Colored lines represent MAX

as Loss per tone loss per tone for a Service

22
23
 Uncorrected errors
are present

good sync rates

24
 Original Dispatch Data Set
allows for 45 days of history,
we now keep the informaIon
indexed in Splunk for 18
months

We changed the clunky

interface

We added Mapping to the

dispatch

25
Side Benet: We are able to review technician performance
Did they perform a test, was it the required tests, and did the tests meet the standard

26
27
 Splunk for Service Management/Eciency
We can now look at dierent akributes from many sources
within our company, to gure out what drives a customer
call-in / poor experience
With Splunk, I know
We narrowed down 175 test items to the handful that drive we have saved
trouble calls expense dollars using
We have adjusted our test standards to what drives a poor cost avoidance by
adjus9ng our tes9ng
customer experience
standards.
Splunk helps us prove where we need to make adjustments
to avoid troubles
Technicians use this informaIon to know what to look for to
x the customers problem
28

 We report the We see the length of
percentage of uploads Ime on any job or
We track the length of crew.
Ime that jobs take as
the day moves
on

We can drill down to the detail of the job and uploads

29
 Splunk Words to the Wise.
You have to have a core understanding of what the
data represents to make it useful
The users have to be a part of the implementaIon
and put in eort to make the data meaningful
Report and dashboard development
IdenIfy independent however Related data sources for
ingesIon into Splunk.

30
 Splunk Successes
Weve improved managements awareness of technician eciency through automated
reporIng.
We know Splunk has reduced cost.
Previous to our Splunk analysis of our test data, we would dispatch two truck rolls for a Video
customer served over a copper loop.
We now only dispatch once for an install
Where a customer may have failed standard and was denied service, we now have standards
that are meaningful to performance and we can evaluate quickly
We now can validate a customers service because we have installaIon history through
uploaded test results.
We are eecIvely mapping our last mile performance using Splunk
Installs and repairs
Gateway and STB automated reporIng
DSLAM performance

31
 Splunking Ahead.
Google mapping
Were going to get more graphical in terms of
laying over installs and trouble Imes
This will beker highlight where we have
performance issues
IdenIfying where to invest for a higher return
on investment
Maintenance versus installing new technology

32
 Looking ahead. What if we combined our test data
to dispatch informaIon? And then have Splunk map it?
We then could idenIfy issues and gure where to spend our Ime in
maintenance.
We could idenIty where to place new technologies
We Could gure out why a certain area takes longer to install or why we
have increased repair acIvity
We could begin to Predict.. To an9cipate results based on
Factual informaIon.

33
Thank You