Michigan Summit

YOU'VE
BEEN
HACKED!

You have been hacked now what?

What do they have?

What should
you do?
Organizations must assume
they are compromised, and
therefore, invest in detective
capabilities that provide
continuous monitoring for
patterns and behaviors
indicative of malicious intent.

Define an incident response

procedure that details the roles
of appropriate business and IT
contacts throughout the
organization and other
departments needed to respond
to security incidents, including
human resources, public
relations, legal and executive
management.

What happened and how we

handled them will follow.

Web Applications Breach

In 2013 just after 6 months of being in the role of CIO, we had what
appeared to be a potential very large data breach.
I was off work, driving to Detriot, when I received a phone call from one of
our department managers. It started off with: I am not sure if we should
be concerned but I think we may have a problem.
She started telling me about an applicant they interviewed for their data
analyst position of their health care program. The applicant showed them
some screen shots of their application and database that the person was
able to access with what they called simple SQL injection commands.
My first question was what application is this.
Then I asked what information did the person claim or show you:

I now know why they tell you not to talk on your cell phone while driving.
She stated the applicant showed / had PI information, Name, Address,
Date of Birth and SS#. The application has over 300,000 records with
this type of information.

Secure the
system
I hang up the phone, called our
Database Admin and Network
Admin and told them to take the
application off line ASAP like
yesterday!
I had the DB Admin start
looking into the what happen,
how it happened and what he
can find out and the same with
the Network Admin.
We also had them copy the
application and start working
on securing it so we can get
operations back to normal.

At this point, I still didnt know

what the person really had and
we didnt have expertise in
house to truly determine what
they had.

Determine
what they got
The next phone calls I made was to
the State Department of
Technology Management and
Budget Security Office and
Michigan State Police Computer
Crimes unit.
Of course, they wanted to know
who the person was. Since we
knew that information we provided
that information.

Then I find out the person is not a

U.S. Citizen and MSP start working
with my staff to determine what
the person got!

The FBI came in as well since we

are now talking about a non U.S.
Citizens.

Can it get worse or better?

Get operations
back up
We have the contractor that wrote
the program for the Health
Department without ITs input rewrite and secure the application
with the IT departments expertise.

State Police interview the

individual and took their computer
equipment.

We got the application secured and

had it pen tested and brought the
application back on-line.

State police forensic the equipment

and found the person only took a
couple of screen shots of the
database and printed them to bring
to the job interview. They ended up
not having all the sensitive data.

Did we do it
right
The person did not see anything
wrong with what they did and had no
ill intent. The person was trying to
show what they could fix what was
wrong with our system. They
exceeded at getting our attention since
this did expose a huge concern for us.

We now have a notification process

defined: what to do, when to do it and
who to notify (most taken out of what
we did and formalized it).

The question from some was did we

over-react bringing in MSP and the
FBI?

I would rather over-react in this type

of situation than not. If there was ill
intent and they had all the data it
would have cost over a million dollars.

What we do now is assume that we are already infected. We changed our

"incident response" mindset to a "continuous detection and response"
process.
We have invested in tools and processes that can detect malware
infections that have evaded traditional blocking and prevention
solutions.
We have to implement preventive controls to "harden" endpoints. These
are highly effective and should be revisited by looking at new tools that
are emerging to simplify this task regularly.
We have to ensure business impact analysis aligns business continuity
and IT disaster recovery plans to the value of the business processes
being protected. This also helps address IT complexity by supporting IT
application and data classification.
Bottom line everyone agreed that it is better to assume the worst and
hope for the best.

What we learned and did

Define an incident response procedure that defines the roles of appropriate
business and IT contacts throughout the organization and others needed to
respond to security incidents, including human resources, public relations,
legal, law enforcement and executive management.
Retain either internal or external resources for executing an incident response
plan: specifically target resources with digital forensics and malware analysis
knowledge.
Security incidents should remain confidential within the incident response
process and proper workflows as well as collaboration need to exist between
involved parties during execution of the incident response procedure.
The number one item was to secure the data first! Determine what and how
the breach occurred then bring the systems back on-line once secured.
Once you know how the breach occurred work on redefining your tools used
and change processes and/or procedures to prevent future ones from
occurring.

Your success or failure will not depend on if you

got breached but how you handled IT

Second Incident

Ransomware or Crypto locker virus

Earlier this year, we got a call on our afterhours number of a potential virus
issues at our Health Department around 5:45pm. Of course I find this out as I
am heading into a board meeting.
The Network admin talked to the end-user who said she had a couple of files
that were encrypted and she had to contact the company to get her file back.
We started our incident response plan at this point.

Crypto Locker
virus
The first thing we did was
to disable the users
account and the infected
computer so her
credential and the laptop
was unable to infect any
further data.
We then looked at what
the user had rights to and
checked the files.
This is where we
discovered that over
1000 files were affected
across several directories
on the SAN.

Get things
back to normal
We moved into a DR process to bring
the business up. We started running
full scans on all volumes of the SAN
and servers. We investigated the time
frame of corruption and started
removing the corrupted files.

We took the infected computer to have

an analysis on it to find out how this
happened. We provided the laptop to
MSP for their help.

Once we got business back into

operations, we then went into
forensics mode to find out the how
and why.

From the time we were notified of the

potential issues to the time we had
operations back to normal was within
1 hour and business was able to
continue otherwise.

How did this

happen
In this phase, we discovered the person
had Dropbox installed on their laptop.
They were also using it to access personal
email from the web.

Since the laptop was not always on the

network, the AV software was outdated.
These all lead to the infected file getting
on to the laptop and going wild on the
network.

We reimaged the laptop, we then started

a full force check on all systems to verify
that the AV was installed and up-to date.
We implemented some better
management protocols and tools to
ensure our Antivirus software is and
continues to be updated.

We are looking at other tools to put in

place to protect the network from the
inside so the network damage cant
continue beyond the one infected
machine.

 While over 1000 files were affected there was minimal

impact to the county and it only cost us some staff time

no real dollars.
We have now implemented a security awareness
monthly training for all users.
Issues was reported at 5:45. By 6:00 PM, the affected
user and system was isolated.
By 6:15 the laptop was in ITs possession and restoration
efforts were started.
By 6:45 the files that were corrupted/encrypted were
restored to their previous state.

How did we do
it
We were prepared and knew what
to do.
The IT staff informed the correct
staff and the correct time to
minimize the affects.
We have DR plans and a DR SAN
which allowed for the quick
restoration of the files.
The only comments we got from
the board and the department were
simple.
How can we prevent it in the
future?
How were you able to get
everything done so quick?
Again it was not an issue of the fact
we got hacked it was more focused
on what we did and how quickly we
did it. Confidence level of IT
operations continue to grow as
they know we can handle critical
situations.

Time is money and having IT services down

cost money and frustrations not only for the
users but the citizens we serve.