Incident Response and Digital Forensics –

Denver Chapter of ISACA

Outline
 Introduction
 The Incident Response Process
 Preparation
 Identification
 Containment
 Eradication
 Recovery
 Lessons Learned
 The Attacker Process
 Reconnaissance
 Scanning
 Exploitation
 Keeping Access
 Covering Tracks
 Conclusion
Introduction
 FishNet Security

 Started in 1996, largest InfoSec VAR

 Support, Consulting, Training, Products
 Locations – Boston, Omaha, Kansas City, St.
Louis, Dallas, Minnesota
 Assessment Team: Policy, Network, WebApp
and DB, Wireless, IR and Digital Forensics
Introduction
 Trey Keifer - (trey.keifer@fishnetsecurity.com)
 Level II Security Engineer

 Sr. Incident Response Analyst

 Coordinator of Incident Response Program
 SANS Certified Incident Handler (GCIH)
Incident Response and Digital Forensics

 One of the least practiced, most stressful, highly

scrutinized areas of Information Security.

 Every incident is unique and can incorporate many

different areas of the affected organization.

 Incident analysts must be able to think quickly,

remain calm and consider all possibilities.
Common Incident Types
 Economic Espionage
 Intellectual Property Theft
 Unauthorized Access
 Stolen Passwords and Data
 Unauthorized Use
 Inappropriate E-Mail and Web Habits
 Malicious Code
 Worms with Backdoors (Sasser)
 Insider Threats
6 Steps of the Incident Handler Methodology

 Preparation
 Identification

 Containment
 Eradication
 Recovery
 Lessons Learned
Preparation:

 The key to a successful response is

preparation.

 Form a strategy.
 Design a procedure.
 Gather Resources.
 Practice, practice, practice.
Preparation:
 Identify the “Core Team”

 Technical (IT, InfoSec and System Owners)

 Management
 Legal Department
 Forensics
 Public Relations
 Human Resources
 Physical Security and Maintenance
 Telecommunications
Preparation:
 Organizing Individuals
 All members of the CSIRT team should know their role
and how they will interact with the other members.

 Outsourced or “third party” members should have

contracts in place.
 Contacts for Law Enforcement should be known and
situations for their involvement discussed.
Preparation:
 Develop a Procedure
 Incident response can be a high-stress time. A well
documented procedure, that is easy to follow, can greatly
reduce the anxiety.

 Develop a call tree and notification procedures

 Brainstorm likely scenarios.
 Identify general information needed in most scenarios
ahead of time.
 Make checklists and forms for as much as possible.
Preparation:
 Communication
 Communication is incredibly important during an incident.
Not only the people involved, but the method which it is
done.

 Updates should be frequent.

 Out-of-Band Communications are very important.
 Faxes
 Cell Phones
 Be careful with the Blackberry’s
Preparation:
 Access Rights
 The incident response team must have access to systems
without the administrators authorization.

 Controversial Issue
 User Accounts, Passwords and Encryption keys
 Third-party storage methods are available
Preparation:
 Policies
 Protect the organization from legal liability and allow
investigators to do their job.

 Warning Banners are readily displayed.

 Search policy is detailed in employee manual.
 Human Resources and Legal have signed off.
 Employees have acknowledged knowing their
expectations on privacy.
 Beware of international laws (European Privacy Directive)
Preparation:
 Gathering Resources
 Incident analysts should have all information ready and
be able to respond to the incident.

 Procedures, Checklists and Forms are ready.

 Access credentials are available or individuals with them
are known.
 System information, network diagrams, software and
intellectual property are documented thoroughly.
Preparation:
 Training
 SANS Institute and GIAC Certifications
 Track 4: Incident Response and Hacker Techniques
 Track ??: Digital Forensics

 Vendor Training
 Guidance Software
 Access Data

 Partners
 Incident Response Scenarios
Identification:

“Incidents can’t always be prevented, but must

always be detected.”
Incident: Intentional or Unintentional

 Multiple failed logins to the domain administrator

account.

 Administrator credentials were cached on a users

workstation and they are attempting to login.

 Someone is actively attempting to brute-force the

account.
Identification:
 Goals

 Determine Scope
 Identify what systems, people and informational assets
are involved in the event.
 Preserve Evidence
 Protect the facts of the incident while determining the
scenario.
Identification: Suspicious Events
 Unexplained Occurrences

 New Accounts or Files

 File Modifications
 IDS Triggers
 Firewall Entries
 Accounting Discrepancies
 Poor Performance/Unresponsive services
 System Instability
Identification: Passive Identification
 Sniffers and Traffic Analysis

 Cyclical Buffers allow full recording of events at the

packet level to a point, depending on size and utilization.
 Target machine evidence is still preserved.
 Assist in determining new attacks for which signatures
have not yet been written.
Identification: Passive Identification
 Intrusion Detection Systems

 Least invasive method

 Target machine evidence is preserved
 Logs must still be protected
 Write-Once, Read-Many Media
Identification: Passive Identification
 Tripwire-style File Modification
 A hash of the file is taken and stored in a secure
database. Any modification to that file results in a change
of the hash.

 Very indicative of a successful compromise.

 Can be noisy during patching and must be tuned after
every software upgrade.
Identification: HoneyPots and HoneyTokens
 Specific systems or accounts with additional logging
and notification to alert on suspicious activity.

 Operators must be careful of entrapment.

 Systems have to be secured and heavily monitored.
 Systems cannot invite intruders –
 No “hackme” accounts
 No “Salary Database” systems
Identification: Chain of Custody
 Evidence must be accounted for from the time it is
collected until the time it is submitted to the court.

 Each piece of evidence must be under the control of one,

identifiable person at all times.
 A change in control of the evidence must be recorded.
 Evidence in storage must be protected from
contamination. (ie… sealed and secured)
Containment -

Now that the events have been identified as an

incident and a chain-of-custody for evidence
has been established, we will take the first step
into system modification by beginning our
containment.
Containment:
 Vendor Coordination
 Work closely with your vendors and know how to open
security-related tickets with high priority.

 ISPs can prevent some Denial of Service situations.

 They are more familiar with attacks because they have seen
them with other clients and are up-to-date on advisories.
 Additional people working towards identification, containment
and recovery.
 We are used to the pressure!
Containment:
 Identifying the Trust Model
 The trust model identifies not only the
technology, but also the people that are
involved in the incident.

 What connectivity does the network or system have to

other areas in the organization?
 What information is contained within it?
 Who needs to be involved and to what extent?
Containment:
 Documentation Strategies
 Documentation should be collected from most
volatile to least volatile and least invasive to
most invasive.

 Volatile evidence includes RAM, running processes and

active connections.
 Be careful of running system commands from anything
but recovery media.
Containment:
 Should we Quarantine?
 Changes to a system may be easily observed by
an active attacker.

 Rootkits may identify a pulled network connection or

extensive system modification and protect the attacker.
 Some exploits are entirely memory resident and will
disappear when the power is pulled.
Containment:
 Initial Analysis

 Keep a low profile

 Never analyze the original
 Make frequent updates to CSIRT
 Acquire log files
 Stick to the facts and avoid blame
 Consider all possibilities but keep it simple
Containment:
 Backups
 Numerous backups allow both investigation and
preservation of evidence.

 Different strategies exist and depend on the situation.

 Original is kept as evidence
 Backup 1 – Placed back in production
 Backup 2 – Forensic Analysis
 Backup 3, 4, etc… separate copies for analysis
Containment:
 Digital Forensics
 Numerous separate analysis all yield the same
results.

 Requires specialty hardware, software and training.

 Bit by Bit copying and analysis of data.
 Recovery of deleted data.
 Identification of altered system files (trojans) and
binaries in a safe environment.
Containment:
 Digital Forensics: Hardware Write Blockers
 No modification to the data itself, we want to
observe and duplicate only.

 Hardware device or driver between acquisition machine

and target system.
 May use NIC, USB, FireWire or IDE/SCSI channels.
 Intercepts write commands and gives logical return
results.
 Allows browsing of the filesystem during acquisition.
Containment:
 Digital Forensics: Forensic Software
 Allows quick and efficient analysis of the
information contained on the device.

 Guidance Software’s EnCase used by law enforcement.

 Linux Forensics CD’s are coming along in maturity.
(still must use write blockers!!!)
 Scripts allow quick searching of keywords in files and
deleted data.
 Hash comparisons verify original files, known dangerous
applications and aid the examiner in avoiding the bad
stuff.
Containment:
 Digital Forensics: What are we looking for?
 Many areas of interesting data are forgotten
about.

 Cached web content

 Email Files (PST’s)
 Recoverable Deleted Files
 Specific Incidents: CAD drawings, Engineering diagrams,
Pornography
 Known file signatures of hacking tools, backdoors, etc…
Containment:
 Digital Forensics: Other devices?
 May not be able to submit as evidence in court,
but can assist the Incident Handler in their
investigation.

 Personal Organizers (PIMs): Blackberry, Palm Pilots,

IPAQ’s.
 SIM Cards/Cell phones
 USB Tokens/Flash Drives
Containment:
 Digital Forensics: Not Perfect!
 Some tools have been written specifically to
defeat forensics software.

 DoD: 7-Pass, random-write method for secure deletion of

magnetic media. (Rainbow Method)
 Windows: Eraser
 Unix: Wipe
Containment:
 Slowing the Attack
 Change passwords and access rights.
 Change hostnames and IPs.
 Null Route suspicious traffic.
 Block IPs or Networks.
 Apply Patches to similar systems.
 Shutdown services.
Eradication -

Once an incident has been contained we attempt

the total removal of malicious applications from
a system or network.
Eradication:
 Remove or Restore
 The decision of whether to remove malicious files or
restore from backups is a difficult task.

 Rootkits almost always demand a rebuild.

 Verification of backups is a must.
 Patches may not be available and a total change of
architectures may be necessary.
Eradication:
 Improve Defenses
 Implement additional detection and protection methods
and strengthen existing technologies and processes.

 Apply firewall and router filters.

 Perform “mini-assessments” using the same tools and
techniques as your attackers.
 Look for the same exploits and backdoors on multiple
machines.
Recovery -

Once the threat has been removed the

organization must begin the process of
returning the business to normal operation.
Recovery:
 Returning to Operation
 System owners make the final call on returning to
production.

 Owners depend on the systems and know their true

value.
 If a disagreement occurs on whether to return to
production or not it should be documented by the
analysts and the owner should acknowledge
responsibility.
Recovery:
 Monitoring
 At this point in the process you should have enough
information to identify the attack if it occurs again.

 Create custom IDS signatures if possible.

 Verify proper operation to baseline configurations.
 Implement additional logging on network, hosts and
applications.
Lessons Learned -

The lessons learned meeting provides a method

for the organization to coordinate knowledge of
an incident, suggest changes in procedures and
policies for the future and justify the
implementation of new safeguards.
Lessons Learned:
 Recap Meeting
 Should occur promptly after eradication of an
incident while details are fresh in the team
members heads.

 Create a timeline of events.

 Provide a consensus of notes and documentation.
 Finalize facts for a final report.
7 Deadly Sins
 Failure to report/ask for help
 Incomplete/Non-Existent Notes
 Mishandling/Damaging Evidence
 Failure to create backups
 Failure to eradicate or contain
 Failure to prevent re-infection
 Failure to apply lessons learned
Attacker Methodology
 Reconnaissance
 Profiling the Target
 Scanning
 Identifying Weaknesses
 Exploitation
 Breaking the Law
 Keeping Access
 Backdoors
 Covering Tracks
 Staying out of Jail
Reconnaissance:
 The target is profiled –

 Employee Information (name, numbers, titles)

 Systems Information (usenet postings, job listings)
 Process Information (vendors and transactions)
 Location Information (external networks, physical locations)
Scanning:
 Port and Vulnerability scanners are run to
identify vulnerable systems.

 Open Ports and Services

 Vulnerable Applications
 Default Usernames and Passwords
 Weak Encryption Implementations
Exploitation:
 Execution of attack – usually the first point at
which the law is broken.

 Goals
 Gaining Access
 Elevating Access
 Extracting Information
 Denying Service (DoS)
Keeping Access:
 Addition of Admin-level User Accounts
 Enabling of default, insecure services
 Installation of “Backdoor” or “root kit”
applications allowing the attacker to retain
access despite system modifications.
 Application Level
 Traditional Rootkit
 Kernel Level Rootkit
Covering Tracks:
 Modification of system logs, applications and
processes to prevent identification by
administrators.
 Hiding files and Directories (… and alt-255 dirs)
 Changes in /var/log
 Changes in shell history
 Removal of events (windows)
Our Example Scenario
 An attacker uses a “0-day” exploit to infiltrate
the target organization, install a backdoor and
retrieve critical intellectual property for a
competitor.
 Normal security procedures alert the
administrators to suspicious activity and the
incident response plan is activated.
Attacker Perspective: Reconnaissance
 Google and the corporate web site are used to
identify the organizational structure of key
personnel including HR managers and
executive management.

 Low-Profile, no data sent directly to

organization.
 Impossible to detect.
Attacker Perspective:
Harvesting
 Freely-available scanning
tools are used to identify
email addresses from the
corporate website.

 Same method as SPAM

groups.
 Many sites do not use
generic web addresses.
Attacker Perspective: Exploitation
 Attacker sends malicious application to email
addresses obtained during scanning.
 Users open emails (possibly through social
engineering) and are immediately infected.
 Attacker can be listening for connections from
infected machines and have immediate control
over systems.
Attacker Perspective: Keeping Access
Incident Timeline
Incident Timeline: Preparation
 IR Team established and roles defined.
 Daily procedures established for log analysis
and identification.
 Containment procedures are outlined in policy.
(Restoration takes priority)
 Roles and Responsibilities are defined
Incident Timeline: Identification
 Bandwidth graphing shows abnormal usage

 Passive sniffing identifies responsible host

Incident Timeline: Containment
 No “watch and learn” policy, power is pulled
from the host.
 System is imaged using forensic tools and
Hardware Write-Blockers which prevent
alteration of data during backup.
 Employee is interviewed to determine method
of infection.
Incident Timeline: Eradication and Recovery
 System is restored from the organizations
hardened base image and patches are applied.
(Analysis can continue through restore)
Incident Timeline: Lessons Learned
 Social Engineering Awareness
 File attachment blocking
 Firewall Rule Revisions
 IDS Signature changes
 Patch Management
 Advisory Alert Services
Questions?