Vous êtes sur la page 1sur 15

Nemesis

disruptive

Shay Holmes
Sr. Eng Dir.
shayh@yahoo-inc.com

Testing

Suresh Visvanathan
Nemesis Architect & Lead
sureshv@yahoo-inc.com

Problem
Outage prevention: Complex SOA architectures
downstream services fail/degrade
cascading effects
+network unreliability

Difficult to test resilience behaviors/features


ad-hoc or pre-launch testing, but
manual, inconsistent, time consuming
specialized
late in process

Strategy
Focus on protocol-level first (HTTP/S, TCP)
the individual app/service POV

Use existing testing/measurement tools


just worry about creating the disruptive conditions

Developer Accessible
integration with NodeJS platform (Fluxible/React)
"out of the box": 1 config file and done!
compatible with tools & tests (JS test library)
self-service, easy CD integration
3

Use Cases & Disruptors


Use Cases
Simulate error conditions using
"disruptors" for these testing cases:

Exploratory testing - early and often


(and easy!)
Regression testing in CD pipelines
Scripted pre-launch testing
Live production tests (future)

"Disruptors"
HTTP/S

Latency injection
Error codes
Request/Response header
Response body

TCP

Connections: reset, hang


Conditions: packet drop, packet
latency, packet transfer speed

System

custom commands
4

Technical Overview

Technical Overview: Sample Configs


Multiple paths

{
"domains": {
"http://aaa.bbb.yahoo.com:4080": {
"protocol": "http",

Disruptor
configurations
per path

"latency": "4000"
},
"aaa.bbb.yahoo.com:4080": {
"protocol": "http",
"res_status_code": "500"
},
"http://aaa.ccc.yahoo.com:4080/fantasy/v2/game/1003/teams": {
"protocol": "http",
"res_header_mod": {
"Cache-Control": "no-cache"
}
}
}
}

Technical Overview: Sample Configs


Disruptor
Condition

{
"Int": {
"loss": {
"percentage": 5.0
},
"delay": {
"delay": 15
}
}
}

Technical Overview: Config Flow


Production Server

Validate Disruptor Config


Distribute Config
Authentication &
Authorization

JS Test Lib - used


with protractor or
similar test
frameworks
Npm library
(nemesis-test-lib)
Integrate with
automated tests

Nemesis Mgmt Daemon

Nemesis
Config APIs

Nemesis Test
Library (JS)

CLI/UI

Daemon

Apply
configurations

Run (or)
system

Ad Hoc
commands
On Demand
Production
execution.

Technical Overview: Real-Time Traffic Flow


Dependency

Production Server

NodeJS App

Nemesis
Routing Lib

Nemesis
Proxy

Core Run-time applies disruptors to


real-time requests
Haproxy
Multiple protocols
Logging

Nemesis Run-Time
Lib bundled with
application
HTTP header to
control Nemesis
on/off
Fallback to default
route if Nemesis fails

Nemesis Mgmt Daemon


9

Architecture (E-E)

Yahoo Confidential & Proprietary

10

@Host

Yahoo Confidential & Proprietary

11

Nemesis Proxy

Leveraging HAProxy and Lua integration.


Haproxy supports multiple protocols(eg: https/tcp) , we leveraged underlying protocol
implementation
Lua integration provides good set of apis which allows to alter/modify the transaction object.
sample lua script to modify/add header to transaction
-- Add additional request header
function req_header_addition(txn,req_header_add)
local buffer = " "
for k, v in pairs(req_header_add) do
if (k ~= "") then
txn:Info("Adding header " .. k .. " : " .. v)
txn.http:req_set_header(k, v)
else
txn:Info("Skipping Request header addition, since header key is empty")
end
end
buffer = buffer .. txn.req:dup()
txn:Debug(buffer)
end

Yahoo Confidential & Proprietary

12

Nemesis SysOps

runs any system level commands (whitelisted)

shaping/unshaping the packet flow .


packet drop, packet latency, packet transfer speed
runs tc,iptables commands

Yahoo Confidential & Proprietary

13

Pipeline

Protractor

Conf
Tests

Libraries

Helper Utils
Protractor Utils

Nemesis-Test Lib

Sauce

Nemesis
API Server
Protractor
NemD

Test

Sauce

FE

Test

Browsers

Nem
Tests

WS
NemD

Protractor Utils
Yahoo Confidential & Proprietary

Configs

Nemesis-Test
Lib

14

Contact
ychaos@yahoo-inc.com