Académique Documents
Professionnel Documents
Culture Documents
What
is a performance metric?
Characteristics of good metrics
Standard processor and system metrics
Speedup and relative change
Duration
Of a time interval
Size
Of some parameter
Time-normalized metrics
Rate
metrics
Throughput
Useful
Seems obvious!
However,
to use
10
11
Clock rate
Faster
But
is it a proportional increase?
What about architectural differences?
What
12
Clock rate
(Faster clock)
(better performance)
A good first-order metric
Linear
Reliable
Repeatable
Easy to
measure
Consistent
Independent
13
MIPS
Measure
of computation speed
Millions of instructions executed per second
MIPS = n / (Te * 1000000)
n = number of instructions
Te = execution time
Physical
analog
14
MIPS
Linear
Reliable
Repeatable
Easy to
measure
Consistent
Independent
15
MFLOPS
Better
GFLOPS,
TFLOPS,
Copyright 2004 David
16
MFLOPS
Linear
Reliable
Repeatable
Easy to
measure
Consistent
Independent
17
SPEC
System Performance Evaluation Coop
Computer manufacturers select
representative programs for benchmark
suite
Standardized methodology
1.
2.
3.
18
SPEC
Geometric mean is
inappropriate (more later)
SPEC rating does not
correspond to execution times
of non-SPEC programs
Subject to tinkering
Linear
Reliable
Repeatable
Easy to
measure
Consistent
Independent
19
QUIPS
Developed
20
QUIPS
HINT
benchmark program
(1 x)
0 (1 x)dx
1
Use
Quality
= 1/(u l)
Copyright 2004 David
21
QUIPS
From: http://hint.byu.edu/documentation/Gus/HINT/ComputerPerformance.html
22
QUIPS
From: http://hint.byu.edu/documentation/Gus/
HINT/ComputerPerformance.html
23
QUIPS
Primary focus on
Floating-point ops
Memory performance
I/O performance
Multiprograming
Instruction cache
Linear
Reliable
Repeatable
Easy to
measure
Consistent
Independent
24
Execution time
Ultimately
25
Execution time
Stopwatch
Start_count = read_timer();
Portion of program to be measured
Stop_count = read_timer();
Elapsed_time = (stop_count start_count)
* clock_period;
Measures
CPU
26
Execution time
Background OS tasks
Virtual to physical page mapping
Random cache mapping and
replacement
Variable system load
Linear
Reliable
Repeatable
Easy to
measure
Consistent
Independent
27
MIPS
MFLOPS
SPEC
Linear
QUIPS
TIME
Reliable
Repeatable
Easy to
measure
Consistent
Independent
28
Other metrics
Response
time
Throughput
Bandwidth
Ad
hoc metrics
29
Ends-based
metrics
metrics
30
Execution time
QUIPS
Ends-based
SPEC
MFLOPS
MIPS
Clock rate
Means-based
31
Speedup
Speedup
R1 = speed of System 1
R2 = speed of System 2
System
32
Speedup
Speed
is a rate metric
Ri = Di / Ti
Di ~ distance traveled by System i
Run
D 1 = D2 = D
Total
33
Speedup
R2 D2 / T2 D / T2
S 2,1
R1 D1 / T1 D / T1
T1
T2
Copyright 2004 David
34
Speedup
35
Relative change
Performance
of System 2 relative to
System 1 as percent change
R2 R1
2,1
R1
Copyright 2004 David
36
Relative change
R2 R1 D / T2 D / T1
2,1
R1
D / T1
T1 T2
T2
S 2,1 1
Copyright 2004 David
37
Relative change
38
Important Points
Metrics
can be
Counts
Durations
Sizes
Some combination of the above
39
Important Points
Good
Linear
Repeatable
Easy to use
Consistent
More natural
Reliable
metrics are
40
Important Points
Execution time
QUIPS
Better metrics
SPEC
MFLOPS
MIPS
Clock rate
Not-so-good metric
41
Important Points
Speedup
= T2 / T1
Relative change = Speedup - 1
42