McGraw-Hill Education

Quality Analysis

January 8, 2016

Prepared by:
Content Solutions Quality

Page 2 of 10

Contents
Comparison of accuracy: Processing date vs verified date ..................................................................... 3
Accuracy Computation ............................................................................................................................ 4
Accuracy Trending (using verification date) ............................................................................................ 5
Accuracy Trending (per Agent) ............................................................................................................... 7
Top Error Contributor ............................................................................................................................. 8
Comparison of internal QC vs MHE QC ................................................................................................... 9
Action Items and Timelines ................................................................................................................... 10

Confidentiality Statement:
The information contained herein is confidential and has been provided with the understanding that it will not be disseminated
outside your organization.

Page 3 of 10

Comparison of accuracy: Processing date vs verified date

Table 1. Comparison of achieved accuracy rating using processing date and verification date
Accuracy
Processing Date
Verified Date
Threshold
Difference

WK 37
98.92%
98.96%
96.99%
-0.04

Accuracy
Processing Date
Verified Date
Threshold
Difference

WK 38 WK 39
99.36%
99.77%
99.32%
99.66%
97.99%
98.99%
0.04
0.11

WK 47
99.84%
99.83%
99.97%
0.01

WK 48
99.81%
99.93%
99.97%
-0.12

WK 40
99.85%
99.89%
99.50%
-0.04

WK 49
99.93%
99.84%
99.97%
0.09

WK 41
99.84%
99.97%
99.60%
-0.13

WK 50
99.88%
99.91%
99.97%
-0.02

WK 42
99.93%
99.90%
99.60%
0.03

WK 51
99.88%
99.88%
99.97%
-

WK 43
99.90%
99.91%
99.97%
-0.01

WK 52
99.92%
99.96%
99.97%
-0.04

WK 44
99.86%
99.87%
99.97%
-0.01

WK 45
99.90%
99.89%
99.97%
0.01

WK 46
99.77%
99.79%
99.97%
-0.02

WK 53
99.49%
99.49%
99.97%
-

The objective of this simple study is to see what would be the achieved accuracy if the orders will
be grouped based on their processing date instead of the verification date of MHE.
In addition, the team would also like to see if flagged errors by MHE is accurately accounted for
based on the date when the orders were processed. This is in the assumption that there might be
orders that were verified late and the flagged errors were already addressed on previous week(s).
Shown in Table 1, the differences of the accuracy rating between the two dates is not significant
as we only recorded an average of 0.01 percent difference. The difference was caused as MHE
verifies the order files 1-2 days after SPi processed the orders.
Based on the given document records all flagged errors are accounted properly, therefore
whether the team will use the processing date or verification date of MHE it will not matter since
no encountered errors relating to the assumption made on previously bullet.

Confidentiality Statement:
The information contained herein is confidential and has been provided with the understanding that it will not be disseminated
outside your organization.

Page 4 of 10

Accuracy Computation
Currently, the team is following the MHE way of accuracy computation.
To get the accuracy rating per order:

Case 1: one error type

(2) Contact Name

Order 1:

100 (no. of error * (weight))

Order 1:

100 (2 * (2.9))
100 5.8

Order 1:

94.2

Case 2: > error type

(1) Contact Name and
(2) Sales Channel

Order 2:

100 (no. of error * (weight))

Order 2:

100 (1 * (2.9))+(2*(0.65))))
100 (2.9+1.3)

Order 2:

95.8

To get the overall accuracy rating:

Just average the accuracy of all orders
Order 1: 94.2
Order 2: 95.8
Order 3: 100
Order 4: 100
Order 5: 100

Overall Accuracy: 98

Take note that MHE overall accuracy computation is more lenient compared to the standard
formula, where:

Accuracy = 100 (no. of error*weight))

Overall Accuracy = 100-((2*(2.9))+((1*(2.9))+((2*(0.65))+0+0+0)))

Overall Accuracy: 92.9
Confidentiality Statement:
The information contained herein is confidential and has been provided with the understanding that it will not be disseminated
outside your organization.

Page 5 of 10

Accuracy Trending (using verification date)

Figure 1. Weekly: Error and Accuracy Trending.

The data is provided by MHE. SPi downloaded the inspected and verified orders in MHE quality
system on a weekly basis to check the achieved overall and per agent accuracy for the week.
SPi also verifies the inspection results on a weekly basis and provides the verification results to
MHE. SPis focuses in the validity of flagged errors based on the provided instructions, updates
and standard operating procedures. Reconcile the discrepancy on the data and accuracy rating if
flagged errors are found invalid.
Since MHE started the live production last September, SPi already processed an approximately
7000 orders.
Based on the provided data, there were 5045 (72%) processed orders went inspections and
verifications by MHE and out of which, 326 (6%) are found with errors.
It is worth to note that the team managed to lessen the errors week-on-week as they acquire
learning curve, yielding to 77% error count reduction.
However, the recorded error reduction doesnt influence the overall accuracy to improve this is
because each error has its own equivalent weight according to criticality.
o MHE provided the list of error opportunities in an order.
o It consist of 46 error types. Each error type has its own weight based on criticality.
o The weight ranges from .32, being the lowest, to 3.55 being the highest.

Confidentiality Statement:
The information contained herein is confidential and has been provided with the understanding that it will not be disseminated
outside your organization.

Page 6 of 10

If an agent processed an average of 20 orders per day, lets say this is equivalent to 100
orders per week. Table 2 suggests how many error count(s) are allowed for the agent
not to get an accuracy lower than 99.97.

Table 2. Simple Simulation: allowable error count per weight (if N=100)

Error types with

Weight of:
Allowed number of errors Accuracy
3.55
0
99.96
3.23
1
99.97
2.9
1
99.97
2.58
1
99.97
2.26
1
99.98
1.94
1
99.98
1.61
2
99.97
1.29
2
99.97
0.97
3
99.97
0.65
5
99.97
0.32
10
99.97

Shown in Figure 1, on the first six weeks SPi achieved and exceeded the overall accuracy
threshold. Please note that MHE set a lower threshold on the said weeks as SPi, at that time, is
still undergoing training and guided live production.
Starting WK 6, the chatter has become stricter in terms of their responses and MHE limit some
assistance to SPi agents as the skills are already built up and the training has already completed.
On WK 7, the threshold was raised to 99.97. Since then, SPi consistently missed to achieve the
threshold please note the WK 16 (99.96 accuracy rate) almost hit the set threshold.

Confidentiality Statement:
The information contained herein is confidential and has been provided with the understanding that it will not be disseminated
outside your organization.

Page 7 of 10

Accuracy Trending (per Agent)

Table 3. Week-on-week Agents Accuracy

Agents
Ibanez, John
Anonuevo, Sandra
Marcos, Von
Alzaga, Jaime
Andula, Amor
Manalang, Martin
Domasig, Sinan
Quinit, Rowell
Agustin, hercel
Reyes, Stephanie
Threshold

WK 1
98.67
97.82
98.49

WK 2
98.77
99.27
99.19
99.62
99.99
99.17
99.11
99.76
99.56
98.74
97.99

WK 3
99.68
99.45
99.78
99.89
99.66
99.56
99.47
99.95
99.62

WK 4
99.93
100
99.9
99.68
99.91
99.83
100
99.8
100

WK 5
99.92
99.67
99.87
99.43
99.67
100
100
100
99.88

WK 6
99.88
99.82
100
99.94
99.9

WK 7
99.93
99.83
99.89
99.96
99.86

WK 8
99.89
100
99.85
99.71
99.89

WK 9
99.85
100
99.8
100
99.88

99.89
99.94

100
100

99.83
100

99.93
99.85

98.99

99.5

99.6

99.6

99.97

99.97

99.97

Agents
Ibanez, John
Anonuevo, Sandra
Marcos, Von
Alzaga, Jaime
Andula, Amor
Manalang, Martin
Domasig, Sinan
Quinit, Rowell
Agustin, hercel
Reyes, Stephanie
Threshold

WK 10
99.59
100

WK 11
100
100
99.94
99.87
99.94

WK 12
99.52
99.95
99.91
99.93

WK 13
99.97
99.46
99.8
99.63
99.67

WK 14
99.93
100
99.86
100
99.78

WK 15
99.94
99.68
99.97
99.75
99.89

WK 16
99.89
100
99.88
99.8
100

WK 17
99.59
99.29
100
98.52
99.86

99.84
100

99.63
99.82

100
100

100
99.93

100
99.92

99.47
99.78

99.79
100

100
100

99.97

99.97

99.97

99.97

99.97

99.97

99.97

99.97

99.49
98.65
99.26
99.33
99.41
100
96.99

Met or exceeded the threshold

Released from MHE QA inspection

Despite that SPi recorded a below par overall accuracy rating. In agent level performance,
shown in Table 3, there are 4 agents that already hitting the accuracy of 99.97 and 100%.
It is also worth to mention that these agents were released from rigorous MHE quality
inspection.
o Only those processed orders that were tagged with hold are the one will undergo
MHE inspection. This will give them a great chance to achieve consistent higher accuracy
rating.
It is highly suggested to continuously coach and mentor the three remaining agents, Amor,
Jaime and John for them to be released in MHE inspection

Confidentiality Statement:
The information contained herein is confidential and has been provided with the understanding that it will not be disseminated
outside your organization.

Page 8 of 10

Top Error Contributor

Table 4. Top 11 Errors (covering WK7 to present)
Row Labels
WK 7 WK 8 WK 9 WK 10 WK 11 WK 12 WK 13 WK 14 WK 15 WK 16 WK 17 Grand Total
Incorrect / No Hold Applied
1
6
4
6
5
1
4
1
1
29
Attachments
1
3
3
2
1
2
4
16
Cust #
3
2
1
2
2
1
11
Bill To Acct #
1
1
4
3
9
Contact Name
1
1
3
2
2
9
Items Left Off
1
2
1
4
1
9
PO Amount
2
1
1
1
1
1
1
8
Wrong Item #
1
1
3
1
6
PO#
1
1
2
1
5
Payment Terms
1
1
2
4
Request Date (Future Ship)
1
3
4

Shown in Table 4, is the top 11 error contributors. It is worth to mention that the error relating
to Incorrect/No Hold Applied, which is consistently topped since week one, has been zero out
for two consecutive weeks already.
On the other side, errors relating to Attachments, payment terms and request date has doubled
their digits since it was last acquired.
Causes of the top errors
o Incorrect / No Hold Applied resulted from misinterpretation of the agent based on the
given information and instruction on how he/she will tag the order if it will be on hold.
o Attachment also resulted from misinterpretation or missed instructions by the agent
on the given information, for example the agent will include specific information which
is needed in the invoice and billing
o Cust # and Bill to Acct # - these errors are attributed to insufficient searching and weak
verification of the customer information thus incorrect number was used in an order.
o Contact name and Request date (future Ship) these errors are attributed to incorrect
decision making of the agent.
Contact name and future ship has different labels in different forms and
sometimes it doesnt have label at all. The agent need to have keen eyes to
important key words for them to identify it correctly
o Items left off this is purely negligence as the information are clearly indicated in the
form.

Confidentiality Statement:
The information contained herein is confidential and has been provided with the understanding that it will not be disseminated
outside your organization.

Page 9 of 10

Comparison of internal QC vs MHE QC

The table 5 shows the MHE verified errors versus the internal QC of SPi in terms of number of
orders. It can be seen that of the total 253 orders with MHE verified errors from Week 4-17 that 66% is
not inspected by SPI. These 167 occurrences are caused by a variety of reasons for example: different
sampling of MHE compared to SPi or due to the Quality operators workload. Only 81(32.02%) of the MHE
verified errors were inspected by SPi. Also Booked is defined as files taken by MHE before SPi could have
inspected the order file but still inspected by SPi.
Table 5: Total Orders with MHE Verified Errors
Total Orders with MHE verified errors

253

Not QC'd by order

167

66.01%

QC'd by order with verified errors

81

32.02%

Booked

2.99%

For further analysis of the internal QC versus the MHE verified errors the table 6 was constructed.
The table represents same error types discovered by SPi and MHE for 4 different orders out of the 81. This
signifies that these operators found the error and revised the error but sent back by MHE with the same
error type.
Table 6: Orders with the same Error Type discovered by MHE and SPi
Sales Order number with same Error Type

QC Operator

Error Type

38303202

Jaime

Incorrect/No Hold Applied

38497924

John

Cust#

38154421

Sinan

PO Amount

38242868

Sinan

PO Amount

For table 7 the internal QC versus the MHE verified errors shows the number of orders flagged by
the quality inspector as no error but verified by MHE to have errors. The difference of the Order flagged
as no error and QCd Total is defined as files flagged with errors but verified by MHE to have different
errors and previously inspected by SPi. It can already be seen in the % Total column that majority of the
orders returned by MHE are not seen by the inspectors except for Von and Jaime.

Confidentiality Statement:
The information contained herein is confidential and has been provided with the understanding that it will not be disseminated
outside your organization.

Page 10 of 10

Table 7: QCd orders but flagged as no error by quality inspector

QC'd orders but flagged as no
error by quality inspector

Order flagged
as no error

QC'd
Total

Sinan

20

24

83.33%

Martin

22

24

91.67%

Von

11

13

84.62%

John

37.50%

Jaime

75.00%

Thom

100.00%

Hercel

100.00%

Amor

100.00%

% Total

Action Items and Timelines

Items
Alignment of internal and external quality
inspection (comparison between internal and
external errors)
Create an intricate FAQ and knowledge
database of the common errors incurred that
will simplify the client procedure
Identify best practice from Hercel and Sandra
will be implemented to the rest of the team
Make QC more proactive; identifying and
containing the problem as quick as possible
Conduct time study
Audit the effectiveness of previous identified
corrective actions

Responsible

Timeline

Status

Miguel Canta

Jan. 22

On-going

Bryan Espiritu

Jan. 22

On-going

Miguel and Bryan

Jan. 22

On-going

Miguel Canta
Miguel Canta

Jan. 29
Jan. 22

On-going
On-going

Miguel Canta

Jan. 15

On-going

Confidentiality Statement:
The information contained herein is confidential and has been provided with the understanding that it will not be disseminated
outside your organization.