Vous êtes sur la page 1sur 26

20/05/2015 07:34:51 CGW

Author:ALANSO BESSMAN
This site is down and not responding to poll please investigate.

INCIDENT CONTACT DETAILS:


---------------------------------------------------Name: 'ALANSO BESSMAN'
Tel No: '1-6517954300-'
Email:
'ALANSO.BESSMAN@ECOLAB.COM'
20/05/2015 07:34:53 CGW
Ticket submitted at 20/05/15 10:34:51 GMT
Service Impact changed to 3 - Medium.
CUSTOMER UPDATE:
'Ticket Created' email sent to: ALANSO.BESSMAN@ECOLAB.COM
emec@ecolab.com
rachel.maskew@bt.com
jason.reed@bt.com
eric.rotunda@bt.com
20/05/2015 07:38:06 ANALISTA 1
Operational Categorisation values Changed to:
Tier 1 - BAU FAILURE
Tier 2 - BAU FAILURE
Tier 3 - UNAVAILABLE/NO RESPONSE
Product Categorisation Values Changed to:
Tier 1 - DEVICE PRODUCT
Tier 2 - ROUTERS
Tier 3 - CISCO 1941
Element changed to na-ecola-car70-dal-am at NA-CARROLTON-TX-1368.
CUSTOMER UPDATE:
'Modify Ticket' email sent to: ALANSO.BESSMAN@ECOLAB.COM
emec@ecolab.com
rachel.maskew@bt.com
jason.reed@bt.com
eric.rotunda@bt.com
20/05/2015 07:38:29 ANALISTA 1
Assignee changed tO ANALISTA 1
20/05/2015 07:40:07 ANALISTA 2
Assignee changed to ANALISTA 2
Severity changed to P3.
20/05/2015 09:00:58 ANALISTA 3
A: Inbound call from Ecolab, spoke with Sca. He informed that there is power on

location
A: Upgraded to P1
N: Engage PTT
Severity changed to P1.
Service Impact changed to 1 - Severe.
CUSTOMER UPDATE:
'Modify Ticket' email sent to: ALANSO.BESSMAN@ECOLAB.COM
emec@ecolab.com
rachel.maskew@bt.com
jason.reed@bt.com
eric.rotunda@bt.com
20/05/2015 11:41:11 ANALISTA 4
Assignee changed to ANALISTA 4
20/05/2015 11:52:24 ANALISTA 5
Assignee changed to ANALISTA 5
20/05/2015 12:30:29 ANALISTA 5
D - Link for the circuit BCRL0493 is down
T PE14-DAL-AM#show interface Serial2/0/1/25:0
Serial2/0/1/25:0 is down, line protocol is down
Hardware is PMC FREEDM, PMC TEMUX, TDK LIU
Description: Service:IGNITE_LL MPLS,CID:01007554,SR:1398381,CE:na-ecola-car70dal-am,Speed:3072,on:14-05-2014,by:indalkv
MTU 1500 bytes, BW 1536 Kbit, DLY 20000 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation FRAME-RELAY, crc 16, loopback not set
Keepalive not supported
Bundle Link of bundle MFR985
Last input 05:27:57, output 05:27:55, output hang never
Last clearing of "show interface" counters 44w5d
Input queue: 0/75/0/7 (size/max/drops/flushes); Total output drops: 2
Interface Serial2/0/1/25:0 queueing strategy: PXF Class-based
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
427279385 packets input, 3839553248 bytes, 0 no buffer
Received 0 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
1143 input errors, 215 CRC, 464 frame, 0 overrun, 0 ignored, 464 abort
531023186 packets output, 1296049121 bytes, 0 underruns
7 output errors, 0 collisions, 19 interface resets
0 output buffer failures, 0 output buffers swapped out
0 carrier transitions alarm present
Timeslot(s) Used: 1-24, subrate: 1536Kb/s, transmit delay is 0 flags
non-inverted data
PE14-DAL-AM#
D - Link is down for for the circuit BCRL0494
T PE14-DAL-AM#show interface Serial2/0/1/26:0
Serial2/0/1/26:0 is down, line protocol is down
Hardware is PMC FREEDM, PMC TEMUX, TDK LIU
Description: Service:IGNITE_LL MPLS,CID:01007554,SR:1398381,CE:na-ecola-car70dal-am,Speed:3072,on:14-05-2014,by:indalkv

MTU 1500 bytes, BW 1536 Kbit, DLY 20000 usec,


reliability 255/255, txload 1/255, rxload 1/255
Encapsulation FRAME-RELAY, crc 16, loopback not set
Keepalive not supported
Bundle Link of bundle MFR985
Last input 05:39:59, output 05:39:59, output hang never
Last clearing of "show interface" counters 44w5d
Input queue: 0/75/0/5 (size/max/drops/flushes); Total output drops: 4
Interface Serial2/0/1/26:0 queueing strategy: PXF Class-based
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
427039352 packets input, 3997747690 bytes, 0 no buffer
Received 0 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
865 input errors, 129 CRC, 368 frame, 0 overrun, 0 ignored, 368 abort
531021461 packets output, 1279368516 bytes, 0 underruns
5 output errors, 0 collisions, 24 interface resets
0 output buffer failures, 0 output buffers swapped out
0 carrier transitions alarm present
Timeslot(s) Used: 1-24, subrate: 1536Kb/s, transmit delay is 0 flags
non-inverted data
PE14-DAL-AM#
C - Ticket has been raised with LEVEL 3 for both circuit
N - Follow up with LEVEL 3 for updates
CUSTOMER UPDATE:
'Modify Ticket' email sent to: ALANSO.BESSMAN@ECOLAB.COM
emec@ecolab.com
rachel.maskew@bt.com
jason.reed@bt.com
eric.rotunda@bt.com
20/05/2015 13:49:48 ANALISTA 5
C - No updates from LEVE 3 yet
111111111111111111111111111111111111111111
C - 1st level escalation since there is no updates from the Level 3 yet
N - Follow up with Level 3 for updates
11111111111111111111111111111111111111111111
CUSTOMER UPDATE:
'Modify Ticket' email sent to: ALANSO.BESSMAN@ECOLAB.COM
emec@ecolab.com
rachel.maskew@bt.com
jason.reed@bt.com
eric.rotunda@bt.com
20/05/2015 16:05:30 ANALISTA 2
A - Inbound call from customer, spoke to TOM, he want BT to escalate the last mi
le PTT.
C - Tom requested to be updated over phone: +16303052404

A - While checking the PE, interface was looped.


S - Link down
A - Customer escalated BT to 1st level.
N - Escalate PTT for updates.
T PE14-DAL-AM#show interface Serial2/0/1/26:0
Serial2/0/1/26:0 is down, line protocol is down
Hardware is PMC FREEDM, PMC TEMUX, TDK LIU
Description: CHAMELEON-LOOPTOOL locked by user 603394964 @ Wed May 20 19:00:41
2015
Internet address is 10.10.11.29/30
MTU 1500 bytes, BW 1536 Kbit, DLY 20000 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation HDLC, crc 16, loopback not set
Keepalive set (10 sec)
Last input 01:21:15, output 01:20:53, output hang never
Last clearing of "show interface" counters 00:00:05
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Interface Serial2/0/1/26:0 queueing strategy: PXF Class-based
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
0 packets input, 0 bytes, 0 no buffer
Received 0 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
0 packets output, 0 bytes, 0 underruns
0 output errors, 0 collisions, 1 interface resets
0 output buffer failures, 0 output buffers swapped out
0 carrier transitions alarm present
Timeslot(s) Used: 1-24, subrate: 1536Kb/s, transmit delay is 0 flags
non-inverted data
PE14-DAL-AM#
Assignee changed to Edion Dias.
CUSTOMER UPDATE:
'Modify Ticket' email sent to: ALANSO.BESSMAN@ECOLAB.COM
emec@ecolab.com
rachel.maskew@bt.com
jason.reed@bt.com
eric.rotunda@bt.com
20/05/2015 16:52:03 ANALISTA 6
Assignee changed to Felipe Jacob.
20/05/2015 17:30:01 ANALISTA 6
############################################################################
############################################################################
D:

- Circuit is still down/down

A:
- BT-NOC SP checked the PE and the technical logs show the circuit still
DOWN/Down (looped)
- BT-NOC SP also checked the tickets raised with Level3 in the web porta
l and found new info.
C:

- Level3 updates:

----------9251305
----------Carrier Escalation Management is reviewing your case at this tim
e and we have reached out to the local carrier to be informed the last mile carr
ier has this set up for a dispatch. I have requested for them to find out if a d
ispatch is even needed or if this can just be dropped remotely. I will be updati
ng this in the next hour.
***************************************
The local carrier is seeing a loop on this that they can not rel
ease. There is a last mile carrier that is being involved at this time. Level 3
will provide another update in about 1 hour.
***************************************
***************************************
***************************************
----------9251337
----------Opened up a ticket with the local exchange carrier/LEC. They were able to test g
ood to the smart jack but not able to loop end user equipment.
N:

Next:
- BT-NOC SP to follow up with Level3 for updates about both tickets rais
ed with them.
=================================================================
----------------------------------------------------------------T:
- Technical Logs:
-------------------------------------------------------------------------------PE14-DAL-AM - Serial2/0/1/25:0
---------------PE14-DAL-AM#show interface Serial2/0/1/25:0
Serial2/0/1/25:0 is down, line protocol is down (Looped)
Hardware is PMC FREEDM, PMC TEMUX, TDK LIU
Description: CHAMELEON-LOOPTOOL locked by user 606198606 @ Wed May 20 20:21:38
2015
Internet address is 10.10.11.21/30
MTU 1500 bytes, BW 1536 Kbit, DLY 20000 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation HDLC, crc 16, loopback not set
Keepalive set (10 sec)
Last input 00:03:21, output 00:03:07, output hang never
Last clearing of "show interface" counters 00:03:31
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Interface Serial2/0/1/25:0 queueing strategy: PXF Class-based
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
1 packets input, 24 bytes, 0 no buffer
Received 0 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort


2 packets output, 48 bytes, 0 underruns
0 output errors, 0 collisions, 1 interface resets
0 output buffer failures, 0 output buffers swapped out
0 carrier transitions alarm present
Timeslot(s) Used: 1-24, subrate: 1536Kb/s, transmit delay is 0 flags
non-inverted data
PE14-DAL-AM#
=================================================================
############################################################################
############################################################################
CUSTOMER UPDATE:
'Modify Ticket' email sent to: ALANSO.BESSMAN@ECOLAB.COM
emec@ecolab.com
rachel.maskew@bt.com
jason.reed@bt.com
eric.rotunda@bt.com
20/05/2015 17:53:14 ANALISTA 6
############################################################################
############################################################################
A:

- Level 3 escalated to 2 level.

C:

2222222222222222222222222222222222
This ticket has been escalated and is currently at a Level 2 escalation. Request
ed escalation is currently being reviewed.
2222222222222222222222222222222222
############################################################################
############################################################################
CUSTOMER UPDATE:
'Modify Ticket' email sent to: ALANSO.BESSMAN@ECOLAB.COM
emec@ecolab.com
rachel.maskew@bt.com
jason.reed@bt.com
eric.rotunda@bt.com
20/05/2015 19:06:05 ANALISTA 6
############################################################################
############################################################################
A:
- Bt-NOC contacted Matt from Level 3 and ask him to get a new update for
this ticket ASAP.
He informed that could not reach an available technician to update us and he wil
l call us back in at least 1 hour with new info.
N:

- BT-NOC SP to follow up with Level 3 for updates.

C:

- Level 3 escalated to 3 level.

3333333333333333333333333333333333

This ticket has been escalated and is currently at a Level 3 escalation. Request
ed escalation is currently being reviewed.
3333333333333333333333333333333333
############################################################################
############################################################################
CUSTOMER UPDATE:
'Modify Ticket' email sent to: ALANSO.BESSMAN@ECOLAB.COM
emec@ecolab.com
rachel.maskew@bt.com
jason.reed@bt.com
eric.rotunda@bt.com
20/05/2015 20:52:54 ANALISTA 6
############################################################################
############################################################################
A:
- BT-NOC tried to contact LCON in order to follow the PTT suggestion and
reboot the modem, but no answer to the phone call.
- BT-NOC contacted Level 3 and schedule a FE dispatch on next business h
ours (9:00 am) to the customer site to get the modem checked.
C:

- Last update in Level 3 ticket:

Circuit status :
The Lec carrier is now good to the smart jack , but cannot loop the customer csu
.
I called customer X , he said the customer power and equipment was verified. I
informed of the Lec testing results. X will contact the end-user to have them r
eset the equipment and check the circuit cabling on their end. Holding the tick
et 24 hours to assist the customer further if needed.
N:

Next:
- BT-NOC SP to follow up with Level 3 for the FE dispatch results.

=================================================================
----------------------------------------------------------------T:
- Technical Logs:
-------------------------------------------------------------------------------PE14-DAL-AM - Serial2/0/1/25:0
---------------PE14-DAL-AM#show interface Serial2/0/1/25:0
Serial2/0/1/25:0 is down, line protocol is down (looped)
Hardware is PMC FREEDM, PMC TEMUX, TDK LIU
Description: Service:IGNITE_LL MPLS,CID:01007554,SR:1398381,CE:na-ecola-car70dal-am,Speed:3072,on:14-05-2014,by:indalkv
MTU 1500 bytes, BW 1536 Kbit, DLY 20000 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation FRAME-RELAY, crc 16, loopback not set
Keepalive not supported
Bundle Link of bundle MFR985
Last input 01:31:57, output 01:31:53, output hang never
Last clearing of "show interface" counters 02:28:57
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Interface Serial2/0/1/25:0 queueing strategy: PXF Class-based

5 minute input rate 0 bits/sec, 0 packets/sec


5 minute output rate 0 bits/sec, 0 packets/sec
30 packets input, 1268 bytes, 0 no buffer
Received 0 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
2 input errors, 0 CRC, 1 frame, 0 overrun, 0 ignored, 1 abort
31 packets output, 1315 bytes, 0 underruns
0 output errors, 0 collisions, 2 interface resets
0 output buffer failures, 0 output buffers swapped out
0 carrier transitions alarm present
Timeslot(s) Used: 1-24, subrate: 1536Kb/s, transmit delay is 0 flags
non-inverted data
PE14-DAL-AM#
=================================================================
############################################################################
############################################################################
CUSTOMER UPDATE:
'Modify Ticket' email sent to: ALANSO.BESSMAN@ECOLAB.COM
emec@ecolab.com
rachel.maskew@bt.com
jason.reed@bt.com
eric.rotunda@bt.com
20/05/2015 20:57:44 ANALISTA 6
Ticket Suspended at 20/05/2015 19:30:22. Reason: Awaiting Downtime from Cust. St
atus changed to SUSPEND.
CUSTOMER UPDATE:
'Modify Ticket' email sent to: ALANSO.BESSMAN@ECOLAB.COM
emec@ecolab.com
rachel.maskew@bt.com
jason.reed@bt.com
eric.rotunda@bt.com
20/05/2015 21:03:06 ANALISTA 6
Assignee changed to DISPATCH.
21/05/2015 10:41:18 ANALISTA 5
Ticket returned from Suspension at 21/05/2015 10:41:12. Status changed to WORK I
N PROGRESS.
Ticket at Escalation Level 6. Customer Update due at 22/05/15 19:41:17 GMT
CUSTOMER UPDATE:
'Modify Ticket' email sent to: ALANSO.BESSMAN@ECOLAB.COM
emec@ecolab.com
rachel.maskew@bt.com
jason.reed@bt.com
eric.rotunda@bt.com
21/05/2015 10:51:44 ANALISTA 5
A - Called Level 3 and spoke with Donna
C - Donna informed that LEVE 3 was able to loop until the smart jack and custome

r need to reboot the equipment


A - Email has been sent to LCON request to reboot the equipment
N - Follow up mailbox for updates from LCON
Assignee changed to Vitor Oliveira.
CUSTOMER UPDATE:
'Modify Ticket' email sent to: ALANSO.BESSMAN@ECOLAB.COM
emec@ecolab.com
rachel.maskew@bt.com
jason.reed@bt.com
eric.rotunda@bt.com
21/05/2015 10:52:41 ANALISTA 5
Ticket Suspended at 21/05/2015 10:52:30. Reason: Cust To Supply More Info. Statu
s changed to SUSPEND.
CUSTOMER UPDATE:
'Modify Ticket' email sent to: ALANSO.BESSMAN@ECOLAB.COM
emec@ecolab.com
rachel.maskew@bt.com
jason.reed@bt.com
eric.rotunda@bt.com
21/05/2015 13:07:47 ANALISTA 7
Assignee changed to ANALISTA 7
21/05/2015 14:43:05 ANALISTA 7
New Subcase XF38112689 submitted for NL_AMS_NCCA_IA-MPLS.
21/05/2015 14:44:41 ANALISTA 7
Update from Parent XF38105843 : A - BT NOC received a call from Customer Tom st
ating NO FE on site.
A - BT NOC called PTT in order to have updates, informed that circuit is looped
and needs further investigation.
A - BT NOC was informed that Level 3 will reach last mile for checks because the
y have only on node in this service and have noremotely managemeng for it.
A - BT NOC need that last mile remove loop that probably they have leaved on the
service.
A - BT NOC received a call from LEvel 3 stating no loop placed from their manage
d environment.
A - BT NOC requested FE on site in order to have HARD LOOP test for entire envir
onment validation.
A - BT NOC RAISED a case with NCCA Team in order to health check PE device with
configuration and physical connections.
N - Follow up with NCCA for PE device health checks.
N - Follow up with PTT for updates.
N - KCI Customer ASAP.
C - Dear NCCA Team,
We have service provider stating no loop and no evideces of errors on th
eir managed environment, we would like to ask for your assistance in orde dro ha
ve the PE equipment health checked physically and configuration because we have
interface hard down with a loop placed, Please refer to the below tech logs.
Please let us know in case you have any concerns.
Device details:

***PE INFORMATION***
PE Node : PE14-DAL-AM
PE Port : Serial2/0/1/25:0
PE Port : Serial2/0/1/26:0
Since now, we would like to thank you in advance for your time and atten
tion.
C LCON:
ALANSO BESSMAN
1-6517954300ALANSO.BESSMAN@ECOLAB.COM
emec@ecolab.com
tdassette@nalco.com - TOM
T PE14-DAL-AM#show interface Serial2/0/1/25:0
Serial2/0/1/25:0 is down, line protocol is down (looped)
Hardware is PMC FREEDM, PMC TEMUX, TDK LIU
Description: Service:IGNITE_LL MPLS,CID:01007554,SR:1398381,CE:na-ecola-car70dal-am,Speed:3072,on:14-05-2014,by:indalkv
MTU 1500 bytes, BW 1536 Kbit, DLY 20000 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation FRAME-RELAY, crc 16, loopback not set
Keepalive not supported
Bundle Link of bundle MFR985
Last input 13:49:02, output 13:48:52, output hang never
Last clearing of "show interface" counters 19:28:17
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Interface Serial2/0/1/25:0 queueing strategy: PXF Class-based
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
62 packets input, 2616 bytes, 0 no buffer
Received 0 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
4 input errors, 0 CRC, 2 frame, 0 overrun, 0 ignored, 2 abort
63 packets output, 2663 bytes, 0 underruns
0 output errors, 0 collisions, 3 interface resets
0 output buffer failures, 0 output buffers swapped out
0 carrier transitions alarm present
Timeslot(s) Used: 1-24, subrate: 1536Kb/s, transmit delay is 0 flags
non-inverted data
PE14-DAL-AM#
PE14-DAL-AM#show interface Serial2/0/1/25:0
Serial2/0/1/25:0 is down, line protocol is down (looped)
Hardware is PMC FREEDM, PMC TEMUX, TDK LIU
Description: Service:IGNITE_LL MPLS,CID:01007554,SR:1398381,CE:na-ecola-car70dal-am,Speed:3072,on:14-05-2014,by:indalkv
MTU 1500 bytes, BW 1536 Kbit, DLY 20000 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation FRAME-RELAY, crc 16, loopback not set
Keepalive not supported
Bundle Link of bundle MFR985
Last input 14:02:49, output 14:02:39, output hang never
Last clearing of "show interface" counters 19:42:04
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Interface Serial2/0/1/25:0 queueing strategy: PXF Class-based
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
62 packets input, 2616 bytes, 0 no buffer

Received 0 broadcasts (0 IP multicasts)


0 runts, 0 giants, 0 throttles
4 input errors, 0 CRC, 2 frame, 0 overrun, 0 ignored, 2 abort
63 packets output, 2663 bytes, 0 underruns
0 output errors, 0 collisions, 3 interface resets
0 output buffer failures, 0 output buffers swapped out
0 carrier transitions alarm present
Timeslot(s) Used: 1-24, subrate: 1536Kb/s, transmit delay is 0 flags
non-inverted data
PE14-DAL-AM#
C CUSTOMER UPDATE:
'Modify Ticket' email sent to: ALANSO.BESSMAN@ECOLAB.COM
emec@ecolab.com
rachel.maskew@bt.com
jason.reed@bt.com
eric.rotunda@bt.com
21/05/2015 14:47:16 ANALISTA 7
Ticket returned from Suspension at 21/05/2015 14:47:12. Status changed to WORK I
N PROGRESS.
Ticket at Escalation Level 6. Customer Update due at 22/05/15 23:47:15 GMT
CUSTOMER UPDATE:
'Modify Ticket' email sent to: ALANSO.BESSMAN@ECOLAB.COM
emec@ecolab.com
rachel.maskew@bt.com
jason.reed@bt.com
eric.rotunda@bt.com
21/05/2015 14:59:34 RESOLVER 1
D:Controller is UP
PE14-DAL-AM#sh controller T3 2/0/1
Load for five secs: 2%/1%; one minute: 6%; five minutes: 6%
Time source is NTP, 17:53:46.636 GMT Thu May 21 2015
T3 2/0/1 is up. Hardware is C10K Half Height CT3 line card
Applique type is Channelized T3
Controller is in channelized mode
D: Multiple T1's are also UP UP on the controller indicating a issue with the in
dividual T1 on the Providers network
PE14-DAL-AM#sh int desc | i 2/0/1
Se2/0/1/1:0
up
up
Service:IGNITE_LL MPLS,CI
D:01006074,SR:1316074,CE:bi-bkleu-dal1-dal-am,Speed:1024,on:29-04-2013,by:darade
a
Se2/0/1/1:0.101
up
up
Service:IGNITE_LL MPLS,CI
D:01006074,SR:1316074,CE:bi-bkleu-dal1-dal-am,Speed:1024,on:29-04-2013,by:darade
a
Se2/0/1/2:0
up
up
Service:IGNITE_LL MPLS,CI
D:1142115,SR:1197446,CE:fx-1086900-cs2-us-dal,Speed:,on:05-08-2010,by:shirsar
Se2/0/1/2:0.100
up
up
System:VCATT,CID:1142115,
SiteID:1086900,CE:fx-1086900-cs2-us-dal,VPN:GREY,Speed:32,OrderID:1218509,on:0902-2011,by:mutcht,VER:43_10
Se2/0/1/2:0.101
up
up
System:VCATT,CID:1142115,
SiteID:1086900,CE:fx-1086900-cs2-us-dal,VPN:FE,Speed:232,OrderID:1420165,on:02-0

7-2014,by:payasir,VER:43_10
Se2/0/1/3:0
up
up
Service:IGNITE_LL,Cust:CA
SENEWHOLLA,CID:1144385,SRN:1058567,Status:configured on 16-01-2008 ,by User:KUMA
RIP
Se2/0/1/3:0.101
up
up
Service:IGNITE_LL,Cust:CA
SENEWHOLLA,CID:1144385,SRN:1058567,Status:configured on 16-01-2008 ,by User:KUMA
RIP
Se2/0/1/4:0
up
up
Service:IGNITE_LL MPLS,CI
D:1141058,SR:1433346,CE:gr-basf-pho71-dal-am,Speed:1536,on:13-10-2014,by:dholakn
Se2/0/1/4:0.101
up
up
Service:IGNITE_LL MPLS,CI
D:1141058,SR:1433346,CE:gr-basf-pho71-dal-am,Speed:1536,on:13-10-2014,by:dholakn
Se2/0/1/5:0
down
down
Service:IGNITE_LL MPLS,CI
D:10361,SR:1198766,CE:gr-nestl-den1-dal-am,Speed:1536,on:17-09-2010,by:kocerm2
Se2/0/1/5:0.101
down
down
Service:IGNITE_LL MPLS,CI
D:10361,SR:1198766,CE:gr-nestl-den1-dal-am,Speed:1536,on:17-09-2010,by:kocerm2
Se2/0/1/6:0
up
up
Service:IGNITE_LL MPLS,CI
D:1144385,SR:1215076,CE:sc-case-spr1-dal-am,Speed:1536,on:29-12-2010,by:shahpa
Se2/0/1/6:0.101
up
up
Service:IGNITE_LL MPLS,CI
D:1144385,SR:1215076,CE:sc-case-spr1-dal-am,Speed:1536,on:29-12-2010,by:shahpa
Se2/0/1/7:0
up
up
Service:IGNITE_LL MPLS,CI
D:01004347,SR:1252053,CE:na-mars-wac71-dal-am,Speed:6144,on:28-09-2011,by:patwar
a5
Se2/0/1/8:0
up
up
Service:IGNITE_LL MPLS,CI
D:01004347,SR:1252053,CE:na-mars-wac71-dal-am,Speed:6144,on:28-09-2011,by:patwar
a5
Se2/0/1/9:0
up
up
Service:IGNITE_LL MPLS,CI
D:01004347,SR:1252053,CE:na-mars-wac71-dal-am,Speed:6144,on:28-09-2011,by:patwar
a5
Se2/0/1/10:0
up
up
Service:IGNITE_LL MPLS,CI
D:01004347,SR:1252053,CE:na-mars-wac71-dal-am,Speed:6144,on:28-09-2011,by:patwar
a5
Se2/0/1/11:0
down
down
Service:IVS MPLS,ICID:103
61,ISID:S1092549,IAID:1-HEKZ21,Status:configured on 07-07-2011 ,by User:calisoi
Se2/0/1/12:0
down
down
Service:IVS MPLS,ICID:103
61,ISID:S1092549,IAID:1-HEKZ21,Status:configured on 07-07-2011 ,by User:calisoi
Se2/0/1/13:0
up
up
Service:IGNITE_LL MPLS,CI
D:01007989,SR:1435598,CE:na-jabil-el70-dal-am,Speed:1024,on:31-10-2014,by:dholev
Se2/0/1/13:0.101
up
up
Service:IGNITE_LL MPLS,CI
D:01007989,SR:1435598,CE:na-jabil-el70-dal-am,Speed:1024,on:31-10-2014,by:dholev
Se2/0/1/14:0
up
up
Service:IGNITE_LL MPLS,CI
D:1142857,SR:1431215,CE:ch-clari-mid70-dal-am,Speed:1536,on:12-05-2015,by:kariek
tm
Se2/0/1/14:0.101
up
up
Service:IGNITE_LL MPLS,CI
D:1142857,SR:1431215,CE:ch-clari-mid70-dal-am,Speed:1536,on:12-05-2015,by:kariek
tm
Se2/0/1/15:0
down
down
Service:IVS MPLS,ICID:103
61,ISID:S1092549,IAID:1-HEKZ21,Status:configured on 07-07-2011 ,by User:calisoi
Se2/0/1/16:0
down
down
Service:IVS MPLS,ICID:103
61,ISID:S1092549,IAID:1-HEKZ21,Status:configured on 07-07-2011 ,by User:calisoi
Se2/0/1/17:0
up
up
Service:IGNITE_LL MPLS,CI
D:1137537,SR:1450930,CE:ch-synge-wes70-dal-am,Speed:128,on:27-03-2015,by:dumrev
Se2/0/1/17:0.101
up
up
Service:IGNITE_LL MPLS,CI
D:1137537,SR:1450930,CE:ch-synge-wes70-dal-am,Speed:128,on:27-03-2015,by:dumrev
Se2/0/1/18:0
up
up
Service:IGNITE_LL MPLS,CI
D:1144147,SR:1268128,CE:na-exide-irv71-dal-am,Speed:256,on:14-03-2012,by:sayyedj
2
Se2/0/1/18:0.101
up
up
Service:IGNITE_LL MPLS,CI
D:1144147,SR:1268128,CE:na-exide-irv71-dal-am,Speed:256,on:14-03-2012,by:sayyedj
2
Se2/0/1/21:0
up
up
Service:IGNITE_LL MPLS,CI

D:1100200,SR:1265271,CE:be-us350-mce1-dal-am,Speed:768,on:21-01-2012,by:nurseca
Se2/0/1/21:0.101
up
up
Service:IGNITE_LL MPLS,CI
D:1100200,SR:1265271,CE:be-us350-mce1-dal-am,Speed:768,on:21-01-2012,by:nurseca
Se2/0/1/22:0
up
up
Service:IGNITE_LL MPLS,CI
D:01007490,SR:1381878,CE:al-prysm-hut01-dal-am,Speed:1536,on:30-12-2013,by:orabo
nl
Se2/0/1/22:0.101
up
up
Service:IGNITE_LL MPLS,CI
D:01007490,SR:1381878,CE:al-prysm-hut01-dal-am,Speed:1536,on:30-12-2013,by:orabo
nl
Se2/0/1/23:0
up
up
Service:IGNITE_LL MPLS,CI
D:1142115,SR:1278282,CE:fx-1110021-ce1-us-dal,Speed:,on:03-04-2012,by:zalarx
Se2/0/1/23:0.100
up
up
System:VCATT,CID:1142115,
SiteID:1110021,CE:fx-1110021-ce1-us-dal,VPN:GREY,Speed:32,OrderID:1278282,on:0304-2012,by:zalarx,VER:54_10
Se2/0/1/23:0.101
up
up
System:VCATT,CID:1142115,
SiteID:1110021,CE:fx-1110021-ce1-us-dal,VPN:FE,Speed:88,OrderID:1278282,on:26-04
-2012,by:enukorg2,VER:56_10
Se2/0/1/24:0
up
up
Service:IGNITE_LL MPLS,CI
D:01007554,SR:1395800,CE:na-ecola-dal70-dal-am,Speed:1536,on:19-03-2014,by:nagar
sa
Se2/0/1/24:0.101
up
up
Service:IGNITE_LL MPLS,CI
D:01007554,SR:1395800,CE:na-ecola-dal70-dal-am,Speed:1536,on:19-03-2014,by:nagar
sa
Se2/0/1/25:0
down
down
Service:IGNITE_LL MPLS,CI
D:01007554,SR:1398381,CE:na-ecola-car70-dal-am,Speed:3072,on:14-05-2014,by:indal
kv
Se2/0/1/26:0
down
down
Service:IGNITE_LL MPLS,CI
D:01007554,SR:1398381,CE:na-ecola-car70-dal-am,Speed:3072,on:14-05-2014,by:indal
kv
Se2/0/1/27:0
up
up
Service:IGNITE_LL MPLS,CI
D:01003084,SR:1406957,CE:aa-tatac-dal70-dal-am,Speed:768,on:05-09-2014,by:paswan
r
Se2/0/1/27:0.101
up
up
Service:IGNITE_LL MPLS,CI
D:01003084,SR:1406957,CE:aa-tatac-dal70-dal-am,Speed:768,on:05-09-2014,by:paswan
r
Se2/0/1/28:0
up
up
Service:IGNITE_LL MPLS,CI
D:37646,SR:1412488,CE:br-emr-new70-dal-am,Speed:1536,on:30-06-2014,by:autim2
Se2/0/1/28:0.101
up
up
Service:IGNITE_LL MPLS,CI
D:37646,SR:1412488,CE:br-emr-new70-dal-am,Speed:1536,on:30-06-2014,by:autim2
N:Please complete loop testing at PE POP site and prove the fault is not on the
providers network
Incident Ended at 21/05/15 17:59:30 GMT and Customer Informed of Clear at 21/05
/15 17:59:31 GMT. Status Changed to AWAITING CLOSURE.
Ticket at Escalation Level 0. Customer Update due at 22/05/15 23:59:32 GMT
*** END OF UPDATE
21/05/2015 15:04:38 ANALISTA 9
Internal Use ONLY:
C - Keep following up with the case
21/05/2015 15:12:06 DUTY MGR 1
======================================= << Duty Manager >> ====================
===================
T -

S - Both links down. One of them is showing looped.


A - SD was on the call with Level 3 and performed many tests. Need to have now a
FE on site for a hard loop.
C - According to Level 3, the FE and only be dispatched tomorrow @ 13:00PM. Not
acceptable.
A - I escalated Level 3 to have the dispatch today.
A - Customer and SRM informed on email chain.
N - Follow up with Level 3 for the FE ETA and keep customer informed.
C - Level 3 tickets escalations levels : NOW
9251305 ---> 4th level
9251337 ---> 2nd level
======================================= << Duty Manager >> ====================
===================
CUSTOMER UPDATE:
'Modify Ticket' email sent to: ALANSO.BESSMAN@ECOLAB.COM
emec@ecolab.com
rachel.maskew@bt.com
jason.reed@bt.com
eric.rotunda@bt.com
21/05/2015 15:41:53 DUTY MGR 1
======================================= << Duty Manager >> ====================
===================
T
S
D
A
N
C

- I Spoke with Terry from Level 3. ( Tech that is working on this case)
- Keep checking updates with Level 3.
- Terry will call their LEC now and try to arrange a best ETA.

======================================= << Duty Manager >> ====================


===================
CUSTOMER UPDATE:
'Modify Ticket' email sent to: ALANSO.BESSMAN@ECOLAB.COM
emec@ecolab.com
rachel.maskew@bt.com
jason.reed@bt.com
eric.rotunda@bt.com
21/05/2015 17:04:42 DUTY MGR 2
======================================= << Duty Manager >> ====================
===================
A - Engaged Global escalation team.
A - Will escalate it further with Level 3
Dear Global Escalation Management Team,
We need your help in order to proceed with further escalation with Level

3 on the tickets below.


"
"

9251305 ---> 4th level


9251337 ---> 3rd level

Facts :
"
"

Site is isolated over 24 hours now;


Level 3 does not provide consistent updates.

Thank you.
======================================= << Duty Manager >> ====================
===================
CUSTOMER UPDATE:
'Modify Ticket' email sent to: ALANSO.BESSMAN@ECOLAB.COM
emec@ecolab.com
rachel.maskew@bt.com
jason.reed@bt.com
eric.rotunda@bt.com
21/05/2015 17:09:20 ANALISTA 10
Assignee changed to ANALISTA 10
21/05/2015 17:11:10 DUTY MGR 2
======================================= << Duty Manager >> ====================
===================
Silvaeo
A - Called Global escalation team talked with Dean and requested help, he will l
ook into the case and and will asist us.
C - Dean requested us to escalate to the next level and copy them so they can pr
oceed with the case.

======================================= << Duty Manager >> ====================


===================
CUSTOMER UPDATE:
'Modify Ticket' email sent to: ALANSO.BESSMAN@ECOLAB.COM
emec@ecolab.com
rachel.maskew@bt.com
jason.reed@bt.com
eric.rotunda@bt.com
21/05/2015 17:30:32 DUTY MGR 2
======================================= << Duty Manager >> ====================
===================
Level 3,
Can you please advise what is the action plan to repair the links under
the tickets below, our customer is down over 24 hours now, and we do not have c
onsistence updates on what is going on, please advise us what is the latest upda

tes on the case.


"
"

9251305
9251337

Thank you.
======================================= << Duty Manager >> ====================
===================
CUSTOMER UPDATE:
'Modify Ticket' email sent to: ALANSO.BESSMAN@ECOLAB.COM
emec@ecolab.com
rachel.maskew@bt.com
jason.reed@bt.com
eric.rotunda@bt.com
21/05/2015 17:31:45 ANALISTA 10
A - Called Level 3 for status, spoke with Terrel, informed they have tech to be
dispatched still today or tomorrow in the morninh to test on the smartjack. I co
ntacted customer, on the phone 1 -972 416 1080, John Garze, but hes not avaialble
, ring no answer.
N - To follow-up with Level 3 for updates, confirmation on the dispatch and info
rm Lcon.
CUSTOMER UPDATE:
'Modify Ticket' email sent to: ALANSO.BESSMAN@ECOLAB.COM
emec@ecolab.com
rachel.maskew@bt.com
jason.reed@bt.com
eric.rotunda@bt.com
21/05/2015 18:09:47 DUTY MGR 2
A - Called customer Tom and left him a message informing the link was restored.
A - Called Alonso and informed him the link was restored, we will monitor the li
nk for 24 hours and get the RFO.
N - Follow up with Level 3 on the RFO and monitor the link for 24 hours.
C - Customer contact details.
Name: 'ALANSO BESSMAN'
Tel No: '1-6517954300-'
Email:
'ALANSO.BESSMAN@ECOLAB.COM'
KCI
Ecolab team,
I just spoke with Alonso and he confirmed the services were restored, I
will pursue the reason for outage and will advise you once carrier provide such
details.
Thank you.
CUSTOMER UPDATE:
'Modify Ticket' email sent to: ALANSO.BESSMAN@ECOLAB.COM

emec@ecolab.com
rachel.maskew@bt.com
jason.reed@bt.com
eric.rotunda@bt.com
21/05/2015 18:15:00 ANALISTA 11
A - Received a call fromTerry/Level 3 about the following circuits:
BCRL0493 - Technician went to the site and found a problem with the router and f
ixed it. He was not able to provide us more information. - Circuit is up and run
ning
BCRL0494 - tech went to the site and found AIS alarm coming from CE. He informe
d we need to check the equipment
N - Follow up for updates
CUSTOMER UPDATE:
'Modify Ticket' email sent to: ALANSO.BESSMAN@ECOLAB.COM
emec@ecolab.com
rachel.maskew@bt.com
jason.reed@bt.com
eric.rotunda@bt.com
21/05/2015 19:19:55 DUTY MGR 2
======================================= << Duty Manager >> ====================
===================
A - Called Level 3 duty manager and they informed the issue is on our router whi
ch I disagree, as the link was up over 45 minutes
I reengaged our Global escalation team, and requested a new router and a case to
be raiase with GCS, X is currently
raising the tickets.
======================================= << Duty Manager >> ====================
===================
CUSTOMER UPDATE:
'Modify Ticket' email sent to: ALANSO.BESSMAN@ECOLAB.COM
emec@ecolab.com
rachel.maskew@bt.com
jason.reed@bt.com
eric.rotunda@bt.com
21/05/2015 19:54:08 ANALISTA 11
A - I have called Level 3 Terrel, he has the Lcon Name and number ( 214 876 5092
), hes going to send a FE engineer to work on the circuit at the smartjack.
N - To follow-up with Level 3 for updates.
CUSTOMER UPDATE:
'Modify Ticket' email sent to: ALANSO.BESSMAN@ECOLAB.COM
emec@ecolab.com
rachel.maskew@bt.com
jason.reed@bt.com
eric.rotunda@bt.com
22/05/2015 00:50:29 ANALISTA 12

Assignee changed to ANALISTA 12.


22/05/2015 01:08:49 ANALISTA 12
+++++++++++++++++++++++++++++++++++++++++
D - Link for the circuit BCRL0493 is down
T:
PE14-DAL-AM#show interface Serial2/0/1/25:0
Serial2/0/1/25:0 is down, line protocol is down
Hardware is PMC FREEDM, PMC TEMUX, TDK LIU
Description: Service:IGNITE_LL MPLS,CID:01007554,SR:1398381,CE:na-ecola-car70dal-am,Speed:3072,on:14-05-2014,by:indalkv
MTU 1500 bytes, BW 1536 Kbit, DLY 20000 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation FRAME-RELAY, crc 16, loopback not set
Keepalive not supported
Bundle Link of bundle MFR985
Last input 06:27:02, output 06:26:58, output hang never
Last clearing of "show interface" counters 07:20:01
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 1
Interface Serial2/0/1/25:0 queueing strategy: PXF Class-based
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
139177 packets input, 29239543 bytes, 0 no buffer
Received 0 broadcasts (0 IP multicasts)
0 runts, 1 giants, 0 throttles
11961 input errors, 432 CRC, 5764 frame, 0 overrun, 0 ignored, 5764 abort
143115 packets output, 103824029 bytes, 0 underruns
0 output errors, 0 collisions, 1 interface resets
0 output buffer failures, 0 output buffers swapped out
0 carrier transitions alarm present
Timeslot(s) Used: 1-24, subrate: 1536Kb/s, transmit delay is 0 flags
D - Link is down for for the circuit BCRL0494
PE14-DAL-AM#show interface Serial2/0/1/26:0
Serial2/0/1/26:0 is down, line protocol is down
Hardware is PMC FREEDM, PMC TEMUX, TDK LIU
Description: Service:IGNITE_LL MPLS,CID:01007554,SR:1398381,CE:na-ecola-car70dal-am,Speed:3072,on:14-05-2014,by:indalkv
MTU 1500 bytes, BW 1536 Kbit, DLY 20000 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation FRAME-RELAY, crc 16, loopback not set
Keepalive not supported
Bundle Link of bundle MFR985
Last input 06:28:15, output 06:28:18, output hang never
Last clearing of "show interface" counters 07:13:58
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Interface Serial2/0/1/26:0 queueing strategy: PXF Class-based
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
109781 packets input, 21659847 bytes, 0 no buffer
Received 0 broadcasts (0 IP multicasts)
0 runts, 1 giants, 0 throttles
12180 input errors, 461 CRC, 5859 frame, 0 overrun, 0 ignored, 5859 abort
115408 packets output, 87127449 bytes, 0 underruns
0 output errors, 0 collisions, 1 interface resets
0 output buffer failures, 0 output buffers swapped out
0 carrier transitions alarm present
Timeslot(s) Used: 1-24, subrate: 1536Kb/s, transmit delay is 0 flags

C:
--- Latest update is that BT Escalation and Service Desk Management has arrange
d with PTT Management for a vendor meet (with the LEC to be onsite) tomorrow mo
rning, ETA by 9am local time.
--- Lcon Name and number ( 214 876 5092 ), hes going to send a FE engineer to wo
rk on the circuit at the smartjack.

N - To follow-up with Level 3 for updates.


++++++++++++++++++++++++++++++++++++++
CUSTOMER UPDATE:
'Modify Ticket' email sent to: ALANSO.BESSMAN@ECOLAB.COM
emec@ecolab.com
rachel.maskew@bt.com
jason.reed@bt.com
eric.rotunda@bt.com
22/05/2015 01:16:09 DUTY MGR 2
======================================= << Duty Manager >> ====================
===================
Team,
I tried several times which different analysts and I finally managed to
create a ticket with Unisys.
"

Talked with Unisys Analyst Akshay REF# 33334939

======================================= << Duty Manager >> ====================


===================
CUSTOMER UPDATE:
'Modify Ticket' email sent to: ALANSO.BESSMAN@ECOLAB.COM
emec@ecolab.com
rachel.maskew@bt.com
jason.reed@bt.com
eric.rotunda@bt.com
CUSTOMER UPDATE:
'Modify Ticket' email sent to: ALANSO.BESSMAN@ECOLAB.COM
emec@ecolab.com
rachel.maskew@bt.com
jason.reed@bt.com
eric.rotunda@bt.com
22/05/2015 05:31:22 DUTY MGR 3
======================= << Duty Manager >> =======================
A - Unisys confirmed part ETA to 6 am local time.
A - Replied to Unisys asking to confirm part will not return as customer confirm
ed 9 am local time for vendor meeting.
N - Get confirmation from Unisys and follow up with vendor meeting with Level 3
and Unisys for circuit troubleshooting at 9 am local time.
======================= << Duty Manager >> =======================

CUSTOMER UPDATE:
'Modify Ticket' email sent to: ALANSO.BESSMAN@ECOLAB.COM
emec@ecolab.com
rachel.maskew@bt.com
jason.reed@bt.com
eric.rotunda@bt.com
22/05/2015 07:56:28 ANALISTA 7
Assignee changed to ANALISTA 7
22/05/2015 10:59:26 ANALISTA 7
A - BT NOC called HM/Unisys in order to get updates, spoke with abishek who stat
ed FE details and ETA.
A - BT NOC called PTT/Level3 spoke Joseph who stated No ETA for FE dispatch avai
lable.
A - BT NOC engaged Duty manager for further assistance.
N - Follow up with Level3 for ETA updates.
C - Level3 FE Details:
PENDING.
C - Unisys FE Details:
Robert Chatter
Phone: +1 2144 2544 61
ETA: 09:00 LT.
C - Level 3 to raise a conference call once on site
Ready-Access Number:
8007471915
Secondary Ready-Access Number:
+1 3032481285
7-Digit Access Code:
4164931
T PE14-DAL-AM#show interface Serial2/0/1/25:0
Serial2/0/1/25:0 is down, line protocol is down
Hardware is PMC FREEDM, PMC TEMUX, TDK LIU
Description: Service:IGNITE_LL MPLS,CID:01007554,SR:1398381,CE:na-ecola-car70dal-am,Speed:3072,on:14-05-2014,by:indalkv
MTU 1500 bytes, BW 1536 Kbit, DLY 20000 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation FRAME-RELAY, crc 16, loopback not set
Keepalive not supported
Bundle Link of bundle MFR985
Last input 16:13:09, output 16:13:05, output hang never
Last clearing of "show interface" counters 00:25:59
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Interface Serial2/0/1/25:0 queueing strategy: PXF Class-based
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
0 packets input, 0 bytes, 0 no buffer
Received 0 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
0 packets output, 0 bytes, 0 underruns
0 output errors, 0 collisions, 1 interface resets

0 output buffer failures, 0 output buffers swapped out


0 carrier transitions alarm present
Timeslot(s) Used: 1-24, subrate: 1536Kb/s, transmit delay is 0 flags
non-inverted data
PE14-DAL-AM#
PE14-DAL-AM#show controller T3 2/0/1
T3 2/0/1 is up. Hardware is C10K Half Height CT3 line card
Applique type is Channelized T3
Controller is in channelized mode
Description: USDALB002-USDALB003-45MB-010(BTGS-WW)
No alarms detected.
MDL transmission is disabled
PE14-DAL-AM#show log | include Serial2/0/1/25:0
May 20 09:15:22.679 GMT: %LINK-3-UPDOWN: Interface Serial2/0/1/25:0, changed sta
te to down
May 20 09:15:23.679 GMT: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial2
/0/1/25:0, changed state to down
May 20 09:17:42.586 GMT: %LINK-3-UPDOWN: Interface Serial2/0/1/25:0, changed sta
te to up
May 20 09:17:57.594 GMT: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial2
/0/1/25:0, changed state to up
May 20 09:48:23.538 GMT: %LINK-3-UPDOWN: Interface Serial2/0/1/25:0, changed sta
te to down
May 20 09:48:24.538 GMT: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial2
/0/1/25:0, changed state to down
May 20 18:05:48.940 GMT: %LINK-3-UPDOWN: Interface Serial2/0/1/25:0, changed sta
te to up
May 20 18:09:42.776 GMT: %LINK-3-UPDOWN: Interface Serial2/0/1/25:0, changed sta
te to down
May 20 18:10:37.457 GMT: %LINK-3-UPDOWN: Interface Serial2/0/1/25:0, changed sta
te to up
May 20 18:11:22.002 GMT: %LINK-3-UPDOWN: Interface Serial2/0/1/25:0, changed sta
te to down
May 20 18:11:49.759 GMT: %LINK-3-UPDOWN: Interface Serial2/0/1/25:0, changed sta
te to up
May 20 18:12:02.711 GMT: %LINK-3-UPDOWN: Interface Serial2/0/1/25:0, changed sta
te to down
May 20 18:12:28.427 GMT: %LINK-3-UPDOWN: Interface Serial2/0/1/25:0, changed sta
te to up
May 20 18:16:41.580 GMT: %LINK-3-UPDOWN: Interface Serial2/0/1/25:0, changed sta
te to down
May 20 18:17:16.617 GMT: %LINK-3-UPDOWN: Interface Serial2/0/1/25:0, changed sta
te to up
May 20 18:30:35.761 GMT: %LINK-3-UPDOWN: Interface Serial2/0/1/25:0, changed sta
te to down
May 20 18:35:19.222 GMT: %LINK-3-UPDOWN: Interface Serial2/0/1/25:0, changed sta
te to up
May 20 18:43:59.432 GMT: %LINK-3-UPDOWN: Interface Serial2/0/1/25:0, changed sta
te to down
May 20 18:44:35.681 GMT: %LINK-3-UPDOWN: Interface Serial2/0/1/25:0, changed sta
te to up
May 20 18:45:37.450 GMT: %LINK-3-UPDOWN: Interface Serial2/0/1/25:0, changed sta
te to down
May 20 20:05:34.131 GMT: %LINK-5-CHANGED: Interface Serial2/0/1/25:0, changed st
ate to administratively down
May 20 20:06:21.412 GMT: %LINK-3-UPDOWN: Interface Serial2/0/1/25:0, changed sta
te to down
May 20 20:13:08.728 GMT: %LINK-3-UPDOWN: Interface Serial2/0/1/25:0, changed sta

te to up
May 20 20:22:22.395 GMT: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial2
/0/1/25:0, changed state to up
May 20 20:22:26.715 GMT: %LINK-3-UPDOWN: Interface Serial2/0/1/25:0, changed sta
te to down
May 20 20:22:27.715 GMT: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial2
/0/1/25:0, changed state to down
May 20 21:18:55.081 GMT: %LINK-3-UPDOWN: Interface Serial2/0/1/25:0, changed sta
te to up
May 20 21:19:08.441 GMT: %LINK-3-UPDOWN: Interface Serial2/0/1/25:0, changed sta
te to down
May 20 21:36:15.005 GMT: %LINK-3-UPDOWN: Interface Serial2/0/1/25:0, changed sta
te to up
May 20 21:39:23.113 GMT: %LINK-3-UPDOWN: Interface Serial2/0/1/25:0, changed sta
te to down
May 21 02:18:02.254 GMT: %LINK-3-UPDOWN: Interface Serial2/0/1/25:0, changed sta
te to up
.May 21 02:21:37.334 GMT: %LINK-3-UPDOWN: Interface Serial2/0/1/25:0, changed st
ate to down
May 21 19:40:19.918 GMT: %LINK-3-UPDOWN: Interface Serial2/0/1/25:0, changed sta
te to up
May 21 19:42:26.080 GMT: %LINK-3-UPDOWN: Interface Serial2/0/1/25:0, changed sta
te to down
May 21 19:44:27.583 GMT: %LINK-3-UPDOWN: Interface Serial2/0/1/25:0, changed sta
te to up
May 21 19:47:11.634 GMT: %LINK-3-UPDOWN: Interface Serial2/0/1/25:0, changed sta
te to down
May 21 20:23:04.761 GMT: %CLEAR-5-COUNTERS: Clear counter on interface Serial2/0
/1/25:0 by autoloop on vty0 (212.31.215.247)
May 21 20:24:28.454 GMT: %LINK-3-UPDOWN: Interface Serial2/0/1/25:0, changed sta
te to up
May 21 20:24:41.855 GMT: %LINK-3-UPDOWN: Interface Serial2/0/1/25:0, changed sta
te to down
May 21 20:28:46.883 GMT: %LINK-3-UPDOWN: Interface Serial2/0/1/25:0, changed sta
te to up
May 21 20:29:01.892 GMT: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial2
/0/1/25:0, changed state to up
May 21 20:46:33.276 GMT: %CLEAR-5-COUNTERS: Clear counter on interface Serial2/0
/1/25:0 by autoloop on vty0 (212.31.215.247)
May 21 21:00:25.841 GMT: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial2
/0/1/25:0, changed state to down
May 21 21:00:28.545 GMT: %LINK-3-UPDOWN: Interface Serial2/0/1/25:0, changed sta
te to down
May 21 21:06:32.205 GMT: %LINK-3-UPDOWN: Interface Serial2/0/1/25:0, changed sta
te to up
May 21 21:06:47.217 GMT: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial2
/0/1/25:0, changed state to up
May 21 21:39:37.252 GMT: %LINK-3-UPDOWN: Interface Serial2/0/1/25:0, changed sta
te to down
May 21 21:39:38.252 GMT: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial2
/0/1/25:0, changed state to down
May 22 13:26:43.323 GMT: %CLEAR-5-COUNTERS: Clear counter on interface Serial2/0
/1/25:0 by autoloop on vty0 (212.31.215.247)
May 22 13:27:15.812 GMT: %LINK-5-CHANGED: Interface Serial2/0/1/25:0, changed st
ate to administratively down
May 22 13:27:42.812 GMT: %LINK-3-UPDOWN: Interface Serial2/0/1/25:0, changed sta
te to down
PE14-DAL-AM#
CUSTOMER UPDATE:

'Modify Ticket' email sent to: ALANSO.BESSMAN@ECOLAB.COM


emec@ecolab.com
rachel.maskew@bt.com
jason.reed@bt.com
eric.rotunda@bt.com
22/05/2015 14:30:19 DUTY MGR 1
======================================= << Duty Manager >> ====================
===================
T
S
D
A
A
N
C

We have now Unisys and Level 3 FE on site working on the issue.


Chat with SRM/ 3rd line and Global Escalation for updates.
Follow up
last email from Level 3 escalation.

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Tony/BT Team,
We've had everyone on site are working together at this time. We are going to va
lidate the demark again while your teams are actively replacing their router/equ
ipment.
I'll update again once we have some new info.
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
======================================= << Duty Manager >> ====================
===================
CUSTOMER UPDATE:
'Modify Ticket' email sent to: ALANSO.BESSMAN@ECOLAB.COM
emec@ecolab.com
rachel.maskew@bt.com
jason.reed@bt.com
eric.rotunda@bt.com
22/05/2015 15:35:11 ANALISTA T2 1
Incident Ended at 22/05/15 18:15:58 GMT. Status changed to CLEARED.
Ticket at Escalation Level 6. Customer Update due at 24/05/15 00:35:10 GMT
CUSTOMER UPDATE:
'Modify Ticket' email sent to: ALANSO.BESSMAN@ECOLAB.COM
emec@ecolab.com
rachel.maskew@bt.com
jason.reed@bt.com
eric.rotunda@bt.com
22/05/2015 15:35:32 ANALISTA T2 1
Assignee changed to ANALISTA T2 1

22/05/2015 15:39:24 ANALISTA T2 1


========================
A:Worked with Unisys FE on site to replace the router which restored service.
A: On site customer advised that service is back online running clean.
A: Validated that service is up and stable.
N: Incident closure.
T:Logs:
RFO: Router chassis failure
RES: Router chassis replaced
Neighbor
/PfxRcd
62.6.6.58

V
4

AS MsgRcvd MsgSent
12641

631

103

TblVer InQ OutQ Up/Down State


2049

0 00:15:47

May 22 18:18:37.561 UTC: %LINK-3-UPDOWN: Interface GigabitEthernet0/0, changed s


tate to down
May 22 18:18:41.597 UTC: %LINK-3-UPDOWN: Interface GigabitEthernet0/0, changed s
tate to up
May 22 18:18:42.597 UTC: %LINEPROTO-5-UPDOWN: Line protocol on Interface Gigabit
Ethernet0/0, changed state to up
May 22 18:18:42.617 UTC: %DUAL-5-NBRCHANGE: EIGRP-IPv4 10: Neighbor 10.9.103.249
(GigabitEthernet0/0) is up: new adjacency

na-ecola-car70-dal-am#sh ip int brief


Interface
IP-Address
ocol
Embedded-Service-Engine0/0 unassigned
GigabitEthernet0/0
10.9.103.250
GigabitEthernet0/1
unassigned
Serial0/0/0:0
unassigned
Serial0/0/1:0
unassigned
Loopback10
147.152.37.248
MFR985
unassigned
MFR985.101
147.152.37.248

OK? Method Status


YES
YES
YES
YES
YES
YES
YES
YES

unset
manual
NVRAM
unset
unset
NVRAM
unset
TFTP

Prot

administratively down down


up
up
administratively down down
up
up
up
up
up
up
up
up
up
up

na-ecola-car70-dal-am#ping 62.6.6.58 r 1000 si 1458 df


Type escape sequence to abort.
Sending 1000, 1458-byte ICMP Echos to 62.6.6.58, timeout is 2 seconds:
Packet sent with the DF bit set
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!
Success rate is 100 percent (1000/1000), round-trip min/avg/max = 20/23/96 ms

================
CUSTOMER UPDATE:
'Modify Ticket' email sent to: ALANSO.BESSMAN@ECOLAB.COM
emec@ecolab.com
rachel.maskew@bt.com
jason.reed@bt.com
eric.rotunda@bt.com
22/05/2015 16:24:58 ANALISTA T2 1
A: Mailed Unisys FE Robert:
@@@@@@@@@@
To:'Chatter, Robert L' <Robert.Chatter@unisys.com>; BTARCE SERVICE G <btarce.ser
vice@bt.com>
Cc:
Subject:RE: txf38105843
Hello Robert,
I have had the chance to collect the traceback logs from defective router, could
you please manage to have someone from Unisys to work and find out the root cau
se for the error?
Attached you have the full collected log, I will expect your feedback upon this.
Thanks!
@@@@@@@@@@
N: Follow up with Unisys FE for updates.
CUSTOMER UPDATE:
'Modify Ticket' email sent to: ALANSO.BESSMAN@ECOLAB.COM
emec@ecolab.com
rachel.maskew@bt.com
jason.reed@bt.com
eric.rotunda@bt.com
22/05/2015 16:40:04 ANALISTA T2 1
A: Advised customer of service restoral:
@@@@@@@@@@
To:Maskew,R,Rachel,JSE R <rachel.maskew@bt.com>; Johnson, Sandra <Sandra.Johnson
@ecolab.com>; Rotunda,ET,Eric,JKL R <eric.rotunda@bt.com>; Ried, Bill <Bill.Ried
@ecolab.com>; dutymanager americas G <dutymanager.americas@bt.com>; EMEC Service
Account (emec@ecolab.com); Custodio,ES,Emerson,JSW3 R <emerson.silva@bt.com>
Cc:Johnson, Jason (IS) <jason.r.johnson@ecolab.com>; Dissette, Thomas <tdissette
@nalco.com>; Denton, Scott <Scott.Denton@ecolab.com>; Ruppel, Sharon <Sharon.Rup
pel@ecolab.com>
SubjectRE: BT Ticket XF38105843 RE: Incident INC0792620 -- opened on your beha
lf
Dear customer,
Currently the incident have been cleared once the defective router have been rep

laced with a new unity which got properly configured thus completely restoring n
etwork availability for the site. Unisys engineer Robert surveyed the whole loca
tion speaking with the majority of users and got a validation that service is up
and running clean, fact that has been also technically validated by captured lo
g's that can be found at the body of the BT incident ticket.
If you have any further questions or concerns please let me know.
Thanks!
ANALISTA T2 1
@@@@@@@@@@
CUSTOMER UPDATE:
'Modify Ticket' email sent to: ALANSO.BESSMAN@ECOLAB.COM
emec@ecolab.com
rachel.maskew@bt.com
jason.reed@bt.com
eric.rotunda@bt.com
24/05/2015 18:15:16 ANALISTA 9
Customer Informed of Clear at 24/05/15 21:15:08 GMT. Status Changed to AWAITING
CLOSURE.
CUSTOMER UPDATE:
'Modify Ticket' email sent to: ALANSO.BESSMAN@ECOLAB.COM
emec@ecolab.com
rachel.maskew@bt.com
jason.reed@bt.com
eric.rotunda@bt.com
24/05/2015 18:16:13 ANALISTA 9
Customer accepted closure at 24/05/2015 22:15:48 GMT . Status changed to CLOSED.
24/05/2015 18:16:29 ANALISTA 9
CLOSURE CONFIRMED
CUSTOMER UPDATE:
'Close Ticket' email sent to: ALANSO.BESSMAN@ECOLAB.COM
emec@ecolab.com
rachel.maskew@bt.com
jason.reed@bt.com
eric.rotunda@bt.com