ROOT CAUSE ANALYSIS

FOR

LOSS OF AC115 POWER SUPPLY TO BOILER

CONTROLLERS

JULY, 2015
 ROOT CAUSE ANALYSIS

1. OBJECTIVE
The main objective of this report is to identify the Root Cause of Power Plant
Failure due to loss of AC115 power supply to Boilers controllers on Monday, 20 th
July, 2015, proffer solutions and make recommendations to avert future
occurrence.

2. BACKGROUND
Power Plant and Utilities have four Turbine Generators (TGs) of 14MW each and
Two Diesel Generators (DGs) of 750KVA each. While the TGs provide the normal
power requirements of the KRPC Complex, the PPMC and the Estate, the DGs are
employed for black start up purposes. The two DGs are normally synchronized to
produce the required power for successful Black Start-up.

The Control power for Boilers and TGs is supplied through two sets of
Uninterrupted Power Supply (UPS) systems called Constant Frequency Constant
Voltage (CVCF-1&2). The systems provide uninterrupted control power in the
event of power loss from TG or DG.

INTRODUCTION:
Power Plant was re-streamed on 16th July, 2015 following a prolonged shutdown
since 28th April, 2015 due to lack of crude oil for processing. Boilers 1&5 and TG-4
were in service with Power successfully supplied to all the sub stations when it
was observed that CVCF-1 was operating with one of its two inverters on by-
pass instead of normal mode for both. This operating condition alarmed the
PPU operators who requested for Maintenance intervention.
However, the requested intervention led to Plant shutdown Monday, 20 th July,
2015. Prior to this incident TG-4 tripped on Saturday 18 th July, 2015 at 4.10hrs
due to estate breaker. However, the TG4 was immediately steam-rolled and
synchronized to the Bus-bar, power was restored to substations.

2|RCA Document
CLASSIFIED
 ROOT CAUSE ANALYSIS

On the other hand the incidents of Monday 20 th July, 2015 at about 14:35 hrs led
entire Power Plant shutdown with far reaching consequences.
A committee was therefore set-up to find the root cause and proffer solutions that
will avert recurrence.

SEQUENCE OF EVENTS
On 19th July, 2015 at about 12.50hrs CVCF-1 inverters were observed with
inverter A on normal and inverter B on by-pass mode. The observation
alarmed the PPU operators who requested Maintenance intervention.
Maintenance made attempts on 20 th July, 2015 at about 14.35 hrs to bring
both CVCF-1 inverters to normal mode but was unsuccessful. During the
Maintenance efforts to normalize the inverters, it was observed that the
power supply to the Boilers Common Panel blinked thrice but came back
ON
About the same time, steam driven equipment (namely: 75K01A,
74P01A&E) in service tripped
At 1440hrs Turbine Generator (TG4) tripped due insufficient cooling water
supply.
This was followed by 70B01 trip due to lack of control power.
Efforts made to keep Boiler 5 in operation and also re-start TG-4 was
unsuccessful due to faults on 52TL (which provides control power to
Demin, Raw Water and Cooling Water Units), as such Power Plant was
brought down at on 20th July 2015.
A CVCF expert was invited the following day, 21 st July 2015 for further
Maintenance intervention and his initial efforts confirmed faults in control
power supply to the Boilers Control Panel as thick smoke exhumed.

ROOT CAUSES ANALYSIS

3|RCA Document
CLASSIFIED
 ROOT CAUSE ANALYSIS

Below is the Root Cause Analysis using Fishbone method to identify possible
causes of the problem and brainstorming to sort out ideas into relevant
categories.
Problem Description:
Power Supply to Boiler Control blinks (Off/On) and subsequent burns of
Boiler fuses and controllers which lead to power plant trip.

Problem Verification:

Electrical and Instrument Engineers/Technicians checked and confirmed that

there was no AC115V instrument power to the Boiler control panels.

Problem Analysis 1:

On 19th July, 2015 at about 12.50hrs CVCF-1 inverters were observed with
inverter A on By-Pass and inverter B on Normal mode. The observation
alarmed the PPU operators who requested Maintenance intervention.

Indicators Auto Start Preventive Starting of

Errors? ON? Maintenance? Major Equipment?

CVCF 1A Bypass
CVCF 1B- Normal

Power CVCF Human Drop in Voltage/

Surge? Malfunction? Error? Frequency?

Fig 1. CVCF 1A on Bypass and CVCF 1B on Normal

From Fig. 1 above, Indicator are confirmed to be in good condition, there was
no Power Surge at the time of the incident, History on the CVCF panel did not
indicate malfunctioning and there was no fluctuation of voltage/frequency.
The problems in circle were not eliminated; hence further analysis was carried
out on why the CVCF is on By-Pass mode.
4|RCA Document
CLASSIFIED
 ROOT CAUSE ANALYSIS

Auto Start Starting of

ON? Major Equipment?
BY PASS

Human Preventive
Error? Maintenance?

Fig 2. By-pass Mode Analysis

Fig. 2 shows that CVCF 1A Auto Start was ON from the software. Auto Start
ON will not allow the system to revert to normal operation if all the
parameters that make it switch to bypass are not met. However, an
investigation was carried out on why the system software was on Auto Start
ON.

Human
Error?

AUTO START ON

Equipment Knowledge
Malfunction? Gap?

Fig 3. Auto Start ON Mode Analysis

Fig. 3: brainstorming revealed knowledge gap is most likely the cause for the
CVCF1A to be in Auto Start On position as many who had fiddled with the
facility were not aware what position On or Off will return the CVCF back to
normal. This was confirmed by the fact that the inverters were not on uniform
Auto Start On/Off position.

What Knowledge gaps exist in the operational principle of CVCF?

5|RCA Document
CLASSIFIED
 ROOT CAUSE ANALYSIS

Staff were trained on Maintenance and operations of this facility, but long
intervals exist between required interventions, thereby creating knowledge
and skill gap.
Operationally when the CVCF goes on bypass operation it is anticipated to
return to normalcy once the fault or overload is corrected.
Normalizing one leg of the CVCF on bypass mode requires that the leg on
normal mode is also brought to the bypass mode, synchronized, before
reverting to normal operation.
CVCF 1A and 1B should always be on Auto Start OFF. This will enable
any of the CVCF that goes on bypass to revert to normal operation.
Problem Description 2
Loss of AC115 power supply to power control panel for the Boilers and
smokes exhumed (from the control panel).
Problem Verification 2.

During the course of troubleshooting and test procedures it was discovered

that there were some damages to Instrument Control Panel which supplies
AC115V to Boiler Controllers.

Problem Analysis 2
The root cause of the damages to the Instrument control panel was
investigated as follows:
Ground Over
Fault (Leakage)? Current ?
DAMAGE TO
INSTRUMENT
CONTROL PANEL
Human Error? High Voltage?

Fig 4. Damage Instrument Panel Analysis

From Fig. 4 above, Ground fault and overcurrent were not unlikely because
the circuit breaker supplying power to the instrument control panel did not
trip. Also there was no any established tempering of electrical circuit or CVCF
6|RCA Document
CLASSIFIED
 ROOT CAUSE ANALYSIS

by maintenance or operation personnel. This necessitated the investigation of

high voltage supply to the instrument control panel.

What are the sources of power (AC115V) to the Boiler Controllers?

Power supply (AC115V) to the controllers is being supplied either from the
CVCF Inverters (normal mode) or through CVCF Static Switch (by-pass
mode). Boilers 1~4 TG1~3 Controllers are fed by CVCF-1 while Boiler 5 and
TG4 Controllers are supplied by CVCF-2. The CVCF supply AC 115V to all the
controllers.

Transformer AC Capacitor
Failure? Failure?
HIGH VOLTAGE

Electronic
Card failure?

Fig 5. High Voltage Analysis

Based on the above analysis the transformer and electronic cards were
energized and are confirmed to be in good condition. However, Twelve (12)
power AC capacitors were damaged.

7|RCA Document
CLASSIFIED
 ROOT CAUSE ANALYSIS

Why did the capacitors to failed?

Preventive Aging/
Maintenance? Life Cycle?

AC Capacitors
Failure

Air Condition?

Inverter Fan?

Dust? Heat? Voltage/

Frequency
Fluctuations?

Fig 6. AC Capacitors Failure Analysis

From Figure 6 above, the following was analyzed.

Dust: Presence of dust on power capacitors can provide a sand layer

that can prevent the capacitor to have enough cooling to operate
optimally. This will increase the temperature of the capacitors. The
CVCF power capacitors are designed to withstand temperature between
-25oC to 75oC. Therefore dust as a cause was eliminated due to the fact
that the ambient temperature did not reach 75oC.
Heat: Heat was eliminated due to the above reason.
Voltage/Frequency Fluctuations: The inverter circuit was designed to
prevent such fluctuations to reach the power capacitors. Series of
components were installed to checkmate excess, and those components
were confirmed to be functioning very well. Therefore
voltage/frequency was eliminated.
Ageing/Life Cycle: Life Cycle of MLR25 capacitors is 30,000hrs (Three
and Half Years). They have spent their life cycle long. To buttress this,
all capacitors there were installed at the same time were confirmed
failed.
8|RCA Document
CLASSIFIED
 ROOT CAUSE ANALYSIS

The ageing/Life Cycle of the capacitors was found to be the root cause of
high voltage supply to the instrument control panels that resulted in
damaging boiler controllers and power supply units.

NOTATIONS:
Based on the above analysis, Management is invited to note as follows:
1. There was knowledge gap in the philosophy of operation of the UPS which
informs the operators apprehension when inverters are found on bypass
mode.
2. Also, knowledge gaps exist on the side of the Maintenance personnel
resulting in the inability to revert the system to normal operating mode.
3. The CVCF capacitors have long out lived their lifespan having been
installed since 2004. The average expected lifespan is 30,000 hours (i.e.
31/2 years). See Attachment 1.

CONCLUSION:
There is a skill gap in the operation and maintenance of the CVCF.
The ageing/Life Cycle of the capacitors was found to be the root cause of
high voltage supply to the instrument control panels that resulted in
damaging boiler controllers and power supply units.

RECOMMENDATIONS:
Periodic Maintenance Checks as stated in the CVCF Manual should be
strictly adhered to. See Attachment 2.
There is a skill gap in the operation and maintenance of the CVCF.
Operation, Maintenance and ETSD personnel should be properly trained.
This will enhance the reliability of the plant and reduce downtime.
Air Condition of all the UPS Rooms and Substations should be put in good
conditions to enhance the lifespan of accessories including the capacitors.
Procurement of spare parts for the maintenance of UPS Systems.
Periodic cleaning of Substations and UPS Rooms.

9|RCA Document
CLASSIFIED
 ROOT CAUSE ANALYSIS

COMMITTEE:

1. Garba T. Muhammad (ETSD) - Chairman .

2. Aminu Ibrahim (PPU) - Member .
3. Garba Bala Mohd (MTCE) - Member .
4. Aliyu Muhammad (ETSD) - Member .
5. Ibrahim Bassey (ETSD) - Member .
6. Aminu Diso (ETSD) - Member .
7. Saleh, Umar A. (REHAB)- Secretary .

10 | R C A D o c u m e n t
CLASSIFIED