THE CASE OF THE MISSING FILE

Johan S. Spijker
Predictive Maintenance Engineer
Rochester Gas & Electric Corporation
Russell Station
1101 Beach Avenue
Rochester, NY 14612
Johan_spijker@rge.com

Tony DeMatteo
Consultant
Computational Systems, Inc.
835 Innovation Drive
Knoxville, TN 37932
585-293-3234

Abstract:

This is a case history of a boiler feed pump vibration problem at a coal-fired power station. The problem
began with an electrical defect on the motor and evolved into a major problem with the pump that resulted
in substantial generation production losses per day. Over a six-month interval, many attempts were made
to identify and correct the problem using corrective and diagnostic technologies. The pump problem was
finally solved, although diagnostic measurements did not identify the defect.

The Case of the Missing File describes the situation with the pump and reviews the data collected on the
machine. As the details of this case history are revealed, consider what more could have been done to find
and fix the problem sooner.

Figure 1 -- Boiler Feed Pump

Introduction (Johan Spijker):
Russell Station consists of four fossil-fired steam generators that have a combined output of 250 Mw. The
plant was built in stages beginning in 1948, with the last unit finished in 1957. The equipment is an Allis
Chalmers 11 stage barrel centrifugal boiler feedpump that delivers 632 gpm of water at a total head of 5300
ft at 3580 rpm. The prime mover is a 1000 hp Allis Chalmers induction motor that is directly coupled to the
pump. Both the motor and pump have sleeve bearings. The boiler feed pump is used to deliver high
pressure, high temperature water to the boiler steam drum to generate steam that transfers its thermal
energy into mechanical energy in the steam turbine. The steam turbine is directly coupled to the generator,
which transfers the mechanical energy into electrical energy. The pump/motor was built in 1957 and has
been in service almost continuously except for low load periods and annual overhauls.

Our Plant Reliability Based Maintenance program started in 1994 with the introduction of vibration
analysis technology and this pump was one of the first 'problems' that we found on our first route. We have
several similar model pumps to compare to and this one was the worst of all, typically running at twice the
amplitude of the other pumps. The pump runner was replaced during a 1993 outage. The readings on the
pump inboard bearing after startup were the highest and were typically around .5 inches per second (ips) at
turning speed (1X) with very few other synchronous or non-synchronous components. We checked for
imbalance and noticed that the coupling had many line-up punch marks that didn't match. We un-coupled
the coupling, rotated the halves to match up the punch marks and ran the pump again with great results. All
readings had decreased with the pump inboard bearing now at .2 ips at turning speed. We continued to
monitor this equipment at quarterly intervals with no notable exceptions for the next five years. In 1999,
we disassembled the pump again because our pump performance test showed the pump was well off the
pump curve, indicating normal 5 years worth of wear. We replaced the pump runner again with a re-built
unit and repaired numerous water cuts on the pump gasket surfaces by filling in with weld material and
filing and honing the surfaces clean and flat. The pump was re-assembled and ran fine after start up.

In May 2001, we performed a motor circuit evaluation test on the feedpump motor and found a 200A
imbalance in the windings. We decided to run the entire summer period using the motor while carefully
monitoring the current imbalance and limiting the starts on the motor to get us to our next scheduled outage
which was due in January 2002. Our plan was to re-wind a similar 2400V-service motor from a retired unit
for 480V service during the outage by our Motor Shop Vendor. I also noticed pre-outage vibration route
readings on this pump's outboard bearing had risen since the last quarterly route. My management decided
to postpone any pump investigative work because the pump was still performing on the pump curve and
the pump had just been overhauled two years earlier.

The Motor Shop re-wound the motor to 480V service and we checked the motor at the shop for vibration
levels and also performed another motor circuit evaluation. Our mechanics re-aligned the 'spare' motor to
the pump using a laser alignment method to our cold setting specification. We test ran the pump during unit
startup and found immediate, very high (> .6 ips) vibrations in the pump inboard at 1X and elevated
vibration levels (> .4 ips) in the outboard bearing, especially in the horizontal direction. We began by re-
checking the alignment and found that the alignment had changed from -.02" to + .04". I asked the
mechanics to re-align the motor, this time to our hot specifications, which they did. The pump was re-
started and at first, seemed to run acceptably until after about an hour, when the vibration levels started to
climb quickly. Next, we decided to run phase checks across the equipment and were baffled by changes in
the phase shift across the pump inboard bearing. The phase changed by over 60 degrees in both directions
and wasn't consistent from data point to data point and run to run. Going back to our 1993 history, we
focussed on the coupling between the motor and the pump and checked for line-up punch marks. These
looked proper but we decided to try to balance the coupling. Even though the phase wasn't steady, we
achieved a modest amount of success. The horizontal readings were now .45 ips inboard and .35 ips
outboard. After operating the pump, we were right back to the high levels within the hour.
The next phase of the investigation centered on the motor re-wind. Our electrical engineering group felt
that using a 2400V rotor from the original motor in a 480V re-wound stator could be part of the problem
and there was some low level 2XLF noise indicated in the motor spectrum. The Motor Shop said that
wasn't the problem (yes, we had been talking all along). They said it was inadvisable to mismatch rotor bar
slots with stator slots. Against the motor shops recommendation, we swapped the rotors and ran the motor
un-coupled. The motor shop was right -- you don't want to mismatch rotor bars and stator slots. We
carefully inspected the correct rotor for damage. We used a core loss tester to heat up the rotor and looked
for hot spots with infrared thermography. We found a large circumferential hot area at the end ring that
extended for almost 180 degrees. The motor shop determined that the laminations at that end ring had
loosened up. The laminations were re-swaged and the motor was again tested. All looked well. The motor
was installed and aligned to the pump. When operated, the unit still had unacceptable vibrations.

At this point in time, management wanted some indication of whether to re-wind the original motor that
had the 200A imbalance or to investigate the pump by an internal inspection. I had no conclusive evidence
to point to either component as the problem so I called a consultant who I hoped would have the tools to
get the answer. Is it the pump or is it the motor?

Consultant (Tony DeMatteo):

I met with Johan and he described the machines problem history. As he spoke, I formulated my plan for
one day of measurement followed by analysis and reporting. We decided on the following tests:

1. Impact testing of the floor, piping, bearing housings and shafts

2. Thermal Growth study of the motor and pump
3. Stroboscopic analysis of the shafts to look for axial motion
4. Housing vibration and phase readings
5. Shaft vibration and phase readings
6. PeakVue readings on the motor to check for rotor bar problems
7. High resolution motor housing vibration readings
8. Operational deflection shape (ODS) of the entire machine, base and floor
9. Repetitive phase and magnitude shaft and bearing housing readings during the day
10. Time capture coast-down readings

Investigation:
The impact testing revealed that the pump inboard bearing housing had a natural frequency1 at 70 Hertz.
On sleeve bearing equipment, impact testing should be done while the shaft is slow rolling (up on the oil
film). When we tried to roll the shaft for the impact test it would not roll. It took one man and a large
strap wrench just to turn the shaft but it would not roll.

In the coast-down captures, performed at the end of the days testing, a resonance just below operating
speed at 59 Hertz was identified on the inboard pump housing (horizontal direction). Figure 2 shows the
transient capture waveform and a peak-hold spectrum of the housing data during coast-down. The shaft
was also measured during the coast-down but did not indicate a resonance at 59 Hertz. The pump housing
should not have a natural frequency so close to operating speed. Perhaps this was an indication that the
shaft is improperly supported.

1
A natural frequency is the frequency at which a part likes to vibrate. Resonance results when forced
vibration, from mechanical defects, coincides with a natural frequency. Resonance is an amplifier. At
resonance, a small amount of vibration from mechanical defects is greatly amplified. The amount of
amplification depends on the system damping characteristics.
 Coast-down begins

Resonance

Resonance Pump Inboard Shaft

at 59 Hz. --
Full Speed
just below
59.85 Hz. 1200 rpm
full speed
at this
another at point
54 Hz.

Figure 2 Coast-Down Capture and Peak-Hold Spectra for the Inboard Pump Housing Horizontal

Vertical thermal growth testing consisted of temperature measurements made from each foot up to the shaft
centerline on each bearing. The temperatures were recorded before the unit was started, 30 minutes after
starting and after four hours of run time. The data in table 1 indicate the machine had 2.5 mils of vertical
growth on the motor and 6.5 mils on the pump outboard bearing.

Table 1 Vertical Thermal Growth (mils)

Position Before 30 min 4 hours
operation after start after start
Motor Outboard 0.4 0.8 1.4
Motor Inboard 0.6 1.8 2.6
Pump Inboard 1.5 4.5 5.7
Pump Outboard 1.6 5.5 6.5

The motor housing vibration readings were less than .25 ips at all positions. A peak at two times line
frequency (120 Hertz) was a large component of the motor vibration. The pump vibration was highest at
the inboard pump bearing where the overall vibration level was .44 ips. The largest peak was at turning
speed. A peak at .5x turning speed, indicating a rub, was also present. Figure 3 shows pump inboard
horizontal housing measurements made over the course of the day. The overall vibration level steadily
increased. The rub would come and go.

11:32 overall = .44 ips 14 :00 overall = .61 ips 17:00 overall = . 82 ips
(start -up)

1x = .81
.5x = .12 1x = .6

1x = .4

Figure 3 Pump Inboard Horizontal Housing Spectra

Shaft vibration was measured with a shaft stick attached to an accelerometer. The readings on the inboard
motor shaft were stable throughout the day at 5-6 mils peak to peak (figure 4a). The motor inboard shaft
orbit was round as you might expect to see from unbalance. The inboard pump shaft readings were about 4
mils at start-up and increased to 6 mils by the end of the day (figure 4b). The motor and pump shafts were
out of phase by about 180 degrees across the coupling.

In addition to the phase shift across the coupling, the outboard pump shaft was out of phase with the
inboard pump shaft by almost 180 degrees. At start-up, the pump outboard shaft reading read 4 mils
horizontal and 7 mils vertical. By the end of the day, the shaft readings had increased to 9 mils horizontal
and 12 mils vertical. The outboard pump bearing might be wiped or have excessive clearance. The shaft
may be bent.

- S3H vs S3V

4a Motor Inboard Shaft Orbit 4b Pump Inboard Shaft Orbit

S3H in Mils
Figure 4 Motor and Pump IB Orbits with Phase Mark Indicating 180 degree Shift

In describing the vibration problem on the machine, the customer expressed a concern that the changing
alignment might be caused by a loose machine foundation. An operational deflection shape2 (ODS) test
was made to determine if the foundation was loose. Shaft stick readings were also included in the ODS.
An ODS provides a view of the shape of the machine as it vibrates during normal operation. ODS testing
can identify many machine faults such as soft foot, looseness, structural weakness and misalignment.

Often, an ODS will speak volumes about what is not wrong with a machine. This was the case with the
boiler feed pump. The ODS test indicated that the machines foundation was not loose to the floor. The
pump housing was fastened to the machine base and the machine base was secure to the concrete. No soft-
foot was found on the motor feet, although, it is not likely that soft foot would be apparent because of the
flexibility of the welded motor frame. The ODS animation shape at turning speed (figure 5), indicates the
machine has shaft-to-shaft misalignment and a 180 degree phase across the pump bearings. If the shafts are
not physically misaligned, it is possible that other forces are moving the rotors around within the bearing
clearance.

2
An Operational Deflection Shape (ODS) is a measurement technique used to analyze the motions of
rotating equipment and structures. An ODS is an extension of phase analysis. In an ODS, a computer
generated model of the machine is animated with phase and magnitude measurement data or
simultaneously measured time waveform captures. ODS is a non-intrusive test made during normal
machine operation.
 Motor

Pump

Machine
Base

Concrete
Floor

Figure 5 ODS Shape of Turning Speed Showing Misalignment

The motor vibration data show clear signs of electrical defects. The largest peak on the vertical inboard
motor bearing was .2 ips at 2X line frequency (120 Hertz). The turning speed vibration was less than .1 ips
(figure 6). A distorted motor frame may be causing some of the 120 Hertz vibration on the motor. The
one-piece motor shims that should be installed under the motor support rails were observed lying next to
the motor. The motor had individual shim packs under each bolt. Welded frame motors are relatively
weak structures compared to cast iron motors. If the support rails are not flat, or improperly shimmed, the
motor housing can distort and create an uneven air gap resulting in electrical vibration at 120 Hertz.

2x rotation

120 Hertz
(2x line frequency)
an indication of
1x rotation
motor frame distortion

Figure 6 Electrical Vibration at 120 Hz. at Inboard Motor Bearing Vibration

A high resolution measurement, made on the side of the motor frame, revealed .96 Hertz sidebands around
each peak in the spectrum. The data in Figure 7 has a log scaled, zoomed insert of the .96 Hertz sidebands
around 2X line frequency. The time waveform shows a one second modulation period (1/.96 = 1.04 sec).
The .96 Hertz sidebands relate to the slip x poles frequency as indicated in the following equations:

Slip Frequency = Magnetic Field Frequency Rotor Frequency Eq. (1)

= 60 Hz. 59.52 Hz.
= 0.48 Hertz
# Motor Poles = (2 x Line Frequency / Magnetic Field Frequency) Eq. (2)
= 120 Hz. / 60 Hz.
= 2
Slip x Poles = Slip Frequency x # Motor Poles Eq. (3)
= .48 x 2
= .96 Hz.

The .96 Hertz, slip x poles sidebands, indicate broken or damaged rotor bars. Faulty rotor bars produce
these sidebands around vibration peaks in the spectrum. When the motor was replaced in December 2001,
a 2400 volt motor rewound to 480 volts was used. It is possible that electrical faults in the motor are acting
on the rotating assembly causing vibration. Sleeve bearings allow the rotors freedom of movement.
Higher loads result in greater forces that might explain the phase and magnitude changes.

2x Line frequency with

Poles x Slip sidebands
of .958 Hz.

1 sec

Figure 7 Motor Housing High Resolution with Slip x Poles Sidebands around 120 Hz.

A PeakVue3 measurement was made on the side of the motor frame (figure 8). PeakVue can be used to
confirm rotor bar problems in the motor. The PeakVue waveform had a peak value of 8.6g (not shown).
The largest peak was 1.3g at 2x line frequency (120 Hz.). Other peaks at slip x poles frequency (.96 Hz.)
and turning speed with harmonics were present. The source of the peak at 32.03 Hertz is not known. All
peaks had sidebands at slip x poles frequency. The sidebands around 120 Hertz are shown in the insert.

3
PeakVue is a patented diagnostic tool on CSI analyzers used to measure stress wave energy. Stress waves
result from metal to metal impacting and friction. Some analysts have learned that PeakVue can also be
used to find rotor bar problems in motors.
 Log Scale Zoom
of 120 Hertz with
.96 side-bands

Figure 8 PeakVue Motor Data Confirms Rotor Bar Defects

Recommendations:
After reviewing the test data, the following recommendations were made.

1. Inspect the pump outboard bearing for wear or damage. Measure the bearing clearance. Verify that
the shaft is properly supported by all four bearings.
Result: The customer determined that the all four bearings were in good condition and showed no
unusual wear patterns or clearance. The outboard pump shaft was bent .007.
2. Remove the individual shim packs and install the one-piece shims under the motor rails.
Result: The customer did not change the motor shims.
3. Check the shaft alignment. Set the motor shaft .003 above the pump shaft.
Result: The alignment was good. The mechanics had been using the correct thermal growth
setting.
4. Check the coupling keys for proper length and 180 degree offset from motor to pump.
Result: The coupling keys were the correct length and orientation.
5. Try balancing the coupling again. This time, use shaft stick readings instead of housing readings for
balance run data.
Result: The coupling balance was inconclusive. Trial weights were removed and the coupling
was returned to its previous condition.
6. Install a portable CSI 4500 on-line monitoring system for two weeks to characterize the variability in
shaft and housing phase and magnitude.
Result: The customer considered doing this but ultimately did not.
7. Measure the rotor-to-stator air gap for concentricity. Test the motor for broken rotor bars.
Result: No problems were found. The motor was sent to a motor shop for inspection. The rotor
bars were reported to be in good condition, although it is not clear if the test was adequate to
identify broken or damaged rotor bars.
Out of all the recommendations provided, the only problem found was a .007 bend on the outboard pump
shaft. The one day of testing was successful in eliminating many suspected problems, however it did not
find the source of the vibration. The customer was faced with making a decision about disassembling the
pump. He looked to me to support that recommendation, but I could not. I wasnt sure that the pump was
the problem.

I would like to believe that two weeks of portable, online monitoring data would have caused me to
recommend an internal pump inspection. Even if I had the phase and magnitude trends, I am still not sure I
would have said the words that my customer wanted to hear.

Conclusion (Johan Spijker):

Since the Motor Shop vendor was adamant that the re-wind was not the problem, our management finally
decided to open up the pump for inspection. Upon disassembly, we noticed a couple of pieces of loose
metal that were lying in the casing of the pump. On closer inspection we discovered that they were pieces
of a metal file, commonly used to dress up gasket surface repairs. There were three pieces found and
comparing it to a whole file, we discovered that there was still one piece that we hadn't found. Upon even
closer inspection of the runner, we discovered the last piece of the file that had lodged itself inside one of
the stage impellers.

There was surprisingly little physical damage to the internals of the pump. We felt that the weld repair
personnel had either bumped the file as he was performing another operation or that he knocked the file
into the casing and forgot to retrieve it. One of the pieces of the file had managed to sometimes get sucked
up into one of the pump stage impellers and then either shake itself out by centrifugal force or vibrations
until it would get sucked up again by another stage impeller. That was why the imbalance kept on moving
and changing phases.

We experienced an extremely expensive lesson in sloppy housekeeping. My management looked to our

Reliability Based Maintenance program to pinpoint the source of the problem but all our technology and
experience did not solve someone's carelessness.

The pump was reassembled and returned to operation. The vibration was less than .14 ips on all measured
points. Im not sure what I would do different if I was faced with a similar type of non-collaborating
evidence equipment problem.