Vision System for Gauging and Automatic Straightening of Steel Bars

Oliver Sidlaa, Ernst Wildlinga, Albert Niela and Herbert Bargb

a

Joanneum Research, Institute for Digital Image Processing, Graz, Austria b Bhler Edelstahl, Kapfenberg, Austria

ABSTRACT
A machine vision application for the fully automatic straightening of steel bars is presented. The bars with lengths of up to 6000 mm are quite bent on exit of the rolling mill and need to be straightened prior to delivery to a customer. The shape of the steel bar is extracted and measured by two video resolution cameras which are calibrated in position and viewing angle relative to a coordinate system located in the center of the roller table. Its contour is tracked and located with a dynamic programming method utilizing several constraints to make the algorithm as robust as possible. 3D camera calibration allows the transformation of image coordinates to real-world coordinates. After smoothing and spline fitting the curvature of the bar is computed. A deformation model of the effect of force applied to the steel allows the system to generate press commands which state where and with what specific pressure the bar has to be processed. The model can be used to predict the straightening of the bar over some consecutive pressing events helping to optimize the operation. The process of measurement and pressing is repeated until the straightness of the bar reaches a predefined limit. Keywords: material modeling, press straightening, dynamic programming, curvature estimation, edge tracing

1. INTRODUCTION
The Austrian company Bhler Edelstahl uses currently three straightening presses for adjustment and final processing of rolled steel bars. Having dimensions of approximately 30mm height, 250mm width and 6000mm length, the bars are quite bent on exit of the rolling mill and need to be straightened prior to delivery to a customer. Up to now this is done by a skilled worker who, based on experience, evaluates the shape of the bar and then applies pressure mostly to those regions on the bar with maximum curvature if necessary on all four sides of the bar. This man supported straightening control loop has been analyzed and transferred to an automatic system which will be able to perform the workers job with comparable accuracy, robustness and speed. As basic elements in the proposed system two cameras transmit images of the steel bar on the roller table to a standard PC. With the help of image processing methods the shape of the bars is measured and evaluated so that a hydraulic press can straighten the bar by applying appropriate force at suitable locations along its contour. Control of the hydraulic press and roller gear bed motors is executed by a Siemens S7 Stored Program Control (SPC) which executes commands received from the PC. The connection between the two communication partners is solved via a PROFIBUS network operating at a data rate of 1.15 MBit/sec. Development and operation of the system is undertaken on a standard industrial graded PC using LabView as the primary programming environment. This graphical programming language allows for easy generation of user interfaces as well as simplified programming of hardware components. As additional benefit the LabView compiler and programming IDE make it possible to have very fast development cycles. Functions or modules may be altered and the effect of the change can be tested immediately without a tedious rebuild of the complete system. In most cases even the values of variables are retained so that a development cycle may be as short as some seconds only.

A typical straightening sequence may be divided into the following steps: Command the SPC to insert a new bar onto the roller bed on a predefined reference position. The shape of the steel bar is then extracted and measured by the two video resolution cameras which are calibrated in position and viewing angle relative to a coordinate system which is located in the center of the roller gear bed. The interesting contours are tracked and located with a dynamic programming method utilizing additional constraints to make the algorithm as robust as possible. 3D camera calibration allows the transformation of image coordinates to realworld coordinates under the assumption that the edge points to be tracked have a (well known) height above the roller table. From the edge points in world coordinates, after smoothing and spline fitting stages, the curvature of the bar is computed. A model of the effect of forces applied to the steel allows the system to generate commands for the SPC. Those commands state where and with what specific pressure the bar has to be bent for shape correction. In order to cope with varying steel materials and deviations from the ideal situation, the model of the bar is updated with information on pressure and deformation recorded during each pressing event. After pressing all necessary and available locations on the bar, it is turned around its longer axis by 180 degrees. The procedure of shape measurement and following correction with the hydraulic jack is then repeated, iterating until the straightness of the bar reaches a predefined limit and it may be pushed of the roll.

Section 2 gives an overview of the system with emphasis on the hardware (HW) configuration. In section 3 all aspects of the straightening system related to image processing and shape evaluation are described. Section 4 explains how the deformation of the bar is controlled using force and deformation measurements. Finally, Section 5 summarizes the performance of the system and gives an outlook for coming improvements and developments.

2. SYSTEM OVERVIEW
The layout of hardware components including the image processing functions and bar manipulation units is depicted in Figure 1.

hydraulic Press

Figure 1: The structure of the bar straightening system. The image in the right upper corner shows an actual photo from the central region of the roller gear bed where the hydraulic jack is located. The jack can be seen in the right half of this image, a flat steel bar rests on the roller table.

Two cameras directed at the center of the roller gear bed take images of the steel bars for the purpose of shape measurement. The images are relayed to the host computer which is a standard industry PC operated under Windows NT 4.0. As a second computational and control resource the SPC is responsible for commanding all mechanical components of

the straightening system: the roller gears, the hydraulic jack and all actuators for getting the bars in and out from the roller bed. It accomplishes this task with the help of a laser distance sensor for sensing the bar position along pressing direction, a force sensor built into the hydraulic jack and several light barriers which are used for positioning the bar precisely along the roller bed. Both computers, the PC and the SPC communicate over a PROFIBUS network at a data rate of more than one MBit per second. Force measurements as well as the bar location are constantly transferred to the PC over the bus and may be accessed on the PC at any time necessary. Only during a press event the SPC stores its sensor measurement values in local memory and transfers it to the PC afterward. This approach is necessary because the protocol implementation of the PROFIBUS PC card is not fast enough for the real time transfer of all measurements.

3. IMAGE ACQUISITION AND PROCESSING

3.1 Image Acquisition and Calibration For measurement purposes the shape of the steel bar is obtained by means of two CCIR standard b/w video cameras which are mounted on both sides of the roller gear. The mutual distance of the two cameras is 20 meters, the distance to the hydraulic jack (which is located in the center of the roller gear bed) is 10 meters . Images from both cameras are captured using a standard PCI framegrabber card. Shutter time, positions, viewing directions and focal length for the cameras are chosen so that they deliver images of the bars with good contrast and maximum spatial resolution. For a precise transformation of image coordinates to world coordinates both cameras have been calibrated using several software and hardware tools available at the Institute of Digital Image Processing. Intrinsic calibration (focal length, main point on the sensor, distortion) has been undertaken in the laboratory with the help of a three dimensional cube shaped point target. The method of Faugeras [1] is applied for this procedure. Extrinsic calibration (camera positions and viewing angles) is accomplished by means of a 6 meter long aluminum calibration target with precisely measured fiducial marks on its surface. For the calibration of the two cameras this tool is placed directly on the roller bed, thereby defining location and orientation of the world co-ordinate system. An image from each camera where at least 10 fiducial marks can be precisely located is sufficient for spatial and orientational definition of extrinsic camera parameters up to an accuracy of 2 millimeters and 0.2 degrees using Sturdys calibration method [2]. 3.2 Image Analysis and Contour Detection After image acquisition from both cameras the next task for the system is to find the bar in those images, detect the location of its front faces and measure the shape of its outline. As a solution for this problem a dynamic programming algorithm has been developed and implemented. It works on a filtered version of the original image because through this preprocessing stage large amounts of redundant and (for this application) useless information can be eliminated. The convolution masks for image processing are chosen so that only the relevant edges of desired direction remain after filtering. In conjunction with proper illumination of the steel bar surface the bar contour detection is made very robust. Figure 2 outlines the geometric alignment between the cameras and light sources in order to optimize the contrast between the bar top surface and its environment.

Steel Bar

Figure 2: Geometric alignment of camera and lights for obtaining optimum contrast between steel bar and roller gear bed.

The resulting contour image is fed into a dynamic programming function which finds an optimum path through the image maximizing both edge strength and minimizing curvature[3]. For this algorithm the image is interpreted as a matrix where each of the gray values is considered separately. Walking from the leftmost columns to the right side of the image and within one column from top to bottom, the optimization task can be formulated as follows: If the current pixel under consideration is part of the optimal path from left to right then its predecessor must also be optimal and be one of the adjacent pixels within its 8-neighborhood on the left side (see Figure 3). The optimality of the pixel location is measured by means of two properties, namely i) the edge strength of the current location and ii) the spatial relationship of the current pixel with respect the optimum pixel on its left neighborhood (straight paths are preferred). Expressing the value of each pixel location, weights are accumulated from left to right, stored in the weight image and the additional information of which neighborhood was chosen is stored for each pixel in a backtrace image. The optimum path may be reconstructed from the backtrace image starting from the rightmost column at the row with the maximum weight, thereby making the identification for the optimum path very easy and computationally efficient. Along the path from left to right the weight information is also used for the purpose of finding the starting and ending points of the bar. This procedure is based on the observation that the path weight should exhibit a more or less sharp change at the location of the bar front side because there is no more clean bar edge available so that the edge weight as well as direction weights degenerate strongly. In order to detect candidates for those locations, the pixel weights from the optimum path are extracted and the second derivative computed. Peaks above a given threshold are taken as candidate side position. A cross check in the actual image is then made, testing at each location if the gray level distribution fits to the expected look of a bar frontside (bright top, darker left and bottom borders). The most promising candidate position is then selected. Figure 3 shows two tracking results with the automatically detected bar edge outlined

pi-1, j-1 pi-1, j pi-1, j+1

pi, j-1 pi, j pi, j+1

Figure 3: Pixel neighborhood relationships for the dynamic programming algorithm. Possible paths are generated from left to right. Each pixel pi,j stores its accumulated weight and the backtrace information to is optimal neighbor on the left, bottom or top border.

Figure 4: Tracking results for two different bar configurations. The small rectangles on the left image sides show all potential bar front sides, the vertical indicate give the finally selected locations.

3.3 Bar Shape and Curvature Analysis

After the edges of the bar have been successfully detected in both images, they are transformed back into world coordinates using the calibration matrices derived from the procedure outlined in Section 3.1. Both world coordinate paths are merged resulting in the overall bar shape consisting of approximately 1000 two-dimensional coordinate pairs. These point coordinates are subsequently smoothed with a Gaussian mean filter of high standard deviation so that small fluctuations and dents in the bar contour are removed. The reconstruction of world coordinates gives unequally spaced points due to inherent perspective in the imaging geometry. A further analysis of the bar shape is more easily undertaken on equally spaced coordinates - for this reason the coordinates are resampled into a grid with spacing of 10 millimeters along the bar axis. Finally a spline is fitted through those points. The bar shape S, as extracted from the two images, has at this stage been reduced to a one dimensional array of approximately 1000 points. For the further processing of the bar shape it has been recognized that the local curvature of the bar edge plays a pivotal role. It is computed as the second derivative from S, yielding S. The result of the bar contour processing is depicted in the plot contained in Figure 5.

Figure 5: The contour of a bar as it has been measured with the two cameras is plotted as thin zigzag line in the left image (the noise in the range of a tenth of a millimeter, which is about a tenth of the pixel resolution). A spline is fitted through the smoothed contour and overlaid as thick solid line. The second derivative of the contour is overlaid as dotted curve over the bar contour. The horizontal axis plots in units of millimeters along the axis of the steel bar. The vertical axis shows the deformation of the bar in millimeters. Note that the deviation of the bar contour from a straight line along its length of 3500 millimeters is only 1.2 millimeter.

At all those locations where S has a local maximum, there is also a significant change of the bar shape a bending. This knowledge and one further observation can be used to compute those locations on the bar where a correction of its shape can most effectively take place: measurement of the bar shape S and its second derivative before (S0) and after (S1) a press event has taken place gives the curvature of the bar at the respective times. The difference of the two curves S0 S1 (despite noise from the measurement process itself) describes the change of the bar shape created by the deformation process. It contains the information of how the curvature of the bar changes locally in response to a distortion with the hydraulic jack. From the geometry of the press it can be deduced that in the middle between the two supports against which the bar rests the deformation must be largest and that it must be slowly decreasing to zero from this point. Outside the supports the bar will not be bent by the jack. Practical experiments have shown that the difference of bar curvatures S0 S1 is quite reproducible over different materials and deformation values so that it may be mathematically modeled. This difference which we call press-template PT can roughly be approximated by a Gaussian curve of specific height and standard deviation. By subtracting the presstemplate from a measured curvature of a bar, it is possible to estimate quite precisely how the bar shape will change after a press correction.

Through the modeling of the deformation process it is possible to predict the straightening of the bar over some consecutive pressing events, resulting in two dominant advantages: i) a simulation of a straightening run may be executed, helping to

find the optimum sequence of pressing events, ii) the straightness of the bar need not be measured after each pressing event, which helps to improve the speed of the system. 3.4 Computation of Press Positions The knowledge of the press-template, which can be generated by means of press-experiments, allows the computation of meaningful press locations. Since the change in curvature for each press event is defined by the construction of the hydraulic jack and its environment, it is necessary to find locations on the bar curve where this change may be applied with most effect. One method to find those locations is the cross correlation function between the second derivation of the bar contour and the press-template PT. Local maximal points in this function can be taken as proposals for press locations, they hint for those positions on the bar where the expected deformation fits best to the local curvature situation. This means that with a press event taking place at this point, most of the curvature will be removed, resulting in a straightening of the bar. The height of the correlation peak can be used for the computation of the necessary correctional deformation which has to be applied to straighten the bar. Using the model PT for simulation it is possible to simulate a press event, calculate the deformation of the bar and then in a further iteration search again for the best possible press location. These steps are repeated several times so that the system generates a list of press positions (usually around 5) for the current measured bar contour.

4. CONTROL OF THE PRESS EVENT

4.1 Sensors for Press Event Monitoring Once the necessary press locations and press forces have been identified, it is important to transform them into real correctional maneuvers on the hydraulic jack. In order to monitor and control the press event as desired, the hydraulic press has been equipped with a force sensor built into its jack head and a laser distance sensor on the opposite of the jack. Whenever a steel is bent with the jack, its movement and the corresponding force is recorded, forming of a forcedeformation diagram over time. 4.2 Force-deformation Diagram of the Press Event From material science it is known how the force-deformation curve of the relaxation process should look like - theoretically it should be linear, depending only on the geometry of the steel bar and the pressing environment. The elastic deformation D of the bar can be calculated as follows, (1) (2)

where F is the applied force, l is the distance of the two supports against which the bar rests, I is the area inertial moment for a rectangular shape of width b and height h. Parameter E is a material specific elasticity constant which has a value of approximately 210*103 N/mm2 for all types of steel (which are currently more than 700) under consideration.

Figure 6: Three press events plotted as force-deformation diagrams. The press event starts at the left lower corner, as the force increases the deformation of the bar continues up to the maximum applied hydraulic pressure. It can be seen that the descending paths show similar slopes, the deviations are at most 0.25mm.

From equations (1) and (2) it becomes clear that an elastic deformation of the bar is only dependant on the applied force, even over all different kinds of steel flavor. The useful property of equation (1) is now that it describes also how the steel bar relaxes after the maximum pressure has been applied via the hydraulic jack. Relaxation is for most of the part elastic with the additional property that it should be identical for steel bars of matching cross sections as explained above. For the purpose of illustrating the constant-relaxation of the steel materials, three recorded press events are depicted in Figure 6. They show the deformation of the bar in millimeters (horizontal axis) against applied force in thousands of Newton (vertical axis). The curves in Figure 6 are called force-deformation diagrams. It can be recognized from the diagrams in Figure 6 that shape and slope of the ascending branches, where force is built up, are more or less different for all three example curves. Nevertheless the slopes of the descending branches are identical with a deviation of at most 0.25mm. This means the relaxation process of the bar after pressing can be forecasted with a model of the descending branch. The current implementation describes the descending branch of the force-deformation diagram as a polynom of the form: s = a0 + a*F + b*F2 + c*F3 (3)

s is the distance which the bar flexes back after force F has been applied on it, a, b and c specify the shape of the model polynom. The difference between an actual measured deformation sc at force Fc on the hydraulic jack and between the computed relaxation path s(Fc) estimates the residual deformation of the bar. 4.3 Generating Defined Residual Deformations With the model of the steel relaxation process it is possible to bend the steel by a requested deformation sdef (limited by the deviation of the actual relaxation curve from the assumed path) by observing the force-deformation diagram during a press action. Beginning from the initial distance measurement strigg, the SPC monitors the real-time deformation value smeas and force measurement F from the sensors and feeds both quantities into formula (3), yielding the relaxation distance scalc for the current system state. The difference smeas scalc strigg can be regarded as the residual deformation of the bar. Figure 7 illustrates the procedure in graphical form. After each press event, force and deformation data is transferred from the SPC to the PC. The resulting curve is analyzed and any deviations from expected values can be taken into regard for the following pressing. This will be explained in the next section.

Figure 7: Evaluation of the force-deformation diagram for press control. The recording of the press event starts at point strig and goes along points 1, 2 and 3. Measurements are undertaken at a frequency of 300 Hz ensuring real-time control of the press event.

4.4 Factors Limiting Pressing Accuracy As it has been shown in Figure 6 the relaxation curves are not perfectly identical even for the same steel bar. This effect is currently not fully understood, nevertheless there exist several known factors explaining this observation at least partially: The so called Bauschinger effect, a change in the crystalline structure of the material, leads to a hardening of the steel bar after it has been deformed. It increases the slope of the relaxation curve for the next press event at the same or a nearby location. The distance of the supports against which the bar is pressed changes over the course of the press event, depending on the shape of the bar. The reason for this change is the fact that at the beginning of the press event the bar rests against the outer edges of the supports with a like shape. Over the course of the bar deformation it is bent inwards and rolls towards the inner side of the supports looking now more or less like a U. The supports have each a width of approximately 150mm so that the change of the effective factor l of equation (1) has a considerable effect on the forcedeformation curve. In finite-element simulations performed by experts from Bhler Edelstahl this effect has been confirmed to be non-negligible. Imperfections in the guide of the hydraulic jack, leading to frictions and a slight resistance against bar relaxation. Friction between the bar and the two supports, taking away energy from the bar during relaxation. This effect is considered to be very small.

All the factors mentioned above limit the precision of a single deformation process with the hydraulic jack to a range of 0.2mm. In order to increase the precision of the single press deformation, a strategy has been developed which tries to adapt to changing material properties and discrepancies between the model and reality. For a requested bending correction, the pressing of the steel is divided into more than one pass. The first press event tries to achieve only a fraction of the requested value, usually 60% to 70%. Since during each pressing the forcedeformation diagram is recorded, it is possible to measure very precisely the actual residual deformation of the steel. From this information an updated value for the pressing of the next iteration can be computed and used. Experimental data suggests that the optimum number of press events for achieving a nominal deformation lies between two and three. Not yet implemented, but regarded as one of the next algorithms to be considered is the usage of the relaxation curve from the previous press event instead of the fixed model polynom (3). With this measure it should be possible to adapt to any changes of the relaxation curve which may be induced for example by the Bauschinger effect.

4.5 Example for a Test Press Event This section describes the result of a typical press event which has been performed by the proposed system. The primary measurement results for this example are depicted in the Figure 8. The brighter line in the upper diagram of Figure 8 outlines the shape of the bar after smoothing and line fitting. Deformation of the bar in millimeters is plotted on the vertical axis. The darker line depicts the calculated second derivative of the bar shape with the small vertical lines depicting automatically generated press positions. Note that any number of press positions may be calculated, only in this example their count is limited to five per iteration. For this example, the leftmost press position has been selected and the steel has been pressed at this point with the help of the hydraulic jack. All measurements after pressing have been recorded in the lower half of Figure 8. The line in the middle represents the contour of the bar after the press event. It can be clearly seen that the curvature at the press location has practically disappeared. The vertically offset dark line shows the difference of the two curvatures before and after the press event. Beside some noise effects, the Gaussian shaped peak at the pressing location may be recognized as the press-template curve PT from Section 3.3.

Figure 8: Recordings of a press event.

5. CONCLUSION AND OUTLOOK

An automatic straightening system for the correction of bent steel bars has been presented. Images taken from two cameras are used for measurement of the bar contour. The image to world co-ordinate transformation relies on calibration matrices calculated by means of calibration targets. The raw contour points are smoothed and subsequently replaced by a spline fit. For the computation of pressing positions and necessary deformations a correlation between the bar curvature and a model of the bending process is computed. Local peaks in the correlation output are used to generate a press list consisting of positions on the bar. A single press event is mastered using a model of the elastic relaxation process of the steel material in conjunction with realtime measurement and control of the hydraulic jack. At the time of this presentation the implementation of the straightening system is fully underway. Practical experiments undertaken so far have shown that the straightening of a bar with arbitrary shape is possible with the proposed system. Several automatic runs have demonstrated that bars can already be corrected to within the limits of customer specifications of 1mm deformation per meter. Current development issues concentrate on making the system faster and still more precise. The automatic acquisition of the model polynoms for the relaxation process is fully automated but not yet robust enough. Although the image processing measurements are quite precise already, there is still room for improvement to reach a desired accuracy of 0.15 millimeters. There is much more to be learned about material properties and effects induced therein by the pressing itself. Despite those important and necessary modifications it is believed that the automatic straightening of steel bars is technically within reach so that the final industrial system can begin operation in Spring 2001.

6. REFERENCES
1. 2. 3. 4. 5. O.D. Faugeras and Toscani, The Calibration Problem for Stereo, Proceedings Conference Computer Vision and Pattern Recognition, Miami beach, FI: IEEE pp. 12-15, 1986. W.I. Grossky and L.A. Tamburino, A unified approach to the Linear Camera Calibration Problem, IEEE Trans. pattern Anal. Machine Intell, Vol 12, No 7m, pp. 663, 1990. Milan Sonka, Vaclav Hlavac, Roger Boyle, Image Processing, Analysis and Machine Vision, Second Edition, PWS Publishing, 1999. National Instruments, LabView User Manuals, July 2000 Edition, National Instruments, Austin, Texas, 2000. Bhler Edelstahl, Edelstahl Handbuch (Handbook of Steel Products and Qualities), Bhler Edelstahl, Kapfenberg, Austria.