ALPR |1

Automatic Licence Plate Recognition (ALPR)

Anshuman

Abstract- Automatic licence plate recognition (ALPR) refers to the extraction process of vehicle licence plate information which may be used, with or without use of a database, in a number of applications, like toll tax collection, monitoring traffic etc. Success of ALPR directly relies on the quality of image captured by the camera. The database or/and algorithm used for ALPR must be generalised enough to recognise number plates from different languages or shapes and to nullify various environmental conditions like indoor, outdoor, day or night time. In this report various state-of-the-art methods for implementing ALPR are discussed with a comparison between them on the basis of their pros, cons, processing speed, recognition accuracy etc.

1. INTRODUCTION UTOMATIC licence plate recognition finds its application in a number of real-life events like road traffic monitoring, toll collections, parking lot control etc. In ALPR vehicle number is extracted from an image or a sequence of images (taken from color, black and white or infra red camera) using a combination of techniques like image processing, object detection and pattern recognition. There are various challenges to the recognition and detection of licence plate, which are summarised below.

I) Variation in plate: a) b) c) d) e) f) g) h) location: number plate may be present at different location of the captured image; quantity: an image may have more than one plates to be recognised; size: due to zoom factor a and distance of camera ,plate size varies; font: different plates may different fonts and may be even in different language; occlusion: plates may get obscured due to dust; color: different plates may have different color or background; inclination: plates might be tilted; others: plates may also contain certain screws and frames etc.

II) Environmental variations: a) illumination: captured image may have different illuminations due to sunlight or vehicle headlights; b) background: in background of the plate there may be patterns similar to plates like numbers stamped on vehicle, textured floors etc. The ALPR system is composed of four stages as shown in fig.1. The first stage is to take the image of the car. Quality of the image depends on the camera distance, zoom ratio, shutter speed etc. Next

ALPR |2 stage is to extract licence plate from the vehicle image, which is done, based on the features like boundary, color or presence of characters. Third stage is to segment this extracted plate image and extract the characters by projecting their colors information or position matching with template. Final stage is to recognise extracted characters using template matching or classifiers.

Fig.1.Four stages of an ALPR system

2. LICENCE PLATE EXTRACTION Licence plate extraction step greatly influence the accuracy of the ALPR. Input to this stage is the image of the vehicle and output is the potential licence plate. Based on the features of the licence plate like its color, plate format or constituent characters. Its rectangular boundary is another noticeable feature that may be used in the extraction process. Various extraction methods based on the feature they use are given below: 2.1 Using edge Information Licence plates are usually rectangular in shape with a fixed aspect ratio. This property is used in edge detection methods like those in hybrid license plate extraction method based on edge statistics and morphology[1], A morphological-based license plate location[1] etc. Sobel filters, combined with the color transition detection between plate and vehicle background is used for edge detection in Saudi Arabian license plate recognition system [2], Feature based recognition of traffic video streams for online route tracing, [1], Car plate character extraction under complicated environment, [1], Development Of vehicle-license number recognition system using real-time image processing and its application to travel-time measurement, [1] etc.

ALPR |3 2.2 Using Templates: In this method the captured vehicle image is cross-correlated with a pre-stored licence plate template to extract the most probable licence plate area. Though this method is independent of the plate position in the image, but is time consuming. It is of the order of n4 for nxn pixels [1]. 2.3 Using Texture Fixtures Due to presence of characters in the licence plate, there is a significant change in the gray-scale level between character color and plate background. This very feature can be deployed through various techniques for plate detection. In scale-line technique [1], change in gray- scale level results in a number of peaks in the scan line, which is equal to the number of characters. The licence plate may also be seen as the irregularity in the texture of the image. So, any abrupt change in the local characteristic of the image may be regarded as the potential licence plate. The sliding concentric windows (SCW) method uses this very concept [3]. In [4], spatial frequency is identified by using discrete Fourier transform (DFT) because it produces harmonics that are detected in the spectrum analysis. The DFT is used in a row-wise fashion to detect the horizontal position of the plate and in a column-wise fashion to detect the vertical position. All these texture based methods have the advantage to detect licence plate even if they are deformed. But, all these methods are computationally very complex especially when numbers of edges are very high due to varying illumination condition or complex background. 2.4 Using Color features Many countries have specific colors for licence plates. This gives a unique color combination to character and the plate region. This principle can be used in the extraction of the licence plate. Like in Chinese licence plates Shi et al. [1] proposed that all pixels in the input image can be classified using hue, lightness, saturation (HLS) color model into 13 categories. In Korea [1], a neural network is used to classify the color of each pixel after converting the RGB image into HLS. Neural network outputs, green, red, and white are the license plate colors in Korea. The same license plate color is projected vertically and horizontally to determine the highest color density region that is the license plate region. Color-based method has problem due to illumination variation, to deal with this a fuzzy logic based method was proposed. The hue, saturation, and value (HSV) color space is employed. Three components of the HSV are first mapped to fuzzy sets according to different membership functions. The fuzzy classification function is then described by the fusion of three weighted membership degrees. Extraction of a license plate using color information has the advantage of detecting inclined and deformed plates. However, it also has several difficulties. Defining the pixel color using the RGB value is very difficult, especially in different illumination conditions. The HLS, which is used as an alternative color model, is very sensitive to noise. Methods that use color projection suffer from wrong detection, especially when some parts of the image have the same license plate color such as the car body. 2.5 Using Character Features Presence of characters in the input image may also be recognised as the potential licence plate.

ALPR |4 In Unconstrained license plate and text localization and recognition [1], instead of using properties of the license plate directly, the algorithm tries to find all character-like regions in the image. This is achieved by using a region-based approach. Regions are enumerated and classified using a neural network. If a linear combination of character-like regions is found, the presence of a whole license plate is assumed. In another method, binary objects that have the same aspect ratio as characters and more than 30 pixels are labelled. The Hough transform is applied on the upper side of these labelled objects to detect straight lines. The same happens on the lower part of these connected objects. If two straight lines are parallel within a certain range and the number of the connected objects between them is similar to the characters, the area between them is considered as the license plate area. These methods of extracting characters from the binary image as defining the license plate region are time consuming because they process all binary objects. Moreover, these methods produce errors when there is other text in the image.

TABLE 1:PROS AND CONS OF EACH CLASS OF EXTRACTION METHOD Methods Using boundary feature Rationale Pros Cons Cant be applied to complex images as too sensitive to unwanted images may be

Licence plate Simplest and fast boundary rectangular

Using templates

Searching connected Independent of Broken object image having licence plate generated dimensions position

Using texture feature

Frequent color Deformed licence Computationally complex transition on licence plate can even be for large number of edges plate detected Color specific licence Inclined and RGB limited to plate deformed plate illumination condition And can be detected HLS sensitive to noise Presence of character Resistant in the licence plate rotation to Time consuming, detection error when other text in image Computationally complex

Using color feature

Using character feature

Combination of two or Combined feature is Reliable more features more effective

2.6 Combining Two or More Features All the methods discussed above have their own limitations as well as advantages. Combining one or more of these give better results. This may be called as hybrid extraction method [1].

ALPR |5 In License plate recognition using DTCNNs, [1] two neural networks are used to detect texture feature and color feature. One is trained for color detection and the other is trained for texture detection using the number of edges inside the plate area. The outputs of both neural networks are combined to find candidate regions. In Automatic license plate location and recognition based on feature salience, [1] the rectangle shape feature, the texture feature, and the color feature are combined to extract the license plate. 1176 images that were taken from various scenes and conditions are used. The success rate is 97.3%. All the extraction methods discussed are tabulated in Table 1 for their pros and cons.

3. LICENCE SEGMENTATION After the extraction of the licence plate, the main task is its segmentation to take out the potential characters from it. Algorithm for segmentation needs to remove all types of errors in extraction like tilt and non-uniform brightness in pre-processing steps. Bilinear translation and least square method [1] are generally used to map the tilted plate to a straight rectangle.

In a yet another method, KarhunenLoeve transform, the coordinates of characters are arranged into a 2-D covariance matrix. The eigenvector and the rotation angle are computed in turn. Then, image horizontal tilt correction is performed. For vertical tilt correction, three methods K-L transform, the line fitting based on K-means clustering, and the line fitting based on least squares are put forward to compute the vertical tilt angle . Various existing segmentation methods based on the feature they use are categorised below.

3.1 Using Pixel Connectivity By labelling the connected pixels in [5], [6] in the binary licence plate image segmentation is achieved. Those labelled pixels which have same dimensions and aspect ratio of the character are considered as the licence plate. This method fails to extract all the characters when there are joined or broken characters. 3.2 Using Projection Profiles Characters and license plate backgrounds have different colors, so they have opposite binary values in the binary image. Therefore, binary extracted license plate may be projected vertically to determine the starting andthe ending positions of the characters, and then projecting the extracted characters horizontally gives each character alone.The advantage of projection method is that the extraction of characters is independent of their positions. However, it depends on the image quality. Any noise affects the projection value. Moreover, it requires prior knowledge of the number of plate characters.

ALPR |6 3.3 Having prior knowledge of character Prior knowledge of characters can help the segmentation of the license plate. In this method [1], the binary image is scanned by a horizontal line to find the starting and ending positions of the characters. When the ratio between characters pixels to background pixels in this line exceeds a certain threshold after being lower than this threshold, then this is considered as the starting position of the characters. The reverse gives the ending position of the characters. In Approach to recognition of license plate numbers using neural networks, [1] the extracted license plate is resized into a known template size. In this template, through scanning all character positions are known. After resizing, the same positions are extracted to be the characters. This method has the advantage of simplicity. However, in the case of any shift in the extracted license plate, the extraction results in background instead of characters. 3.4 Using Combined Features For efficient segmentation two or more feature of the licence plate are combined. For e.g. [7] describes an adaptive morphology based segmentation approach for seriously degraded extracted images. It consists of an algorithm which is based upon histogram that detects fragments and merges these fragments.

TABLE 2: PROS AND CONS OF LICENCE SEGMENTATION METHODS Method Using pixel connectivity Pros Cons

Simple and robust to licence Joined or broken characters plate rotation cant be extracted Independent of character Affected by noise and prior position, can deal with some knowledge of number of rotation character in the plate is required of Simple Limited by prior knowledge, any deviation may cause error

Using projection profile

Prior knowledge characters

Using character contour

Exact character boundary is Slow and may produce achievable distorted or incomplete contour More reliable Computationally complex

Using combined feature

4. CHARACTER RECOGNITION Recognising characters from the segmented plate image marks the final stage of ALPR. Character recognition in ALPR systems may have some difficulties. Due to the camera zoom factor, the extracted characters do not have the same size and the same thickness [1]. Resizing the characters into one size before recognition helps overcome this problem. The characters font is not the same all the time since different countries license plates uses different fonts. Extracted characters may have some noise or they may be broken [1]. The extracted characters may also be tilted [1].

ALPR |7 Existing character recognition methods based on their features are categorized below. 4.1 Using Templates It is the simplest and the straightforward method for recognition. Similarity between various templates and the character is made, template which is most similar is then considered as the character. Usually binary images are used for this purpose as grey-scale level are susceptible to lightening changes. Optical recognition of motor vehicle license plates,[1] uses normalized cross correlation to match the extracted characters with the templates. Each template scans the character column by column to calculate the normalized cross correlation. The template with the maximum value is assumed to be the most similar one. Template matching can detect non-broken, non rotated ,fixed size characters only If a character is different from the template due to any font change, rotation, or noise, the template matching produces incorrect recognition. This problem may be solved by having templates of several inclinations.[8] 4.2 Using Extracted features Not all the pixels of the extracted character are important in distinguishing characters, so an alternate, feature extraction technique is used. It reduces the processing time as required in template matching because not all pixels are involved. It also overcomes template matching problems if the features are strong enough to distinguish characters under any distortion [1]. The extracted features are taken in the form of a feature vector which is compared with the prestored feature vectors to measure the similarity. The feature vector in [8] is generated through the Hotelling transform of each character, which is very sensitive to the segmentation result. Binary characters can be divided into blocks of 3 x 3 pixels to generate the feature vector[9]. Then, the number of black pixels in each block is counted. In [10], after obtaining feature vector as described ,the number of elements that have 0, 45, 90, and 135 inclination are counted. The character may be scanned along a central axis[1]. This central axis is the connection between the upper bound horizontal central moment and lower bound horizontal central moment. Then the number of transitions from character to background and spacing between them form a feature vector for each character. This method is invariant to the rotation of the character because the same feature vector is generated. Gabor filter may also be used for feature extraction. The character edges whose orientation has the same angle as the filter will have the maximum respond to the filter. This can be used to form feature vector for each character. In [11], Kirsch edge detection is applied on the character image in different directions to extract features. Using Kirsch edge detection for feature extraction and recognition achieved better results than other edge detection methods, such as Prewitt, Frei Chen, and Wallis [1]. The topological features of characters the number of holes, endpoints, three-way nodes, and four-way nodescan be used[1]. These features are invariant to spatial transformations. Classifiers are used to recognise characters, after the feature extraction. Various classifiers used are Support Vector Machine (SVM), ANN, HMM etc. A combination of classifiers may also be used.

ALPR |8 TABLE 3: PROS AND CONS OF CHARACTER RECOGNITION METHODS Methods Using pixel values Template matching Pros Cons

Straightforward and Slow, vulnerable to simple font change, rotation, noise and thickness change tilted Requires more processing time

Several templates character Using features

of

angled Can detect each characters

extracted Horizontal and Recognition fast, Feature extraction Vertical projections robust to distortion takes time Hotelling transform Number of black pixels in 3x3 pixels block Number of elements that hve certain degree of inclination Gabor filter Topological of character features

Kirsch edge detection

5. FUTURE PROSPECT Though, a significant progress has been made in the field of ALPR. But still there is need of a robust system that could deal with varying environmental and plate conditions. Different nations and even regions have different styles of plates and event fonts of the plates vary from one to another. Very less research has been done regarding this problem, which can be exploited in the future. At present almost all ALPR methods input is either a still image or some frame of sequence of images. However, taking advantage of the temporal information of a video can highly improve the system performance. Basically, using the temporal information consists of tracking vehicles over time to estimate the license plate motions and thus to make the recognition step more efficient. There are two kinds of strategies to achieve this goal. One strategy is to use the tracking output to form a high resolution image by combining multiple, sub pixel shifted, low-resolution images. This technique is known as super-resolution reconstruction [1]. Alternately, high level outputs of the recognition can be merged to make a final decision.

ALPR |9 Segmentation and recognition are two important tasks in ALPR. Traditionally, these two tasks are implemented in a cascade fashion independently and sequentially [1]. Recently, there has been an increasing interest in exploring the interaction between the two tasks. For example, the prior knowledge on the characters to be recognized is employed for segmentation and the recognition outputs are fed back to the segmentation process. Many issues that are open for research regarding ALPR are given below. i. ii. iii. iv. For video based ALPR capturing the correct frame containing the number plate is major challenge, especially when vehicle are moving at high speed. Improving image enhancement techniques to remove lighting effect on plates. New sensing system, robust to illumination changes. Improving recognition rate of ambiguous charaters , like (B-8), (O-0), (I-1), (A-4),(C-G), (D-O), (K-X), and broken characters.

REFERENCES
Shan Du, Mahmoud Ibrahim, Mohamed Shehata and Wael Badaway, Automatic Licence Plate Recognition(ALPR): A State-of-the-Art Review, IEEE Transactions on Circuits and Systems for Video Technology, vol. 23, no. 2,pp. 311-325, February 2013. [2] M. Sarfraz, M. J. Ahmed, and S. A. Ghazi, Saudi Arabian license plate recognition system, in Proc. Int. Conf. Geom. Model. Graph, pp. 3641, 2003. [3] C.-N. E. Anagnostopoulos, I. E. Anagnostopoulos, V. Loumos, and E. Kayafas, A license plate -recognition algorithm for intelligent transportation system applications, IEEE Trans. Intell. Trans. Syst., vol. 7, no. 3, pp. 377392, Sep. 2006. [4] R. Parisi, E. D. D. Claudio, G. Lucarelli, and G. Orlandi, Car plate recognition by neural networks and image processing, in Proc. IEEE Int. Symp. Circuits Syst., vol. 3., pp. 195198, Jun. 1998. [5] K. Kanayama, Y. Fujikawa, K. Fujimoto, and M. Horino , Development of vehicle-license number recognition system using real-time image processing and its application to travel-time measurement, in Proc. IEEE Veh. Tech. Conf, pp. 798804., May 1991. [6] K. Miyamoto, K. Nagano, M. Tamagawa, I. Fujita, and M. Yama moto, Vehicle license-plate recognition by image analysis, in Proc. Int. Conf. Ind. Electron. Control Instrum. , vol. 3., pp. 17341738, 1991. [7] S. Nomura, K. Yamanaka, O. Katai, H. Kawakami, and T. Shiose, A novel adaptive morphological approach for degraded character image segmentation, Pattern Recognit., vol. 38, no. 11, pp. 19611975, 2005. [8] T. Naito, T. Tsukada, K. Yamada, K. Kozuka, and S. Yamamoto, Robust license -plate recognition method for passing vehicles under outside environment, IEEE Trans. Veh. Tech., vol. 49, no. 6, pp. 2309 2319, Nov. 2000. [9] F. Aghdasi and H. Ndungo, Automatic license plate recognition system, in Proc. AFRICON Conf. Africa, vol. 1, pp. 4550, 2004. [10] T. Nukano, M. Fukumi, and M. Khalid, Vehicle license plate character recognition by neural networks, in Proc. Int. Symp. Intell. Signal Process. Commun. Syst., pp. 771775, 2004. [11] S. N. H. S. Abdullah, M. Khalid, R. Yusof, and K. Omar, License plate recognition using multicluster and multilayer neural networks, Inform. and Commun. Tech., vol. 1, pp. 18181823, Apr. 2006. [1]