See discussions, st ats, and author pr ofiles f or this public ation at : https:www .researchgate.ne tpublic ation257694469 [623368]

See discussions, st ats, and author pr ofiles f or this public ation at : https://www .researchgate.ne t/public ation/257694469
Real-time automatic license plate recognition for CCTV forensic applications
Article    in  Journal of R eal-Time Imag e Pr ocessing · Sept ember 2013
DOI: 10.1007/s11554-011-0232-7
CITATIONS
27READS
1,225
6 author s, including:
Some o f the author s of this public ation ar e also w orking on these r elat ed pr ojects:
Micr owave Br east Imaging View pr oject
BIOELECPRO View pr oject
M. Saquib Sarfr az
Karlsruhe Instit ute of T echnolog y
35 PUBLICA TIONS    340 CITATIONS    
SEE PROFILE
Atif Shahz ad
National Univ ersity of Ir eland, Galw ay
28 PUBLICA TIONS    113 CITATIONS    
SEE PROFILE
Muhammad Adnan Elahi
National Univ ersity of Ir eland, Galw ay
31 PUBLICA TIONS    111 CITATIONS    
SEE PROFILE
M. F raz
Loughbor ough Univ ersity
12 PUBLICA TIONS    71 CITATIONS    
SEE PROFILE
All c ontent f ollo wing this p age was uplo aded b y Atif Shahz ad on 15 A ugust 2016.
The user has r equest ed enhanc ement of the do wnlo aded file.

SPECIAL ISSUE
Real-time automatic license plate recognition
for CCTV forensic applications
M. S. Sarfraz •A. Shahzad •Muhammad A. Elahi •
M. Fraz •I. Zafar •E. A. Edirisinghe
Received: 30 September 2010 / Accepted: 21 October 2011 / Published online: 19 November 2011
/C211Springer-Verlag 2011
Abstract We propose an efficient real-time automatic
license plate recognition (ALPR) framework, particularly
designed to work on CCTV video footage obtained fromcameras that are not dedicated to the use in ALPR. At
present, in license plate detection, tracking and recognition
are reasonably well-tackled problems with many successfulcommercial solutions being available. However, the exist-
ing ALPR algorithms are based on the assumption that the
input video will be obtained via a dedicated, high-resolu-tion, high-speed camera and is/or supported by a controlled
capture environment, with appropriate camera height,
focus, exposure/shutter speed and lighting settings. How-ever, typical video forensic applications may require
searching for a vehicle having a particular number plate on
noisy CCTV video footage obtained via non-dedicated,medium-to-low resolution cameras, working under poor
illumination conditions. ALPR in such video content faces
severe challenges in license plate localization, tracking andrecognition stages. This paper proposes a novel approach
for efficient localization of license plates in video sequence
and the use of a revised version of an existing technique for
tracking and recognition. A special feature of the proposed
approach is that it is intelligent enough to automaticallyadjust for varying camera distances and diverse lighting
conditions, a requirement for a video forensic tool that may
operate on videos obtained by a diverse set of unspecified,distributed CCTV cameras.Keywords License plate recognition /C1CCTV video
footage /C1Traffic monitoring /C1Video indexing /C1Surveillance
1 Introduction
The number of on-road motor vehicles has increased with
the rapid growth of world’s economy and with this aug-mentation the need for security and monitoring of vehicles
has also increased. It is necessary for officials to continu-
ously examine the traffic to avoid/control congestion,overspeeding and unlawful activities that involves a vehi-
cle. Many successful commercial systems that employ
dedicated camera systems, providing video input capturedunder control environments to ALPR algorithms, exist at
present [ 1,3,6,9–11,19,24,25]. However, application
scenarios in video surveillance and forensics such astracking down a stolen vehicle or searching for a vehicle
involved in a crime, as identified by a bystander to be of a
particular registration number, requires the painstaking taskof manual search, because the existing ALPR systems are
not capable of efficiently working on video footage
obtained via non-dedicated (for ALPR) CCTV systems The
non-deterministic camera positioning (height and angle),
specifications (speed, focus, aperture), lighting conditions,presence of compression artifacts, high levels of noise etc.
in CCTV systems pose a significant challenge to computer
vision and pattern recognition algorithms used in existingALPR systems.
This paper presents an efficient and robust framework
that can perform localization, tracking and recognition ofmultiple vehicle license plates in a real-time scenario (i.e.,
incoming video stream from low-resolution surveillance
cameras). The major aim of carrying out this work is tomake significant contribution to the efficacy improvement
M. S. Sarfraz /C1A. Shahzad /C1M. A. Elahi /C1M. Fraz
Computer Vision Research Group, COMSATS Instituteof Information Technology Lahore, Lahore, Pakistan
I. Zafar /C1E. A. Edirisinghe ( &)
Digital Imaging Research Group, Loughborough University,LE11 3TU Loughborough, UKe-mail: E.A.Edirisinghe@lboro.ac.uk
123J Real-Time Image Proc (2013) 8:285–295
DOI 10.1007/s11554-011-0232-7

of video indexing/annotation applications. The system is
robust enough to learn any scenario and adjust itself
according to any camera angle, height and distance from
the road.
The main contribution of this paper is a novel tech-
nique of detecting license plates in real time from low-
quality surveillance video. The method detects the licenseplate of moving vehicles based on the geometry of its
contours present in the foreground. The detection proce-
dure is supplemented by a conventional template match-
ing procedure in the initial lea rning phase to automatically
learn and adjust itself to the plate size in the initialfew frames. The detected plate is tracked by using a
dynamic displacement method and finally the characters
of the license plate are recognized using a simple nearestneighbor classifier. The main intention of tracking a
vehicle’s license plate throughout the video stream is to
enhance the efficiency of lic ense plate character recog-
nition using majority votin g on a set of detected samples
of the same license plate and to eliminate the fallaciously
detected license plates b y continuous assessment of
the tracked plates. The idea of tracking a license plate
instead of the whole vehicle relies on the fact that there
are less chances of occlusion for license plate than forvehicle because of its smaller s ize when tracked in several
frames.
The rest of this paper is organized as follows. After a
review of related works, the main framework is described
in three sections, license plate localization, tracking and
recognition. The experimental results are provided in Sect.4, followed by discussion and conclusion in Sect. 5.
2 Related work
In general, license plate recognition systems consist of two
major parts, localization within a single frame of traffic
video and character recognition.
Donoser et al. [ 10] addressed the issues of detection,
tracking and recognition together. They introduced a real-
time framework that enabled detection, tracking and rec-
ognition of license plates from video sequences. Theirdetection algorithm is based on the analysis of a maximally
stable extremal region (MSER) detection that differentiates
the region of interest on the basis of intensity of the regionas against the boundary of the region. However, MSER
detection approach fails when the intensity of the license
plate and/or characters on the plate are akin to the outerregion. This effect is very common in variant outdoor
conditions and motion blurred videos.
Hough transform is another well-known technique for
license plate detection that can prove useful in finding the
boundary box of a license plate regardless of characters[11,23]. This is a quite efficient method, but it has high
computational complexity and therefore not suitable for
real-time applications.
Chang et al. [ 3] proposed a license plate detection
algorithm using color edge and fuzzy logic. However,
their algorithm can only b eu s e dt od e t e c tt h el i c e n s e
plates with specific colors. Techniques based on learning
such as Adaboost [ 9,24] are also used for license plate
detection. Simplicity and spe ed are the attractive features
for Adaboost learning with respect to other classifiers.
However, in comparison to edge-based methods, Ada-
boost is slow. Adaboost method fails to detect a licenseplate when the range of variations for distance or viewing
angle increases.
In the area of vehicle tracking, the standard approach for
robust tracking in traffic consists of adopting sophisticated
instruments like radars, as for example in [ 20]. However,
this has the drawback of being very expensive in com-parison to standard video cameras. Another method is the
blob tracking [ 12]. In this approach, a background model
without moving objects is generated for the scene. Eachframe is compared with the background model by com-
puting the absolute difference between them and conse-
quently obtains a foreground blob representing thevehicles. The vehicle tracking literature almost universally
relies on variants of Kalman filters [ 7], although particle
filters and hybrid approaches have been widely used inother tracking applications. One of the earlier prominent
works has been done by Koller et al. [ 14] who proposed a
deformable contour-based vehicle tracking algorithm. Thealgorithm does not work well if the vehicle entering the
scene is partially occluded. This is why the approach
proposed in this paper directly focuses on license platedetection instead of detecting a vehicle and then localizing
its license plate.
Several other real-time tec hniques have been proposed
in literature, for example us ing a mixture of sonar and
vision information [ 13]o ru s i n gs t e r e ov i s i o n[ 15].
Another method proposed by [ 17] suggested mounting a
few artificial landmarks on the car to be followed, while
[4] using templates of a ca r’s back to perform the
tracking.
A number of approaches for recognizing the characters
on a license plate after successful detection have been
proposed in literature. Shapiro et al. [ 21] use adaptive
iterative thresholding and analysis of connected compo-
nents for segmentation. The classification task is then
performed with two sets of templates. Rahman et al. [ 18]
used horizontal and vertical intensity projection for
segmentation and template matching for classification.
Dlagnekov and Belongie [ 8] used the normalized cross-
correlation for classification by analyzing the whole plate,
hence skipping segmentation.286 J Real-Time Image Proc (2013) 8:285–295
123

3 Proposed framework
This section describes the proposed framework comprising
detection/localization, tracking and recognition of licenseplates in CCTV surveillance videos, as shown in the block
diagram in Fig. 1. As a first step, the license plate is
localized in the incoming frames. This requires somebackground learning and pre-processing to differentiate
between license plates and other plate-like regions. The
located plate is further tracked in each frame by continuousupgradation of background and finding the new location of
the license plate. The detected plate(s) is/are enhanced and
character recognition procedure is applied to recognize thecharacters of the license plate in each frame.
3.1 Background learningEfficient background extraction is a key step for moving
object detection in a video sequence. A background imageis required to represent the base state of the area under
examination for detection purposes. The extracted back-
ground is subtracted from each frame to obtain theforeground object (moving vehicle in this case). However,
it is hardly possible to acquire an image of the observation
area that does not contain any vehicles or other foreground
objects. Thus, it is crucial to extract the background imagefrom the video stream itself. The accuracy of the detection
procedure depends on precise and rapid background
estimation.
In the proposed approach, background learning is per-
formed by using the exponential forgetting technique [ 22].
The system begins its background computation by con-
sidering the very first frame as background and updates it
with impending frames. Every updated background is aweighted sum of the previous background and the new
frame. In this way, the background dynamically adapts the
changes in the movement of objects or luminance condi-tions in the frame. Our detailed experiments revealed that
the exponential forgetting technique performs efficiently in
CCTV footage, in which movement of objects can berather complex and no control exists over luminance
variations.
Mathematically, the background learning procedure can
be described as follows:
Fig. 1 Overview of the
proposed frameworkJ Real-Time Image Proc (2013) 8:285–295 287
123

Bnț1¼1/C0l ðȚ BnțlFn ð1Ț
where ‘ Fn’ is the current frame, ‘ B’ is the background and
‘l’ is the background learning coefficient.
Background learning procedure is illustrated in Fig. 2.
The value of background learning coefficient ‘ l’i s
empirically set at ‘0.1’, i.e., for each new background
estimation, 90% of the previously estimated backgroundand 10% of the new frame values are incorporated. In the
same way, for the subsequent frame, 90% of the previously
calculated background values are considered that have 90%effect of their previous background values and so on. In
this way, the effect of previously calculated backgrounds is
not voided straightaway; instead, their effect is abridgedframe by frame in an exponential manner.
The first row shows a sequence of video frames on
which background learning is applied. The second rowshows the calculated background by using exponential
background learning. The third row illustrates the fore-
ground as calculated by subtracting the background fromthe original frame.
3.2 Pre-processing
The current system is designed taking surveillance cam-
eras, which usually record less frames per second and havelow resolution, into consideration. For this reason, pre-
processing is required to minimize noise and for sharpen-
ing edge information in the frame. This enhances plate
regions and improves detection efficiency.
Figure 3illustrates the effect of pre-processing. It can be
seen that the original frame contains significant amount of
salt and pepper noise and has poor edge information.
Noise present in the foreground is removed by using
well-known median filtering: a nonlinear technique that
applies a sliding window on the frame pixels and removes
noise while preserving the edge information. Morphologi-
cal operators (erosion and dilation) are applied afterward tofurther refine the foreground.
3.3 License plate detection/localizationAfter pre-processing, candidate regions (for license plate)
are detected by finding the contours and boundaries ofconnected components in the frame. These connected
components are further processed based on template
matching criteria to void the false regions and decide aboutthe true region of interest ‘ROI’. The ROI selection is
carried out in two steps. First, the identified candidate
regions are judged on the basis of their size and aspectratio. If the size of any region is smaller or larger than a
certain threshold, then that region is classified as a false
Fig. 2 Background learning and subtraction by using exponential forgetting method. a–eSequence of original video frames. f–jCalculated
backgrounds for the respective frames. k–oBackground subtraction from the respective frames288 J Real-Time Image Proc (2013) 8:285–295
123

region and is discarded from further processing. If the
‘width-to-height’ aspect ratio of any region is less than or
greater than a certain threshold, then that region is also
discarded for further processing.
3.3.1 Region of interest selection
After neglecting the false regions on the basis of their size,
the remaining candidate regions are checked on the basis of
their texture similarity to license plate-like areas. Since wetarget the application of our approach to videos captured
with CCTV camera installations of unknown specification,
it is imperative that the method must cope with varyingcamera distances and/or height from the plane. To achieve
this in real time, we propose using a new initial learning
mechanism that accounts for and learns automatically theexpected size of the number plate in the video. This is
important since it reduces a possibly large set of candidate
regions to be evaluated, to a very few, which results notonly in improved detection accuracy but also in greater
computational advantage.
3.3.1.1 Initial learning The initial learning is performed
on the first few incoming frames by narrowing down the
possible sizes of expected plate-like regions fulfilling the
initial geometrical constraints (aspect ratio etc.). The rangeof possible plate sizes is taken between some upper and
lower threshold that is the function of the frame size. The
regions formed in this range are then checked on the basisof their texture similarity in a pure template matching
fashion and this information is further used to refine the
size range for that particular video.
More specifically, we set the initial lower and upper
thresholds for a rectangular region to be a license plate as5–20% of the frame size, respectively. The upper limit is
set to 20% of the frame size considering the application
requirements that the camera view should be at least as
wide as the width of a single lane on the road; if thedetected plate is less than 5% of the frame size, it is almost
impossible to extract the characters with acceptable accu-
racy. However, this initial threshold is automaticallyupdated to narrow down the range on the basis of learning
performed by the system.
The learning procedure starts from the very first, frame
in which the lower and upper threshold values are set at
0.05 and 0.2, respectively, of the frame size. For the initial
few frames, these threshold values are kept unchanged byconsidering the possibility that true candidate regions may
be of any size in this range. The candidate regions formed
on the basis of these threshold values are processed bymatching with the stored templates of example plates. The
regions for which the respective normalized cross-corre-
lation is above a defined threshold are considered and theirsize information is further used for updating upper and
lower bound values. The process keeps updating the
threshold values with each new candidate that fulfills thecorrelation criteria. It stops when the size does not change
between few successive frames.
Figure 4illustrates the effect of the learning procedure.
It is apparent that license plate sizes are learned and the
range is narrowed with passing frames.
For any candidate region, if the cumulative sum of
correlation is more than a certain threshold, then that
region is finalized as a license plate. Offline training has
been done for finding the optimized correlation threshold(see Sect. 4.1).
Figure 5illustrates the output of various stages of the
ROI selection stage. The implemented technique is robust
Fig. 3 Effect of pre-processing
(median filtering andmorphological operators). a,
dOriginal frames. b,eBinary
images of foreground objectscontaining salt and pepper
noise. c,fPre-processed noise-
free framesJ Real-Time Image Proc (2013) 8:285–295 289
123

enough that it automatically decides and adapts the
threshold for the size of license plate in a video with
respect to the overall frame size on the basis of learning.This results in improved computational efficiency, as only
a few plate-like regions, formed on the basis of the learned
size, will be evaluated for the final detection/localizationpurpose in the rest of the video frames.
3.3.2 License plate localizationThe final selection of the license plate is achieved by
extracting relevant features and classification. For thispurpose, we use a histogram of oriented gradients HOG [ 5]
for feature description. The HOG features are extracted
from the candidate plate-like regions in a slightly adaptedmanner for our purpose. Each candidate region is parti-
tioned into 16 non-overlapping blocks. For each block, wecompute a nine-bin histogram of gradient magnitudes over
gradient orientations; concatenating the histograms of all
the partitions, we obtain a 144-dimensional feature vector
for each candidate region. The final classification is carriedout using a simple nearest mean classifier as given in Eq. 2,
trained on an offline training set of license plate and non-
license plate positive and negative examples.
C¼signffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiX
iðxi/C0/C22xi/C0Ț2r
/C0ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiX
iðxi/C0/C22xițȚ2r()
ð2Ț
where xis the HOG feature vector of the test region and
/C22xțand/C22x/C0are the mean vectors of the positive and negative
examples, learned from the training set. The term under
square root determines the Euclidean distance.
This relatively naive classification strategy is adopted
for simplicity keeping in view the real-time requirements
of the intended application. The overall detection schemegives very promising results in terms of accurate locali-
zation and computational efficiency.
3.4 License plate tracking
After detection of the license plate in the video sequence,
the next step is to keep track of that license plate with the
movement of a vehicle in the consecutive frames. The real
idea of applying a tracker is to provide a supplementary
resource for the localization and extraction processes. The
tracking of license plate can serve wider practical issues aswell. The most important of them is the traffic parameter
estimation. The tracking of license plates also helps in
video indexing by using the plate characters.
When a new car appears in the frame, its license plate is
detected and passed on to an efficient tracker to keep track
of the vehicle by its license plate in the video sequence.The tracker returns a number of shifted samples of the
Fig. 4 Threshold learning in various videos. In video 1, the camera
was mounted quite close to the passing vehicles on the road. In video2, the camera was mounted far away from on-road vehicles
Fig. 5 False candidate region
rejection. a,dShow that a
number of false candidateregions detected. b,eAre
obtained by applying the
limiting criteria of windowsizes. c,fAre finalized true
candidate regions after applying
the correlation criteria290 J Real-Time Image Proc (2013) 8:285–295
123

same license plate that are testified on the same correlation
criteria used in localization. If the plate retains the mini-
mum threshold limit, it is extracted and the result is
finalized by majority voting; otherwise, the region is con-sidered as a false candidate and rejected in the next frames.
In every frame, the license plates are detected inde-
pendently. Tracking in this situation is simply connectingthese detections in successive frames. For every license
plate, the coordinates in the current frame are passed to a
modified Lucas Kanade’s tracker that predicts the position
of the respective license plate in the next frame by com-
puting the displacement using a constant accelerationdynamic model [ 2,16] as follows:
Ix ;y;tțs ðȚ ¼ Ix/C0n;y/C0g;t ðȚ ð 3Ț
where, ‘ I’ is a window of pixels, ‘ x’ and ‘ y’ are new xand
ycoordinates of the object (license plate), and ‘ n’ and ‘ g’
are the displacement values for the previous xandycoor-
dinates, respectively.
The later image taken at time t?scan be obtained by
moving every point in the current image, taken at time t,b y
a suitable amount. The amount of motion d=(n,g)i s
called the displacement of the point at X=(x,y) between
time instants tandt?s, and is in general a function of x,
y,tands[2,16].
Aw i n d o wo fs i z e Nis used to gather more information of
the texture around the feature point, as the value of a single
pixel can change due to noise. In our approach, the detected
license plate’s top left coordinate is the feature point used.In this way, the gradient matrix Mis computed as:
M¼X
N
i¼1I2
xIxIy
IxIyI2
y/C20/C21
ð4Ț
Note that the gradient matrix Mdenotes the standard Hessian
matrix of the window centered on the point to be tracked (the
top left corner of the detected license plate in our case). Ix
andIyin Eq. ( 4) are therefore the directional derivatives at
each pixel in that window. The sum over subscript irepre-
sents that the final Hessian Mis obtained by summing these
directional derivatives on all the ipixels in that window.
The displacement d=(n,g) of a feature is computed to
minimize the residue error ( e). The displacement d is cal-
culated by iteratively solving the following equation for Dd
(where Dd=d):
M:e¼d ð5Ț
where
e¼XN
i¼1ðIi/C0JiȚ½Ixi/C0Iyi/C138Tð6Ț
Iand Jare the two consecutive images. This is done to
minimize the nonlinear error. Each time a new Ddis
calculated, the value for dis updated by:diț1 ðȚ ¼ diðȚ țDd ð7Ț
Until the condition norm ( Dd)\eis satisfied. Then the
coordinate of the feature point is updated by adding dto the
formal coordinate x(t):
xtț1 ðȚ ¼ xtðȚț d ð8Ț
where, x(t?1) is the updated coordinate of the feature
point for the next frame.
Figure 6illustrates the operation of tracking in six
consecutive frames of a surveillance video sequence, where
multiple license plates are accurately detected.
3.5 License plate recognition
The final step of the framework is to recognize the char-
acters of the detected plates. The localized plate is pre-
processed for noise removal and enhancement of edge
information. After pre-processing, connected regions are
identified in the enhanced image, and bounding boxes aredrawn around them using minimum boundary rectangle
‘MBR’. This way, each character is separately enclosed by
a bounding box. Each separated character is then classifiedusing a simple nearest neighbor classifier. Note that the
classifier requires the binary maps of the segmented char-
acters. The recognition process is illustrated in Fig. 7.
The classifier is trained using an offline training set
made from a set of standard characters as well as manually
cropped characters (10 examples each) from few of thevideos (not used in the evaluation).
A key point of consideration is, due to poor lighting
conditions or low-quality video, significant chances oferror may arise in the recognition of the plate characters in
a single iteration. To reduce the probability of error, each
number plate is read in every frame and compared with itspreviously recognized result; if a recognition error is found
in the previous result, then it is corrected. Since we are
tracking the license plate, the result is finalized by majorityvoting of results from all the instances of the license plate
images. Figure 8shows the majority voting for license
plates, as shown in Fig. 7.
4 Experiments and results
The proposed framework is evaluated on a set of CCTV
road surveillance videos obtained for general purpose andmanual inspection, i.e., the cameras used to capture footage
are not dedicatedly selected or set up for automatic license
plate detection and recognition. The resolution variesfrom a maximum of (1,024 9768) to a minimum of
(360 9288). The framework processes the specified vid-
eos for all three operations (detection, tracking andJ Real-Time Image Proc (2013) 8:285–295 291
123

recognition of license plates) at a maximum of 35 ms per
frame, which means approximately 28 fps. This includesthe majority voting of detected license plates in each frame
for improving recognition efficiency. The system is
implemented using Con a 2.3-GHz dual core machine.
The comparison of execution time with contemporary
methods is presented in Table 1. It is noted that theperformance time figures of the benchmark algorithms
were obtained from the relevant articles and scaled to thesame frame size (360 9288) and frame rate (25 fps). The
results achieved by the proposed approach are significantly
better when compared with those reported by existingstate-of-the-art approaches. For instance, when the pro-
posed framework’s execution time is compared with that of
[10] which reports minimal detection, tracking and recog-
nition time, it can be seen that the execution time for
tracking and recognition are almost identical but the
localization performed by the proposed framework is sig-nificantly faster. This is due to the fact that the proposed
approach has the ability to learn operating conditions, due
to which overall detection operation becomes faster withtime. It takes about 1 s for the system to learn and adapt
operating conditions after initializing. Also, in [ 10], the
reported resolution of test videos is 352 9288, which is
lesser than our test videos.
4.1 Detection/recognition results
To evaluate the performance accuracy of our framework, we
execute it for [10 good to worst quality videos with [200
vehicles passing through them. A number of these videos
contained pedestrians and other unwanted objects as well.
Fig. 6 Tracking of license
plates in six consecutive videoframes
Fig. 7 Recognition of license plate character: abinary map of detected license plate; bcharacter segmentation using minimum boundary
rectangle on connected components; crecognized characters using nearest neighbor search
02468101214161820
CharactersOccurrence
N J LS 5U6E8 9 R B HZ A
Fig. 8 Effect of majority voting. Various characters are recognized in
30 consecutive frames, and the recognition result is finalizeddepending on the occurrence of each character292 J Real-Time Image Proc (2013) 8:285–295
123

Figure 9shows the ROC curve for different operating
points (correlation threshold). The number of false posi-tives increases with the decrease in the value of the nor-
malized threshold. We have selected 0.4 as the operatingpoint threshold in our initial learning stage. With this value,
approximately 80% true positives and 3% false positivesoccur that are rejected afterward by the successive detec-
tion steps. The overall average detection/localization
accuracy is 94%.
For good-quality videos, the license plate recognition
results are 100% which is promising. For extremely low-
quality videos with high blur factor in which the license
plates are even hard to be read by a human observer, the
recognition procedure has shown 91% efficiency. Theresults for low-quality videos with high blur factor are
shown in Fig. 10. Figure 10shows false candidate detec-
tions due to low-quality video. This is why majority votingis used to eliminate such false candidate regions.
In recognition of characters, a common problem is the
similarity of appearance of letters/characters. Some letterslook similar to numbers, e.g., O and 0, B and 8, I and 1, S
and 5, etc. If letters and numbers have designated positions
on the license plates, which is usually the case for licenseplate formats worldwide, for example, there are seven
characters on a license plate and the first two and last two
characters contain only letters and the remainder containsnumbers. The problem stated is illustrated in Fig. 11.Table 1 Comparison of the
proposed framework with
various methodsMethod Localization Tracking Recognition
Donoser et al. [ 10] 0.070 s 0.005 s 0.006 s
Zhange et al. [ 24] 0.05 s – –
Rastegar et al. [ 19] 2.3 s – 0.4 s
Zheng et al. [ 25] 5.03 s – –
Arth et al. [ 1] – 0.039 s 0.0198 s
Line-wise filters DFTs [ 19] 2.1 s – –
Edge image improvement [ 19] 1.97 s – –
Row-wise and column-wise [ 19] 2.44 s – –
The proposed method 0.025 s 0.006 s 0.0045 s
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 100.10.20.30.40.50.60.70.80.91
False Positives (1-specificity)True Positives (sensitivity)0.10.20.3
0.4
0.5
0.6
0.7
0.80-0.1
Fig. 9 ROC curve for license plate detection. The graph shows that
false positives increase with the increase in correlation threshold
Fig. 10 False detection due to
low-quality video
Fig. 11 Recognition results of
detected plates fromconsecutive frames. aDetected
plates from consecutive video
frames. bRespective recognized
charactersJ Real-Time Image Proc (2013) 8:285–295 293
123

The proposed system has been thoroughly trained, so
that the chance of misclassification of a character is very
low.
5 Conclusion
An intelligent license plate localization, tracking and rec-
ognition method has been implemented for real-time video
streams captured from road surveillance cameras. We have
addressed these issues for a range of low- to high-resolu-
tion video streams with variant conditions and motion blur.The realized system intelligently performs all three oper-
ations in 35 ms per frame with exceptional accuracy. It
means that the method can easily work at 28 fps. This isdue to its capability to learn and adjust itself with different
camera positions and distances. The framework uses a
novel and comparatively faster technique for license platedetection that is a key operation for the whole system. The
system works well for multiple license plates in a single
frame. The method can be improved further by incorpo-rating super-resolution techniques on the obtained multiple
low-resolution instances of the license plate for enhanced
recognition
References
1. Arth, C., Limberger, F., Bishof, H.: Real-time license plate rec-
ognition on an embedded DSP-platform. Institute for ComputerGraphics and Vision, Graz University, Austria (2006)
2. Baker, S., Matthews, I.: Lucas–Kanade 20 years on: a unifying
framework. Int. J. Comp. Vis. 56, 3221–3255 (2004)
3. Chang, S.L., et al.: Automatic license plate recognition. IEEE
Transact. Intell. Transp. Sys. 5(1), 42–53 (2004)
4. Chateau, T., Lapreste, J.: Robust real time tracking of a vehicle
by image processing. IEEE Intell. Vehicles Symp. 315–318(2004). doi: 10.1109/IVS.2004.1336401
5. Dalal, N., Triggs, B.: Histograms of oriented gradients for human
detection. CVPR IEEE Comp. Soc. Conf. 1, 886–893 (2005)
6. Deb, K., Chae, H.U., Jo, K.H.: Vehicle licence plate detection
method based on sliding concentric windows and histogram. Int.J. Comp. 4(8), 771–777 (2009)
7. Dellaert, F., Thorpe, C.: Robust car tracking using Kalman fil-
tering and Bayesian templates. Conference on Intelligent Trans-portation Systems (1997)
8. Dlagnekov, L., Belongie, S.: Recognizing cars. Technical Report
CS2005–0833. UCSD University of California, San Diego (2005).Accessed 19 June 2010
9. Dlagnekov, L.: license plate detection using AdaBoost, computer
science & engineering dpt., La Jolla, San Diego. http://www.
cse.ucsd.edu/classes/fa04/cse252c/projects/louka.pdf
10. Donoser, M., Arth, C., Bischof, H.: Detecting, tracking and rec-
ognizing license plates ACCV 2007, Part II. LNCS 4844 ,
447–456 (2007)
11. Kamat, V., Ganesan, S.: An efficient implementation of the
hough transform for detecting vehicle license plates using DSPs.
In: Proceedings of the IEEE International Conference RTAS.
58–59 (1995)12. Kanhere, N.K., Birchfield, S.T., Sarasua, W.A.: Vehicle seg-
mentation and tracking in the presence of occlusions. Transp.
Res. Rec. 1944 , 89–97 (2006)
13. Kim, S., Oh, S. Y., Kang, J., Ryu, Y. K., Park, C., Park, K.: Front
and rear vehicle detection and tracking in the day and night-times
using vision and sonar sensor fusion. IEEE/RS J. Int. Conf. Intell.Robots Sys. 2173–2178 (2005)
14. Koller, D., Weber, J., Malik, J.: Robust multiple car tracking with
occlusion reasoning. In Proc. Third Europeon Conference on
Computer Vision, Stockholm, Sweden. LNCS 800, pp. 189–196.Springer, 2–6 May 1994
15. Li, X., Yan, X., Murphey, Y., Karlsen, R., Gerhart, G.: A real-
time vehicle detection and tracking system in outdoor traffic
scenes. ICPR 2004, Proceedings of the 17th International Con-ference 2. (2004) 761-764
16. Lucas, B. D., Kanade, T.: An iterative image registration tech-
nique with an application to stereo vision. In: Proceedings of the
7th International Joint Conference on Artificial Intelligence.(1981)
17. Marmoiton, F., Collange, F., Alizon, J., Derutin, J.: 3d localiza-
tion of a car observed through a monocular video camera. In:Proceedings of the IEEE International Conference on IntelligentVehicles. (1998)
18. Rahman, C., Badawy, W., Radmanesh, A.: A real time vehicle’s
license plate recognition system. In: AVSS, Proceedings of theIEEE Conference on Advanced Video and Signal Based Sur-veillance, IEEE Computer Society Press, Los Alamitos 163–166.
(2003)
19. Rastegar, S., Ghaderi, R., Ardeshir, J., Asadi, N.: An intelligent
control system using an efficient license plate location and rec-
ognition approach. Int. J. Image Process. (IJIP) 3(5), 252–264
(2009)
20. Sergi, M., Shankwitz, C., Donath, M.: Lidar-based vehicle
tracking for a virtual mirror. Intelligent Vehicle Symposium.
Proc. IEEE. 333–338 (2003). doi: 10.1109/IVS.2003.1212932
21. Shapiro, V., Gluhchev, G., Dimov, D.: Towards a multinational
car license plate recognition system. Mach. Vis. Appl. 17(3),
173–183 (2006)
22. Tan, X., Li, J., and Liu, C. A video-based real-time vehicle
detection method by classified background learning. WorldTransact. Eng. Technol. Educ. 6(1) (2007)
23. Yanamura, Y., Goto, M., Nishiyama, D., Soga, M., Nakatani, H.,
Saji, H.: Extraction and tracking of the license plate using Hough
transform and voted block matching. IEEE IV2003 IntelligentVehicles Symposium, Proceeding (Cat. No. 03TH8683), 243–246
(2003)
24. Zhang, H. et al.: Real-time license plate detection under various
conditions. Springer -Verlag Berlin, Heidelberg, 192–199 (2006)
25. Zheng, D., Zhao, Y., Wang, J.: An efficient method of license
plate location. Pattern Recogn. Lett. 26, 2431–2438 (2005)
Author Biographies
Dr M. Saquib Sarfraz born in 1977, received an M.S. degree in
Electrical and Computer Engineering from National University of
Sciences and Technology, Pakistan, in 2003 and his PhD degree in
Computer Vision at Technische Universita ¨t Berlin, Germany in 2008.
He is currently working as head of Computer Vision Research Group,COMSATS Institute of Information Technology Lahore, Pakistan.
His research interests include pattern recognition, statistical machine
learning, face recognition, and multimodal biometrics.
Atif Shahzad is currently working as Research staff in Computer
Vision Research Group, COMSATS Institute of Information294 J Real-Time Image Proc (2013) 8:285–295
123

Technology Lahore, Pakistan. He received his MSc degree in
Electrical & Electronics Engineering; United Kingdom. His researchinterests include Object Identification and Classification in Videos.
Muhammad Adnan Elahi is currently working as Research staff in
Computer Vision Research Group, COMSATS Institute of Informa-tion Technology Lahore, Pakistan. He received his MSc degree in
Embedded Digital Systems, United Kingdom. His research interests
include Microprocessor & FPGA based design, GPS based Tracking/Navigation systems.
Muhammad Fraz is currently working as Research staff in Computer
Vision Research Group, COMSATS Institute of Information Tech-
nology Lahore, Pakistan. He received his MSc degree in EmbeddedDigital Systems, United Kingdom His research interests includeVideo Processing on DSPs.
Iffat Zafar is currently working as a Research Associate in the
Department of Computer Science at Loughborough University UK.She received her MSc degree in Computer Sciences from the
International Islamic University, Islamabad, Pakistan, in October2000, her degrees in MSc Multimedia and Internet Computing (with
distinction) in October 2004 and her PhD in Computer Science in
2008 at Loughborough University. Her research interests includeComputer and Machine Vision, Pattern Matching and Recognition,
Image Analysis, machine learning and Automated Surveillance.
Eran. A. Edirisinghe is a Reader in the Department of Computer
Science at Loughborough University UK. He is the researchcoordinator and a member of the Visual, Imaging & Autonomous
system Research Division (VIAS) within the Department. His
research interests include image processing, computer vision, patternrecognition, video coding, signal processing, stereo image coding,image and video watermarking.J Real-Time Image Proc (2013) 8:285–295 295
123
View publication statsView publication stats