MINISTRY OF EDUСATION OF MOLDOVA Free International University of Moldova Faсulty of Informatics, Engineering, Design Department of Informational… [309855]

MINISTRY OF EDUСATION OF MOLDOVA

Free International University of Moldova

Faс[anonimizat], Design

Department of Informational Technologies and Engineering

Accepted for defense Accepted for defense

Dean of the Faculty Head of the Department

Dubсovețс[anonimizat]., conf. univ.

___________________________ ____________________________

„____”________________2016 „____” __________________2016

[anonimizat]: [anonimizat]. C-42

Project Supervisor

Perju L. Veaceslav, dr. hab., prof. univ.

Сhișinău 2016

© Bordian Vlad, 2016

Head of the Department of Informational Technologies and Engineering

Dubсovețс[anonimizat]., prof. univ.

_________________________

”___” ________________ 201_

TASK

for license project of the student: [anonimizat]. C-42

Bordian Vlad

Theme: „Objects detection system based on optical correlator and intelligence analysis”

approved by the order nr. ________ from ”___” _____________ 201_

Explanatory notes contents: 1. The analysis of existing methods and systems of objects decetion; 2. Elaboration, realization and investigation of an original algorithm of objects detection; 3. Elaboration and research of the objects detection system.

Graphic material list: 1. [anonimizat]; 2. The structure of the original algorithm of objects detection; 3. Block-scheme of the elaborated software of objects detection; 4. The results of investigation of objects detection algorithm; 5. The structure of objects detection system; 6. The results of investigation of objects detection system.

Date of task handing:”___” _____________ 201_

Author

Student: [anonimizat]. C-42, Bordian Vlad

Project Supervisor

Perju L. Veaceslav, dr. hab., prof. univ.

ABSTRAСT

Bordian Vlad, „Objects detection system based on optical correlator and intelligence analysis” license project for speсialty Computers, Сhisinau, 2016.

According to the guideline the license project is structured: [anonimizat], [anonimizat], bibliography that quantifies 25 [anonimizat]. [anonimizat], Curriculum Vitae and other requested data. The license project consists of 67 pages of the main text including 37 figures, 1 table and 35 formulas.

The keywords of the project license are the following: optic, correlator, [anonimizat], recognition, analysis, filter.

Field of study of the thesis is signal proсessing.

The main goal of the given paper is to create an original algorithm for objects detection in order to develop suitable software for objects detection by means of optical correlator.

The objectives of the license project:

[anonimizat], etc.;

to investigate the theoretical perpectives on methods and systems for objects detection;

to identify the suitable method for object detection;

to create an original algorithm for object detection on the basis of other algorithms and chosen method;

to develop and describe the software for objects detection based on the original algorithm;

to test the developed software for objects detection;

to elaborate a user guide for software and a Programmer Developer guide.

Novelty and originality of this work is the use of an original algorithm for object detection by optical correlator.

The theoretiсal signifiсanсe is the developing a software that uses improved algorithms for object detection.

Appliсative value of the work is that this system сan be used in various fields and areas.

Implementation results. The developed system has been сonfigured and tested on multiple images.

ADNOTARE

Bordian Vlad, „ Sistemul de detectare a obiectelor în baza corelatorului optic și analizei inteligente”, proiect de liсență la speсialitatea Calculatoare, Сhișinău, 2016.

În conformitate cu regulamentul de perfectare a proiectelor de licență, lucrarea data este structurată în felul următor: lista abrevierilor utilizate în textul lucrării, introducerea, trei capitole, concluzii generale și recomandări, bibliografie ce include 25 de titluri de literature de specialitate print și on-line. De asemenea proiectul de licență include anexe ce prezintă codul sursă, Curriculum Vitae și alte documente solicitate. El este perfectat pe 67 pagini, incluzând 3 7 figuri, 1 tabel și 35 formule.

Сuvinte-сheie ale proiectului de licență sunt următoarele: optic, corelator, detectarea obiectelor, sistem, recunoastere, analysis, filtru.

Domeniul de studiu al proiectului este preluсrarea semnalelor.

Sсopul de bază a lucrprii date este de a crea un algoritm nou de detectare a obiectelor pentru a crea un soft potrivit de detectare a obiectelor cu ajutorul corelatorului optic.

Obieсtivele proiectului de licență sunt următoarele:

de a studia și sintetiza literatura de specialitate în ceea ce privește metodele și sistemele de detectare a obiectelor, etc.;

de a investiga abordările teoretice cu privire la metodele și sistemele de detectare a obiectelor;

de a identifica metoda potrivită de detectare a obiectelor;

de a crea un algoritm nou de detectare a obiectelor în baza altor algoritme și metodei alese;

de a scrie și expune soft-ul de detectare a obiectelor bazat pe algoritmul nou;

de a testa soft-ul de detectare a obiectelor;

de a elabora ghidul utilizatorului pentru soft precum și ghidul programatorului.

Noutatea și originalitatea luсrării сonstă în folosirea unui algoritm nou pentru a detecta obiectele în baza corelatorului optic.

Semnifiсația teoretiсă a luсrării сonstă în dezvoltarea unui software сare implementează algoritme mai eficiente pentru detectarea obiectelor.

Valoarea apliсativă a luсrării сonstă în faptul сă aсest sistem poate fi implementat în diverse domenii.

Implementarea rezultatelor obținute. Sistemul realizat a fost сonfigurat și testat pe mai multe imagini.

ABREVIATION LIST

OD – Objects detection

SLM – Spatial light modulator

JPS – Joint power spectrum

FT – Fourier transforms

SNR – Signal-to-noise ratio

NR – Noise removing

SIFT – Scale invariant feature transform

DoG – Difference of Gaussian

LTI – Linear time invariant

IDE – Integrated development environment

SAR – Synthetic aperture radar

SVM – Support vector machine

SUV – Sport utility vehicle

INTRODUСTION

Object detection is a computer technology related to computer vision and image processing that deals with detecting instances of semantic objects of a certain class (such as humans, buildings, or cars) in digital images and videos. Well-researched domains of object detection include face detection and pedestrian detection. Object detection has applications in many areas of computer vision, including image retrieval and video surveillance.

A basic problem that often occurs in object detection is to determine the position of a given patter in an image, or part of an image, the so-called region of interest. This problem is closely related to the determination of a received digital signal in signal processing using e.g. a matched filter.

Two basic cases can be differentiated:

the position of the pattern is unknown

an estimate for the position of the pattern is given

Usually, both cases have to be treated to solve the problem of determining the position of a given pattern in an image. In the latter case, the information about the position of the pattern can be used to reduce the computational effort significantly. It is also known as feature tracking in a sequence of images.

For both feature tracking and the initial estimation of the position of the given pattern, many different, well-known algorithms have been developed. One basic approach that can be used in both cases mentioned above, is template matching. This means that the position of the given pattern is determined by a pixel-wise comparison of the image with a given template that contains the desired pattern. For this, the template is shifted u discrete steps in the x direction and v steps in the y direction of the image, and then the comparison is calculated over the template area for each position (u; v). To calculate this comparison, normalized cross correlation is a reasonable choice in many cases. Nevertheless, it is computationally expensive and therefore a fast correlation algorithm that requires less calculation than the basic version is of interest.

The main goal of the given paper is to create an original algorithm for objects detection, in order to develop a suitable software for objects detection by means of optical correlator.

The objectives of the license project are the following:

to study the specialty literature regarding the methods and systems for objects detection, algorithm and systems for object detection, etc.;

to investigate the theoretical perpectives on methods and systems for objects detection;

to identify the suitable method for object detection;

to create an original algorithm for object detection on the basis of other algorithms and chosen method;

to develop and describe the software for objects detection based on the original algorithm;

to test the developed software for objects detection;

to elaborate a user guide and a programming developer guide.

The theoretiсal signifiсanсe is the developed algorithm that uses correlation function for a more accurate detection of the object in the given image.

Appliсative value of the work is that this system сan be used in various fields and areas, like UAV (both civilian and military).

In Chapter 1 is made an analysis of the existent object detection systems. There is shown the general idea and description of each method and module it uses. Also, there are presented their advantages and disadvantages.

Chapter 2 begins with the algorithm description, of the correlation function and cross-correlation.

There is presented the software scheme, the information about optical correlator and matched filter. Chapter 2 concludes with the results of the software testing (one image, which was processed under different circumstances with the purpose of software testing) and a conclusion on the Chapter.

Chapter 3 is composed of a user guide, which helps any user understand and efficiently use this software, programmer developer guide and the conclusion on Chapter 3.

The project itself ends with a General Conclusion and Recommandations.

ANALYSIS OF METHODS AND SYSTEMS FOR OBJECTS DETECTION

In the field of computer vision, object detection describes the task of finding and identifying objects in an image or video sequence [1]. Humans recognize a multitude of objects in images with little effort, despite the fact that the image of the objects may vary somewhat in different view points, in many different sizes and scales or even when they are translated or rotated. Objects can even be recognized when they are partially obstructed from view. This task is still a challenge for computer vision systems. Many approaches to the task have been implemented over multiple decades.

Object Detection and Localization

Object detection is mainly concentrated to detect the target position in each frame with coordinates, scale, and orientation [12]. In object detection phase, the feature vectors are derived using 2D correlation. Correlation is one of the statistical approaches, which provide a direct measure of the similarity between two video frames, and it will not be influenced by illumination variation and object translations. However, it cannot cope with image rotation and scaling. The proposed model can further be extended to deal with an image rotation and scaling by incorporating the sophisticated object detection algorithm such as multiresolution analysis. In proposed Simulink model, the 2D correlation block computes the two-dimensional cross correlation between compressed frame and template frame. At each location, the cross correlation coefficient has inflated scores for matching feature points and deflated scores for others. Let S(x, y) be the compressed video frame with a dimension (M × N) and T(x, y) be the template frame with a dimension (P × Q). Cross correlation C(i, j) is calculated by using the following equation:

C(i,j) =, (1.1)

where 0 ≤ i < (M + N − 1); 0 ≤ j < (P + Q − 1).

In this phase, an effective classifier is constructed to classify the matched features points into foreground and background using Bayesian rule. Let S(k, t) be the input image frame at time t in the position k, T(k, t) be the template frame, and fv,t be the feature vector of target in the position k at time t. The posterior probability of feature vector that appears from the background at position k is calculated as follows:

P(fback/fv,t) = (P(fv,t/fback)P(fback)) / P(fv,t), (1.2)

where fback is the background and P(fv,t/fback) is the probability of feature vector fv,t being observed as background. The prior probability of feature vector being identified at the position k is denoted by P(fv,t) and P(fback) is the prior probability of feature vector belonging to background. Similarly, the posterior probability of feature vector that appears from foreground at position k is calculated as follows:

P(ffore/fv,t) = (P(fv,t/ffore)P(ffore)) / P(fv,t), (1.3)

where ffore represents the foreground and P(fv,t/ffore) is the probability of feature vector fv,t being observed as foreground. The prior probability of feature vector belonging to foreground is P(ffore). Thus, a probability map is constructed over the compressed video frames and the target is localized by searching the maximum likelihood density. When the template is centered at coordinates (x, y), the peak cross correlation coefficient indicates a good match of target location between compressed frame and the target template. Thus, the maximum likelihood density for foreground is estimated using the following equation:

ρ(fv,t, fobj) = max . (1.4)

Thus, the posterior probability P(fv,t/ffore) can be estimated as follows:

P(fv,t/ffore) = C1ρ(fv,t,fobj), (1.5)

where C1 denotes normalization factor. Similarly, the matching feature points other than peak correlation coefficients in each image frame are considered as maximum likelihood densities for the background and are calculated as follows:

ρ(fv,t, fback) = . (1.6)

Hence, the posterior probability P(fv,t/fback) can be estimated as follows:

P(fv,t/fback) = C2 max ρ(fv,t,fback), (1.7)

where C2 represents normalization constant.

Object Detection Methods

Appearance Based Methods of Object Detection

– Use example images (called templates or exemplars) of the objects to perform recognition [15]

– Objects look different under varying conditions:

Changes in lighting or color

Changes in viewing direction

Changes in size / shape

– A single exemplar is unlikely to succeed reliably. However, it is impossible to represent all appearances of an object.

The central idea behind appearance-based methods is the following [11]. Having seen all possible appearances of an object, can recognition be achieved by just efficiently remembering all of them? Could recognition be thus implemented as an efficient visual (pictorial) memory? The answer obviously depends on what is meant by “all appearances”. The approach has been successfully demonstrated for scenes with unoccluded objects on black background. But remembering all possible object appearances in the case of arbitrary background, occlusion and illumination, is currently computationally prohibitive.

Appearance based methods typically include two phases. In the first phase, a model is constructed from a set of reference images. The set includes the appearance of the object under different orientations, different illuminants and potentially multiple instances of a class of objects, for example faces. The images are highly correlated and can be efficiently compressed using e.g. Karhunen-Loeve transformation (also known as Principal Component Analysis – PCA).

In the second phase, “recall”, parts of the input image (subimages of the same size as the training images) are extracted, possibly by segmentation (by texture, colour, motion) or by exhaustive enumeration of image windows over whole image. The recognition system then compares an extracted part of the input image with the reference images (e.g. by projecting the part to the Karhunen-Loeve space).

A major limitation of the appearance-based approaches is that they require isolation of the complete object of interest from the background. They are thus sensitive to occlusion and require good segmentation. A number of attempts have been made to address recognition with occluded or partial data.

The family of appearance-based object recognition methods includes global histogram matching methods [16]. Some scholars proposed to represent an object by a colour histogram. Objects are identified by matching histograms of image regions to histograms of a model image. While the technique is robust to object orientation, scaling, and occlusion, it is very sensitive to lighting conditions, and it is not suitable for recognition of objects that cannot be identified by colour alone. The approach has been later modified by other researchers to exploit illumination invariants. Recently, the concept of histogram matching was generalised by Schiele, where, instead of pixel colours, responses of various filters are used to form the histograms (called then receptive field histograms).

To summarise, appearance based approaches are attractive since they do not require image features or geometric primitives to be detected and matched. But their limitations, i.e. the necessity of dense sampling of training views and the low robustness to occlusion and cluttered background, make them suitable mainly for certain applications with limited or controlled variations in the image formation conditions, e.g. for industrial inspection.

Edge matching

Uses edge detection techniques, such as the Canny edge detection, to find edges [20].

Changes in lighting and color usually don’t have much effect on image edges

Strategy:

Detect edges in template and image

Compare edges images to find the template

Must consider range of possible template positions

Measurements:

Good – count the number of overlapping edges. Not robust to changes in shape

Better – count the number of template edge pixels with some distance of an edge in the search image

Best – determine probability distribution of distance to nearest edge in search image (if template at correct position). Estimate likelihood of each template position generating image

Divide-and-Conquer

Strategy:

Consider all positions as a set (a cell in the space of positions)

Determine lower bound on score at best position in cell

If bound is too large, prune cell

If bound is not too large, divide cell into subcells and try each subcell recursively

Process stops when cell is “small enough”

Unlike multi-resolution search, this technique is guaranteed to find all matches that meet the criterion (assuming that the lower bound is accurate)

Finding the Bound:

To find the lower bound on the best score, look at score for the template position represented by the center of the cell

Subtract maximum change from the “center” position for any other position in cell (occurs at cell corners)

Complexities arise from determining bounds on distance

Greyscale matching

Edges are (mostly) robust to illumination changes, however they throw away a lot of information

Must compute pixel distance as a function of both pixel position and pixel intensity

Can be applied to color also

Gradient matching

Another way to be robust to illumination changes without throwing away as much information is to compare image gradients

Matching is performed like matching greyscale images

Simple alternative: Use (normalized) correlation

Histograms of receptive field responses

Avoids explicit point correspondences

Relations between different image points implicitly coded in the receptive field responses

Large modelbases

One approach to efficiently searching the database for a specific image to use eigenvectors of the templates (called eigenfaces)

Modelbases are a collection of geometric models of the objects that should be recognized

Geometry-based Methods of Object Detection

In geometry- (or shape-, or model-) based methods, the information about the objects is represented explicitly. The recognition can than be interpreted as deciding whether (a part of) a given image can be a projection of the known (usually 3D) model of an object [10].

Generally, two representations are needed: one to represent object model, and another to represent the image content. To facilitate finding a match between model and image, the two representations should be closely related. In the ideal case there will be a simple relation between primitives used to describe the model and those used to describe the image. Would the object be, for example, described by a wireframe model, the image might be best described in terms of linear intensity edges. Each edge can be then matched directly to one of the model wires. However, the model and image representations often have distinctly different “meanings”. The model may describe the 3D shape of an object while the image edges correspond only to visible manifestations of that shape mixed together with “false” edges (discontinuities in surface albedo) and illumination effects (shadows).

To achieve pose and illumination invariance, it is preferable to employ model primitives that are at least somewhat invariant with respect to changes in these conditions [13]. Considerable effort has been directed to identify primitives that are invariant with respect to viewpoint change.

The main disadvantages of geometry-based methods are: the dependency on reliable extraction of geometric primitives (lines, circles, etc.), the ambiguity in interpretation of the detected primitives (presence of primitives that are not modelled), the restricted modelling capabilities only to a class of objects which are composed of few easily detectable elements, and the need to create the models manually.

Feature-based Methods of Object Detection

The general idea of this group of object detection methods are the following:

– a search is used to find feasible matches between object features and image features [20].

– the primary constraint is that a single position of the object must account for all of the feasible matches.

– methods that extract features from the objects to be recognized and the images to be searched:

surface patches

corners

linear edges

Types of Optical Correlator

A device for comparing two signals by utilizing the Fourier transforming properties of a lens is optical correlator [22]. It is commonly used in optics for target identification and tracking. The correlator has an input signal, which is multiplied by some filter in the Fourier domain. An example filter is the matched filter which uses the cross correlation of the two signals. An optical correlator automatically identifies or recognizes the contents of an image by combining an incoming image with a reference image, and the degree of correlation after combining the images determining the intensity of an output light beam. Optical correlators can perform complex pattern recognition tasks more rapidly than known digital techniques. Optical correlators are capable of processing large amounts of data in a data stream that can be useful in the detection, extraction and classification of desired information included in the data. Optical correlators are employed for many optical signal-processing applications, including pattern and character recognition, implementation of optical interconnections in hybrid optoelectronic parallel computers, and in artificial neural network technologies. Optical correlators are typically found in optical communication systems for signal detection applications involving data that are conveyed by light carriers whose frequency or operating wavelength is increasing as the technology continues to advance. The optical correlator can be used in many different applications including quality control, defect detection, target tracking and pattern recognition.

There are two main families of optical correlator, the matched Filter and the Joint Transform Correlator. Both types achieve largely the same results, but process the information in different ways [25].

The Matched Filter

The Matched Filter is the most commonly used type of correlator, but also the most complex and sensitive to build, owing to its strict alignment criteria and long optical train [6].

The process involves taking the individual Fourier Transforms of the “Input” and “Reference” images, taking their product and then Fourier Transforming the result. Usually, the reference transform is done off-line in electronics to provide the “filter”, which is usually displayed on a transmissive Spatial Light Modulator (SLM).

Typically, the input transform is produced optically, using an input SLM. The input FT is then optically multiplied with the reference filter and the result optically Fourier transformed to produce the correlation. The advantages of this method are a high space-bandwidth product and extremely fast process time.

The Joint Transform Correlator

The Joint Transform Correlator is traditionally the less commonly used type of correlator than the Matched Filter [9].

The process differs significantly from the Matched Filter process, since no reference filter is required. Instead, the “Input” and “Reference” images are aligned and displayed alongside each other on the SLM. The two images are then Fourier Transformed together. A nonlinearity then captures the intensity distribution of transform, producing the Joint Power Spectrum (JPS). The JPS is then Fourier Transformed itself, producing an output of symmetrically identical pairs of correlation peaks, per match.

Invariant Pattern Recognition and Filter Design

Optical Invariant Filter

The invariance to rotation, scaling, view angle change and partial occlusion of the input image remains as an important issue in all pattern recognition approaches [23]. For optical correlator, Hsu and Arsenault have proposed the circular harmonic matched filters in 1982, which correlates the input image with its one circular harmonic component. The synthetic discriminant function may be designed to produce equal correlation peaks for all the Ninput training images, including all possibly distorted images, for instance, N = 12 training images each rotated by 30 degrees20. However, the number N of the distorted training images, which may be composed into a single synthetic discriminant function, is limited. Moreover, the image distortions are often continuous, such as the rotation angle and scale of the image are usually continuous. The discrete sampling of the image continuous distortions can lead to a decrease of the output signal-to-noise ratio of the composite filter.

Scale Invariant Feature Transform

In the computer vision community, Lowe has proposed the Scale Invariant Feature Transform (SIFT) in 2004 [8]. The algorithm is not designed to extract and describe the key-points in the image. Based on pairing of the keypoints detected in two images of the same scene by two calibrated cameras one can compute a 3D vision by the epipolar geometry, or compute the distortions between these two images for image registration and image fusion. The SIFT features may be also used for object classification and pattern recognition. The SIFT is invariant to translation, rotation, scaling and illumination changes. It has been built by massive experimental trial and testing, and proved to be minimally affected by noise and small distortion and particularly efficient to process natural outdoor images, compared with other mathematically based image invariant features, such as the geometrical image moment invariants, which are mostly suitable for describing 2D geometrical patterns. The SIFT is based on the scale space analysis with the scale coordinate σ discretized in logarithmic steps as σ = σ02i+s/S with the octave scale index i and the sub-scale index s = 1, 2 …S with the sub-scale resolution S, where σ0 = 1.6×21/S is the basic scale. Firstly, the Gaussian scale image G(x, σ) and the Difference of Gaussian (DoG) scale images D(x, σ (s,i)) of an input image I(x) are computed as

G(x, σ) = (gσ ∗ I )(x) and D(x, σ (s,i)) = G(x,σ (s +1,i)) − G(x, σ (s,i)) (1.8)

where gσ is the Gaussian kernel of the width σ.

The SIFT keypoints (x, σ) are then extracted as local minima/maxima in the (DoG) scale images, compared with their 26 neighbour pixels in the regions of 3×3 pixels at the current subscale space s and the upscale (s+1) and the downscale (s-1) adjacent sub-scale spaces, respectively [4]. The keypoints with low contrast and located along edges are next discarded. Then, the keypoints (x,σ) is assigned by a dominant orientation as the maximum in an interpolated histogram of the gradient orientations calculated in the Gaussian scale image G(x,σ) within a Gaussian support of width 1.5σ around the keypoint. The contribution of each pixel in the Gaussian support to the orientation histogram is weighted by its gradient magnitude and by the Gaussian function centered at the keypoint. In addition to the location and the orientation, a SIFT feature descriptor is computed for a keypoint as a set of 4×4 histograms of gradient orientations. Each histogram is based on a mσ × mσ pixels region with m=3 in the neighbourhood of the keypoint and consists of 8 orientation bins with the magnitudes of that histogram entry. The histograms are interpolated and weighted in the same way as that for computing the dominant orientation. Consequently, each SIFT keypoint is described by 8×4×4 = 128 features, plus the coordinates of its location, the scale of the image and its dominant orientation [5].

Component-based Approach

The component-based object detection is another interesting technique of pattern recognition, which can be not sensitive to image distortions [18]. As a whole object appearance can change easily by changes in view angle, object scale, orientation and illumination and by occlusion, the component-based approach can provide a robust detection by first detecting small and characteristic parts of the object and then combining the object parts using an object model. Detection of small object patches may be faster and less sensitive to the distortions, than that of the whole object. Also, the description of the object components can be based on the invariant image features. Furthermore, once the object components are detected the object model to combine the object parts to the object can be deformable and is not sensitive to image distortions. The algorithm for regrouping the components to the object according to the unique spatial relations among the parts can be robust to image distortion and to the uncertainties in the detection of the object parts. More importantly, the component-based object detection does not aim to detect one specific object based on its appearance, but to detect a class of objects, described by the same object model. In addition, the approach is of cause not sensitive to the occlusion of the object. The component-based object detection has been particularly successful in the face recognition applications. Figure 1.1 shows the detection of cars in the aerial images by the component-based approach. All the cars were correctly classified regardless their orientations and colors, except a white SUV whose components were not present in the training set for the SVM.

Fig. 1.1. Sixty-eight cars in parking lot are detected with only one SUV missing

Filter Design

Many filters have been designed to be used with an optical correlator [21]. Some have been proposed to address hardware limitations, others were developed to optimize a merit function or to be invariant under a certain transformation.

Matched filter

The matched filter maximizes the signal-to-noise ratio and is simply obtained by using as a filter the Fourier transform of the reference signal r(x, y).

(1.9)

Phase-only filter

The phase-only filter is easier to implement due to limitation of many SLMs and has been shown to be more discriminant than the matched filter.

(1.10)

Feature Extraction in the Fourier Transform Plane

Let Sij be the intensity of the digitized Fourier [7] transform image and Fij(r, θ) be the intensity function of the basic geometrical feature of the desired object. Using an intuitive approach, we seek the values of r and θ that minimize the error function

L(r, θ) = ij – Fij(r, θ)2 (1.11)

If we select a rectangular feature with the dimensions 2w x 2l, the intensity function Fij becomes

Fij(w, l, θ) = []2 []2 , (1.12)

where

Iij(θ) = icosθ + jsinθ, Jij(θ) = -isinθ + jcosθ, (1.13)

and K is the appropriate scaling constant times π. Therefore, given a digitized Fourier transform intensity image Sij, the rectangular information regarding the object scale and orientation can be estimated as

L(w*, l*, θ*) = ij – Fij(w, l, θ)]2 (1.14)

This nonlinear parametric optimization problem is easy to solve; many efficient iterative algorithms are available [2]. Furthermore, since the rectangular feature (object dimension) is in the low-frequency range, only a small section of the intensity plane is needed for evaluating the object's scale and orientation.

Analysis of the Object Detection Methods

In this table, we represented the advantages and disadvantages of the known methods of object detection. Also, we briefly explained the general idea behind each method.

Table 1.1. Advantages and disadvantages of object detection methods

Conclusion on Chapter 1

Objects detection systems are increasingly required as the time goes. They are used in urban video surveillance (object detecton and tracking, face/pattern recognition, etc.), army, DAS (drive assistance system), etc. After studying the existent methods, we can see that each of them has their own advantages and disadvantages (Table 1.1), so we pick the mehod based on our requirements and needs. Also, we can improve the system through the use of various filters, to cover method weaknesses.

ALGORITHM AND SYSTEM FOR OBJECTS DETECTION

Algorithm for Objects Detection

Correlation Function

A correlation function is a statistical correlation between random variables at two different points in space or time, usually as a function of the spatial or temporal distance between the points. If one considers the correlation function between random variables representing the same quantity measured at two different points then this is often referred to as an autocorrelation function being made up of autocorrelations. Correlation functions of different random variables are sometimes called cross correlation functions to emphasise that different variables are being considered and because they are made up of cross correlations [4].

Fig.2.1. Visual comparison of convolution, cross-correlation and autocorrelation

For random variables X(s) and X(t) at different points s and t of some space, the correlation function is:

(2.1)

where {\displaystyle \operatorname {corr} } corr is described in the article on correlation. In this definition, it has been assumed that the stochastic variable is scalar-valued. If it is not, then more complicated correlation functions can be defined. For example, if X(s) is a vector, then a matrix of correlation functions is defined as:

(2.2)

or a scalar, which is the trace of this matrix. If the probability distribution has any target space symmetries, i.e. symmetries in the value space of the stochastic variable (also called internal symmetries), then the correlation matrix will have induced symmetries. Similarly, if there are symmetries of the space (or time) domain in which the random variables exist (also called spacetime symmetries), then the correlation function will have corresponding space or time symmetries. Examples of important spacetime symmetries are:

translational symmetry yields C(s,s') = C(s − s') where s and s' are to be interpreted as vectors giving coordinates of the points [14].

rotational symmetry in addition to the above gives C(s, s') = C(|s − s'|) where |x| denotes the norm of the vector x (for actual rotations this is the Euclidean or 2-norm).

Higher order correlation functions are often defined. A typical correlation function of order n is:

(2.3)

If the random variable has only one component, then the indices ij are redundant. If there are symmetries, then the correlation function can be broken up into irreducible representations of the symmetries — both internal and spacetime.
The case of correlations of a single random variable can be thought of as a special case of autocorrelation of a stochastic process on a space which contains a single point.

Cross Correlation

In signal processing, cross-correlation is a measure of similarity of two series as a function of the lag of one relative to the other [4]. This is also known as a sliding dot product or sliding inner-product. It is commonly used for searching a long signal for a shorter, known feature. It has applications in pattern recognition, single particle analysis, electron tomography, averaging, cryptanalysis, and neurophysiology.

For continuous functions f and g, the cross-correlation is defined as:

(2.4)

where {\displaystyle f^{*}}f* denotes the complex conjugate of {\displaystyle f}f and {\displaystyle \tau }r is the lag.

Similarly, for discrete functions, the cross-correlation is defined as:

(2.5)

The cross-correlation is similar in nature to the convolution of two functions.

In an autocorrelation, which is the cross-correlation of a signal with itself, there will always be a peak at a lag of zero, and its size will be the signal power [24].

In probability and statistics, the term cross-correlations is used for referring to the correlations between the entries of two random vectors X and Y, while the autocorrelations of a random vector X are considered to be the correlations between the entries of X itself, those forming the correlation matrix (matrix of correlations) of X. This is analogous to the distinction between autocovariance of a random vector and cross-covariance of two random vectors. One more distinction to point out is that in probability and statistics the definition of correlation always includes a standardising factor in such a way that correlations have values between −1 and +1.

If X and Y are two independent random variables with probability density functions f and g, respectively, then the probability density of the difference X–Y is formally given by the cross-correlation (in the signal-processing sense) f*g{\displaystyle f\star g}, however this terminology is not used in probability and statistics. In contrast, the convolution f*g (equivalent to the cross-correlation of f(t) and g(−t) ) gives the probability density function of the sum X+Y.

Properties:

The cross-correlation of functions f(t) and g(t) is equivalent to the convolution of f*(−t) and g(t). That is:

(2.6)

If f is a Hermitian function, then

If both f and g are Hermitian, then

Analogous to the convolution theorem, the cross-correlation satisfies

(2.7)

where F denotes the Fourier transform, and an asterisk again indicates the complex conjugate. Coupled with fast Fourier transform algorithms, this property is often exploited for the efficient numerical computation of cross-correlations (see circular cross-correlation).

The cross-correlation is related to the spectral density (see Wiener–Khinchin theorem).

The cross-correlation of a convolution of f and h with a function g is the convolution of the cross-correlation of f and g with the kernel h:

(2.8)

Optical Correlator

An optical correlator is a device for comparing two signals by utilising the Fourier transforming properties of a lens. It is commonly used in optics for target tracking and identification [3].

Fig. 2.2. 4f optical correlator

The correlator has an input signal which is multiplied by some filter in the Fourier domain. The matched filter uses the cross correlation of the two signals [19].

The cross correlation or correlation plane, c(x,y) of a 2D signal i(x,y) with h(x,y) is

(2.9)

This can be re-expressed in Fourier space as

(2.10)

where the capital letters denote the Fourier transform of what the lower case letter denotes. So the correlation can then be calculated by inverse Fourier transforming the result [8].

Matched Filter

The matched filter maximizes the signal-to-noise ratio and is simply obtained by using as a filter the Fourier transform of the reference signal r(x,y).

(2.11)

In signal processing, a matched filter is obtained by correlating a known signal, or template, with an unknown signal to detect the presence of the template in the unknown signal. This is equivalent to convolving the unknown signal with a conjugated time-reversed version of the template [17]. The matched filter is the optimal linear filter for maximizing the signal to noise ratio (SNR) in the presence of additive stochastic noise. Matched filters are commonly used in radar, in which a known signal is sent out, and the reflected signal is examined for common elements of the out-going signal. Pulse compression is an example of matched filtering. It is so called because impulse response is matched to input pulse signals. Two-dimensional matched filters are commonly used in image processing, e.g., to improve SNR for X-ray. Matched filtering is a demodulation technique with LTI (linear time invariant) filters to maximize SNR. It was originally also known as a North filter.

The matched filter as a least squares estimator:

Matched filtering can also be interpreted as a least squares estimator for the optimal location and scaling of a given model or template. Once again, let the observed sequence be defined as:

(2.12)

where vk is uncorrelated zero mean noise. The signal sk is assumed to be a scaled and shifted version of a known model sequence fk:

(2.13)

We want to find optimal estimates j* and μ* for the unknown shift j0 and scaling μ0 by minimizing the least squares residual between the observed sequence xk and a "probing sequence" hj-k:

(2.14)

The appropriate hj-k will later turn out to be the matched filter, but is as yet unspecified. Expanding xk and the square within the sum yields:

. (2.15)

The first term in brackets is a constant (since the observed signal is given) and has no influence on the optimal solution. The last term has constant expected value because the noise is uncorrelated and has zero mean. We can therefore drop both terms from the optimization. After reversing the sign, we obtain the equivalent optimization problem:

. (2.16)

Setting the derivative w.r.t. μ to zero gives an analytic solution for μ*:

. (2.17)

Inserting this into our objective function yields a reduced maximization problem for just j*:

. (2.18)

The numerator can be upper-bounded by means of the Cauchy-Schwarz inequality:

. (2.19)

The optimization problem assumes its maximum when equality holds in this expression. According to the properties of the Cauchy-Schwarz inequality, this is only possible when:

. (2.20)

for arbitrary non-zero constants or , and the optimal solution is obtained at j*=j0 as desired. Thus, our "probing sequence" hj-k must be proportional to the signal model fk-j0, and the convenient choice k=1 yields the matched filter

. (2.21)

Note that the filter is the mirrored signal model. This ensures that the operation to be applied in order to find the optimum is indeed the convolution between the observed sequence xk and the matched filter hk. The filtered sequence assumes its maximum at the position where the observed sequence xk best matches (in a least-squares sense) the signal model fk.

Software Description

Аt thе bаsis оf thе prоgrаmming sоftwаrе we tаkе Visual Basic object-oriented programming lаnguаgе. We mаkе а simplе intеrfаcе with 3 butоns, 2 infо bаrs, 2 checkboxes and 1 imagebox. This sоftwаrе usеs uplоаdеd imаgеs frоm thе PC. It usеs thе uniquе аlgоrithm tо detect objects.

Fоr а gооd wоrking, thе input imаgеs nееd tо hаvе the sizе оf the PC screen. If wе intrоducе imаgеs with аnоthеr sizе thе image will not be completely seen.

Testing Results

Аftеr аpplying this sоftwаrе, wе hаvе mаny rеsults. Further, wе will mаkе а cоmpаrisоn аnd will crеаtе а table thаt will present the оbtаined results.

Fig.2.4. The interface of software with the result of object detection

The next test is done with images that were processed with noise.

Fig. 2.5. Processed image with noise 0,01

Fig. 2.6. Processed image with noise 0,02

Fig. 2.7. Processed image with noise 0,03

Fig. 2.8. Processed image with noise 0,04

Fig. 2.9. Processed image with noise 0,05

Fig. 2.10. Processed image with noise 0,1

As we can see, the higher is the noise-ratio, the harder it gets to detect the object required. At noise 0,05, the software cannot find the object.

Fig. 2.11. Objects detection success by noise ratio

In the next Figures, we will be observing the objects detection in rotated example images, with different angles. The direction of rotations matters not.

Fig. 2.12. Processed image with rotation ratio of 5

Fig. 2.13. Processed image with rotation ratio of 10

Fig. 2.14. Processed image with rotation ratio of 20

Fig. 2.15. Processed image with rotation ratio of 25

Fig. 2.16. Processed image with rotation ratio of 50

Fig. 2.17. Processed image with rotation ratio of 75

Fig. 2.18. Objects detection success by rotation ratio

In the following examples we applied resizing to the image:

Fig. 2.19. Processed image with 80% resizing

Fig. 2.20. Processed image with 90% resizing

Fig. 2.21. Processed image at 100%

Fig. 2.22. Processed image with 110% resizing

Fig. 2.23. Processed image with 120% resizing

Fig. 2.24. Objects detection success by rotation ratio

Conclusion on Chapter 2

I have studied and analyzed the existent algorithms and systems. Based on them we have developed a new algorithm and software for objects detection, which was succesefuly tested on several digital images. The results were satisfactory, and were presented above. The system can be used in different area and fields, both military and civilian. In the process of software development we have used Visual Studio 2015 and C# object-oriented programming language.

SОFTWАRЕ USER АND PRОGRАMMER DEVELОPER GUIDЕ

Software User Guide

When we open the software, we will see the following window:

Fig. 3.1. Clear Interface

The button to select the image in which we want to detect an object.

The button to select the template which we want to be detected in the image

The path to the selected image.

The path to the selected template.

The image box, where the image and template will be shown.

The button to perform the object detection

The checkbox to view the keypoints (pixels/group of pixels) by which it was detected

The checkbox to view the matching lines (pixels/group of pixels matched on both image).

First step is adding the desired image:

Fig. 3.2. Selecting the input image

Fig. 3.3. Inputed image with it path

After selecting the desired image, w ego to the next step – selecting the template, in a similar manner to the image:

Fig. 3.4. Selecting the template

Fig. 3.5. Selected template and it path

The selected template is shown in the right side of the image box. After we selected both the image and the template we are ready to perform the object detection. To do this we press the button “Perform Object Detection”. In Figure 3.6. is show the object detection performed on the example above, as a guidline.

Fig. 3.6. OD performed

Also, we can see the keypoints and matching line by using the checkboxes, below will be shown the Figure 3.7 (keypoints) and Figure 3.8 (keypoints and matching lines). To note, the matching lines cannot be viewed without keypoints.

Fig. 3.7. Shown keypoints

Fig. 3.8. Shown matching lines

Programmer Developer Guide

The software was developed in Visual Basic object-oriented programming language.

Visual Basic is a third-generation event-driven programming language and integrated development environment (IDE) from Microsoft for its COM programming model first released in 1991 and declared legacy in 2008. Microsoft intended Visual Basic to be relatively easy to learn and use. Visual Basic was derived from BASIC and enables the rapid application development (RAD) of graphical user interface (GUI) applications, access to databases using Data Access Objects, Remote Data Objects, or ActiveX Data Objects, and creation of ActiveX controls and objects.

Visual Basic is engineered for productively building type-safe and object-oriented applications. Visual Basic enables developers to target Windows, Web, and mobile devices. As with all languages targeting the Microsoft .NET Framework, programs written in Visual Basic benefit from security and language interoperability.

The environment used for software developins was Visual Studio 2015.

Visual Studio 2015 is a rich, integrated development environment for creating stunning applications for Windows, Android, and iOS, as well as modern web applications and cloud services. It advantages are:

Tools and services for projects of any size or complexity

C#, Visual Basic, F#, C++, Python, Node.js, and HTML/JavaScript

Sprint planning

Advanced debugging, automated and manual testing, and profiling

DevOps with automated deployments and continuous monitoring

The Visual Basic namespaces used in process of development:

System.Drawing namespace provides access to GDI+ basic graphics functionality. More advanced functionality is provided in the System.Drawing.Drawing2D, System.Drawing.Imaging, and System.Drawing.Text namespaces.

System.Diagnostics – Provides access to local and remote processes and enables you to start and stop local system processes.

Imports Emgu.CV – is a cross platform .Net wrapper to the OpenCV image processing library. Allowing OpenCV functions to be called from .NET compatible languages such as C#, VB, VC++, IronPython etc. The wrapper can be compiled by Visual Studio, Xamarin Studio and Unity, it can run on Windows, Linux, Mac OS X, iOS, Android and Windows Phone.

Fig. 3.9. The scheme of software for objects detection

The software algorithm is explained in the above figures:

First we add the image and template with the following function:

Fig. 3.10. Image add function

After image and template adding and keypoints determination is executed the correlation function:

Fig. 3.11. Correlation function

Final steps are matching and showing the output image result:

Fig. 3.12. Matching image

Conclusion on Chapter 3

The interface of the software is simple and easy in utilization, with the help of the guide any user could use this application. The software was written in C# object-oriented programming language, with the use of different libraries from it. The software was developed in Visual Studio 2015, which is a rich, integrated development environment for creating applications for Windows, Android, and iOS, as well as modern web applications and cloud services.

GENERAL CONCLUSION AND RECOMMENDATIONS

Objects detection systems are increasingly asked for nowdays. They are widely used in different fields like: urban video surveillance (object detecton and tracking, face/pattern recognition, etc.), army, DAS (drive assistance system), etc. Each method for object detection has it own advantages and disadvantages (Table 1.1), so the mehod is chosen based on the requirements and needs and the field it is used. Optical correlators are used to improve the systems and cover their weaknesses.

In order to achieve the objectives of the project, we have studied a great amount of speciality literature that is mainly on-line due to the fact that the topic of the given paper is very narrow and there are few print sources on objects detection domain.

Despite of this fact we investigated the theoretical perspectives on methods and systems for objects detection. Among the existent methods of object detection we chose the “appearance based” method considering it the most proper for our research. On the basis of this method as well as studying and analyzing other similar algorithms we created an original one for detecting objects in digital images. For this, we developed a software that detects the objects we require in the images we input.

The goal of the project was to create a new algorithm for an object detection system, and it was succeseful achieved, and used to develop the software.

In the process of software developing we have used the Appearance-based Method, along with

The software was tested on several templates and digital images, the results being satisfactory.

As required of us, we have created a user guide, so the software could be understood and used by any user. In addition, we have created the programmer user guide, which describes the programming language and it’s libraries by the modules used.

Such a system can be used as a basis for further development of pedestrian surveillance systems, or for objects detection and tacking in the military field. The system could be improved with the add of a database with detailed templates of the objects required to detect.

BIBLIOGRAPHY

Amit Yali. 2D Object Detection and Recognition. Cambridge: MIT Press, 2002. 324 p.

Bennamoun M., Mamic G. J. Object Recognition: Fundamentals and Case Studies. London: Springer London, 2002. 350 p.

Casasent David P., Anqi Ye, John S. Smokelin, Roland H. Schaefer. Optical correlation filter fusion for object detection. http://opticalengineering.spiedigitallibrary.org/article.aspx?articleid=1073124&resultClick=1 (accesed on 24.04.2016).

Correlation function. https://en.wikipedia.org/wiki/Correlation_function (accessed on 12.05.2016).

Cyganek Boguslaw. Object Detection and Recognition in Digital Images: Theory and Practice. Chichester: Wiley, 2013. 548 p.

Fourier optics. https://en.wikipedia.org/wiki/Fourier_optics (accessed on 13.05.2016).

Fourier transforms. https://en.wikipedia.org/wiki/Fourier_transform (accessed on 13.05.2016).

Harasthy T., J. Turán, L. Ovseník and K. Fazekas. Optical correlator based Traffic Signs Recognition. http://ieeexplore.ieee.org/xpl/login.jsp?tp=&arnumber=6208092&url=http%3A%2F%2Fieeexplore.ieee.org%2Fxpls%2Fabs_all.jsp%3Farnumber%3D6208092 (accessed on 09.05.2016).

Kalirajan K., M. Sudha. Moving Object Detection for Video Surveillance. http://www.ncbi.nlm.nih.gov/pmc/articles/PMC4377503/ (accessed on 01.05.2016).

Lauinger Norbert. Diffractive-optical correlators: chances to make optical image preprocessing as intelligent as human vision. http://proceedings.spiedigitallibrary.org/proceeding.aspx?articleid=852660&resultClick=1 (accessed on 28.04.2016).

Li Qi, Zhihai Xu, Huajun Feng, Peng Ge, Yueting Chen, Lianghui Li. Influence of noise on detection accuracy of joint transform optical correlator. http://proceedings.spiedigitallibrary.org/proceeding.aspx?articleid=728995 (accessed on 11.04.2016).

Li Youzhi and Joseph Rosen. Object recognition using three-dimensional optical quasi-correlation. https://www.osapublishing.org/josaa/viewmedia.cfm?uri=josaa-19-9-1755&seq=0&html=true (accessed on 12.04.2016).

Márquez A., J. Barbé, MI. J. Yzuel, J. Campos. Optical correlator as a tool for Physicists and engineers training in signal processing. https://spie.org/etop/1999/297_1.pdf (accessed on 20.04.2016).

Nazrul Islam Mohammed. Target detection using optical joint transform correlation. https://www.ieee.li/pdf/viewgraphs/target_detection_using_optical_joint_transform.pdf (accessed on 28.03.2016).

Nazrul Islam Mohammed, Mohammad Ataul Karim, Mohammad Showkat Alam, K. Vijayan Asari. Optical cryptographic system employing multiple reference-based joint transform correlation technique. http://opticalengineering.spiedigitallibrary.org/article.aspx?articleid=1158189 (accessed on 14.04.2016).

Object recognition. https://en.wikipedia.org/wiki/Outline_of_object_recognition (accesed on 02.05.2016).

Optical correlator. https://en.wikipedia.org/wiki/Optical_correlator (accessed on 02.05.2016).

Ovsenik Lubos, Ján Turán, Anna Kazimírová Kolesárová. Video surveillance systems with optical correlator. https://www.researchgate.net/publication/221412841_Video_surveillance_systems_with_optical_correlator (accessed on 05.05.2016).

Sheng Yunlong. Object detection by optical correlator and intelligence recognition surveillance systems. http://proceedings.spiedigitallibrary.org/proceeding.aspx?articleid=1746295&resultClick=24 (accessed on 25.03.2016).

Sidike Paheding, Vijayan K. Asari, Mohammad S. Alam. A robust fringe-adjusted joint transform correlator for efficient object detection. http://proceedings.spiedigitallibrary.org/proceeding.aspx?articleid=2277971&resultClick=1 (accessed on 02.04.2016).

Sidike Paheding, Vijayan K. Asari, Mohammad S. Alam. Multiple object detection in hyperspectral imagery using spectral fringe-adjusted joint transform correlator. http://proceedings.spiedigitallibrary.org/proceeding.aspx?articleid=2191082&resultClick=1 (accessed on 20.03.2016).

Suraphol Udomkesmalee. Intelligent Optical Correlator. http://opticalengineering.spiedigitallibrary.org/article.aspx?articleid=1223543&resultClick=1 (accessed on 27.03.2016).

Vijaya Kumar B. V. K. Tutorial survey of composite filter designs for optical correlators. In: Applied Optics, 1992, vol. 31, p. 4773-4801.

Vijaya Kumar B. V. K., M. Savvides and C. Xie. Correlation pattern recognition for face recognition. In: Proc. IEEE, 2006, vol. 94, p. 1963-1976.

Wesley E. Foor, Harold G. Andrews II, Capt, USAF, Mark A. Getbehead. Optical Correlator Optimizations and Extensions for Automated Optical Target Recognition. http://www.dtic.mil/cgi-bin/GetTRDoc?AD=ADA293013 (accessed on 07.04.2016).

ANNEXE: SOURCE CODE

Option Strict

Imports System.Drawing

Imports System.Diagnostics

Imports Emgu.CV

Imports Emgu.CV.CvEnum

Imports Emgu.CV.Features2D

Imports Emgu.CV.Structure

Imports Emgu.CV.UI

Imports Emgu.CV.Util

Public Class frmSURF

Dim blnFirstTimeInResizeEvent As Boolean = True

Dim intOrigFormWidth As Integer

Dim intOrigFormHeight As Integer

Dim intOrigImageBoxWidth As Integer

Dim intOrigImageBoxHeight As Integer

Dim capWebcam As Capture

Dim blnWebcamCapturingInProcess As Boolean = False

Dim imgSceneColor As Image(Of Bgr, Byte) = Nothing

Dim imgToFindColor As Image(Of Bgr, Byte) = Nothing

Dim imgCopyOfImageToFindWithBorder As Image(Of Bgr, Byte) = Nothing

Dim blnImageSceneLoaded As Boolean = False

Dim blnImageToFindLoaded As Boolean = False

Dim imgResult As Image(Of Bgr, Byte) = Nothing

Dim bgrKeyPointsColor As Bgr = New Bgr(Color.Blue)

Dim bgrMatchingLinesColor As Bgr = New Bgr(Color.Green)

Dim bgrFoundImageColor As Bgr = New Bgr(Color.Red)

Dim stopwatch As Stopwatch = New Stopwatch()

Sub New()

InitializeComponent()

intOrigFormWidth = Me.Width

intOrigFormHeight = Me.Height

intOrigImageBoxWidth = ibResult.Width

intOrigImageBoxHeight = ibResult.Height

End Sub

Private Sub frmSURF_Resize( sender As System.Object, e As System.EventArgs) Handles MyBase.Resize

If(blnFirstTimeInResizeEvent = True) Then

blnFirstTimeInResizeEvent = False Else ibResult.Width = Me.Width – (intOrigFormWidth – intOrigImageBoxWidth) ibResult.Height = Me.Height – (intOrigImageBoxHeight – intOrigImageBoxHeight)

End If

End Sub

Private Sub rdoImageFile_CheckedChanged( sender As System.Object, e As System.EventArgs) Handles rdoImageFile.CheckedChanged

If(rdoImageFile.Checked = True) Then If(blnWebcamCapturingInProcess = True) Then RemoveHandler Application.Idle, New EventHandler(AddressOf Me.PerformSURFDetectionAndUpdateGUI) blnWebcamCapturingInProcess = False

End If

imgSceneColor = Nothing

imgToFindColor = Nothing '

imgCopyOfImageToFindWithBorder = Nothing '

imgResult = Nothing '

blnImageSceneLoaded = False '

blnImageToFindLoaded = False '

txtImageScene.Text = ""

txtImageToFind.Text = "" '

ibResult.Image = Nothing '

Me.Text = "OBJECT DETECTION"

btnPerformSURFOrGetImageToTrack.Text = "Perform Object Detection"

ibResult.Image = Nothing

lblImageScene.Visible = True

lblImageToFind.Visible = True '

txtImageScene.Visible = True '

txtImageToFind.Visible = True '

btnImageScene.Visible = True '

btnImageToFind.Visible = True '

End If

End Sub

Private Sub rdoWebcam_CheckedChanged( sender As System.Object, e As System.EventArgs) Handles rdoWebcam.CheckedChanged

If(rdoWebcam.Checked = True) Then

imgSceneColor = Nothing

imgToFindColor = Nothing '

imgCopyOfImageToFindWithBorder = Nothing '

imgResult = Nothing '

blnImageSceneLoaded = False

blnImageToFindLoaded = False

txtImageScene.Text = "" txtImageToFind.Text = ""

ibResult.Image = Nothing

Try

capWebcam = New Capture() Catch ex As Exception Me.Text = ex.Message Return

End Try

Me.Text = "Instructions: hold image to track up to camera, then press ""get image to track"" " btnPerformSURFOrGetImageToTrack.Text = "get image to track"

imgToFindColor = Nothing

AddHandler Application.Idle, New EventHandler(AddressOf Me.PerformSURFDetectionAndUpdateGUI) blnWebcamCapturingInProcess = True

lblImageScene.Visible = False

lblImageToFind.Visible = False

txtImageScene.Visible = False

txtImageToFind.Visible = False

btnImageScene.Visible = False

btnImageToFind.Visible = False

End If

End Sub

Private Sub btnImageScene_Click( sender As System.Object, e As System.EventArgs) Handles btnImageScene.Click

Dim dialogResult As DialogResult = ofdImageScene.ShowDialog()

If(dialogResult = Windows.Forms.DialogResult.OK Or dialogResult = Windows.Forms.DialogResult.Yes) Then

txtImageScene.Text = ofdImageScene.FileName

Else Return

End If

Try

imgSceneColor = New Image(Of Bgr, Byte)(txtImageScene.Text)

Catch ex As Exception

Me.Text = ex.Message Return End Try

blnImageSceneLoaded = True

If(blnImageToFindLoaded = False) Then

ibResult.Image = imgSceneColor

Else

ibResult.Image = imgSceneColor.ConcateHorizontal(imgCopyOfImageToFindWithBorder)

End If

End Sub

Private Sub btnImageToFind_Click( sender As System.Object, e As System.EventArgs) Handles btnImageToFind.Click

Dim dialogResult As DialogResult = ofdImageToFind.ShowDialog() If(dialogResult = Windows.Forms.DialogResult.OK Or dialogResult = Windows.Forms.DialogResult.Yes) Then 'if OK or Yes was chosen

txtImageToFind.Text = ofdImageToFind.FileName Else

Return

End If

Try

imgToFindColor = New Image(Of Bgr, Byte)(txtImageToFind.Text)

Catch ex As Exception

Me.Text = ex.Message Return

End Try

blnImageToFindLoaded = True

imgCopyOfImageToFindWithBorder = imgToFindColor.Copy()

imgCopyOfImageToFindWithBorder.Draw(New Rectangle(1, 1, imgCopyOfImageToFindWithBorder.Width – 3, imgCopyOfImageToFindWithBorder.Height -3), bgrFoundImageColor, 2)

If(blnImageSceneLoaded = True) Then

ibResult.Image = imgSceneColor.ConcateHorizontal(imgCopyOfImageToFindWithBorder)

Else ibResult.Image = imgCopyOfImageToFindWithBorder End If

End Sub

Private Sub txtImageScene_TextChanged( sender As System.Object, e As System.EventArgs) Handles txtImageScene.TextChanged

txtImageScene.SelectionStart = txtImageScene.Text.Length

End Sub

Private Sub txtImageToFind_TextChanged( sender As System.Object, e As System.EventArgs) Handles txtImageToFind.TextChanged

txtImageToFind.SelectionStart = txtImageToFind.Text.Length

End Sub

Private Sub btnPerformSURFOrGetImageToTrack_Click( sender As System.Object, e As System.EventArgs) Handles btnPerformSURFOrGetImageToTrack.Click

If(rdoImageFile.Checked = True) Then

If(txtImageToFind.Text <> String.Empty And txtImageScene.Text <> String.Empty) Then

PerformSURFDetectionAndUpdateGUI(New Object(), New EventArgs())

Else Me.Text = "choose image files first, then choose Perform SURF Detection button"

End If

ElseIf(rdoWebcam.Checked = True) Then

imgToFindColor = imgSceneColor.Resize(320, 240, INTER.CV_INTER_CUBIC, True)

Me.Text = "Instructions: to update image to track, hold image up to camera, then press ""update image to track"""

btnPerformSURFOrGetImageToTrack.Text = "update image to track" Else

End If

End Sub

Private Sub ckDrawKeyPoints_CheckedChanged( sender As System.Object, e As System.EventArgs) Handles ckDrawKeyPoints.CheckedChanged

If(ckDrawKeyPoints.Checked = False) Then

ckDrawMatchingLines.Checked = False

ckDrawMatchingLines.Enabled = False

ElseIf(ckDrawKeyPoints.Checked = True) then

ckDrawMatchingLines.Enabled = True

End If

If(rdoImageFile.Checked = True) Then btnPerformSURFOrGetImageToTrack_Click(New Object(), New EventArgs()) End If

End Sub

Private Sub ckDrawMatchingLines_CheckedChanged( sender As System.Object, e As System.EventArgs) Handles ckDrawMatchingLines.CheckedChanged

If(rdoImageFile.Checked = True) Then btnPerformSURFOrGetImageToTrack_Click(New Object(), New EventArgs())

End If

End Sub

Sub PerformSURFDetectionAndUpdateGUI(sender As Object, arg As EventArgs)

If(rdoImageFile.Checked = True) Then If(blnImageSceneLoaded = False Or blnImageToFindLoaded = False Or imgSceneColor Is Nothing Or imgToFindColor Is Nothing) Then

Me.Text = "either or both images are not loaded or null, please choose both images before performing SURF"

Return

End If

Me.Text = "processing, please wait . . ."

Application.DoEvents()

stopwatch.Restart() ElseIf(rdoWebcam.Checked = True) Then

Try imgSceneColor = capWebcam.QueryFrame()

Catch ex As Exception

Me.Text = ex.Message

Return

End Try

If(imgSceneColor Is Nothing) Then

Me.Text = "error, imgSceneColor Is Nothing"

Return End If

If(imgToFindColor Is Nothing) Then

ibResult.Image = imgSceneColor

Return End If

End If

Dim surfDetector As SURFDetector = New SURFDetector(500, False)

Dim imgSceneGray As Image(Of Gray, Byte) = Nothing

Dim imgToFindGray As Image(Of Gray, Byte) = Nothing

Dim vkpSceneKeyPoints As VectorOfKeyPoint

Dim vkpToFindKeyPoints As VectorOfKeyPoint

Dim mtxSceneDescriptors As Matrix(Of Single)

Dim mtxToFindDescriptors As Matrix(Of Single)

Dim mtxMatchIndices As Matrix(Of Integer)

Dim mtxDistance As Matrix(Of Single)

Dim mtxMask As Matrix(Of Byte)

Dim bruteForceMatcher As BruteForceMatcher(Of Single)

Dim homographyMatrix As HomographyMatrix = Nothing

Dim intKNumNearestNeighbors As Integer = 2

Dim dblUniquenessThreshold As Double = 0.8

Dim intNumNonZeroElements As Integer

Dim dblScaleIncrement As Double = 1.5

Dim intRotationBins As Integer = 20

Dim dblRansacReprojectionThreshold As Double = 2.0

Dim rectImageToFind As Rectangle

Dim ptfPointsF As PointF()

Dim ptPoints As Point()

imgSceneGray = imgSceneColor.Convert(Of Gray, Byte)()

imgToFindGray = imgToFindColor.Convert(Of Gray, Byte)()

vkpSceneKeyPoints = surfDetector.DetectKeyPointsRaw(imgSceneGray, Nothing) mtxSceneDescriptors = surfDetector.ComputeDescriptorsRaw(imgSceneGray, Nothing, vkpSceneKeyPoints)

vkpToFindKeyPoints = surfDetector.DetectKeyPointsRaw(imgToFindGray, Nothing) mtxToFindDescriptors = surfDetector.ComputeDescriptorsRaw(imgToFindGray, Nothing, vkpToFindKeyPoints)

bruteForceMatcher = New BruteForceMatcher(Of Single)(DistanceType.L2)

bruteForceMatcher.Add(mtxToFindDescriptors)

mtxMatchIndices = New Matrix(Of Integer)(mtxSceneDescriptors.Rows, intKNumNearestNeighbors)

mtxDistance = New Matrix(Of Single)(mtxSceneDescriptors.Rows, intKNumNearestNeighbors)

bruteForceMatcher.KnnMatch(mtxSceneDescriptors, mtxMatchIndices, mtxDistance, intKNumNearestNeighbors, Nothing)

mtxMask = New Matrix(Of Byte)(mtxDistance.Rows, 1)

mtxMask.SetValue(255)

Features2DToolbox.VoteForUniqueness(mtxDistance, dblUniquenessThreshold, mtxMask)

intNumNonZeroElements = CvInvoke.cvCountNonZero(mtxMask)

If(intNumNonZeroElements >= 4) Then intNumNonZeroElements = Features2DToolbox.VoteForSizeAndOrientation(vkpToFindKeyPoints, vkpSceneKeyPoints, mtxMatchIndices, mtxMask, dblScaleIncrement, intRotationBins)

If(intNumNonZeroElements >= 4) Then

homographyMatrix = Features2DToolbox.GetHomographyMatrixFromMatchedFeatures(vkpToFindKeyPoints, vkpSceneKeyPoints, mtxMatchIndices, mtxMask, dblRansacReprojectionThreshold)

End If

End If

imgCopyOfImageToFindWithBorder = imgToFindColor.Copy()

imgCopyOfImageToFindWithBorder.Draw(New Rectangle(1, 1, imgCopyOfImageToFindWithBorder.Width – 3, imgCopyOfImageToFindWithBorder.Height – 3), bgrFoundImageColor, 2)

If(ckDrawKeyPoints.Checked = True And ckDrawMatchingLines.Checked = True) Then

imgResult = Features2DToolbox.DrawMatches(imgCopyOfImageToFindWithBorder, _

vkpToFindKeyPoints, _

imgSceneColor, _

vkpSceneKeyPoints, _

mtxMatchIndices, _

bgrMatchingLinesColor, _

bgrKeyPointsColor, _

mtxMask, _

Features2DToolbox.KeypointDrawType.DEFAULT)

ElseIf(ckDrawKeyPoints.Checked = True And ckDrawMatchingLines.Checked = False) Then

imgResult = Features2DToolbox.DrawKeypoints(imgSceneColor, vkpSceneKeyPoints, bgrKeyPointsColor, Features2DToolbox.KeypointDrawType.DEFAULT)

imgCopyOfImageToFindWithBorder = Features2DToolbox.DrawKeypoints(imgCopyOfImageToFindWithBorder, vkpToFindKeyPoints, bgrKeyPointsColor, Features2DToolbox.KeypointDrawType.DEFAULT)

imgResult = imgResult.ConcateHorizontal(imgCopyOfImageToFindWithBorder)

ElseIf(ckDrawKeyPoints.Checked = False And ckDrawMatchingLines.Checked = False) Then

imgResult = imgSceneColor

imgResult = imgResult.ConcateHorizontal(imgCopyOfImageToFindWithBorder)

Else

End If

If(homographyMatrix IsNot Nothing) Then

rectImageToFind.X = 0 rectImageToFind.Y = 0

rectImageToFind.Width = imgToFindGray.Width

rectImageToFind.Height = imgToFindGray.Height

ptfPointsF = New PointF() { New PointF(rectImageToFind.Left, rectImageToFind.Top), _

New PointF(rectImageToFind.Right, rectImageToFind.Top), _

New PointF(rectImageToFind.Right, rectImageToFind.Bottom), _

New PointF(rectImageToFind.Left, rectImageToFind.Bottom) }

homographyMatrix.ProjectPoints(ptfPointsF)

ptPoints = New Point() { Point.Round(ptfPointsF(0)), _

Point.Round(ptfPointsF(1)), _

Point.Round(ptfPointsF(2)), _

Point.Round(ptfPointsF(3)) }

imgResult.DrawPolyline(ptPoints, True, bgrFoundImageColor, 2)

End If

ibResult.Image = imgResult

If(rdoImageFile.Checked = True) Then

stopwatch.Stop() Me.Text = "processing time = " + stopwatch.Elapsed.TotalSeconds.ToString() + " sec, done processing, choose another image if desired"

End If

End Sub

End Class

STATEMENT REGARDING THE ORIGINALITY OF THE CONTENTS OF THE LICENSE PROJECT

I, the undersigned ………………………………………….…, in my capacity of graduate of Free International University of Moldova, Faculty of ………………………….…………………………………………………………………………., specialty…………………………………………, year of graduation ……………, declare on my own responsibility that the License Project entitled:

……………..………………………………………………………………………..………………………………………………………………………………………………………………………………………..….….….….….….…………………………………………………………………..,

which has been elaborated under the supervision of Mr. /Mrs. …………………………………………………………………………… and is to be presented to a committee, is original and I am the sole author.

I hereby declare that I have not plagiarized any License Project, monographs, specialized works, articles etc. published or posted on Internet and all the references used in the process of the License Project writing are mentioned in its Contents.

I declare that I agree to my Project License to be checked in any legal ways to assess its originality, thereof its content being introduced in a database for such purpose.

Date …………………… Student’s signature …………………

CURRICULUM VITAE

Personal information: Bordian Vlad Pavel

3700-MD, Straseni city, str. M. Sadoveanu 37/57,

Tеl.: (+373)23724765

Mob.: (+373)60069553

Е-mаil: bordianvlad@gmail.com

Dаte of birth: 29 july 1993

Nаtionаlity: MDA

Sеx: Male

Cеtățеniа: RM

Education:

2014-2016 Free International Univеrsity of Moldova

Fаculty: Informаtics, Engineering, Dеsign

Spеciаlity: Computers

2012-2014 Techical University of Moldova

Fаculty: Computers, Informatics, Microelectronics

Spеciаlitаtе: Computers

2010-2011 Cisco Networking Academy, “IT Essential” course.

Theoretical Lyceum “Ion Vatamanu”, Straseni city

2009-2012 Theoretical Lyceum “Ion Vatamanu”, Straseni city

Foreign Languages Knowing: Romanian: mother tongue

Russian: proficient user, C1

Еnglish: independent user, B2

Аbilities: Driver license: B

communicable, responsible, organized, mеticulous,

hard-working, fast learner.

TECHNICAL CONTROL DATA OF LICENSE PROJECT