In the last decades w e can observ e a con tin uous gro wth of the remote sensing data, [615320]
Chapter 1
Introduction
In the last decades w e can observ e a con tin uous gro wth of the remote sensing data,
acquired with a large v ariet y of sensors, b oth m ultisp ectral (MS) and syn thetic ap erture
radar (SAR), in dieren t acquisition mo des. This has lead to large collections of Earth
Observ ation (EO) image data that m ust b e understo o d and analyzed.
Multisp ectral Earth Observ ation images, from lo w to v ery high resolution are no w
a v ailable from a large v ariet y of sensors. Also, the last generation m ultisp ectral satellite
systems are oering image pro ducts that co v ers a wide area of the electromagnetic (EM)
sp ectrum. On the other hand, in the last t w o decades the SAR imagery has b ecome
increasingly p opular as some of its prop erties are fa v orable and complemen tary to optical
imagery .
It is kno wn that along with appropriate metho ds able to w ork with great amoun ts
of data, information retriev al pro cesses requires algorithms to cop e with a range of input
imagery . Ev en if the geometry and radiometry of SAR images are totally dieren t than
m ultisp ectral data, there are attempts of nding a common ground suc h that optical
image indexing algorithms can b e applied for SAR data and vice v ersa [23]. Moreo v er,
new concepts m ust b e de ned in order to obtain satisfying results, enabling measuremen ts
and comparisons b et w een the extracted features.
Regarding this idea, our goal is to dev elop an application framew ork capable to join
feature extraction metho ds and classi cation algorithms for a wide range of remote sensing
images, acquired from b oth MS and SAR satellite sensors. Its success will sustain the
in tegration of a reliable EO data searc h engine in to an op erativ e data mining system.
1.1 Motiv ation
Ha ving in to accoun t that b oth the satellite sensors and remote sensing imagery are
ev olving so fast, the metho ds and tec hniques of pro cessing and analyzing EO data usually
are sta ying one step b ehind. One w a y to analyze these data is to in tegrate the mo del
kno wledge obtained from collections of pre-recorded ph ysical target data. This leads to a
comparison of the acquired data with represen tativ e mo dels. Similarities and deviations
rev ealed during the comparison allo w a detailed high resolution in terpretation of the image
data and lead to a full image understanding. On the other hand, giv en the large collections
of EO data and the necessit y of the users to get certain information as quic kly as p ossible
1
Chapter 1. Intr o duction
ha v e led to the need of ha ving automated indexing systems. Suc h systems can increase
the pro ductivit y and it can b e of vital imp ortance in critical infrastructures monitoring.
In Fig. 1.1 is illustrated a comparison b et w een thematic classes that can b e automati-
cally extracted from remotely sensed data. In the top side of the gure are presen ted some
samples from a W orldView-2 m ultisp ectral image, ha ving t w o meters spatial resolution,
while in the b ottom of the gure can b e observ ed their corresp ondence in a T erraSAR-X
image at one meter spatial resolution.
Figure 1.1: A few generic classes observ ed in m ultisp ectral and syn thetic ap erture radar images
In the frame of image understanding, metho ds and tec hniques for feature extraction
and automatic image classi cation and indexing ha v e b een prop osed. Unfortunately , no
general information retriev al pro cedure has b een dev elop ed to e cien tly p erform, regard-
less of the analyzed data. Best results require sp eci c algorithms for sp eci c t yp es of
data. Most of the high resolution image indexing metho ds are based on iden ti cation and
classi cation of image texture, image in tensit y or b y using statistical mo dels. The results
are then group ed in a few generic classes (3-6) lik e crops, buildings, streets, v egetation,
forest etc.
1.2 Goals
The thesis aims at the theoretical study and elab oration of adv anced metho ds for
m ultisp ectral (MS) and syn thetic ap erture radar (SAR) image understanding based on
2
Metho ds and algorithms for Earth Observation image information mining
in teractiv e and automatic algorithms. The metho ds are in tended for the analysis and
image con ten t mining of m ultisp ectral images and detected SAR images, ha ving the goal
to dev elop a framew ork for feature extraction and automatic classi cation and indexing
of remotely sensed images, that can b e in tegrated in to a data mining system.
The main directions follo w ed in this thesis are:
1. T o dev elop a framew ork for feature extraction metho ds that can b e applied on b oth
m ultisp ectral and SAR images, taking in to accoun t particularities of eac h t yp e of
image, from data acquisition to data understanding.
2. T o analyze and adapt state of the art feature extraction metho ds that can b e in te-
grated in to a data mining system.
3. T o dev elop new feature extraction metho ds able to extract the maxim um infor-
mation needed in the classi cation pro cess and automatic image annotation. Also
follo wing the idea of feature extraction standardization our in terest is fo cused on
dev eloping new image descriptors that can b e in tegrated within a standard.
4. T o dev elop a data mining system that can b e used in kno wledge extraction from
m ultisp ectral and SAR images, based on the prop osed feature extraction and clas-
si cation framew ork. F urthermore, the dev elop ed system will in tegrate to ols that
allo ws the user a qualitativ e and quan titativ e analysis of the results.
1.3 Con tributions of the thesis
T o ac hiev e these goals, in terdisciplinary kno wledge w ere applied. Information ex-
traction from remote sensing images requires a deep understanding on ho w the recording
platform w orks, what ranges of the electromagnetic sp ectrum are used and ho w the signal
is pro cessed to create the image data. Th us, domains lik e remote sensing, signal pro-
cessing, information theory , estimation theory and mac hine learning w ere approac hed in
order to ac hiev e our goal of information extraction from Earth observ ation image data. In
the rst steps of our framew ork of kno wledge mining, feature extraction metho ds are ap-
plied to extract image descriptors that highligh t sp eci c color/sp ectral, texture or shap e
features. This w a y , using information and estimation theory concepts w e obtain a math-
ematical mo del of the image con ten t, that is analyzed b y the means of mac hine learning
algorithms.
The pro cess of information extraction con tin ues with the quest of disco v ering patterns
and applying mac hine learning algorithms on the extracted features. In Fig. 1.2 w e
presen t ho w the concept of information extraction, whic h is the main topic of this thesis
is in teracting with other disciplines. There is a strong relationship b et w een the main topic
of the thesis and other theoretical domains. When tak en separately , the wisdom of eac h
discipline can pro vide v aluable p ersp ectiv es of the data b eing analyzed, but when tak en
together, eac h of the disciplines acts as a part of system, completing eac h other.
The actual con tributions of this thesis are follo wing the ob jectiv es presen ted in the
previous section and can b e summarized as follo ws:
3
Chapter 1. Intr o duction
Figure 1.2: Thesis concept and applications
1. In the actual con text of remote sensing data analysis and understanding, where
enormous amoun ts of data need to b e pro cessed, w e prop ose a framew ork for feature
extraction and automatic classi cation and indexing for b oth m ultisp ectral and SAR
images. The concept is presen ted in Fig. 1.3.
In order to ac hiev e our goal, in the rst steps, the remote sensing data b eing an-
alyzed is read and separated in m ultisp ectral or SAR images. F urthermore, in the
prepro cessing step, subsets of sp eci ed size are extracted from the data, b eing pre-
pared for the feature extraction. When feature extraction is completed, the results
are sa v ed in a database. In the next step, feature classi cation is p erformed using
unsup ervised or sup ervised classi cation metho ds. As nal step, the classi cation
results are sa v ed and prepared for visualization.
2. Based on a deep understanding of the remote sensing data that last generation
m ultisp ectral and SAR satellite sensors pro vides, w e tested and adapted state of
the art feature extraction metho ds that can b e used to extract maxim um informa-
tion from EO images. Ev en though our assessmen t is fo cused on a patc h based
approac h, w e also p erformed pixel lev el analysis to ac hiev e a deep understanding of
the relationship b et w een pixel distributions.
3. As a consequence of the fact that ob jects are represen ted using sev eral sp ectral
bands that equally in uence the classi cation pro cess, w e prop ose new patc h-based
4
Metho ds and algorithms for Earth Observation image information mining
Figure 1.3: F eature Extraction framew ork
approac hes of feature extraction from m ultisp ectral data. Also, ha ving the goal to
nd a common ground b et w een m ultisp ectral and syn thetic ap erture radar image
analysis, some of the metho ds w e prop ose can b e successfully used in SAR image
information extraction.
Using b oth the ra w texture data and the high sp ectral resolution pro vided b y the
latest satellite sensors, w e prop ose enhanced image descriptors based on Gab or,
sp ectral histograms, sp ectral indices, bag-of-w ords framew ork and p olar co ordinates
5
Chapter 1. Intr o duction
computation. Our approac h leads to a scene classi cation that outp erforms the re-
sults obtained when emplo ying the initial image features. Also, b eing highly moti-
v ated to dev elop patc h-based image descriptors, w e prop ose a fast BRIEF approac h
that can b e used in remote sensing image classi cation, instead of the initial use of
image matc hing.
4. T o assess our metho ds of feature extraction w e created test databases for b oth m ul-
tisp ectral and syn thetic ap erture radar that can b e used for qualit y and quan tit y
ev aluation. In the case of m ultisp ectral images, the test database w as dev elop ed
taking in to accoun t all the a v ailable sp ectral bands, while for SAR images w e con-
sidered the magnitude of the image whic h w as compared with an optical image.
5. The progress w e realized in the con text of feature extraction and classi cation of
remotely sensed data, w as in tegrated in to a data mining application, that can b e
used for thematic map generation, patc h-based image classi cation and indexing
of m ultisp ectral and SAR images and also to assess and visualize the results. In
the data mining to ol w e dev elop ed, the analyzed images can b e observ ed using a
geographical information system in terface whic h allo ws to o v erla y m ultiple la y ers
and visualize the results of the classi cation.
6. The c onc epts that ar e pr esente d within this thesis have b e en inte gr ate d into data
mining applic ations that c an b e use d by other r ese ar chers.
1.4 Outline of the thesis
In Chapter 2 are presen ted general principles of Remote Sensing, ha ving the goal
to pro vide a b etter understanding on ho w image acquisition is realized and what kind
of information is enco ded in Earth Observ ation images acquired with dieren t imaging
sensors. Also, the in terest is to rev eal some of the particularities of remotely sensed
images, concerning the ranges of electromagnetic sp ectrum used in EO data, sources of
errors and image resolutions. A closer lo ok to these in trinsic prop erties of remotely sensed
images, can pro vide a b etter understanding of the analyzed data.
Chapter 3 ha v e the purp ose to pro vide an insigh t on the lifecycle of remotely sensed
image data, from image acquisition to data understanding and in terpretation. In order to
do so, w e consider EO images as random pro cesses, and mo del their con ten t according to
information and estimation theory . F urthermore, ha ving a mathematical represen tation
of the image data mo del, w e apply image pro cessing tec hniques and extract features of
in terest suc h as sp ectral, texture and shap e information. F urthermore w e apply mac hine
learning algorithms in order to giv e seman tic meaning to the analyzed data.
In Chapter 4, the in terest is to presen t feature extraction metho ds and algorithms w e
dev elop ed in the frame of Earth Observ ation data analysis and understanding. Metho ds
for sp ectral and texture analysis are prop osed along with bag of w ords and feature p oin t
descriptors. Moreo v er, in this c hapter w e presen t our progress in adapting and creating
new feature extraction metho ds, and also w e presen t some case study applications that
can b e used in EO data understanding scenarios.
In Chapter 5 w e fo cused on the actual status of the data mining systems used in EO
data analysis and understanding. Concepts regarding h uman mac hine comm unication are
6
Metho ds and algorithms for Earth Observation image information mining
de ned, ha ving the goal to optimize user in teraction in data mining systems. Moreo v er,
in this c hapter w e presen t the Data Mining T o ol soft w are w e dev elop ed based on the
framew ork prop osed in this thesis and w e presen t some of the data mining to ols and
protot yp es that emerged from our researc h.
Finally , Chapter 6 dra ws the conclusions regarding the metho ds and algorithms w e
prop osed, pro viding an analysis of our study .
7
Chapter 2
Remote Sensing principles
In this c hapter, w e will presen t basic kno wledge of remote sensing that will allo w a
b etter understanding of the data b eing analyzed. F urthermore, w e will pro vide relev an t
information ab out the relationship b et w een the acquisition mo des and data represen tation.
2.1 General understanding
Ev en though m ultiple de nitions are a v ailable, according to [85], remote sensing is
expressed as the measuremen t of ob ject prop erties on the earth's surface using data ac-
quired from aircrafts and satellites, b eing an attempt to measure something at a distance,
rather than in situ. Usually , the goal of these measuremen ts is to create an image of the
observ ed area or phenomenon, but sometimes an adv anced analysis is required.
In Fig. 2.1 is presen ted a simple sc hema in whic h emitted or re ected energy from the
earth's surface, dra wn with red lines, is recorded b y an aerial or satellite remote sensing
platform, using sp ecialized sensors and then is transmitted to a ground reception station
for storage and pro cessing. A t this p oin t the recorded signal is transformed in to a sp eci c
image data format, according to the sensor used in the acquisition pro cess. F urthermore,
this data can b e used in dieren t applications suc h as en vironmen tal assessmen t and
monitoring, c hange detection, agriculture, meteorology , mapping, military surv eillance
and critical infrastructure monitoring and ev en in news and media trough illustrations
and analysis.
Figure 2.1: Remote sensing image formation and analysis
8
Metho ds and algorithms for Earth Observation image information mining
The images obtained trough remote sensing means can b e v ery similar with what
w e see when ying with an aeroplane, although the w a v elengths used in remote sensing
are often outside the range of h uman vision as it is explained in [81]. F urthermore, to
obtain a remote sensing measuremen t w e need an energy source, in the form of electro-
magnetic radiation, either to illuminate the scene or to b e emitted b y the in terest target.
Dep ending on the radiation source, t yp e of the sensor, and acquisition mo de, the recorded
measuremen ts will ha v e sp eci c prop erties and require sp eci c pro cessing tec hniques.
In Fig. 2.2 w e presen t the electromagnetic radiation as a w a v e comp osed b y an
electrical eld ( E ) and an electromagnetic eld ( M ) that propagates with the sp eed of
ligh t (c ). F urthermore, these elds v aries in magnitude in a direction p erp endicular to the
direction in whic h the radiation is tra v eling, b eing c haracterized b y a w a v elength and a
frequency . Those t w o c haracteristics are imp ortan t for a b etter understanding of remote
sensing and are b ounded together b y the form ula c=f , in whic hc is the sp eed of ligh t,
is the w a v elength and f is the frequency of the radiation. Therefore, w e can observ e
that are in v ersely related with eac h other. The shorter the w a v elength, the higher the
frequency .
Figure 2.2: Electromagnetic radiation
Ev en though electromagnetic (EM) sp ectrum ranges from shorter w a v elengths (gamma
and X-ra ys) to longer w a v elengths (micro w a v es and radio w a v es), only sev eral p ortions of
the electromagnetic sp ectrum are usefull in remote sensing. In Fig. 2.3 can b e observ ed
the dep endency b et w een frequency and w a v elength of the EM sp ectrum.
Figure 2.3: Electromagnetic sp ectrum
Dep ending on the energy source used in image formation pro cess, the remote sensing
systems can b e group ed in to passiv e and activ e sensing systems. In the case of pas-
siv e sensing systems, the recorded energy can b e re ected sunligh t or thermal radiation
emitted b y the earth's surface, while in the case of activ e sensing systems, the energy
detected can b e scattered from the earth as the result of illumination with an arti cial
energy source suc h as laser or radar carried on the platform [81]. A comparison b et w een
the image pro ducts obtained with passiv e and activ e sensors can b e seen in Fig. 2.4,
where t w o images co v ering the same area, acquired with the last generation m ultisp ectral
(MS) (Sen tinel-2) and Syn thetic Ap erture Radar (SAR) (Sen tinel-1) satellite sensors are
assessed. Ev en though w e can nd similarities b et w een the images presen ted in Fig. 2.4,
the measured radiation is in the visible range of the sp ectrum for the MS image, and in
the micro w a v e range for the SAR image.
9
Chapter 2. R emote Sensing principles
Figure 2.4: Sen tinel-2 m ultisp ectral image (Left) vs Sen tinel-1 SAR image (Righ t)
2.2 Remote sensing instrumen ts
Imaging in remote sensing can b e carried out using platforms situated on the ground,
on an aircraft or balo on or on a spacecraft or satellite, outside the Earth's athmosphere.
In remote sensing, a platform is referring to the structures or v ehicles on whic h a sensor is
installed. Th us, w e ma y nd sp eci c c haracteristics of the sensors dep ending on the plat-
form they are installed [81]. Of high imp ortance is also the range of the electromagnetic
sp ectrum that the sensors can sense. If w e classify the remote sensing instrumen ts ac-
cording to the sensor prop erties, w e can distinguish passiv e and activ e sensors. When the
criteria of selection is c hanged to the platform that houses the sensor, the main categories
of systems are group ed in ground sensors, airb orne sensors and space-b orne sensors.
Using ground-based sensors can b e recorded detailed information ab out the Earth's
surface. Usually the data obtained from ground platforms is compared with other infor-
mation collected from aerial and satellite sensors, b eing used for a b etter understanding
of the observ ed phenomenon or surface.
The sensors installed on aerial and satellite platforms ha v e ob vious adv an tages o v er
the ground-based sensors, b eing able to capture v ery detailed images that co v er large
surfaces whic h sometimes are unaccessible. Usually , aerial platforms are using stable wing
aircrafts, although helicopters and unmanned air v ehicles (UA V) are used. In the case
of satellite platforms, remote sensing is conducted from space sh uttles or from satellites.
By de nition, satellites are ob jects that rev olv e around another ob ject, th us in remote
sensing, satellites are man-made ob jects that rev olv e around the Earth, allo wing rep etitiv e
co v erage of its surface. Dep ending on the p erformed measuremen ts, in the case of aerial
and satellite imaging platforms, limitations suc h as atmospheric conditions and da y or
nigh t a v ailabilit y are o v ercome b y thermal and syn thetic ap erture radar sensors. In most
of the cases, the platform used is imp osing some c haracteristics of the sensor, lik e w eigh t,
size, p o w er supply , t yp e of detector, igh t altitude etc.
In Fig. 2.5 are presen ted the main categories of airb orne and space-b orne remote
sensing platforms, based on the platform igh t altitude. W e can distinguish this w a y
ultraligh t and lo w altitude airb orne sensors that are represen ted b y UA V s, ha ving the
igh t heigh t b et w een 100m and3km , follo w ed b y high altitude airb orne sensors ( 3km to
10km ) installed on aircrafts and spaceb orne sensors installed on space-sh uttles or satellites
that orbit the earth in near-p olar orbits ranging from 600km to1000km or are placed on
a geostationary orbit at 36000km .
10
Metho ds and algorithms for Earth Observation image information mining
Figure 2.5: Remote sensing platforms
Remote sensing measuremen ts are done in all the ranges of electromagnetic sp ectrum,
but most of them are made in the visible domain, infrared and micro w a v e (radar). A large
v ariet y of sensors for b oth m ultisp ectral and syn thetic ap erture radar data acquisition ha v e
b een dev elop ed in the last decades and can observ e the earth's surface in a wide sp ectrum
range at dieren t spatial resolutions ranging from a few cen timeters to tens of meters.
A ccording to the t yp e of sensors used in image formation, w e can group the remote sensing
platforms in to passiv e sensing platforms, whic h captures the radiation re ected b y earth's
surface and activ e sensing platforms, whic h are using their o wn pulses of electromagnetic
radiation as it is describ ed in [80].
The sensors used in remote sensing can b e non-imaging or imaging. In the frame of
non-imaging sensors, measured radiation is receiv ed from all p oin ts of the sensed target
and con v erted to an electrical signal strength or other quan titativ e attribute, lik e radiance.
In the case of imaging sensors, the measured energy will b e con v erted in to an image or
a raster data. F urthermore, the sensors can b e in tegrated in to non-scanning or scanning
systems. F or non-scanning systems, the scene or target is sensed for a v ery brief momen t
lik e in a photographic camera, while in scanning systems is implied a motion across the
scene o v er a time in terv al. An example of a scanning system that is used to collect data
o v er a v ariet y of sensors is called a m ultisp ectral scanner, b eing one of the most commonly
used scanning system.
11
Chapter 2. R emote Sensing principles
The common sources of errors in remote sensing:
geometric distortions – all remote sensing images are aected b y this t yp e of errors.
The main causes for geometric distortions are:
– the p ersp ectiv e of the sensor optics
– the motion and orien tation of the scanning system
– the stabilit y of the platform
– the platform altitude, attitude and v elo cit y
– the terrain relief
– the curv ature and rotation of the Earth
radiometric distortions – is a consequence of the fact that the measured energy is
not the same with the energy emitted or re ected b y an ob ject. The main causes of
radiometric distortions are:
– sensor sensitivit y
– atmosphere
– Sun angle and top ograph y
In T able 2.1 are presen ted the main t yp es of radiation used in remote sensing, with
their w a v elength ranges, the source of radiation and the surface prop ert y of in terest that
is measured b y the sensor.
T able 2.1: Sp ectral regions used b y v arious remote sensing platforms.
Radiation t yp e W a v elength range Radiation sourceSurface prop ert y
of in terest
Ultr aviolet (UV) 100-400 nm solar re ectance
Visible (V) 400-700 nm solar re ectance
Ne ar Infr aR e d (NIR) 700-1100 nm solar re ectance
Short W ave Infr ar e d
(SWIR)1100-1350 nm
1400-1800 nm
2000-2500 nmsolar, thermal re ectance
MidW ave Infr aR e d
(MWIR)3000-4000 nm
4500-5000 nmthermalre ectance,
temp erature
Thermal or L ongW ave
Infr aR e d (TIR or L WIR)8000-9500 nm
10000-14000 nmthermal temp erature
Micr owave, R adar 1mm – 1mthermal (p assive),
arti cial (active)temp erature (passiv e)
roughness (activ e)
2.3 P asiv e sensing
The main source of radiation in passiv e remote sensing is the Sun, whic h illuminates
the Earth's surface and atmosphere. In this case, part of the radiation recorded b y a sensor
has b een re ected at the earth's surface and part has b een scattered b y the atmosphere,
without ev er reac hing the Earth. In other cases, lik e in the thermal infrared sensing, the
sensor is recording the thermal radiation emitted directly b y the materials on the Earth's
surface whic h is com bining with the thermal radiation emitted b y the Earth atmosphere.
12
Metho ds and algorithms for Earth Observation image information mining
The basic principle of passiv e sensing image acquisition is illustrated in Fig. 2.6. In
the left side of Fig. 2.6 w e can observ e ho w the energy from the Sun is re ected b y the
clouds or b y the Earth's surface, and recorded b y an optical or m ultisp ectral sensor as
re ectance. Also, in the righ t side of Fig. 2.6 is sho wn the thermal imaging principle, in
whic h the thermal energy emitted b y the ob jects is recorded b y the sensor.
Figure 2.6: P asiv e sensing principle
In Fig. 2.7 are illustrated the results of a m ultisp ectral image acquisition, where is
sho wn a scene acquired with the new Sen tinel-2 satellite from Decem b er 23rd, 2015 and
illustrates an area o v er Buc harest, Romania. In the left side of Fig. 2.7 can b e observ ed
the red-green-blue band com binations, while in the righ t side is presen ted the thermal
infrared band, in whic h the areas colored with blue are represen ting cold surfaces while
the areas colored with red are represen ting w arm surfaces.
Figure 2.7: Sen tinel-2 m ultisp ectral image. Left: Red-Green-Blue bands. Righ t: Thermal IR band
In remote sensing, a common w a y of classifying optical and thermal imaging sensors
is done according to the n um b er of sp ectral bands or the size of the pixel, lik e it is
presen ted in T able 2.2. P assiv e sensors can b e group ed in panc hromatic, m ultisp ectral,
sup ersp ectral, and h yp ersp ectral sensors.
T able 2.2: Classi cation of optical/thermal imaging sensors
P assiv e sensor Sp ectral Bands Spatial resolution
Panchr omatic 1 0.31 meters or mor e
Multisp e ctr al 3 or more 0.30 meters or mor e
Sup ersp e ctr al tens of sp ectral bands mor e than 1 meter
Hip ersp e ctr al h undreds of sp ectral bands mor e than 30 meters
13
Chapter 2. R emote Sensing principles
2.3.1 Image acquisition
No matter the tec hnology used in the optical image acquisition pro cess, w e can iden-
tify as common comp onen ts the optical system (lens, mirrors, ap ertures, mo dulators and
disp ersion devices, detectors), detectors (electronic devices able to measure sp eci c w a v e-
lengths and con v ert them in to electrical signals) and signal pro cessor (p erform sp eci c
op erations on the measured signal to pro vide desired output data). Dep ending on the
p erformances of eac h comp onen t, can b e obtained images with dieren t tec hnical sp eci-
cations, the spatial, radiometric, sp ectral and temp oral resolutions b eing directly in u-
enced.
The tec hniques used in remote sensing acquisitions are v ery imp ortan t, allo wing
measuremen ts that lead to b oth image and non-image pro ducts. A classi cation of passiv e
sensors divided in scanning and non-scanning systems can b e observ ed in T able 2.3. In our
researc h, w e fo cus on image pro ducts, ha ving the goal to extract kno wledge from remote
sensed images. Th us, dep ending on the t yp e of acquisition, geometrical and radiometric
errors ma y o ccur in the image formation pro cess, that ha v e to b e iden ti ed and corrected.
T able 2.3: P assiv e sensing non-scanning and scanning systems
Non-Sc anning Sc anning – Imaging
Non-Imaging Imaging (camera) Image plane scanning Ob ject plane scanning
Micro w a v e Radiometer Mono c hrome TV Camera Optical Mec hanical Scanner
Magnetic Sensor Natural Color Solid Scanner Micro w a v e Radiometer
Gra vimeter Infrared
F ourier Sp ectrometer Color Infrared
etc etc.
In Fig. 2.8 is presen ted ho w the image acqusition is made in the case of non-scanning
instrumen ts. The most common non-scanning instrumen ts used in remote sensing are
phothogrammetric cameras, b oth analogic and digital. Usually this instrumen ts are in-
stalled on aerial platforms.
Figure 2.8: Scanning image acquisition
14
Metho ds and algorithms for Earth Observation image information mining
F urthermore, in Fig. 2.9 can b e observ ed ho w image is formed in the case of scanning
instrumen ts. The most common scanning instrumen ts are the m ultisp ectral scanner and
push bro om, whiskbro om and h yp ersp ectral linear arra ys.
Figure 2.9: Non-Scanning image acquisition
2.3.2 Resolution
The resolution of an image is a prop ert y of the sensor used in image acquisition, but
in most of the cases it is asso ciated with the image qualit y . If the resolution of an image
is higher, the image is more clear, also it b ecomes more sharp, more de ned and ha v e
more details. Th us, in terms of remote sensing imagery , the resolution can b e extended
to spatial, sp ectral, radiometric and temp oral resolutions.
Spatial resolution
Considering the case of passiv e sensing images, the spatial resolution is referring to
the observ able details in the image, b eing fo cused on determining the smallest p ossible
feature that can b e detected. Also, spatial resolution is dep enden t on the instan taneous
eld of view (IF O V) of the platform, whic h is the angle trough whic h a detector is sensitiv e
to radiation. Th us, a lo w altitude imaging instrumen t will ha v e a higher spatial resolution
than a higher altitude instrumen t with the same IF O V.
In the digital represen tation of the remote sensing images, the information is stored
as a grid of pixels, ha ving the size dep enden t on the sensor t yp e and IF O V. Due to the
latest dev elopmen ts, the spatial resolution of the remotely sensed images v ary from a few
cen timeters to sev eral kilometers. Dep ending on the area co v ered b y the pixels, w e can
classify the spatial resolution as:
Lo w resolution: larger than 30m
Medium resolution: 2-30m
High resolutio: 2-1m
15
Chapter 2. R emote Sensing principles
V ery high resolution: less than 1m
Fig. 2.10 illustrates a scene acquired with W orldView-2 ( 2m ), Sen tinel-2 ( 10m )and
a LandSA T8 ( 30m ) satellite sensors, o v er Buc harest, Romania, ha ving dieren t spatial
resolutions. As it can b e observ ed, spatial resolution is v ery imp ortan t if the analysis is
fo cused on ob ject or phenomenon iden ti cation.
Figure 2.10: Spatial resolution in dieren t remote sensing images.
Sp ectral resolution
Sp ectral resolution describ es the abilit y of a sensor to de ne ne w a v elength in terv als.
The ner the sp ectral resolution, the narro w er the w a v elength range for a particular
c hannel or band. The higher the sp ectral resolution, the narro w er is the w a v elength range
for a sp eci c band, and therefore, the more bands there are. With a higher sp ectral
resolution single ob jects can b e p erceiv ed b etter and sp ectrally distinguished.
Figure 2.11: Sp ectral resolution in dieren t remote sensing images.
Radiometric resolution
In radiometric resolution, is sp eci ed ho w w ell the dierences in brigh tness in an
image can b e p erceiv ed, using the n um b er of gra y lev el v alues. Common represen tations
of the remotely sensed data are using 8 bits (256 gra y lev el v alues) or 16 bits (65536 gra y
lev el v alues) to store image data. In Fig. 2.12 w e presen t an example of a LandSA T8
image, o v er Buc harest, Romania that is computed using b oth 8 and 16 bits represen tation.
W e can observ e that in the 16 bits represen tation, the image is ner that the image with
only 256 gra y lev els (8 bits represen tation).
16
Metho ds and algorithms for Earth Observation image information mining
Figure 2.12: Radiometric resolution exmaples. 16 bits vs 8 bits
Dep ending on the radiometric resolution, the small dierences in the re ected or
emitted radiation can b e measured. A higher radiometric resolution will enable ner
measuremen ts, but will increase the v olume of data needed to store the image acquisition.
T emp oral resolution
The temp oral resolution is giv en as the time in terv al b et w een t w o iden tical igh ts
o v er the same area, also called rep etition rate. T emp oral resolution is determined b y
altitude and orbit of the satellite as w ell as its sensor c haracteristics (viewing angle).
In Fig. 2.13 can b e observ ed t w o acquisitions of LandSA T8 satellite sensor, tak en at a
t w o y ears dierence, o v er the cit y of Dubai, United Arab Emirates. In the gure can b e
observ ed the ev olution of the cit y in a t w o p erio d.
Figure 2.13: T emp oral resolution example
The temp oral resolution allo w m ultitemp oral or c hange detection analysis lik e sea-
sonal c hanges of v egetation or the expansion of the cities o v er y ears. This is p ossible using
rep etitiv e acquisitions of the same area at dieren t times and orbits.
17
Chapter 2. R emote Sensing principles
2.4 A ctiv e sensing
Despite the m ultisp ectral sensors, whic h are using optical systems to capture visible
ligh t and near-infrared radiation, in activ e remote sensing, the image is formed using
radiation in the micro w a v es domain. The radar images can b e group ed in to circularly
scanning plan-p osition indicator (PPI) images and side-lo oking images. The rst category
of images are limited to certain applications lik e monitoring of airp orts and air or maritime
tra c [57], while the second one can b e used in remote sensing applications.
Of great in terest for remote sensing is the syn thetic ap erture radar (SAR) whic h is a
side-lo oking radar (SLR) that use the igh tpath of the platform to sim ulate an extremly
large an tenna or ap erture electronically [99]. When the sensor is placed on an aeroplane,
it is usually refereed as side-lo oking airb orne radar (SLAR) or SLR, omitting a reference
to "airb orne" [57].
With the increase of the SAR sensor p erformance up to sub-meter resolution, a
more detailed analysis and a ner description of SAR images o v er scenes, mainly man-
made structures and critical infrastructures, are needed. The analysis and information
extraction from SAR images co v ering a high div ersit y of man-made structures, com bined
with the complexit y of the scattering pro cesses is not an easy task.
In Fig. 2.14 is illustrated the principle of SAR image acquisition. The scene is
illuminated b y the sensor using pulses of micro w a v es at regular in terv als whic h are fo cused
b y the radar an tenna in to a b eam. The radar b eam illuminates the scene obliquely , at a
righ t angle to the motion of the platform. And the radar an tenna is measuring a p ortion
of the transmitted energy that is re ected/bac kscattered from v arious ob jects within the
scene.
Figure 2.14: A ctiv e sensing
The time dela y b et w een the transmission and reception of the radar ec ho es from
dieren t targets is measured, determining this w a y the distance from the radar to the
lo cation of the observ ed ob jects. SAR image acquisition, is indep enden t of the Sun illu-
mination and atmospheric conditions.
2.4.1 Image acquisition
T o form an image, in tensit y measuremen ts m ust b e tak en in t w o orthogonal direc-
tions. In the SAR con text, one dimension is parallel to the radar b eam, as the time
18
Metho ds and algorithms for Earth Observation image information mining
dela y of the receiv ed ec ho is prop ortional to the distance or range along the b eam to the
scatterer.The second dimension of the image is giv en b y the tra v el of the sensor itself. As
the sensor mo v es along in a nominally straigh t line ab o v e the Earth's surface, the radar
b eam sw eeps along the ground at appro ximately the same sp eed [24].
Fig. 2.15 illustrates a comparativ e lo ok on ho w the optical images and SAR data are
formed. In the case of optical image acquisition w e observ e that the data is recorded in
a cen tral geometry while in the SAR image acquisition, the data is measuring slat ranges
from the sensor to the ob ject and the return time of the signal. Due to the tec hnological
progress, most of the SAR sensors are no w oering images that ha v e spatial resolution
comparable with the ones pro vided b y the m ultisp ectral image satellites.
Figure 2.15: Comparison b et w een optical image acquisition and SAR image acquisition
A classi cation of activ e sensors in scanning and non-scanning systems can b e ob-
serv ed in T able 2.4. In this thesis, our in terest is fo cused on syn thetic ap erture radar
systems.
T able 2.4: A ctiv e sensing non-scanning and scanning systems
Non-Scanning Scanning – Imaging
Non-Imaging Image plane sc anning Obje ct plane sc anning
Micro w a v e Radiometer P assiv e Phased Arra y Radar Real Ap erture Radar
Micro w a v e Altimeter Syn thetic Ap erture Radar
LASER W ater Depth Meter
LASER Distance Meter
A SAR can b e op erated in dieren t mo des, sometimes with dieren t systems, or
sometimes as dieren t mo des within a single system. Some of the op eration mo des used
in SAR image acquisition are the stripmap, scanSAR, sp otligh t and stare. An illustration
of this acquisition mo des is presen ted in Fig. 2.16.
In Fig. 2.17 can b e observ ed the geometric eects that o ccur in a SAR image, suc h
as foreshortening, la y o v er and shado wing. In the case of foreshortening the radar b eam
reac hes the base of a tall feature tilted to w ards the radar b efore it reac hes the top. Due
to the fact that radar distances are measured in slan t-range, the slop e of p oin ts 1 2
will app ear compressed and the length of the slop e will b e represen ted incorrectly at the
image plane as it can b e observ ed in the top of the gure.
The la y o v er o ccurs when the radar b eam reac hes the top of a tall feature b efore it
reac hes the base. The return signal from the top of the feature will b e receiv ed b efore the
signal from the b ottom. As a result, the top of the feature is displaced to w ards the radar
from its true p osition on the ground, and la ys o v er the base of the feature. The la y o v er
19
Chapter 2. R emote Sensing principles
Figure 2.16: SAR acquisition mo des. (c) T erraSAR-X Image Pro duct Guide
eects can b e observ ed for p oin ts 3 4 in the gure. F urthermore, the shado wing eect
increases with greater inciden t angle and can b e observ ed for p oin ts 4 5 and6 7 .
2.4.2 Resolution
In radar systems, unlik e optical systems, the spatial resolution can b e de ned as a
function of sp eci c prop erties of the micro w a v e radiation and geometrical eects. If the
radar system is used for image formation, lik e in the case of SLR, a single pulse and its
bac kscattered signal are used to generate the image.
Ha ving this considerations, the resolution is dep enden t on b oth the length of the
pulse in the slat range direction and the width of illumination in the azim uth direction.
Th us, in radar systems, t w o t yp es of resolutions can b e iden ti ed, the range or across-
trac k resolution and the azim uth or along-trac k resolution. In Fig. 2.18 can b e observ ed
ho w the range and azim uth resolutions are obtained, considering the parameters on whic h
those t w o resolutions are dep enden t of: the high t ( H ), the range to target ( R ), the pulse
duration ( ), the ph ysical length of the an tenna ( L ) and the length of the syn thesized
an tenna (Ls ).
The minim um separation of t w o resolv able targets inside the SAR fo otprin t along
the radiation b eam is determining the range resolution, while the azim uth resolution is
referring to the resolv able area along the direction of the sensor motion [68].
2.5 Conclusions
In this c hapter, w e presen ted some of the basic remote sensing principles, ha ving the
goal to pro vide a b etter understanding of the data w e in tend to analyze. As w e sho wn in
this c hapter, remote sensing data can b e obtained from passiv e and activ e sensors, and
the ob jects prop erties in remote sensed images are dep enden t on ho w the measuremen ts
are realized.
20
Metho ds and algorithms for Earth Observation image information mining
Figure 2.17: SAR image acquisition
Figure 2.18: Range vs. Azim uth resolution
21
Chapter 3
Data representation and data analysis
The main purp ose of the researc h describ ed in this man uscript is do pro vide new
metho ds of kno wledge extraction from Earth observ ation image data, b y p erforming a
high lev el seman tic mo deling of the analyzed data con ten t and to obtain thematic maps
according to a real meaning of the imaged area with a minim um h uman in teraction. State
of the art feature extraction metho ds and new dev elop ed ones are in tro duced for a scene
lev el in terpretation.
The prop osed metho dology is relying on the standard comm unication c hannel mo del,
in whic h a signal, or an image lik e in our case, represen ting the input data is mo deled
according to certain needs, in order to obtain at the output of the system a classi cation,
p erformed at patc h lev el, of the original image. It is desirable that the transformation
la ws of the input image to b e w ell de ned in order to highligh t certain features of the
image, lik e texture, color and shap e.
3.1 F rom data to kno wledge
The kno wledge extraction pro cess is requiring a full understanding on ho w the data
is measured and ho w the image is formed. F urthermore, w e need to nd the main c har-
acteristics of the image and giv e them a mathematical meaning that the algorithms used
in the analysis can use. Also, in order to obtain relev an t results, the user in teraction is
necessary either for training or v alidation. In Fig. 3.1 is presen ted the pro cess of kno wl-
edge extraction from EO data that in v olv es image acquisition, image formation, lo w lev el
features extraction, high lev el feature extraction and classi cation.
Figure 3.1: F rom data to kno wledge
22
Metho ds and algorithms for Earth Observation image information mining
3.1.1 Image acquisition
As w e presen ted in the previous c hapter, in remote sensing w e use dieren t data
pro ducts from a large v ariet y of sensors. The pro cess of understanding the remote sensing
data requires to ha v e a deep kno wledge on ho w the data is computed. As it is presen ted
in Fig. 3.2, the data acquired with dieren t sensors, that are capable to sense sp eci c
regions of the electromagnetic sp ectrum are pro viding dieren t image pro ducts that needs
to b e understo o d and analyzed.
Figure 3.2: Dieren t image formats and their sp ectrum
Ev en though SAR data is complemen tary to m ultisp ectral data, the image con ten ts
for b oth t yp es of data ha v e dieren t geometric eects and are measuring dieren t energy
signatures – micro w a v es in the SAR image and visible sp ectrum in the m ultisp ectral data.
An example of suc h complemen tary data sets can b e observ ed in Fig. 3.2. In the left
side of the gure is presen ted a SAR magnitude image, acquired with Sen tinel-1 satellite
sensor, in the cen ter of the gure can b e observ ed the same scene as a m ultisp ectral image,
acquired with Sen tinel-2 satellite sensor, and in the righ t is an aerial m ultimedia image
that is cop yrigh ted to Microsoft Corp oration.
Multisp ectral imaging
In the b eginning of remote sensing, simple cameras w ere emplo y ed b ecause of their
simplicit y . Some of the rst aerial photographs of the earth w ere made in hot air ballo ons
starting with Nadar in 1858, in F rance, and later with the tec hnical impro v emen ts, some
cameras w ere moun ted in unmanned ying ob jects and ev en on pigeons in the early 1900's.
F urther, the aerial photograph y devices and metho ds w ere ev olving fast during the W orld
W ar I, b eing moun ted on aircrafts, and later on satellite platforms.
Curren tly , w e ha v e at disp osal a large v ariet y of sensors and platforms to c ho ose
according to our needs. Optical imaging systems, whic h are passiv e sensing platforms,
are relying on an external source of energy to form images. The range of electromagnetic
sp ectrum that most of the remote sensing platforms can sense is somewhere b et w een 300
nm to 2500 nm, b eing able to co v er the electromagnetic sp ectrum from ultra violet to
short-w a v e infrared. Also there are other passiv e imaging sensors that can sense in the
thermal infrared in terv al.
23
Chapter 3. Data r epr esentation and data analysis
Syn thetic ap erture radar imaging
W e can mark the b eginning of syn thetic ap erture radar imaging in 1865, with the
publication of Maxw ell's pap er on the dynamical theory of the electromagnetic eld, whic h
w as used b y Hertz in 1887 to generate electromagnetic w a v es. F urther, in 1899, Marconi
disco v ered that radio w a v es can b e re ected bac k to the transmitter when encoun tering
an y ob jects and in 1904 Hulsmey er, dev elop ed the rst functional radar device that used
radio ec ho es. Also, the time of W orld W ars I and I I had a strong impact on the dev el-
opmen t of radar systems, and in 1950's ha v e emerged the side-lo oking radar principle,
whic h is used in to da y's syn thetic ap erture radar platforms.
Being complemen tary to m ultisp ectral images, SAR image data can pro vide extra
information to an image analyst or to an image information mining system. Due to
the prop erties of micro w a v es to p enetrate clouds, and sometimes v egetation, ice and
extremely dry sand, the radar is the only w a y to explore inaccessible regions of the Earth's
surface [37]. F urthermore, b eing an activ e sensing system, the syn thetic ap erture radar
is computing the image b y using the micro w a v e energy re ected bac k to w ard the radar
an tenna ha ving the p ossibilit y of da y/nigh t imaging.
3.1.2 Image formation
Dep ending on the tec hnology used for remote sensing measuremen ts, w e can obtain
either an image lm, a digital image or a signal to b e pro cessed. In the case of image lms,
a c hemical pro cess is required, while in the case of digital images and syn thetic ap erture
radar, some prepro cessing steps are in v olv ed. It is w ell kno wn that an image recorded on
a photographic lm can b e con v erted in to digital format using digitizing equipmen t. In
the follo wing w e will consider the digital image case, whic h w e will refer to as image.
In [37], an image is de ned as a t w o-dimensional function f(x;y) , wherex andy
are the spatial co ordinates and the amplitude of f at an y pair of co ordinates (x;y) is
represen ting the in tensit y or gra y lev el of the image at that p oin t. The ph ysical meaning
of the in tensities is dieren t in remotely sensed images, its v alues b eing prop ortional to
the radiated energy of a ph ysical source [37]. This can b e expressed using Equation 3.1.
0f(x;y)<1 (3.1)
The amoun t of source illumination inciden t on the scene b eing observ ed (illumina-
tion), and the amoun t of illumination re ected b y the ob jects in the scene (re ectance),
are v ery imp ortan t factors in the image formation pro cess. Ha ving the illumination i(x;y)
and re ectance r(x;y) comp onen ts, the image function f(x;y) can b e considered to b e a
pro duct of illumination and re ectance as expressed in Equation 3.2. The nature of i(x;y)
is determined b y the illumination source, and r(x;y) is determined b y the c haracteristics
of image ob jects. Also, in Equation 3.2, the fact that r(x;y)2(0;1) dep ends on the
prop ert y of the ob jects to totally absorb the source energy or to totally re ect it.
f(x;y) =i(x;y)r(x;y) , where 0i(x;y)<1 and0r(x;y)<1 (3.2)
Considering the gra y lev el l of a mono c hrome image equal to the in tensit y at an y
co ordinates (x0;y0) , thenl=f(x0;y0) , results that l is b ounded b y a minim um Lmin
24
Metho ds and algorithms for Earth Observation image information mining
and a maxim um Lmax gra y lev el v alues, as presen ted in Equation 3.3. Also, the nite
and p ositiv e in terv al [Lmin;Lmax] , called gra y scale in terv al, is usually shifted to t the
in terv al [0;L 1] , wherel= 0 is considered blac k, l=L 1 is white on the gra y scale
and all the v alues in b et w een are shades of gra y v arying from blac k to white.
LminlLmax , whereLmin=iminrmin andLmax=imaxrmax (3.3)
Figure 3.3: P anc hromatic image and its gra yscale represen tation of f(x;y)
The mathematical mo del w e presen ted is common for the m ultimedia and m ultisp ec-
tral images. In the case of SAR image formation pro cess, the illumination source is the
micro w a v e generator moun ted on the platform, while the re ectivit y is de ned as a func-
tion of ob jects size, shap e, p osition and orien tation, as w ell as its comp osition and radar
w a v elength. Moreo v er, the radar image is a complex image, c haracterized b y a real and
an imaginary part.
In [24] is stated that SAR image formation and data pro cessing are dieren t from
man y other remote sensing tec hniques, since the imaging pro cess is coheren t. The most
natural w a y to describ e suc h a system and its signals is b y complex-v alued functions.
Hence, signal pro cessing, rather than image pro cessing, pro vides the appropriate to ols.
SAR pro cessing needs DSP , but DSP also pro ts from SAR. The SAR imaging principle
and the pro cessing algorithms ha v e on their o wn b ecome an attractiv e class of DSP
metho ds that can b e transferred to other elds.
In the SAR image formation pro cess, the ob jects or targets ha v e a v ery imp ortan t
role. Dep ending on the target prop erties, the radar re ectivit y , or radar cross section
ma y b e in uenced. As men tioned earlier, the radar cross section is in uenced b y the
incidence angle i and a material co e cien t M , as it can b e observ ed in Equation 3.4.
=M3cosi
(sini+Mcosi)3(3.4)
Considering a matrix of pixels that describ e the remotely sensed data, w e need to
establish the n um b er of bits used to store the gra y lev el v alues. Using m bits giv es a range
of2mv alues, ranging from 0 to2m 1 . In most of the cases an 8 bit image represen tation
is used, but when w orking with remotely sensed data, 8 bit p er sample is not enough. In
Fig. 3.4 is illustrated an m ultisp ectral image with with dieren t gra y lev el in terv als due
to the n um b er of bits m used for image represen tation.
25
Chapter 3. Data r epr esentation and data analysis
Figure 3.4: P anc hromatic image represen tation at dieren t n um b er of bits m= 16 ,8 ,4 and2
In [67] is explained that the ideal v alue of m is related to the signal to noise ratio
(bandwidth) of the camera, b eing of appro ximately 45 dB and since there are 6 dB p er bit,
then 8 bits will co v er the a v ailable range. As w e can observ e in Fig. 3.4, the dierences
b et w een the 16 and 8 bits represen tations are v ery lo w.
3.1.3 Lo w and high lev el features
The feature extraction pro cess can b e de ned as an essen tial pre-pro cessing step
to pattern recognition and mac hine learning problems. Its main goal is to transform
redundan t information from an image to a reduced represen tation set of features.
Ha ving in to accoun t this considerations, in [67], lo w-lev el features are considered
those basic features that can b e extracted automatically from an image without an y shap e
information (information ab out spatial relationships). As suc h, thresholding is actually a
form of lo w-lev el feature extraction p erformed as a p oin t op eration. Naturally , all of these
approac hes can b e used in high-lev el feature extraction, where w e nd shap es in images.
Man y image analysis approac hes are based on lo w-lev el features lik e edge detection,
corner detection, blob detection and ridge detection. Those are usually obtained using
sp eci c detectors sensible to dierences of con trast in the image. The high lev el features
concerns nding shap es in digital images or to nd a spatial dep endency b et w een lo w-
lev el features. In the case of high-lev el features, the goal is to nd in v arian t prop erties of
ob jects.
F eature extraction metho ds for lo w and high lev el features will b e detailed in the
section regarding data represen tation.
3.1.4 Data analysis and in terpretation
Considering the exp onen tial gro wth of the EO image data collections obtained from
satellite and aerial sensors, in the last decades a lot of eort has b een made for scene
understanding and analysis.
In the classical remote sensing approac h, a trained image analyst is required to iden-
tify sp eci c targets and phenomenons, based on the kno wledge gained in time. Also
in this approac h, m ultiple sources of information are required for the kno wledge extrac-
tion pro cess and sometimes complemen tary data sets are needed. Th us, to increase the
p erformances and minimize the time required for image analysis, there are attempts of
automatic image analysis systems.
Most of the image analysis approac hes use feature extraction tec hniques and classi-
26
Metho ds and algorithms for Earth Observation image information mining
cation algorithms to automatically group input data similarities. Also, b eing generally
inspired from the h uman visual system sp ecialized in detecting sp eci c image prop er-
ties suc h as texture, color and shap e, these metho ds usually require h uman-based data
annotation [3], either for the training or v alidation pro cess.
3.2 Earth observ ation images as sto c hastic pro cesses
An y statistical analysis of an ev en t or phenomenon m ust b e de ned using a math-
ematical mo del that links the observ ed realit y with a mec hanism that generates the ob-
serv ations. In the de nition pro cess w e should tak e in to accoun t that the functional form
of the ev en t b eing analyzed m ust b e simple and the n um b er of parameters and com-
p onen ts should b e minimized. When applying this idea on earth observ ation images, a
parametrization of the mo del ha v e to b e realized so that eac h parameter can b e easily
iden ti ed with certain asp ects of the realit y .
A ccording to information theory and statistical analysis, w e can consider earth obser-
v ation images to b e either pure deterministic mathematical mo dels or sto c hastic mo dels.
In the deterministic approac h, the data is dep enden t on a systematic and noise free com-
p onen t, suc h that eac h time an image is tak en, w e will obtain the same measured in tensi-
ties. In realit y , in an earth observ ation image, a giv en pixel in tensit y or gra y-scale v alue,
deriv ed from the measured radiance at a satellite sensor is nev er exactly repro ducible b e-
cause of the instrumen t noise, atmospheric conditions, observ ation angle or illumination
c hange [12]. In this situation w e ma y refer to the earth observ ation imagery as sto c hastic
realizations of random v ariables.
When an observ ation is realized, w e are sampling from the probabilit y distribution,
meaning that w e are observing a dieren t p ossible realization of a random pro cess eac h
time. Similarly , in the Earth Observ ation imagery , ev ery acquisition is unique ev en though
the formed images are almost alik e. Th us, image formation is a result of a sto c hastic
pro cess and the pixels are a realization of a random v ariable that caries information in
form of uncertain t y . Therefore, in the quest of image information extraction, w e should
determine ho w to estimate the parameters that c haracterize the distribution functions
of the random v ariables and also their means, v ariances and co v ariance matrices from
observ ations.
F or a b etter understanding of remote sensing images as sto c hastic pro cesses, a series
of concepts suc h as probabilit y , random v ariablem sto c hastic pro cess, random signal and
information should b e de ned.
3.2.1 De ning probabilit y and ev en ts
The main de nition of term "probabilit y" is referring to the qualit y or state of b eing
probable. Th us, w e ma y a rm that the probabilit y P is a measure of the lik eliho o d that
an ev en t E can o ccur.
De nition 1 The pr ob ability of one event is the r atio of the numb er of favor able c ases
to the numb er of al l c ases p ossible when nothing le ads us to exp e ct that every one of these
27
Chapter 3. Data r epr esentation and data analysis
T able 3.1: Op eration sets in probabilit y and corresp onding probabilit y statemen ts
Op eration set Probabilit y statemen t
E1[E2 A t least one of the ev en ts E1 orE2 o ccurs.
E1\E2 Both ev en ts E1 orE2 o ccurs.
E The ev en t E do es not o ccur
? The imp ossible ev en t
E1\E2=? The ev en ts E1 andE2 are m utually exclusiv e.
E1E2 The ev en t E1 implies the ev en t E2 .
T able 3.2: Probabilit y axioms
Axiom Description
P(E)0 P ositivit y axiom
P(I) = 1 Certain ev en t
P(E1[E2) =P(E1) +P(E2) ifE1[E2=? Sum rule
P(E1\E2) =P(E2jE1)P(E1) =P(E1jE2)P(E2) Pro duct rule
c ases should o c cur mor e than any other, which r enders them, for us, e qual ly p ossible.
(L aplac e 1819)
De nition 2 The pr ob ability is a r epr esentation of de gr e es of plausibility by r e al numb ers.
(Jer ey, mo dern interpr etation)
The rst de nition is the classical, "frequen tist" approac h, em bracing the idea of
frequency of en ev en t in a series of ev en ts, while the second de nition represen ts the
mo dern "Ba y esian" in terpretation. In T able 3.1 can b e found some of the op eration sets
used to de ne the probabilit y scenarios w e need.
A ccording to Laplace de nition, if w e assume all the outcomes of an ev en t are equally
lik ely , w e can compute the probabilit y of an ev en t with the equation 3.5, where NE is the
n um b er of outcomes that are fa v orable to the ev en t E andN is total n um b er of p ossible
outcomes.
P(E) =NE
N(3.5)
In T able 3.2 can b e observ ed the probabilit y axioms that represen ts the rules used
in probabilit y theory , where the notation P(AjB) refers to the probabilit y of ev en t A
conditioned b y ev en t B . As an observ ation, the axiom regarding the pro duct rule do es
not assume an y exclusivit y of the ev en t, th us w e can conclude that the imp ossible ev en t
has zero probabilit y (Equation 3.6), ho w ev er it do es not follo w that an ev en t of zero
probabilit y is imp ossible, as expressed in Equation 3.7.
P(?) = 0 (3.6)
0P(E)1 (3.7)
28
Metho ds and algorithms for Earth Observation image information mining
T w o ev en ts, E1 andE2 are statistical indep enden t if Equation 3.8 is true, meaning
that the probabilit y of an ev en t is not in uenced b y the fact that another ev en t tak es
place.
P(E1\E2) =P(E1)P(E2) (3.8)
As a consequence, w e can deduce the follo wing:
P(E1jE2) =P(E1) (3.9)
P(E2jE1) =P(E2) (3.10)
Considering further a m utually exclusiv e and complete set of ev en ts fH1;H2;::;Hng
that is not indep enden t of an ev en t E in a certain exp erimen t, w e call the ev en ts Hi
h yp otheses and in terpret them as h yp othetical causes of the ev en t E . Th us, the follo wing
decomp osition can b e written:
P(E) =nX
i=0P(EjHi)P(Hi) (3.11)
and the form ula of Ba y es:
P(HijE) =P(EjHi)P(Hi)
P(E)(3.12)
The probabilit y of P(HijE) is the probabilit y satisfying the h yp otheses Hi kno wing
that the ev en t E w as eectiv ely pro duced. This is called " a p osteriori " probabilit y of Hi
andP(Hi) is called " a priori " probabilit y .
3.2.2 Random v ariables
In general terms, a random v ariable can b e used to represen t a quan tit y . In the case
of remote sensing imagery , that quan tit y can b e a gra yscale v alue, whic h c hanges in an
unpredictable w a y , due to v arious factors, eac h time it is observ ed.
If w e consider a random exp erimen t sp eci ed b y the space
of p ossible outcomes
(sample space), a eld K of subsets of
(ev en ts) and the probabilit y P assigned to
these ev en ts, w e ma y sa y an ev en t ha v e o ccurred whenev er the outcome of the random
exp erimen t is con tained within it [12].
De nition 3 A r andom variable X:
7 !R is a function which maps al l the outc omes
! onto the set R of r e al numb ers, such that the set
f!2
jX(!)xg (3.13)
is an event, that is, a subset of
. This subset is usual ly abbr eviate d as fXxg
29
Chapter 3. Data r epr esentation and data analysis
De nition 4 The pr ob abilities of the events fX=1g andfX= 1g e quals 0:
PfX=1g= 0 (3.14)
PfX= 1g = 0 (3.15)
F or example, the fact that an image pixel has a certain gra y lev el can b e considered
an elemen tary ev en t c haracterized b y its probabilit y . Th us when extending to the full
image, the ensem ble of pixel gra y lev els, its in tensit y , is a random v ariable.
De nition 5 The distribution function ,F(x) , of a r andom variable X2( 1;+1)
is de ne d as the pr ob ability of the event f!2
jX(!)xg :
F(x) =P(Xx) (3.16)
Pr op erties of the distribution function:
1.0F(x)1 ,(8)x2R
2.lim
x! 1F(x) = 0
3.lim
x!+1F(x) = 1
4.P(a<Xb) =F(b) F(a) , ifa;b2R anda<b
5.F(x) is a monotonic non de cr e asing function, thus F(a)F(b) ifa<b
De nition 6 The pr ob ability density function (p.d.f.) of a r andom variable X is
de ne d as the derivative in x of the distribution function:
p(x) =dF(x)
dx(3.17)
and is interpr ete d as
P(x<Xx+dx) =p(x)dx (3.18)
Pr op erties of the pr ob ability density function:
1. p(x)>0, ac c or ding to the monotony of F(x)
2. Considering p(x) is p ositive and inte gr able on ( 1;+1) , thenF(x) =Rx
1p(u)du
3.lim
x!1F(x) =R+1
1p(u)du
4.P(a<X <b ) =F(b) F(a) =Rb
ap(u)du
W e can use the p.d.f. to determine the orien tation of a line in an image. F or example,
in Fig. 3.5 is presen ted a region from an image that con tains a line formed b y the pixels
groupf 1; 5; 9g . Considering the neigh b orho o d of pixel 5 , w e can de ne the p.d.f. of
the line on the neigh b orho o d v ariables. The join t distribution p( 1; 5; 9) captures the
orien tation of the image.
30
Metho ds and algorithms for Earth Observation image information mining
Figure 3.5: Example of a group of 3×3 pixels describing a line
De nition 7 Given an -dimensional r andom variable, fr om which only k < n c omp o-
nents ar e of inter est, a mar ginal p.d.f. is de ne d as:
p(x1;::;xk) =Z+1
1dxk+1::Z+1
1p(x1;::;xk;xk+1;::;xn)dxn (3.19)
Assuming the kno wledge of a join t p.d.f. c haracterizing edges in a 9-pixel neigh-
b ourho o d, the p.d.f. of the gra y lev el in the cen tre of a neigh b ourho o d is obtained as
a marginal p.d.f. b y in tegrating o v er 8 random v ariables attac hed to the surrounding
neigh b ours.
De nition 8 A nn -dimensional joint p.d.f. is c al le d c onditional r elative ton k
variables if these n k variables have pr e de ne d values.
p(x1;::;xkjxk+1;::;xn) =P(x1<X 1<x 1+dx1;::
::;xk<Xk<xk+dxkjXk+1=xk+1;::
::;Xn=xn)(3.20)
Using the Ba y es form ula w e obtain:
p(x1;::;xkjxk+1;::;xn) =p(x1;::;xk;xk+1;::;xn)
p(xk+1;::;xn)(3.21)
De nition 9 Two events ar e statistic al ly indep endent if the o c curr enc e of one do es
not ae ct the pr ob ability of the other. The gener alization of the statistic al indep endenc e
gives:
p(x1;::;xkjxk+1;::;xn) =p(x1;::;xk) (3.22)
and
p(x1;::;xk;xk+1;::;xn) =p(x1;::;xk)p(xk+1;::;xn) (3.23)
31
Chapter 3. Data r epr esentation and data analysis
De nition 10 Considering a function f(X1;::Xn) of the r andom variables X1;::Xn the
exp e ctation op er ator "[] is intr o duc e d:
"[f(X1;::Xn)] =Z+1
1dx1::Z+1
1dxnf(x1;::xn)p(x1;::xn) (3.24)
The exp ectation op erator allo ws the de nition of the random v ariable momen ts.
Th us, for the t w o dimensional case, can b e used in computation of the mean, the mean
square, the v ariance and co v ariances.
De nition 11 The me an:
m1=Z+1
1dx1Z+1
1dx2x1p(x1;x2) =Z+1
1dx1x1p(x1) (3.25)
m2=Z+1
1dx1Z+1
1dx2x2p(x1;x2) =Z+1
1dx2x2p(x2) (3.26)
De nition 12 The me an squar e:
R1=Z+1
1dx1Z+1
1dx2x2
1p(x1;x2) =Z+1
1dx1x2
1p(x1) (3.27)
R2=Z+1
1dx1Z+1
1dx2x2
2p(x1;x2) =Z+1
1dx2x2
2p(x2) (3.28)
R12=Z+1
1dx1Z+1
1dx2x1x2p(x1;x2) (3.29)
De nition 13 The varianc e:
2
1="[(X1 m1)2] =R1 m2
1 (3.30)
2
2="[(X2 m2)2] =R2 m2
2 (3.31)
De nition 14 The c ovarianc e:
C12="[(X1 m1)(X2 m2)] =R12 m1m2 (3.32)
If the random v ariables are indep enden t, their co v ariance is zero. Otherwise, if the
co v ariance is zero this do es not mean that the random v ariables are indep enden t.
IfX= [X1;X2;::;Xn]Tis a v ector of random v ariables, with the mean mX=
[m1;m2;::;mn]T, then the co v ariance matrix CX of X is de ned as:
CX="[(X mX)(X mX)T] (3.33)
32
Metho ds and algorithms for Earth Observation image information mining
and has the extended form as a symmetric, squared and non-negativ e matrix:
CX=0
BBBB@2
1C12::: C 1n
C212
2::: C 1n
……. . ….
Cn1Cn2::: 2
X1
CCCCA(3.34)
If the comp onen ts of a random v ector are indep enden t the co v ariance matrix is diago-
nal. Ho w ev er if the co v ariance matrix is diagonal there is non implication of indep endence
of the t w o random v ectors only their comp onen ts are m utually uncorrelated.
3.2.3 P arameter estimation
By de ning the distribution functions for random v ariables and v ectors w e can no w
mo del random ev en ts suc h as image acquisition. F urthermore, w e need a w a y to estimate
the parameters that c haracterize the distribution functions, their means, v ariances and
co v ariances matrices from observ ations.
The goal of parameter estimation is the ev aluation of a parameter generated b y
a source of information in noisy conditions. Considering a remote sensing image x=
[x1;x2;::;xN] the source of information that is observ ed in the presence of noise n=
[n1;n2;::;nN] and the measured pixel in tensities y= [y1;y2;::;yN] , in Equation 3.35 w e
can observ e the measuremen t mo del in case of the additiv e noise, while in Equation 3.36
w e can nd the m ultiplicativ e noise mo del.
y=x+n (3.35)
y=xn (3.36)
In remote sensing imagery , it is kno wn that optical images are aected b y additiv e
noise, and SAR images are aected b y m ultiplicativ e noise. The noise that disturbs the
measuremen ts in the earth observ ation images can b e due to the atmospheric disturbances,
sensor artifacts, solar radiation etc. In Fig. 3.6 can b e observ ed a sample of additiv e and
m ultiplicativ e noise.
The image x is a random signal, a realization of sto c hastic pro cess and the noise n is
a t w o dimensional random signal. The problem that ha v e to b e solv ed using parameter
estimation theory is to nd a guess of x , giv en the observ ations y and p ossibly some
kno wledge ab out x andn .
If w e consider the linearly ordered, observ ed pixel in tensities of a random image
y= [y1;y2;::;yN] that are c haracterized b y the conditional p.d.f, then:
p(y1;y2;::;yNjx) =p(yjx) (3.37)
Also, if w e consider ^x to b e an estimate of the unkno wn gra y-lev el x , then the
estimator error can b e computed using Equation 3.38. The error "x is only h yp othetically
33
Chapter 3. Data r epr esentation and data analysis
Figure 3.6: Example of additiv e noise (Left) and m ultiplicativ e noise (Righ t)
de ned, since the true v alue is unkno wn.
"x=x ^x(y) (3.38)
T o measure the imp ortance of the estimator error is needed a cost function c("x)
whic h can b e quadratic (Equation 3.39) or uniform (Equation 3.40)
cq("x) ="2
x (3.39)
cu("x) =8
<
:0 , ifj"xj
2
1 , ifj"xj>
2(3.40)
Figure 3.7: Quadratic cost function cq(x;^x) (Left). Uniform cost function cu(x;^x) (Righ t)
De nition 15 The exp e ctation of the c ost r elative to the joint p.d.f. p(x;y) is c al le d
34
Metho ds and algorithms for Earth Observation image information mining
Bayes risk R:
R="[c(x ^x)] =Z Z
dxdyc (x ^x)p(x;y) (3.41)
T o solv e the parameter estimation problem, the minimization of the Ba y es risk R
should b e p erformed. Th us, the estimators will use the quadratic and uniform cost func-
tions.
De nition 16 Minimum me an squar e err or estimators (MMSE)
Using the quadr atic c ost function, the risk is:
Rq=Z Z
dxdy (x ^x)p(x;y) (3.42)
When applying the Bayes formula for c onditione d p.d.f. we wil l obtain:
Rq=Z
dyp(y)Z
dx(x ^x)2p(xjy) (3.43)
Since b oth in tegrals are non-negativ e, the minim um risk is obtained b y minimizing
equation 3.44 with resp ect to ^x .
I(^x;y) =Z
dx(x ^x)2p(xjy) (3.44)
@
@^xI(^x;y) = 2^xZ
p(xjy)dx 2Z
xp(xjy)dx= 0 (3.45)
Kno wing thatR
p(xjy)dx= 1 , w e obtain the MMSE estimator as the conditional
mean of the desired parameter x :
^xMMSE (y) =Z
xp(xjy)dx (3.46)
De nition 17 Maximum a p osteriori estimator (MAP)
In the c ase of MAP, a uniform c ost function is assume d. In this c ase the risk function
wil l b e:
Ru=Z
dyp(y)Z
dxcu(x ^x)p(x;y)
=Z
dyp(y)
1 Z^x+
2
^x
2dxp(xjy)!(3.47)
35
Chapter 3. Data r epr esentation and data analysis
In con trast with the MMSE estimator, the minimization of the risk Ru requires the
maximization of the in tegral:
I(^x;y) =Z^x+
2
^x
2dxp(xjy) (3.48)
Due to lim
!0I(^x;y) =p(^x;y) , the maximization of I is obtained b y maximization of
the p osterior densit y p(xjy) :
@
@^xp(xjy)jx=^xMAP= 0 (3.49)
The p osterior probabilit y can b e ev aluated using the Ba y es form ula:
p(xjy) =p(yjx)p(x)
p(y)(3.50)
Due to the fact that man y p.d.f.'s used in practice ha v e exp onen tial forms the MAP
estimator is ev aluated under a logarithmic transform.
@
@xlogp(xjy) =@
@xlogp(yjx) +@
@xlogp(x) = 0 (3.51)
^xMAP = arg max
xlogp(xjy) (3.52)
W e can observ e that b oth estimators, MMSE and MAP are using a p osterior p.d.f.
p(xjy) , ho w ev er the estimators extract dieren t information and do not result in the same
solution.
Figure 3.8: MMSE and MAP estimators ev aluated using a p osteriori p.d.f. p(x|y).
The cost function used b y the MAP estimator leads to an inference v ery sensitiv e
to ho w accurately the prior mo dels the data. The generalizations of the cost function are
36
Metho ds and algorithms for Earth Observation image information mining
alternativ e solutions b ecause it is able to capture information in the mass of the prior
distribution.
As a remark, in the particular case of symmetric p osterior p.d.f. the MMSE and
MAP estimators are equal. (Fig. 3.9)
Figure 3.9: In the particular case of symmetric p osterior probabilit y , the MAP and MMSE estimators are iden tical.
3.3 Image pro cessing fundamen tals
F or the purp oses of image analysis and pattern recognition, image pro cessing algo-
rithms are v ery imp ortan t. Most of the tec hniques are based on color, texture and shap e
analysis. In order to extract sp eci c information from an image, in most of the cases sp e-
ci c transformations of the image b eing analyzed are required. Th us, in order to obtain
a quan titativ e description of geometric structures and shap es, mathematical morphology
tec hniques are used. F urthermore this metho ds are pro viding a mathematical description
of algebra, top ology , probabilit y and in tegral geometry regarding the con ten t of an image.
3.3.1 Sp ectral analysis metho ds
The assessmen t of color information pla ys an imp ortan t role in image pro cessing.
Metho ds and tec hniques for color analysis ha v e b een used since the b eginning of digital
image represen tation. Considering the color information, digital images can b e dieren-
tiated in gra y-lev el images and color images. In gra y-lev el images, a scalar gra y lev el is
assigned to a pixel, while in color images a pixel is describ ed using a color v ector, is a
certain color space.
In remote sensing imagery , suitable sp ectral transformations of the observ ed data
ha v e to b e computed in order to highligh t sp eci c image features. In this man uscript,
sp ectral transformations lik e h ue-saturation-v alue (HSV) and normalized dierence v ege-
tation index (ND VI) are considered to compute sp ectral indexes for ob ject extraction.
37
Chapter 3. Data r epr esentation and data analysis
HSV color space
In computer vision, for digital image represen tation it is common to use the R GB
(red-green-blue) color space, whic h is an orthogonal (Cartesian) co ordinate system, with
three axes, represen ting the quan tit y of red, green and blue of an image pixel. In the case
of m ultisp ectral remotely sensed images, in whic h w e ha v e more than three sp ectral bands
that usually exceed the visible sp ectrum, the R GB color space is used for visualization
purp oses, b y com bining three dieren t bands, obtaining this w a y an image in real colors
(when the bands com bination is R GB) or false colors (the band com binations is other
than R GB) [8].
Ev en though HSV color space is a p opular c hoice for manipulating color, R GB color
space is used for visualization purp oses b ecause most of the televisions, computer displa ys
and pro jectors are using com binations of red, green and blue ligh t in v arying in tensities. If
the goal is to extract information from a digital image, it is advisable to use a p erceptual
color space, lik e HSV (h ue-saturation-v alue) color space whic h is making color comp onen ts
p erceptually indep enden t and uniform [76]. The transformation from R GB color space to
HSV color space is realized using Equations 3.53 – 3.55, in whic h R0=R=255 ,G0=G=255 ,
B0=B=255 ,Cmax= max(R0;G0;B0) ,Cmin= min(R0;G0;B0) and=Cmax Cmin are
used to compute h ue (H), saturation (S) and v alue (V), as presen ted in [61].
H=8
>>>>><
>>>>>:0o, if= 0
60o G0 B0
mod 6
, ifCmax=R0
60o B0 R0
+ 2
, ifCmax=G0
60o B0 R0
+ 4
, ifCmax=B0(3.53)
S=8
<
:0 , ifCmax= 0
Cmax, ifCmax6= 0(3.54)
V=Cmax (3.55)
The transformation from R GB to HSV, presen ted in Equations 3.53 – 3.55 is non-
linear, but rev ersible. The HSV color space is dev elop ed to pro vide a more in tuitiv e
represen tation of color, closer to the w a y h uman b eings p erceiv e and in terpret color. In
[61], the h ue (H) represen ts the dominan t sp ectral comp onen t, the color in its pure form,
the saturation (S) is dep enden t on the p ercen tage of white added to the pure color and
the v alue (V) corresp onds to the brigh tness of color.
Normalized dierence v egetation index
In m ultisp ectral remote sensing images, the normalized dierence v egetation index
(ND VI) is a n umerical indicator used to analyze ho w an in terest surface is re ecting
the ligh t from the sun. It is kno wn that health y v egetation absorbs most of the visible
ligh t that hits it and re ects a large quan tit y of the near-infrared ligh t. As expressed
38
Metho ds and algorithms for Earth Observation image information mining
in Equation 3.56, the ND VI index is computed from the visible (VIS) and near-infrared
(NIR) ligh t re ected b y v egetation.
NDVI =NIR VIS
NIR VIS(3.56)
In computation of ND VI index, the result will consists of a n um b er that ranges
b et w een (0;1) . A v alue closer to 1 highligh ts the presence of v egetation, while a lo w er
v alue is indicating the absence of v egetation.
3.3.2 W a v elet analysis
W a v elets are to ols for decomp osing signals, suc h as images, in to a hierarc h y of in-
creasing resolutions. Th us, in a w a v elet transform, the image is decomp osed in to a set
of subimages or subbands whic h represen t details at dieren t scales. In order to do so,
a series of lter banks should b e computed, b eing the elemen tary building blo c ks in the
construction of the w a v elets [7]. In signal pro cessing, a lter bank is de ned as an arra y
of band-pass lters that separates the input signal in to m ultiple comp onen ts, eac h one
carrying a single frequency comp onen t subband of the original signal.
Considering images to b e linear com binations of elemen tary images whic h can b e rep-
resen ted as discrete w a v elets and scaling functions, w a v elet transforms pro vide a complete
image represen tation and p erform a decomp osition on b oth the scale and orien tation.
3.3.3 Morphological image pro cessing
As explained in a previous section, an image is a t w o-dimensional function f(x;y) ,
and the v alue of the function at (x;y) is the v alue of in tensit y at that p oin t. In the
eld of image analysis, to highligh t features lik e texture, color and shap e, the image
function should b e transformed dep ending on the features w e w ould lik e to extract. Suc h
transformation is named morphological image pro cessing and is mo difying the spatial
form or structure of ob jects within an image [87].
The fundamen tal morphological op erations are dilation, erosion and sk eletonization.
With dilation, an ob ject gro ws uniformly in spatial exten t, whereas with erosion an ob ject
shrinks uniformly and sk eletonization results in a stic k gure represen tation of an ob ject
[76]. F urthermore, in image pro cessing, lo w-lev el features extraction is v ery imp ortan t
for subsequen t higher lev el vision tasks and can lead to some inference ab out ph ysical
prop erties of the analyzed image or phenomenon.
In morphological image analysis, edges can b e de ned as abrupt in tensit y c hanges
of an image and are some of the basic lo w-lev el features. In the edge detection problem,
op erations suc h as smo othing, dieren tiation and decision are required. Smo othing is
used mainly to suppress the noise in the image and decomp ose edges at m ultiple scales,
dieren tiation is amplifying the edges creating this w a y more easily detectable simple
geometric patterns, while decision is using thresholds to consider certain edges.
Usually in the edge detection op erations, a con v olution is applied to the image, as
expressed in Equation 3.57, where k can b e either Prewit, Sob el or Kirsc h edge-enhancing
39
Chapter 3. Data r epr esentation and data analysis
con v olution masks and f0is the result of the con v olv ed image.
f0=kf , where k is a con v olution k ernel (3.57)
F urther dev elopmen ts of the edge detection metho ds implies the use of isotropic
Gaussian functions, as expressed in Equation 3.58, to optimally lo calize edges b oth in the
space and frequency domains.
G(x;y) =e (x2+y2)
22
22(3.58)
In the quest of nding relationships b et w een pixels in an image, lo w-lev el features
w ere computed. The next logical step is to extend neigh b orho o ds op erations to more
complex structures. W e can analyze simple patterns that are encapsulated in to lo cal
neigh b orho o ds. In [46] is stated that the h uman visual system cannot recognize ob jects
that do not dier from a bac kground b y their mean gra y v alue but only b y the orien tation
or scale of a pattern. Also, to p erform this recognition task with a digital image pro cessing
system, w e need to de ne op erators that determine the orien tation and the scale of the
pattern. A common result of suc h op erators will transform a gra y scale image in to a feature
image in whic h pattern that dier b y orien tation and scale can b e easily distinguished.
In Fig. 3.10 can b e observ ed ho w the image is transformed after applying dieren t
con v olutions k ernels that highligh t edges in the image. This transformation of the image
in to a feature space is simplifying the task of observing patterns in an analyzed scene.
Figure 3.10: Edge detection on a m ultisp ectral EO image.
New lo cal discriminativ e features, ha v e rstly b een dev elop ed for m ultimedia image
pro cessing and then their applicabilit y w as extended to remotely sensed image analysis.
The most p opular and widely used tec hniques are referred to scale in v arian t feature trans-
form [58], sp eeded up robust feature detector [5], rotation in v arian t feature transform [56],
census transform histogram [101] and lo cal binary pattern [69]. These tec hniques are us-
ing lo w lev el features to compute v ery discriminativ e and p o w erful image features that
can b e used for image matc hing, image retriev al and classi cation.
3.4 Mac hine learning for Earth Observ ation image data
analysis
As a consequence of the con tin uous gro wth of EO data collections, the dev elopmen t
of remote sensing data analysis to ols and metho ds is required. In most of the cases,
40
Metho ds and algorithms for Earth Observation image information mining
when dealing with large amoun ts of EO image data, the searc h for a sp eci c image or
phenomenon is a w ell kno wn problem. Assuming that the remote sensing image data is
random and ha v e a probabilit y distribution function dep enden t on some parameters of
in terest, w e can mo del the data trough the means of mathematical pattern recognition.
In pattern recognition, whic h is a branc h of mac hine learning and estimation theory ,
the atten tion is fo cused on recognition of patterns and regularities in data b eing ana-
lyzed using in most of the cases training datasets and sup ervised learning algorithms.
Also, sometimes, when no training data is a v ailable, the patterns can b e disco v ered using
unsup ervised learning metho ds.
There are t w o approac hes for solving pattern recognition problems, one in v olving
estimation theory in whic h the v alues of the parameters can b e determined based on em-
pirical measuremen ts that ha v e a random comp onen t and the other approac h is cen tered
on the syn tactical de nition of the phenomenon b eing analyzed.
A simple form recognition system is presen ted in Fig. 3.11. Because in most of the
cases the n um b er of extracted features is v ery big, a feature selection step is required. Also
to reduce the dimensionalit y of the feature space, some compression algorithms can b e
applied. In the feature classi cation step, v arious sup ervised or unsup ervised algorithms
can b e used to group the features according to sp eci c needs. F urthermore, a decision
regarding the classi cation ha v e to b e made.
Figure 3.11: F orm recognition system
3.4.1 F eature extraction and data represen tation
A ccording to [76], image analysis is concerned with the extraction of measuremen ts,
data or information from an image b y automatic or semiautomatic metho ds and is dis-
tinguished from other t yp es of image pro cessing, suc h as co ding, restoration, and en-
hancemen t, in that the ultimate pro duct of an image analysis system is usually n umerical
output rather than a picture. The eld of image analysis has b een named b y v arious
authors as image data extraction, automatic photo in terpretation, image understanding,
scene analysis or image description [65].
T exture analysis
By computing lo w-lev el features w e obtain the building blo c ks of more complex fea-
tures, lik e textures and shap es. Ev en though there are v arious de nitions of texture, most
of them are referring to h uman p erception of the feel or the app earance of ob jects. Th us,
ev ery one has their o wn in terpretation of texture and there is no mathematical de nition
of texture, as is explained in [67]. When extending this concept to image pro cessing, the
texture can b e de ned as a group of pixels that is c haracterized b y prop erties suc h as
rep etitiv eness, directionalit y , roughness, smo othness, etc [73].
41
Chapter 3. Data r epr esentation and data analysis
Ha ving in to accoun t the fact that there is no univ ersal de nition for texture, in or-
der to dev elop algorithms for texture feature extraction it is required to establish some
in v arian t measuremen ts suc h as p osition, scale and rotation. Th us, the p osition in v ari-
ance assumes that the measuremen ts of texture should not v ary with the p osition of
the analyzed image section [66]. Also scale and rotation in v ariance can b e imp ortan t in
determining texture prop erties of ob jects, but they are not mandatory .
The most kno wn statistical approac h for texture analysis is the grey lev el co-o ccurrence
matrix (GLCM), prop osed in [41], it w as the rst approac h to describ e and classify the
texture in an image. Haralic k's GLCM approac h assumed that texture can b e represen ted
using pure statistical description, while in [15] is suggested that for a b etter description
of texture is advisable to com bine geometrical structures with statistical ones, lik e it is
prop osed in the Statistical Geometric F eatures metho d.
It is kno wn that most of the EO data analysis tec hniques are based on m ultimedia
image pro cessing metho ds. Ev en so, there are attempts of using statistical text mo deling
approac hes [1], suc h as author-topic-mo del [59] and author-genre-topic-mo del [60]. These
metho ds are using laten t Diric hlet allo cation to treat the topic mixture parameters as
v ariables dra wn from a Diric hlet distribution [59]. Also, [18], [17] and [16] presen t new
tec hniques based on libraries of pretrained part detectors used for midlev el visual elemen ts
disco v ery in VHR remote sensing images [28].
Sp ectral analysis
Sp ectral analysis metho ds are based on the pure sp ectral or color information enco ded
in a digital image and ha v e b een used since the early y ears of image pro cessing. In the anal-
ysis pro cess, the color is exploited in a particular color space or mo del. Some of the color
spaces used in image pro cessing are R GB (red-green-blue), HSV (h ue-saturation-v alue),
LUV ( ligh tness L with c hrominances U and V) and HMMD (h ue-max-min-dierence).
F urthermore, no matter the color space used, sp ectral features are not robust to signi -
can t app earance c hanges in the image due to the lac k of spatial information. Also, the
main adv an tage presen ted b y the sp ectral analysis tec hniques is represen ted b y the ease
of computation when compared with texture and shap e features, and are used on a large
scale in scene classi cation and con ten t based image retriev al applications.
Some of the color features mostly used in remote sensing image analysis are color
histograms [96] and color momen ts [45]. Other metho ds are based on color coherence
v ectors [72], color correlograms [44] and ev en on the dynamic color distribution en trop y of
neigh b orho o ds [2]. F urthermore, in EO image analysis, sp eci c feature extraction metho ds
based on the sp ectral indexes [62] ha v e emerged due to the high sp ectral resolutions
pro vided b y the m ultisp ectral remote sensing sensors.
Bag of W ords framew ork
Curren tly ev olving texture analysis and lo cal feature extraction tec hniques ha v e led
the w a y to the Bag of W ords (BoW) metho d. Ev en though BoW w as initially used for
video searc h, a lot of deriv ate metho ds that emerged from it could solv e problems lik e
image classi cation, image retriev al and ob ject recognition.
In the remote sensing comm unit y this tec hnique has b een recen tly in tro duced for
42
Metho ds and algorithms for Earth Observation image information mining
image annotation, ob ject classi cation, target detection and land use classi cation, and it
has already pro v en its discrimination p o w er in image classi cation [22], b y p erforming a
v ector quan tization of the sp ectral descriptors in an image against a visual co deb o ok. In
the BoW framew ork, there are sev eral w a ys to generate the visual co deb o ok. Ev en though
K-means is the most common clustering pro cedure used [96], there are some attempts
in using random dictionaries [22]. Dep ending on the features used for the co deb o ok
generation, dieren t classi cation results ma y b e obtained.
3.4.2 F eature selection
In data mining applications, feature selection is a v ery imp ortan t step due to the
fact that real data sets often ha v e high-dimensional features. Man y existing feature se-
lection metho ds rank features b y optimizing certain feature ranking criterions, suc h that
correlated features often ha v e similar rankings [97].
In most of the cases, after feature extraction step is completed, the n um b er of features
obtained is v ery large. The feature selection step of a classi cation system is needed to
reduce the computational complexit y b y minimizing the n um b er of features. Also, in the
case of t w o feature v ectors that carry go o d information for the classi cation algorithms
when are tak en separately , sometimes they do not impro v e the classi cation when are
tak en together [92]. Th us, feature selection is p erformed in order to select relev an t fea-
tures from the high-dimensional space, pla ying an imp ortan t role in man y scien ti c and
practical applications, b ecause it can sp eed up the learning pro cess, impro v e the mo de
generalization capabilit y and decrease the algorithm running time in real applications
[97].
There are three dieren t t yp es of feature selection metho ds: em b edded metho d [10],
[9], [93] , lter metho d [47], [51], [79] and wrapp ed metho d [50]. In em b edded metho ds,
feature searc h and classi cation mo del is incorp orated in to a single optimization problem,
the lter metho ds can b e seen as a particular case of the em b edded metho ds, in whic h
feature selection is used as a prepro cessing while for wrapp ed metho d ev en though usually
ha v e go o d p erformance, are using classi cation results to select features.
No matter the approac h used for feature selection, all the metho ds are using dieren t
w a ys to rank features, suc h as score function, classi cation results, w eigh ts from the mo del
parameter matrix [97]. This means that correlated features will ha v e similar rankings,
and can b e considered redundan t. Th us, in the case of redundan t features, no extra
information will b e made a v ailable to the classi er.
3.4.3 F eature classi cation
Classi cation is the pro cess b y whic h w e attribute a class lab el to a set of measure-
men ts [67] and if the desired output consists of one or more con tin uous v ariables, then this
pro cess is referred to as a regression [6]. Moreo v er, the applications in whic h the training
data comprises examples of the input v ectors with their corresp onding target v ectors are
kno wn as sup ervised learning problems.
In other classi cation problems, the training data consists of a set of input v ectors
that do es not ha v e an y corresp onding target lab els. In this case, w e deal with an un-
43
Chapter 3. Data r epr esentation and data analysis
sup ervised learning problem, in whic h the goal is to disco v er groups of similar examples
within the data. Suc h unsup ervised learning problem is called clustering. F urthermore,
if the goal is to determine the distribution of data within the input space, the problem
is called densit y estimation. Also, if w e desire to pro ject a high-dimensional space do wn
to t w o or three dimensions ha ving the goal to obtain a visual represen tation of the data
distribution, w e deal with a visualization problem.
Ev en if w e w an t to solv e a sup ervised or unsup ervised learning problem, the classi er
function is to decide in whic h class the training data will b e assigned. Fig. 3.12 illustrates
a simple comparison b et w een sup ervised and unsup ervised learning metho ds, where the
same data is group ed according to a certain class, in the case of sup ervised learning, or
in a cluster, in the case of unsup ervised learning.
Figure 3.12: Sup ervised vs. unsup ervised learning
Moreo v er, the classi cation tec hniques can b e group ed dep ending on the statistical
prop erties of the data in to parametric and nonparametric classi cations. In the case of
parametric classi cation it is assumed that some of the statistical prop erties of the data
ar kno wn, while for nonparametric tec hniques no assumption on the b eha vior of the data
is made.
Common classi cation algorithms are using minim um distance, maxim um lik eliho o d,
Mahalanobis distance, parallelepip ed algorithm, exp ectation-maximization and so on.
These algorithms are simple and can p erform v ery fast. In complex classi cation prob-
lems, lik e in remote sensing image classi cation, the use of classical approac hes do es not
pro vide b est results.
F urthermore, new classi cation algorithms based on fuzzy logic ha v e emerged. Ex-
p erimen ts ha v e sho w ed that fuzzy algorithms can deal with mixed pixels and impro v e
the accuracy of the classi cation [102]. Some of the most kno wn fuzzy classi cation algo-
rithms are fuzzy equal relationship metho d, fuzzy ISOD A T A metho d, fuzzy syn thesized
judgemen t metho d, fuzzy language metho d. Another sp ecial category of classi cation
algorithms is using neural net w orks, suc h as m ulti-la y er p erceptron net w ork (MLPN), ra-
dial basis function neural net w ork, fuzzy self organizing neural net w ork and so on. These
classi cation algorithms are more complex, but pro vide enhanced accuracies of classi ed
data.
44
Metho ds and algorithms for Earth Observation image information mining
3.5 Data understanding and in terpretation
In EO data understanding and in terpretation, dep ending on the problem that has to
b e solv ed, w e ma y c ho ose b et w een optical, SAR images and ev en data fusion pro ducts.
In most of the cases, there are sp eci c tasks that requires either optical or SAR data.
In Fig. 3.13 is presen ted a m ultisp ectral image, acquired with Pleiades satellite
sensor, ha ving four sp ectral bands and 0:5m spatial resolution, that co v ers Naples region
of Italy . F urthermore, this gure presen ts ho w the extracted features are analyzed from
pixel lev el, to seman tic lev el. As explained in a previous section, the feature extraction
pro cess is the most basic op eration required in automation of image analysis. After the
image features are extracted, the analysis adv ances to iden ti cation of ob jects or regions
that will pro vide imp ortan t information for a scene lev el understanding.
Figure 3.13: Data understanding and in terpretation from pixel lev el to seman tic lev el.
A t a rst lev el of analysis, a user will rely on its kno wledge and exp erience in order
to extract information from a remotely sensed image. If there is a necessit y to extract
information from a h uge collection of Earth Observ ation images, the skills of the image
analyst will b e exceeded. In order to solv e this problem, automatic or semiautomatic
algorithms for feature extraction, image indexing and classi cation ha v e emerged.
It is kno wn that the t yp e of analyzed image, either optical or SAR, and its spatial and
radiometric resolutions ma y in uence the data understanding and in terpretation pro cess,
for b oth h uman and computer based analysis. Th us, same ob jects will ha v e dieren t
represen tations for dieren t acquisition mo des, spatial and radiometric resolutions of the
remotely sensed images. In Fig. 3.14 can b e observ ed a comparison b et w een dieren t
satellite images with spatial resolutions ranging from 30m to0:5m , co v ering the same
area o v er a region in Constan ta, Romania.
The spatial resolution ha v e a great impact on ho w details are p erceiv ed. In lo w er
resolution images w e can obtain information regarding the landco v er, while in high reso-
lution w e can idendify building and industry structures. Also in Sen tinel-1 SAR image,
the details are not as expressiv e as in the Sen tinel-2 image, ev en though the images ha v e
the same spatial resolution. F urthermore, in the SAR images, in b oth Sen tinel-1 and
T erraSAR-X, the structures with high re ectivit y/bac kscatterer are highligh ted.
F rom Fig. 3.15 w e can conclude that dep ending on the analyzed image w e can apply
certain algorithms to highligh t sp eci c image features.
45
Chapter 3. Data r epr esentation and data analysis
Figure 3.14: Comparison b et w een dieren t t yp es of remotely sensed data, at dieren t spatial resolutions and acquisition
mo des.
Figure 3.15: Comparison b et w een dieren t t yp es of remotely sensed data, at dieren t spatial resolutions and acquisition
mo des.
3.6 Conclusions
In this c hapter, the goal w as to pro vide a b etter understanding of the remote sensing
lifecycle, starting from data acquisition and image formation and con tin uing with feature
extraction and kno wledge extraction. Also in this c hapter w as presen ted ho w remote
sensing images can b e mo deled using estimation theory in order to generate seman tic
46
Metho ds and algorithms for Earth Observation image information mining
kno wledge.
47
Chapter 4
F eature extraction methods and
algorithms for Earth Observ ation data
understanding
In the con text of fast gro wing EO data arc hiv es, it is imp ortan t to ha v e at disp osal
a large v ariet y of to ols that can extract the maxim um relev an t information from images.
Also, with con tin uous c hanges in v olume and div ersit y , information mining has pro v en
to b e a di cult, y et highly recommended task [95] . The rst and p erhaps the most
imp ortan t part of the pro cess b eing data represen tation.
In most of the metho dologies for data represen tation, sp eci c classes in the image
b eing analyzed are group ed according to some dominan t c haracteristics lik e coarseness,
con trast, color distribution or directionalit y . Ha ving in to accoun t the div ersit y of prop-
erties that an image has, feature extraction metho ds sensible to sp ectral, texture, and
shap e information ha v e b een prop osed. The extracted features are usually stored in to
m ultidimensional feature v ectors.
As it w as presen ted in the previous c hapter, due to the fact that remote sensing
data is measured within sp eci c w a v elength in terv als, there is no general rule that can
b e applied to create a univ ersal information retriev al pro cedure regardless of the data
b eing analyzed. In most of the cases w e m ust use sp eci c algorithms for sp eci c t yp es
of data. F urthermore, in the con text of image indexing, the metho ds used are based on
iden ti cation and classi cation of image texture, image in tensit y or b y using statistical
mo dels. The results are then group ed in a few generic classes (3-6) lik e crops, buildings,
streets, v egetation, forest etc. [75]
Ev en though there are man y approac hes for primitiv e feature analysis for con ten t
classi cation that are computed at the pixel lev el, our atten tion is fo cused on larger
areas. Th us, in our approac h w e consider patc h-based feature extraction metho ds and
algorithms, that can pro vide useful information regarding a basic seman tic in terpretation
of the scene.
T o classify EO images, the pro cessing c hain w e prop ose requires that steps lik e patc h
extraction, image feature extraction and classi cation to b e completed. A t the b eginning,
a set of lo cal descriptors is extracted to p ortra y the image con ten t. The image prepro cess-
ing step assumes that the analyzed image is geometrically and radiometric correct. Also,
48
Metho ds and algorithms for Earth Observation image information mining
in this step, the data is mo di ed with certain transformations, according to the features
w e w an t to extract and then is cut in to image patc hes of con v ey able size.
In this c hapter, w e presen ted our con tribution regarding patc h-based feature extrac-
tion metho ds for sp ectral, texture and shap e analysis. In the frame of sp ectral feature
descriptors, w e prop ose new metho ds based on the sp ectral histogram, sp ectral indexes
and p olar co ordinates represen tation of the sp ectral v alues, while for texture features, w e
fo cused on Gab or, W eb er Lo cal Descriptors and Edge Histogram Descriptors. Also, in
the con text of dev elopmen t of new image feature descriptors, w e prop ose a mo di ed bi-
nary robust indep enden t elemen tary features descriptor (mo di ed BRIEF), that is able to
describ e the patc h con ten t. Moreo v er, using a bag of w ords (BoW) represen tation of the
feature space w e dev elop ed new BoW metho ds based on dictionaries computation from
sp ectral features, sp ectral indices and the p olar represen tation of the sp ectral v alues.
4.1 Sp ectral analysis metho ds
The image features based on the sp ectral v alues of the images are e cien t and easy to
compute, compared with other feature extraction metho ds, b eing used on a large scale in
scene classi cation and con ten t based image retriev al applications. The most imp ortan t
adv an tages of sp ectral features are the simplicit y of extracting color information from
images and the p o w er of represen ting visual con ten t.
Our con tributions regarding the analysis of the sp ectral comp onen ts in the remote
sensing images are materialized in to the dev elopmen t of descriptors based on the sp ectral
histogram, sp ectral indices and on the computation of the sp ectral v alues using a p olar
co ordinates transformation.
4.1.1 Sp ectral Histogram
The Sp ectral Histogram is one of the most frequen tly used and basic descriptors
that c haracterize the sp ectrum distribution in an image. Being inspired from the color
histogram, w e extend this terminology to the high sp ectral resolution of a m ultisp ectral
image. A generic Sp ectral Histogram (SH) descriptor [32] should b e able to capture the
sp ectral v alues distribution for image searc h and retriev al applications with resonable
accuracy [61].
By de nition, a generic histogram of a gra y lev el image can b e de ned as an appro x-
imation of the probabilit y densit y of pixel v alues. Considering a gra y lev el image f(x;y) ,
ha vingM ro ws andN columns, the histogram v alue for a gra y lev el k (k= 1;::;L , where
L is the n um b er of gra y lev els in the image) is presen ted in Equation 4.1, in whic h the
statemen t (f(x;y) ==k) will ha v e the v alue 1 if its true and 0 if its false.
h(k) =1
MNMX
y=1NX
x=1(f(x;y) ==k) (4.1)
Fig. 4.1 illustrates a m ultisp ectral image with its band decomp osition and asso ciated
histograms for eac h band. As it can b e observ ed, the sp ectral v alues are distributed on all
49
Chapter 4. F e atur e extr action metho ds and algorithms for Earth Observation data understanding
the gra y lev el in terv al. This suggest that image patc hes of a sp eci c thematic class will
ha v e a distinct sp ectral histogram signature, that will help in the classi cation pro cess.
Figure 4.1: W orldView-2 m ultisp ectral image and asso ciated sp ectral histograms for eac h sp ectral band.
In our approac h, w e compute the sp ectral histogram at a patc h lev el, considering
that the analyzed patc h is encapsulating the seman tics w e w an t to extract. In Fig. 4.2
w e presen t a few generic classes and the sp ectral histogram descriptor asso ciated with
these classes. Also, it can b e easily observ ed that at a patc h lev el the sp ectral signatures
distribution is v ery sp eci c for dieren t thematic classes.
Figure 4.2: Example of thematic classes and asso ciated Sp ectral Histogram feature v ector
F or this simple, but e cien t image descriptor, w e compute for eac h band of a m ulti-
sp ectral image patc h the Sp ectral Histogram with hb n um b er of histogram bins for eac h of
thenb n um b er of bands of the image, obtaining in this w a y a feature v ector with the size
ofhbnb . The feature v ector that is computed this w a y is represen ting a concatenated
sp ectral histogram.
4.1.2 Sp ectral Indices
In remote sensing, sp ectral indices are refering to the com binations of surface re-
ectance from m ultiple sp ectral bands to highligh t a sp eci c feature that indicates the
presence of v egetation, w ater, m ud, ice, geologic co v erage, etc. In computation of the sp ec-
tral indices, simple algebraic form ulations, lik e sums, dierences and ratios are applied
b et w een dieren t band com binations.
Sp ectral indices can b e used in applications where information ab out the terrain co v-
erage is needed. Some of the sp ectral indices are pro viding information ab out v egetation,
lik e Simple Ratio (SR) or Ratio V egetation Index (R VI), Normalized Dierence V ege-
tation Index (ND VI), P erp endicular V egetation Index (PVI), Soil A djusted V egetation
50
Metho ds and algorithms for Earth Observation image information mining
Index (SA VI), A tmospherically Resistan t V egetation Index (AR VI), etc. In other cases,
sp ectral indices are pro viding geological information ab out the analyzed surface (Cla y
Ratio – CR; F errous Minerals Ratio – CMR; Iron Oxide Ratio – IOR) and other sp ectral
indices can detect burned land b y using the red and near-infrared sp ectrum (Burn Area
Index – BAI; Normalized Burn Ratio – BNR).
Using all the m ultisp ectral attributes, the sp ectral indices are a sp ecial category of
image features that can b e applied on m ultisp ectral images only . In our approac h, w e
consider the radiance v alues for eac h band and all the p ossible (b1 b2)=(b1 +b2) band
ratios, where b1 andb2 refer to dieren t band com binations, the n um b er of computed
sp ectral attributes is nb(nb+1)=2 , considering nb is the n um b er of bands in a m ultisp ectral
image, as describ ed in [62]. This leads to a v ery fast and easy to compute feature descriptor
for m ultisp ectral images.
F or our patc h-based feature extraction emplo ying the Sp ectral Indices computation,
w e mo di ed the descriptor to use the same n um b er of features as the original one, pre-
sen ted in [62]. F urthermore, w e compute the feature v ector with the size nb(nb+ 1)=2 to
b e the mean of all Sp ectral Indices v alues from within the analyzed patc h. By doing so,
w e obtain a general represen tation of the sp ectral con ten t of the patc h.
In Fig. 4.3 can b e observ ed ho w dieren t classes are represen ted using a patc h-based
approac h of sp ectral indices. The rst ratios that comp oses the descriptor ha v e bigger
v alues that the last ratios. This fact is a consequence of the high correlation b et w een the
sp ectral bands. Also, ev en though some of the feature v ectors ha v e similar app earance,
the v alues that are represen ted are v ery dieren t.
Figure 4.3: Example of thematic classes and asso ciated Sp ectral Indices feature v ector
4.1.3 P olar co ordinates based features
Most of the tec hniques used in the m ultisp ectral image analysis and understanding
are based on iden tifying the general c haracteristics of the pixels or image patc hes b eing
analyzed. One of the con v en tional w a ys of represen ting m ultisp ectral data is to plot
the image features in a m ultisp ectral v ector space with as man y dimensions as there are
sp ectral comp onen ts [70]. Ha ving the m ultisp ectral features space as a reference system,
in [70] is presen ted an impro v ed m ultisp ectral image analysis using p olar co ordinates.
51
Chapter 4. F e atur e extr action metho ds and algorithms for Earth Observation data understanding
Figure 4.4: Left: Angular vs. Euclidean Distance in feature space. Righ t: P olar vs Cartesian Represen tation of R GB
feature space
P olar co ordinates features
In pap er [90], the authors are relying on the Sp ectral Angular Distance for measuring
distances in feature space and also to compute m ultisp ectral image classi cation and
clustering. The angular distance is pro v en to b e in v arinat to linearly scaled v ariations
of the image features. If t w o sp ectral signatures de ned b y the feature v ectors V1 and
V2 are considered, the sp ectral distance (euclidean distance) can b e extressed using (4.2)
and represen t the length of the segmen t d=V1V2 that connects the end p oin ts of the
t w o v ectors, while in the case of sp ectral angular distance (4.3) is giv en b y the angle
b et w een the t w o v ectors as it can b e observ ed in Fig. 4.4.
d=p
(r2 r1)2+ (g2 g1)2+ (b2 b1)2 (4.2)
= arccos !V1 !V2
k !V1kk !V2k(4.3)
Using the p olar co ordinates transformation of the image features (Fig. 4.4, righ t),
the authors of [70] ha v e succeeded to obtain an illumination in v arian t descriptor. This is
pro v en to b e of high imp ortance when classifying regions with p o w erful shado w and cloud
co v erages presen t in m ulti-sp ectral EO images.
In our approac h w e con v ert the radiances v alues of eac h sp ectral band from the
m ultisp ectral image in to p olar co ordinates and c haracterize the image trough a distance
andN 1 angles as presen ted in equations (4.4)-(4.8).
=q
x2
N+x2
N 1+:::+x2
2+x2
1 (4.4)
1= arctanq
x2
N+x2
N 1+:::+x2
2
x1(4.5)
2= arctanq
x2
N+x2
N 1+:::+x2
3
x2(4.6)
. . .
N 2= arctanq
x2
N+x2
N 1
xN 2(4.7)
52
Metho ds and algorithms for Earth Observation image information mining
N 1= 2 arctanxN
xN 1+q
x2
N+x2
N 1(4.8)
In Equations (4.4)-(4.8), N represen t the n um b er of the sp ectral bands presen t in
the m ulti-sp ectral image while the terms x1;x2;:::;xN are referring to the radiance v alues
of the band i , wherei=1::N .
Using the con v ersion of the radiance v alues in to p olar co ordinates w e aim to create
a syn thetic m ultisp ectral space that can b e used in the SCD computation, as presen ted
in the MPEG-7 standard.
Scalable color descriptor from p olar co ordinates
This new approac h of scalable color descriptor (SCD), is an illumination in v arian t
descriptor, based on p olar co ordinates transformation of the sp ectral v alues. The p olar
feature space w e create will pro vide the necessary supp ort to compute in a high dimen-
sional sp ectral space the scalable color descriptor describ ed in the MPEG-7 standard.
In the default setup of SCD, the sp ectral space is transformed from R GB (Red-Green-
Blue) to HSV (Hue-Saturation-V alue). In the case of high dimensional m ultisp ectral data,
this transformation is not p ossible. In our approac h, w e describ e the sp ectral v alues using
the p olar co ordinates v alues, represen ted b y a magnitude and a couple of angles . This
represen tation will pro vide a sp ectral space that is b oth illumination in v arian t and HSV
complian t.
As describ ed in [61], in our approac h w e consider a uniform quan tization of the p olar
co ordinates color space to 256 bins. F urthermore, w e compute the histograms in the
transformed space on whic h w e apply the Haar transform as is presen ted in the MPEG-7
sp eci cation.
Figure 4.5: Example of thematic classes and asso ciated p olar SCD feature v ector
The feature v ector presen ted in Fig. 4.5 suggest a high p oten tial in separating features
in to classes. Results of the classi cation of p olar SCD will b e presen ted further.
4.1.4 Ev aluation and Discussion
Sp ectral features are fast and easy to compute. This is the main reason wh y a lot
of con ten t based image retriev al applications are using features based on the color and
53
Chapter 4. F e atur e extr action metho ds and algorithms for Earth Observation data understanding
sp ectral distribution in images. Unfortunately , the sp ectral features main disadv an tage is
that it lac ks an y spatial information.
Due to the dep endency on the color information, the sp ectral features ma y not pro vide
satisfactory results on single band images, lik e panc hromatic and SAR data. In Fig. 4.6
are presen ted the results of a k-Means classi cation in to 5 thematic classes. In the frame
of feature extraction, for all the assessed images w e used square patc hes that co v ers a
ground surface of 400m2.
Figure 4.6: K-Means ev aluation of sp ectral features for dieren t EO data.
As it can b e seen in Fig. 4.6, the sp ectral features are pro viding a go o d separation in
thematic classes ev en for the single band images in whic h the color information is replaced
with the maxim um re ectance v alues in the case of panc hromatic images and signal mag-
nitude v alues for SAR data. Sp ectral indices and p olar-SCD cannot b e computed for
panc hromatic and SAR images, b ecause their computation needs at least three sp ectral
bands.
4.2 T exture analysis metho ds
T exture analysis goal is to quan tify in tuitiv e qualities of an image describ ed b y terms
suc h as rough, smo oth, silky , or bump y as a function of the spatial v ariation in pixel gra y
lev els. The metho ds for texture analysis can b e helpful when ob jects in an image are more
c haracterized b y their texture than b y in tensit y , and traditional thresholding tec hniques
cannot b e used eectiv ely .
In our approac h of texture analysis, w e computed Homogeneous T exture Descrip-
tors (HTD) or Gab or, W eb er Lo cal Descriptors (WLD) and Edge Histogram Descriptors
(EHD) in a patc h-based approac h. These descriptors are used in m ultimedia image pro-
cessing for texture extraction and w e extend their functionalities for remote sensing image
analysis. F urthermore, Gab or and EHD are presen t in the MPEG-7 standard.
54
Metho ds and algorithms for Earth Observation image information mining
4.2.1 Homogeneous texture descriptor – Gab or features
Based on a w a v elet transform, m ulti scale Gab or lter is amongst the most used
texture descriptors, b eing describ ed in the MPEG-7 standard as a homogeneous texture
descriptor (HTD) [89]. The Gab or represen tation has b een pro v en to b e optimal in the
sense of minimizing the join t t w o-dimensional uncertain t y in space and frequency [27],
b eing w ell suited for texture detection and classi cation.
Considering the texture c haracteristics and other related studies, one can conclude
that the h uman visual system resp onds to texture prop erties suc h as rep etition direction-
alit y and complexit y [38], so the 2D Gab or lter [71] can b e expressed using Equation 4.9,
where is the w a v elength of the sin usoidal factor, is the orien tation angle (in radians),
' is the phase oset, is the standard deviation and
represen ts the scale of the lter
[34].
g;;';;
(x;y) =e x02+
2y02
22cos (2x0
+') (4.9)
=1
r
ln 2
22b+ 1
2b 1(4.10)
b= log2
+q
ln2
2
q
ln2
2(4.11)
x0=xcos+ysin
y0= xsin+ycos(4.12)
In Gab or lter computation can b e used dieren t parameter setups, but for b est
results, most of the authors are using 2-6 frequencies and 2-8 orien tations, as presen ted
in [30], [28] and [82]. In Fig. 4.7 can b e seen an example of a Gab or lter with 4 dieren t
orien tations =0o,45o,90oand135oand the scale parameters x=y= 3 pixels. [54]
Figure 4.7: Gab or lter example in spatial domain, with 4 orien tations and 1 scale.
An example on Gab or ltering can b e observ ed in Fig. 4.8, where is computed a
Gab or con v olution for a single scale and 4 orien tations , as presen ted in Fig. 4.7, that
aects the rst band of a m ultisp ectral image (Fig. 4.8 – T op) and the magnitude of a
SAR image (Fig. 4.8 – Bottom).
The computation of Gab or features is similar for eac h t yp e of analyzed EO image, no
matter if its SAR, m ultisp ectral or a data fusion pro duct, the main dierence b eing in the
size of the feature v ector whic h is dep enden t on the n um b er of sp ectral bands. Fig. 4.9
illustrates ho w the Gab or feature v ector lo oks lik e for a m ultisp ectral image with eigh t
sp ectral bands, computed for = 6 orien tations and '= 4 scales. This means that w e
will ha v e to lter eac h sp ectral band for ev ery parameter com bination. F or eac h computed
55
Chapter 4. F e atur e extr action metho ds and algorithms for Earth Observation data understanding
Figure 4.8: Gab or ltering applied on the rst band of a m ultisp ectral image, and on the magnitude of a SAR image, using
four orien tations
patc h w e extract the mean and standard deviation and k eep them as Gab or features. The
size of a Gab or feature v ector computed for a m ultisp ectral image patc h with nb n um b er
of bands will ha v e the size 'nb2 .
Figure 4.9: Example of thematic classes and asso ciated HTD feature v ector
4.2.2 W eb er Lo cal Descriptor
The W eb er's la w (Ernst W eb er) state that the barely p erceptible dierence b et w een
t w o stim ulus is prop ortional with the stim ulus amplitude [14]. In Equation 4.13, I rep-
resen ts the barely p erceptible dierence of t w o stim ulus, I represen ts the initial in tensit y
of the stim ulus and k meaning is that the ratio sta ys constan t regardless of I v ariations.
This equation is kno wn as W eb er ratio. [20]
I
I=k (4.13)
WLD is a lo cal descriptor based on the fact that h uman p erception of a texture
dep ends on the stim ulus c hange and its in tensit y and is describ ed b y t w o comp onen ts:
dieren tial excitation (Equation 4.16) and orien tation (Equation 4.17).
v00
s=p 1X
i=0xi=p 1X
i=0xi xc (4.14)
Gratio(xc) =v00
s
v01
s(4.15)
56
Metho ds and algorithms for Earth Observation image information mining
(xc) = arctanv00
s
v01
s= arctanp 1X
i=0xi xc
xc(4.16)
(xc) = arctan 2v11
s
v10
s(4.17)
v10
s=x5 x1 andv11
s=x7 x3 (4.18)
The equations 4.16 and 4.17 are used only for optical data. In the case of SAR images
is computed an adapted WLD that tak es in to accoun t the ratio of the mean of t w o no-
o v erlapping neigh b ourho o d on the opp osite sides of the p oin t xc b eing analysed [19]. T o
detect all p ossible edges, the ratio detector m ust b e applied in all p ossible directions (d)
[20]. By taking in to accoun t those c hanges, the dieren tial excitation is computed using
Equation 4.19 and the orien tation b y using Equation 4.20, where 1 and2 are the lo cal
means,d is the n um b er of directions considered.
(xc) = arctandX
j=0dX
i=1i
j xc
xc(4.19)
(xc) = arctan 21
1 1
2
3
1 3
2(4.20)
In the top of Fig. 4.10 is presen ted the decomp osition of the rst sp ectral band of
a m ultimedia image in to dieren tial excitation and orien tation, using WLD equations,
while in the b ottom of the image can b e observ ed ho w the same features extracted using
Equations 4.19 and 4.20 are aecting a SAR magnitude image. Also, in the righ t of
Fig. 4.10 are illustrated the represen tations of the dieren tial excitation and orien tation
histograms for b oth t yp es of images.
Figure 4.10: Dieren tial excitations and orien tations of dieren t remote sensing data
The feature v ector obtained using WLD features, for a single patc h will b e dep enden t
on the n um b er of bands nb in the image, the n um b er of dieren tial excitation bins C and
the n um b er of orien tation bins T . Th us, the feature v ector size will b e nbTC . A
graphical represen tation of the WLD feature v ectors of sev eral thematic classes can b e
observ ed in Fig. 4.11 along with patc h samples of eac h class.
57
Chapter 4. F e atur e extr action metho ds and algorithms for Earth Observation data understanding
Figure 4.11: Example of thematic classes and asso ciated WLD feature v ector
4.2.3 Edge Histogram Descriptor
The edge histogram descriptor (EHD) is de ned in the MPEG-7 standard as a tex-
ture descriptor [100]. In this feature extraction metho d, the o ccurrence probabilit y of 5
orien tations, as presen ted in Fig. 4.12, is computed [78] for sev eral areas in the image.
Despite Gab or and WLD, there is no con v olution applied to this descriptor, the statistics
are computed directly on the pixels in tensities.
Figure 4.12: EHD orien tations
In our approac h, w e extract the patc hes in the image, and for eac h patc h w e com-
pute EHD. After eac h patc h is extracted, it is split in to 44 subregions, as it can b e
observ ed in Fig. 4.13. Eac h of the 44 subregions are further decomp osed in to smaller
regions on whic h is determined the dominan t orien tation. This is used in the histogram
computation for eac h subregion. Eac h of the resulting histograms of the 44 subregions
are concatenated, obtaining this w a y the edge histogram feature v ector.
Figure 4.13: Edge Histogram Descriptor region split
The size of the EHD descriptor for a single patc h it will b e EHDSize=nbpr ,
wherenb is the n um b er of bands in the image, r= 5 regions describ ed in Fig. 4.12
andr= 16 initial partitions of the patc h as it is illustrated in Fig. 4.13. The texture
describ ed using EHD at patc h lev el will b e a prop ert y of the patc h, and not a prop ert y of
the image as it w as initially designed in the MPEG-7 standard. In Fig. 4.14 is illustrated
the graphical form of the descriptor for a single patc h.
58
Metho ds and algorithms for Earth Observation image information mining
Figure 4.14: Example of thematic classes and asso ciated EHD feature v ector
4.2.4 Ev aluation and Discussion
In remote sensing imagery , the texture analysis is one of the most common tec hniques
for image information mining. Due to the EO data represen tation and b ecause the sensor
is p erforming measuremen ts from the top of the scene, the analyzed images presen t a
lac k of deepness information [35]. Ev en so, the phenomenons that are illustrated in
an EO image presen t a lot of c haracteristics that are useful in texture analysis suc h as
rep etitiv eness, roughness and smothness [43].
Dep ending on the analyzed images, some metho ds of texture feature extraction can
pro vide b etter results and can compute faster than others. Of course, the computation
time is hardly in uenced b y the complexit y of the algorithms used for the feature extrac-
tion pro cess.
A w a y to measure the p erformances of feature extraction metho ds is to p erform a vi-
sual insp ection of the results o v er the analyzed image. Unfortunately , this w a y of analysis
do es not pro vide an y information regarding the qualitativ e and quan titativ e assessmen t.
In this section w e will presen t ho w the texture features can b e separated in to clusters,
using unsup ervised k-Means for dieren t EO image data. F urthermore, only the k-Means
classi cation without p erforming an y qualitativ e and quan titativ e assessmen t.
Figure 4.15: k-Means classi cation of texture features extracted from a W orldView-2 m ultisp ectral image
Figure 4.16: k-Means classi cation of texture features extracted from a W orldView-2 panc hromatic image
The data sets b eing analyzed are co v ering the same area, but ha v e dieren t spatial
resolution. In order to mak e a visual comparison of the results, w e extracted texture
59
Chapter 4. F e atur e extr action metho ds and algorithms for Earth Observation data understanding
Figure 4.17: k-Means classi cation of texture features extracted from a T erraSAR-X image
features for dieren t patc h sizes that are co v ering the same area of 50m2in all three
analyzed images.
F or the W orldView-2 m ultisp ectral image, presen ted in Fig. 4.15, w e ha v e 8 sp ectral
bands and a spatial resolution of 2 meters, th us, w e used a patc h size of 2525 pixels.
In Fig. 4.16 and 4.17 are presen ted k-Means classi cation results for texture feature
extraction from panc hromatic and SAR images. In those t w o cases, the patc h is also
co v ering the same surface of 50m2, but the patc h size used for feature extraction is in this
case 100100 pixels, due to the spatial resolution of 0:5 meters.
As it can b e seen in Figs. 4.15 to 4.16 for the same category of features, dieren t
results are obtained. The size of the patc hes, n um b er of sp ectral bands and the correlation
b et w een bands, t yp e of the image, acquisition p erio d are just a few of the factors that
ma y in uence the classi cation pro cess.
A closer insp ection of what the three descriptors rev eals that in most of the cases
Gab or features, or HTD as it is referred in the MPEG-7 standard, pro vides the most
relev an t results, follo w ed b y WLD and EHD. This is exp ected b ecause of the m ultiple
scales and orien tations in v olv ed in the feature extraction pro cess. A cceptable results
are also obtained using WLD, whic h is computed diferen tly for the m ultisp ectral and
panc hormatic cases than for SAR images. In the case of EHD, for the m ultisp ectral and
panc hromatic images, the classi cation results pro vides information ab out orien tation
distributions within patc hes, while for SAR images the results are highly in uenced b y
the eects of the sp ec kle.
4.3 F eature p oin t descriptors
F eature p oin t descriptors are referring to sp eci c structures lik e p oin ts, edges or
ob jects, b eing the result of a neigh b orho o d op eration or feature detection applied on the
analyzed image. This t yp e of descriptors are able to matc h pairs of images v ery fast with
a reduced cost of used memory . [11]
Despite the classical approac h used for feature extraction, in the case of feature p oin t
descriptors, the goal is to enco de distinctiv e lo cal structure trough a collection of image
p oin ts that are used for image matc hing despite an y c hanges of the viewing conditions.
This sp ecial t yp e of image descriptors can b e used in a wide range of applications from
image matc hing and registration to 3D reconstruction and ob ject recognition.
60
Metho ds and algorithms for Earth Observation image information mining
4.3.1 Binary Robust Indep enden t Elemen tary F eatures
The Binary Robust Indep enden t Elemen tary F eatures (BRIEF), is a binary descriptor
and a v oids the use of dimensionalit y reduction b y computing binary strings from image
patc hes. The individual bits are obtained b y comparing the pairs of in tensit y of the c hosen
p oin ts from the considered image patc h. [11]
F or example, in Fig. 4.18 w e can observ e ho w classical BRIEF descriptors are used
to matc h Earth Observ ation image data based on the computed feature p oin ts, b eing able
to determine if a patc h is included in to a test image. Also in Fig. 4.18 is sho wn ho w the
BRIF descriptor is able to iden tify the feature p oin ts, mark ed with y ello w in the big image
and with green in the analyzed patc h, while the blue lines are the connections b et w een
the feature p oin ts iden ti ed in the t w o images.
Figure 4.18: Multisp ectral image matc hing using BRIEF descriptors
Extraction of BRIEF feature p oin t detectors relies on the computation of binary
strings, whic h is done using the Equation 4.21 to compute the binary test (p;x;y) for a
patc hp , for ev ery pixel lo cations x andy . Also in the binary test describ ed in Equation
4.21, the v alues of p(x) andp(y) are referring to the v alues of the pixels inside the patc h p
at lo cations x andy . F urthermore, the binary string ca b e con v erted in to a decimal v alue
using Equation 4.22, where fnD(p) is the decimal v alue of the binary string obtained for
nD distinct binary tests (pixel lo cations).
(p;x;y) =(
1; ifp(x)<p(y)
0; else(4.21)
fnD(p) =nDX
i=12i 1 (p;xi;yi) (4.22)
By c ho osing a n um b er of nD(x;y) lo cations it can b e uniquely iden ti ed a binary test
set [11] whic h is the feature descriptor computed for the ev aluated patc h. Also in [11]
61
Chapter 4. F e atur e extr action metho ds and algorithms for Earth Observation data understanding
the considered v alues for nD are128 ,256 and512 whic h yield go o d compromises b et w een
sp eed, storage e ciency and recognition rate.
4.3.2 Mo di ed BRIEF for fast EO classi cation
In the con text of EO image classi cation, w e prop ose a mo di ed BRIEF descriptor
that can b e used for patc h classi cation instead of image matc hing as describ ed in [11].
Unlik e the classical BRIEF approac h, in whic h feature p oin ts computation is required
for the analyzed scene, w e compute a mo di ed patc h-based BRIEF descriptor that will
c haracterize a full patc h instead a lo cal feature p oin t.
The binary descriptor, for our patc h based approac h, is formed using the mean in-
tensities dierences of regions as de ned in Fig. 4.19. Th us for eac h pair p(0;1) , the
sub-patc h 1 is compared with sub-patc h 0 and the v alue of the binary test is stored to
the binary string asso ciated with the patc h. In our example nD =16 whic h means that w e
use only 16 bits to describ e a patc h.
Figure 4.19: Sub-patc h computation for mo di ed BRIEF. The binary string is computed using the mean in tensities dierence
b et w een sub-patc h 0 and 1 for eac h represen tation.
The descriptor computation for a image patc h in v olv es the de nition of the 4
4 regions used for computing the mean in tensities dierences as sho wn in Fig. 4.19,
completing the binary string after eac h test and applying the Hamming distance b et w een
the analyzed patc hes.
By using the mean in tensities dierences instead of pixels in tensities w e succeeded to
compute the binary descriptors for the whole image v ery fast, and as it is exp ected, when
the patc h size increases, the computation time decreases. As similarit y measure used in
the k-Means classi er w e used the Hamming distance b et w een patc hes whic h is easy to
compute using a X OR op erator b et w een the binary strings.
4.4 Bag of W ords features framew ork
Dep ending on the extracted features, the p erformances of a classi cation ma y v ary .
Ev en though there are feature extraction metho ds that rely on texture, sp ectral and shap e
information, a lot of eort has b een made to reduce the complexit y of the algorithms. So,
tec hniques lik e Bag-of-W ords (BO W) emerged.
In the BO W framew ork, the image is considered an orderless collection of patc hes
whic h can b e extracted b y an in terest p oin t detector or can b e densely sampled. A c-
62
Metho ds and algorithms for Earth Observation image information mining
cording to [21], computing BO W features requires v e steps: feature detection, lo cal
feature extraction, dictionary learning, feature co ding and feature p o oling. In Fig. 4.20
is illustrated the BO W framew ork for feature extraction and classi cation.
Figure 4.20: The framew ork for Bag of W ords feature extraction
A ccording to the BoW mo del, a v ector quan tization of the descriptors in an image
against a visual co deb o ok is p erformed. Dep ending on the features used in the co deb o ok
generation, dieren t classi cation results ma y b e obtained.
Our approac h is based on BO W framew ork to dev elop new enhanced feature ex-
traction metho ds. W e used p o w erful image descriptors that c haracterize the sp ectrum
information of the EO data to compute BoW dictionaries. In the frame of dictionary
learning w e used k-Means clustering to group our features in to a sp eci ed n um b er of
w ords. A ccording to [22], the use of more than 100 w ords do es not impro v e the classi -
cation results.
Figure 4.21: Examples of seman tic meaning for BoW histograms
Some examples of thematic classes and their asso ciated BoW descriptor can b e seen
in Fig. 4.21. W e can remark that the w ords histogram of the patc hes within a certain
class ha v e unique signatures. This certify the fact that the ob ject separation can easily
b e done using BO W framew ork.
4.4.1 New BoW-based descriptors
Considering some of the features prop osed for sp ectral analysis, w e dev elop ed new
BO W based descriptors suc h as Bag of Sp ectral Indexes (BSI) and Bag of P olar Co ordi-
nates (BPC). Our goal w as to impro v e the feature extraction step of the BO W framew ork
b y pro viding b etter image descriptors. Also, kno wing that feature extraction and dictio-
nary learning steps are time exp ensiv e in BO W pro cessing c hain, our descriptors impro v e
63
Chapter 4. F e atur e extr action metho ds and algorithms for Earth Observation data understanding
the computation time, b eing fast and easy to compute, ev en when using EO images with
m ultiple sp ectral bands.
The metho ds w e prop ose, BSI and BPC, are designed to handle m ultisp ectral images
only , their purp ose b eing to optimize the pro cessing time of the BO W descriptors for high
dimensional m ultisp ectral images.
In the feature extraction step, BSI is extracting sp ectral indexes while BPC is fo cused
on pro cessing the p olar co ordinates v alues. As pro v ed in the previous section, the features
based on band ratios and p olar co ordinates computation are pro viding go o d results in a
shorter time ev en as stand alone descriptors.
Figure 4.22: BSI and BSV w ords visualization and histograms
In Fig. 4.22 is illustrated the generated dictionaries and the w ords distributions for
BSI and BPC descriptors, in a W orldView-2 scene.
4.5 F eature extraction assessmen t using mac hine learn-
ing algorithms
In the follo wing, our atten tion is fo cused on the classi cation algorithms w e used in
our data mining approac h, suc h as k -Means,k -Nearest Neigh b ors (k-NN) and Supp ort
V ector Mac hines (SVM). The classi cation algorithms w e presen t are b oth unsup ervised
(k -Means) and sup ervised ( k -NN and SVM).
k -Means
Being one of the simplest unsup ervised learning algorithms, k -Means can b e used to
solv e clustering problems. In its initial form, the algorithm w as prop osed b y Stuart Llo yd
in 1957 [42] to nd an ev enly spaced set of p oin ts in an Euclidean space. In the con text
of Llo yd's algorithm the input represen ts a con tin uous geometric space, while in k -Means
clustering the input is a discrete set of p oin ts. F urthermore, k -Means, corresp onds to a
nonprobabilistic limit of exp ectation maximization applied to mixtures of Gaussians [6].
k -Means clustering is solving the problem of iden tifying groups or clusters of data
p oin ts in a m ultidimensional space. Considering a data set X=fX1;X2;::;XNg , consist-
ing ofN observ ations of d -dimensional random v ariables the goal is to group the input
data in tok clusters [6]. F urthermore, a cluster can b e de ned as a group of data p oin ts
whic h ha v e the minim um in ter-p oin ts distances when compared with the distances to
64
Metho ds and algorithms for Earth Observation image information mining
p oin ts outside the group. In order to accurately iden tify the minim um distance b et w een
all the p oin ts in the clusters, w e need to in tro duce a new d -dimensional v ariable i , with
i= 1;::;k , whic h is asso ciated with the cen troid of i -th cluster. The goal is to nd the
optimal assignmen t of data p oin ts to clusters and the optim um set of v ectors i to mini-
mize the within-cluster sum of squares, as presen ted in Equation 4.23. In order to ac hiev e
this optim um a series of iterations ha v e to b e p erformed.
arg min
cNX
n=1kX
i=1kXn ik2(4.23)
In Algorithm 4.1 is presen ted the pseudo co de used for k-Means computation.The
algorithm ev en tually con v erges to a p oin t, although it is not necessarily the minim um of
the sum of squares. That is b ecause the problem is non-con v ex and the algorithm is just
a heuristic, con v erging to a lo cal minim um. The algorithm stops when the assignmen ts
do not c hange from one iteration to the next.
Algorithm 4.1 k-Means clustering pseudo co de
1: Setk . Num b er of clusters
2: Setit . Num b er of iterations
3: InitializeK
4: rep eat
5: forn 1;N do
6: fori 1;k do
7: Compute distance from Xn toi
8: end for
9: Assign p oin t Xn to the closest cluster, based on minim um distance
10: end for
11: fori 1;k do
12: Compute new cen troids i
13: end for
14: un til con v ergence . Con v ergence is considered when either the n um b er of iterations
is completed or the cen troids are no longer c hange their p osition
k -Nearest Neigh b ors
Being a non-parametric metho d, k -Nearest Neigh b ors (kNN) metho d is used for
classi cation and regression. Also kNN represen t a sup ervised learning metho d, whic h
requires a training set in order to classify new data, b eing based on computation of
distances to all the feature p oin ts in the training set, but selecting the class whic h has
the closest k neigh b ors.
In kNN classi cation, is solv ed the problem of determining the densit y of probabilit y
that a p oin t x has in order to b e included in a v olume V . In Equation 4.24 is computed
the probabilit y , that a sample x has to b e included in a v olume V .
=Z
Vp(x)dx (4.24)
65
Chapter 4. F e atur e extr action metho ds and algorithms for Earth Observation data understanding
F or a v ery small v olume, Equation 4.24 can b e written using Equation 4.25, whic h is
an appro ximation of the probabilit y densit y function using a n um b er of samples included
in the v olume V.
p(x)V (4.25)
Considering that k is the n um b er of samples from a total n um b er n of samples inside
the v olume V , then w e can rewrite Equation 4.25. In Equation 4.26, the probabilit y
b ecomes the ration b et w een the p ossible samples k and the maxim um n um b er of samples
n .
=k
n(4.26)
Using Equation 4.25 and 4.26 w e will obtain the probabilit y densit y ^p(x) , expressed
in Equation 4.27. Th us, kNN is nding the b est k neigh b ors of a p oin t x from a total of
n samples inside a v olume V .
^p(x) =k
nV(4.27)
In Algorithm 4.2 is a simple pseudo co de that can b e used to implemen t the kNN
classi cation metho d. As it can b e observ ed, a training set Xi and asso ciated classes
Yi is needed, ha ving i= 1;::;n , wheren is the n um b er of classes in the training set.
F urthermore, in the b eginning of the pro cessing, the n um b er of neigh b ors k is required.
Algorithm 4.2 k-Nearest Neigh b ors Classi cation pseudo co de
1: Setk . k – Num b er of neigh b ors
2: SetX . X =fX1;::;Xng – T raining data
3: SetY . Y =fY1;::;Yng – Class lab els
4: . n = Num b er of classes
5: function kNNClassify (X;Y;x ). Letx b e an unkno wn sample that need to b e
classi ed
6: fori 1;n do
7: Compute distance d(Xi;x) , fromXi tox
8: end for
9: Assign tox the class lab el Yi , whose distance to at least k neigh b ors is minim um
10: end function
Some of the distances that can b e used in selecting the optimal k Nearest Neigh b ors
are presen ted in T able 4.1
Supp ort V ector Mac hines
Supp ort v ector mac hines (SVM) is a sup ervised learning metho d, b ecoming a ref-
erence algorithm to solv e classi cation problems due to its exibilit y and computational
e ciency of handling m ultidimensional data. SVM metho ds ha v e similar functionalities
with other p opular tec hniques used in classi cation and data mining, lik e neural net w orks
66
Metho ds and algorithms for Earth Observation image information mining
T able 4.1: Distances in Lk metrics
Distane name F orm ula
EuclideanqPk
i=1(xi yi)2
ManhattanPk
i=1jxi yij
Mink o vskiPk
i=1jxi yijp1
p
(NN) and radial basis functions (RBF) [63]. Also, SVM can b e used to mo del real w orld
problems lik e text classi cation [48], facial expression recognition [64], gene analysis [40],
remote sensing image classi cation [84] and man y others.
Considering the case of separating the analyzed data, consisting of L training p oin ts,
referred asxi , in to t w oN -dimensional v ectorial data sets, yi= 1 or+1 , wherei= 1;::;L ,
SVM will generate a separation h yp erplane that will maximize the margin b et w een the
t w o classes. Assuming that the data is linearly separable, t w o separation h yp erplanes are
build and then pushed to w ards the t w o data sets, as it can b e observ ed in Fig. 4.23, in a
t w o dimensional represen tation. Also, the h yp erplanes are de ned b y the Equation 4.28,
wherew is the normal to the h yp erplane andb
kwkis the p erp endicular distance from the
h yp erplane to the origin.
wx+b= 0 (4.28)
Figure 4.23: Hyp erplane through t w o linearly separable classes
In tuitiv ely , a go o d separation is obtained b y the h yp erplane that ha v e the maxim um
distance from the elemen ts of the t w o classes. The v ectors that comp ose the h yp erplane
can b e c hosen to b e linear com binations of the Lagrange m ultipliers i , solving this w a y
the optimization problem expressed in Equation 4.29
max
LX
i=1 i 1
2 TH
whereH =yiyjxixj j and i08i andLX
i=1 iyi= 0 (4.29)
67
Chapter 4. F e atur e extr action metho ds and algorithms for Earth Observation data understanding
In the case of a m ulti-class SVM, determining optim um h yp erplane for class sepa-
ration is equiv alen t to solving the optimization problem presen ted in Equation 4.30, in
whic hw is the h yp erplane normal, is a slac k v ariable whic h allo ws a sample to exist
in the margin ( where 10 is the margin error) or outside the margin as a wrong
classi cation ( 1 ) andC0 is a margin maximization constan t.
min
w;b1
2kwk2+CLX
ii (4.30)
The inputs of the algorithm are the training sets (stored in to a database) and a
test set (the computed patc hes). In order to obtain go o d results the input data m ust b e
normalised. The Algorithm 4.3 is presen ting the pseudo co de of a SVM classi cation.
Algorithm 4.3 SVM Classi cation pseudo co de
1: SetX . X =fX1;::;Xng – T raining data
2: SetY . Y =fY1;::;Yng – Class lab els
3: function SVMClassify (X;Y;x ). Letx b e an unkno wn sample that need to b e
classi ed
4: Compute separation h yp erplanes – solv e optimization problem
5: Determine what class should b e assigned to x
6: end function
4.6 Ev aluation and Discussion on F eature extraction
In this section are presen ted applications of feature extraction metho ds for data
classi cation and kno wledge extraction. The scenarios w e presen t are referring to land
co v er classi cation using satellite imagery .
4.6.1 Enhancing feature extraction for m ultisp ectral Earth ob-
serv ation image classi cation
In order to impro v e the p erformances of the existing feature extraction metho ds,
w e prop ose joining HTD or WLD features computed on the pure texture band with the
Sp ectral Histogram features computed for all the sp ectral bands. W e generate the texture
band b y using an a v erage of the en tire sp ectral bands a v ailable in the m ultisp ectral image
[36].
Fig. 4.24, on the left side, sho ws the band correlations b et w een t w o of the visible
bands while the righ t side illustrates the correlation b et w een a visible and an infra-red
band of a W orldView-2 m ultisp ectral image. As w e can easily observ e, the visible bands
are highly correlated with eac h other, and strongly uncorrelated with the infra-red bands.
This ma y suggest that using the infra-red information to compute the image features will
help us dev elop more e cien t image descriptors that can represen t the analyzed regions
more accurately .
68
Metho ds and algorithms for Earth Observation image information mining
Figure 4.24: Correlated (costal-blue vs. blue band – left) and Uncorrelated sp ectral bands (coastal-blue vs. infrared-2 –
righ t) of a W orldView-2 image.
Figure 4.25: W orldView-2 m ultisp ectral scene (Left). Reference annotation (Righ t)
F or this assessmen t, w e consider the scene presen ted in the left side of Fig. 4.25, on
whic h the feature extraction is p erformed. Also, in the righ t side of the gure can b e
found the man ual annotation used qualit y and quan tit y ev aluation. The color co de used
for man ual annotation represen tation is explained in Fig. 4.26, where are presen ted some
patc hes of asso ciated classes.
Figure 4.26: Man ual annotation legend and represen tativ e patc hes for eac h class.
Enhancing Gab or features
Ha ving as a start p oin t the texture measuremen ts done b y computing the a v erage
texture band used for Gab or feature extraction, and con tin uing with Sp ectral Histogram
computation of the m ultisp ectral image, w e observ ed that the results of the classi cation
can b e impro v ed.
Also, the p ossibilit y of increasing p erformance of the texture or sp ectral descriptors
alone can b e easily seen from the band correlation images presen ted in Fig. 4.24. This
69
Chapter 4. F e atur e extr action metho ds and algorithms for Earth Observation data understanding
has led to the dev elopmen t of a Gab or-Histogram feature descriptor that computes the
texture using Gab or metho d only for the a v erage texture band and Sp ectral Histogram
for all the m ultisp ectral bands.
Figure 4.27: Left: Gab or descriptor results obtained for texture analysis of band mean. Cen ter: Histogram descriptor
computation for m ultisp ectral image. Righ t:Gab or-Histogram descriptor computation for m ultisp ectral image Gab or applied
for mean of bands and histogram for eac h band
During classi cation step w e used supp ort v ector mac hines (SVM) and k-Nearest
Neigh b ors. F or the SVM setup w e used radial basis function (RBF) k ernel that ha v e the
gamma co e cien t
= 3:051810 4and the regression parameter C= 5 . In the case
of k-NN classi cation w e used k= 10 neigh b ours. Also, the qualitativ e and quan titativ e
ev aluation w as made b y computing confusion matrixes using as reference the man ual
annotated map. In 4.27 w e can observ e the results of the sup ervised classi cation using
SVM and k-NN.
This new feature extraction metho d, based on b oth Gab or and Sp ectral Histogram
features (GH), will compute for a m ultisp ectral image patc h a feature v ector with the
size of 2'+hbnb elemen ts, where represen ts the n um b er of orien tations and
' the n um b er of frequencies of the Gab or lter. The parameters nb andhb represen t
the n um b er of bands in the m ultisp ectral image and the n um b er of bins for eac h of the
computed histograms resp ectiv ely .
Impro v ed W eb er Lo cal Descriptor
Using the same considerations lik e in the previous example, w e fo cused also on im-
pro ving WLD features capabilities. W e compute a sim ulated texture band from the
m ultisp ectral scene, on whic h w e extract WLD features. F urthermore, w e compute the
Sp ectral Histogram descriptor for eac h sp ectral band that is presen t in the m ultisp ectral
image. After the computation of the t w o t yp es of feature, w e realize a reunion of the
feature spaces, obtaining this w a y a new descriptor for b oth texture and sp ectral analysis.
Figure 4.28: Left: WLD descriptor results obtained for texture analysis of band mean. Cen ter: Histogram descriptor
computation for m ultisp ectral image. Righ t:WLD-Histogram descriptor computation for m ultisp ectral image – WLD applied
for mean of bands and histogram for eac h band
70
Metho ds and algorithms for Earth Observation image information mining
The feature extraction metho d obtained b y com bining WLD features computed for
the pure texture band with Sp ectral Histogram features from all the sp ectral band will
generate a feature v ector with the size of CT+hbnb . The parameters nb andhb
represen t the n um b er of bands in the m ultisp ectral image and the n um b er of bins for eac h
of the computed histograms resp ectiv ely .
4.6.2 Benc hmarking feature extraction metho ds for m ultisp ectral
Earth Observ ation image analysis
Considering this paradigm of image understanding, the dev elopmen t of e cien t fea-
ture extraction metho ds for m ultisp ectral data analysis and classi cation is not an easy
task and no general approac hes for e cien t classi cation of satellite images are pro vided.
F ollo wing the idea of nding a common ground for measuremen ts and comparisons
b et w een extracted features, w e p erform a b enc hmarking to test the p erformances of the
prop osed descriptors. In the frame of this assessmen t, w e consider the satellite image from
the W orldView-2 sensor and its man ual annotation, presen ted in Fig. 4.25. Our goal is to
set up a b enc hmarking en vironmen t for the prop osed feature extraction metho ds in order
to determine the b est p erforming algorithm for m ultisp ectral image classi cation.
T able 4.2 and 4.3 presen t the precision-recall (P-R) scores obtained for eac h of the
assessed metho ds in the form of confusion matrixes for SVM and k-NN classi cation.
T able 4.2: P-R on SVM classi cation, WV2, Scenario 2
C1 C2 C3 C4 C5 C6 Mean
GP 69.6% 73.6% 72.4% 70.8% 42.4% 95.7% 70.7 %
R 80.1% 71.6% 65.0% 73.4% 53.7% 96.8% 73.4%
HP 64.6% 68.8% 63.7% 69.8% 47.1% 95.3% 68.2 %
R 84.5% 65.0% 60.4% 73.5% 53.3% 97.5% 72.4%
GHP 68.4% 71.8% 68.6% 72.7% 49.7% 96.0% 71.2 %
R 85.8% 67.7% 64.5% 76.2% 58.8% 97.2% 75.0%
BSIP 67.9% 72.1% 69.3% 72.3% 49.7% 97.2% 71.4 %
R 83.8% 70.9% 65.6% 76.0% 59.7% 96.5% 75.4%
BSVP 65.2% 70.2% 70.3% 66.4% 52.0% 95.9% 70.0 %
R 87.1% 70.9% 62.6% 75.7% 53.5% 96.3% 74.3%
SIP 55.6% 68.6% 57.8% 58.2% 40.6% 94.5% 62.5 %
R 78.9% 60.2% 57.2% 71.7% 39.6% 97.6% 67.5%
Ev en though the classi cation results are v ery tigh t for sp eci c feature extraction
metho ds and also for the same classi ers, w e can observ e from T able 4.2 and T able
4.3, that for b oth SVM and k-NN classi cations the BoW features ha v e the highest
a v erage accuracy scores BSI ha v e an accuracy of 71.4% for SVM classi cation while
BSV accuracy is 72.3% for k-NN. The BoW based features are follo w ed b y the prop osed
GH descriptor with an accuracy of 71.2% for SVM classi cation and 71.3% for k-NN
classi cation.
A few initial setups ha v e b een made in the frame of the sup ervised classi cation. The
training samples w ere randomly selected from the man ual annotation le and the n um b er
of samples for eac h class represen t 20% from the n um b er of man ually annotated classes.
Inspired from m ultimedia image classi cation, the v alues sho wn in the T ables 4.2-4.5
are obtained using the mean v alues of the P-R scores obtained b y classifying the scene
with 10 randomly generated training sets.
71
Chapter 4. F e atur e extr action metho ds and algorithms for Earth Observation data understanding
T able 4.3: P-R on k-NN classi cation, WV2, Scenario 2
C1 C2 C3 C4 C5 C6 Mean
GP 69.3% 84.0% 64.5% 73.8% 43.3% 93.4% 71.4 %
R 71.4% 68.9% 70.5% 76.2% 60.7% 97.5% 74.2%
HP 55.3% 88.7% 69.2% 73.8% 39.8% 94.1% 70.1 %
R 90.2% 66.2% 77.6% 77.4% 68.6% 98.1% 79.7%
GHP 60.5% 89.7% 67.1% 74.8% 41.5% 94.4% 71.3 %
R 89.0% 67.2% 78.8% 77.2% 67.4% 98.1% 79.6%
BSIP 63.8% 85.2% 67.7% 73.3% 42.7% 96.0% 71.4 %
R 83.2% 67.5% 73.5% 77.5% 65.0% 96.4% 77.2%
BSVP 64.3% 78.0% 74.2% 76.7% 44.5% 96.0% 72.3 %
R 83.4% 72.8% 68.5% 74.9% 65.4% 96.2% 76.9%
SIP 64.1% 80.4% 70.2% 74.6% 37.6% 94.1% 70.2 %
R 72.1% 69.5% 68.9% 75.4% 57.2% 97.8% 73.5%
Figure 4.29: Sen tinel-2 database of thematic classes.
With the purp ose to demonstrate the usabilit y of feature extraction metho ds on v ar-
ious EO m ultisp ectral images, w e also assessed the prop osed metho ds on data from the
recen tly released Sen tinel-2 satellite. F or this assessmen t, w e use only 4 of the sp ectral
bands pro vided b y this satellite that has 10m spatial resolution. With this setup w e man u-
ally selected 2500 patc hes, of 25×25 pixels, group ed in v e thematic classes of 500 patc hes
eac h (D1 – W ater, D2 – Urban, D3 – F orest, D4 – AgricultureLo w, D5 – AgricultureHigh),
presen ted in Fig. 4.29.
T able 4.4: P-R on SVM classi cation, S2
D1 D2 D3 D4 D5 Mean
GP 98.53% 97.08% 98.61% 98.61% 98.76% 98.32%
R 98.82% 97.58% 97.87% 98.78% 98.62% 98.33%
HP 97.88% 97.02% 96.34% 96.66% 97.44% 97.07%
R 98.96% 93.15% 97.05% 97.97% 98.51% 97.13%
GHP 98.18% 97.18% 97.72% 98.14% 98.14% 97.87%
R 99.02% 96.16% 97.48% 98.64% 98.15% 97.89%
BSIP 97.60% 98.24% 99.28% 97.60% 98.42% 98.23%
R 99.44% 96.51% 98.42% 98.43% 98.44% 98.25%
BSVP 98.28% 95.06% 98.44% 98.50% 98.80% 97.82%
R 98.84% 96.17% 96.20% 99.68% 98.31% 97.84%
SIP 98.56% 96.00% 99.38% 95.68% 98.38% 97.60%
R 99.18% 95.14% 99.12% 96.02% 98.69% 97.63%
High classi cation scores can b e seen in T ables 4.4 and 4.5, b ecause of the patc hes
uniformly distributed microtextures. Also, the purp ose of this test database is to v erify
the image feature p erformances and not the classi cation accuracy .
A ccording to the exp erimen tal results, common texture and color descriptors can b e
adapted and successfully used for m ultisp ectral image analysis. Moreo v er, b y com bining
texture and sp ectral features w e can obtain more p o w erful descriptors. W e also ac hiev ed
in imp ortan t results b y using Sp ectral Indices descriptors, whic h are v ery fast and easy
to compute.
72
Metho ds and algorithms for Earth Observation image information mining
T able 4.5: P-R on k-NN classi cation, S2
D1 D2 D3 D4 D5 Mean
GP 96.14% 93.92% 98.20% 99.00% 93.38% 96.13%
R 98.71% 92.98% 93.17% 97.14% 99.17% 96.23%
HP 96.04% 96.72% 96.14% 98.40% 94.80% 96.42%
R 99.44% 92.85% 95.61% 95.63% 99.04% 96.52%
GHP 96.66% 96.00% 96.38% 98.70% 95.58% 96.66%
R 99.55% 93.01% 94.91% 96.98% 99.23% 96.74%
BSIP 97.82% 94.72% 97.96% 96.80% 96.06% 96.67%
R 97.25% 95.68% 98.26% 94.28% 98.00% 96.69%
BSVP 97.18% 92.64% 98.54% 97.42% 96.90% 96.54%
R 97.95% 93.53% 93.78% 99.47% 98.26% 96.60%
SIP 97.90% 96.18% 99.14% 95.32% 95.24% 96.76%
R 98.71% 92.87% 98.26% 94.67% 99.61% 96.82%
Ev en though the most suitable image descriptors for m ultisp ectral image analysis
pro v e to b e the ones based on the BoW framew ork, the classical ones pro vide similar
results with a shorter computation time. Moreo v er, the Gab or-Histogram descriptor,
whic h computes b oth texture and sp ectral information leads to similar a v erage accuracy
rates.
4.6.3 F eature classi cation on Sen tinels-1 and 2 data
In this scenario, w e assessed feature extraction metho ds on recen tly released Sen tinel-
1 and Sen tinel-2 satellite image data. Eac h Sen tinel mission is based on a constellation of
t w o satellites to ful ll revisit and co v erage requiremen ts, pro viding robust datasets. The
Sen tinel-1 mission is pro viding SAR images in C-band, while the Sen tinel-2 satellites are
imaging the Earth using 13 sp ectral bands. In Fig. 4.30 are sho wn t w o scenes from b oth
satellites, co v ering the same area o v er Buc harest, Romania. The SAR image is acquired
on 05th of April 05, 2016 while the m ultisp ectral data is acquired on Decem b er 23, 2015.
Figure 4.30: Sen tinel-1 (GRD) vs Sen tinel-2 (MSI) image pro ducts
W e extracted HTD and WLD image features from patc hes of 2525 pixels, co v ering
a surface of 250250m2eac h. This patc h size ensures the computation of image features
in optim um conditions and the extraction of the relev an t information from the analyzed
images.
Using the man ual annotation presen ted in Fig. 4.31, w e realized the qualitativ e and
73
Chapter 4. F e atur e extr action metho ds and algorithms for Earth Observation data understanding
Figure 4.31: Man ual annotation with legend and sample patc hes
quan titativ e ev aluation of the feature classi cation results for SAR, MSI and DFI images.
In T ables 4.6 and 4.7 are presen ted the classi cation accuracies for b oth feature extraction
metho ds, HTD and WLD, for eac h of the analyzed image pro duct.
T able 4.6: SVM Classi cation of Sen tinel image data. Precision (P) – Recall (R) rates
C1 C2 C3 C4 C5 Mean
Sen tinel-1HTDP 70.7% 60.7% 70.9% 78.2% 61.9% 68.5 %
R 50.4% 70.8% 73.0% 90.5% 75.3% 72.0%
WLDP 24.0% 46.3% 24.0% 80.3% 10.4% 37.0 %
R 30.6% 42.6% 26.8% 78.6% 3.8% 36.5%
Sen tinel-2HTDP 91.4% 90.3% 94.6% 94.7% 76.8% 89.6 %
R 97.8% 93.1% 83.9% 95.9% 58.6% 85.8%
WLDP 61.5% 63.9% 64.7% 91.3% 58.8% 68.1 %
R 74.7% 75.9% 48.2% 93.1% 19.0% 62.2%
T able 4.7: kNN Classi cation of Sen tinel image data. Precision (P) – Recall (R) rates
C1 C2 C3 C4 C5 Mean
Sen tinel-1HTDP 57.4% 80.2% 71.1% 50.8% 40.2% 59.9%
R 55.3% 66.7% 55.1% 96.0% 98.0% 74.2%
WLDP 43.6% 26.6% 24.2% 82.8% 10.8% 37.6%
R 29.8% 43.7% 25.1% 79.8% 4.5% 36.6%
Sen tinel-2HTDP 87.1% 95.9% 73.3% 89.1% 53.8% 79.8%
R 95.3% 77.5% 89.3% 97.8% 91.7% 90.3%
WLDP 43.7% 46.9% 77.4% 85.9% 54.0% 61.6%
R 79.9% 68.7% 36.5% 95.6% 17.6% 59.7%
W e can observ e from T ables 4.6 and 4.7 that the b est classi cation results are obtained
using Gab or features. Also, from these tables w e can conclude that kNN classi cation
pro vides b etter accuracies than SVM.
In Fig. 4.32 are presen ted in a comparativ e w a y the results of SVM classi cation of
Gab or and WLD features. As it can b e seen in T able 4.6 and Fig. 4.32, if w e represen t
the classi cation results in a thematic map, w e will obtain a v ery similar output with the
man ual annotation.
The results w e obtained when extracting Gab or and WLD image features from SAR
and m ultisp ectral images demonstrate that Sen tinel-1 and Sen tinel-2 data can b e used
with success in land co v er classi cation applications.
74
Metho ds and algorithms for Earth Observation image information mining
Figure 4.32: SVM classi cation results – Gab or vs. WLD features for a) SAR and b) MSI
4.6.4 Multisp ectral image classi cation assessmen t of Sen tinel-2
data
Ev en though dieren t patc h sizes can b e used, some of patc h-based feature extraction
metho ds are imp osing a minim um patc h size that can b e used. Also, when c ho osing the
patc h size is imp ortan t to tak e in to accoun t the spatial resolution of the analyzed image
in order to co v er the ob jects of in terest. Therefore, for observ ation of Sen tinel-2 images
using the 10m spatial resolution sp ectral bands, a patc h of 25×25 pixels will co v er relev an t
ob jects of dieren t thematic classes suc h as agriculture, urban areas, w ater b o dies, etc.
T o establish the optim um n um b er of classes that can b e obtained using unsup ervised
classi cation, w e used rate distortion (RD) theory [94], as it is presen ted in [31]. In
order to compute the RD algorithm w e ha v e tak en in to accoun t the k-Means clustering
of the en tire scene (Fig. 4.33) using the extracted BoW features. The reason b ehind
c ho osing these features is lying in the prop ert y of the BoW mo del of preserving b oth the
texture and color distribution in the analyzed patc hes. As it can b e seen in Fig. 4.35, the
optim um n um b er of classes to b e analyzed for a patc h size of 25×25 pixels is b et w een 5
and 30 thematic classes [29].
The results presen ted in this pap er are computed for 5 generic classes suc h as agricul-
75
Chapter 4. F e atur e extr action metho ds and algorithms for Earth Observation data understanding
Figure 4.33: En tire Sen tinel-2 analyzed scene. The red rectangle represen t the region used for qualit y assessmen t.
ture elds with abundan t v egetation (Agriculture-HV C1), agriculture elds with less
v egetation (Agriculture-L V C2), forests (C3), urban areas (C4) and w ater b o dies (C5),
as presen ted in Fig. 4.34.
Figure 4.34: Sen tinel-2 scene used for qualit y assessmen t (Left). Man ual Annotation in to 5 thematic classes (Righ t)
F or quan titativ e and qualitativ e assessmen t w e mak e use of a small region, represen t-
ing almost 10% of the original scene, whic h con tains all the analyzed classes of the initial
scene. As it is stated in [4], our computation assumes a sample image for whic h w e ha v e
ground truth kno wledge and a set of w ell kno wn classi ers lik e SVM and k-NN.
Figure 4.35: Left: Rate Distortion (X axis represen ts the n um b er of analyzed classes and Y axis represen ts the asso ciated
MSE). Righ t: Mean Precision of SVM and k-NN
Using a sample image co v ering a surface of 2500 km2, illustrated in Fig. 4.34, w e
realized the man ual annotation of 40000 patc hes with a size of 25×25 pixels in to 5 thematic
76
Metho ds and algorithms for Earth Observation image information mining
classes, taking in to accoun t all the sp ectral bands in order to assign to eac h patc h the b est
suited class. This annotation is used further for automatic generation of the training sets
and also to compute the confusion matrixes needed in the qualitativ e and quan titativ e
ev aluation of the classi cations.
Figure 4.36: SVM Classi cation of the scene. Left: HTD features; Righ t: WLD features
Figure 4.37: k-NN Classi cation of the scene. Left: SH features; Righ t: BO W features
The mean accuracy of the ev aluated sup ervised classi cation algorithms is computed
using 10 random training sets obtained from the man ual annotation. The training sets are
20% of the sample image and 1.9% of the full scene b eing analyzed. The results obtained
for Gab or, BoW and for Sp ectral Histogram feature extraction metho ds, while for WLD
the accuracies obtained are b et w een 50% and 65%. (Fig. 4.35 – Righ t)
4.6.5 Earth Observ ation feature extraction standardization
In the frame of feature extraction standardization, our in terest is fo cused on the visual
descriptors pro vided b y MPEG-7 standard, for b oth texture and color understanding
and analysis, b eing highly motiv ated to adjust descriptors pro vided in this standard for
m ultisp ectral Earth Observ ation data analysis and understanding.
MPEG-7 standard, formally named "Multimedia Con ten t Description In terface" is
aiming to pro vide metho ds and tec hniques of describing the m ultimedia con ten t data and
should supp ort a high degree of in terpretation of the information meaning in order to b e
accessed b y a device or a computer co de [13]. Also, ha ving the goal to pro vide supp ort for
a large range of m ultimedia applications, for b oth audio and visual data, one c hallenging
asp ect of this standard is to pro vide capabilities of exploiting remote sensing images.
In MPEG-7, the color descriptors pro vided are optimized for m ultimedia image pro-
cessing and are usually applied on a p erceptual color space and consist of a n um b er of
histogram descriptors, a Dominan t Color Descriptor (DCD) and a Color La y out Descrip-
tor (CLD) [13]. On the other hand, the texture descriptors prop osed in the standard,
T exture Bro wsing Descriptor (TBD), Homogeneous T exture Descriptor (HTD) and lo cal
Edge Histogram Descriptor (EHD), are follo wing attributes suc h as directionalit y , regu-
larit y , coarseness and homogeneit y . [61].
77
Chapter 4. F e atur e extr action metho ds and algorithms for Earth Observation data understanding
As sho wn in Fig. 4.38, MPEG-7 visual descriptors are designed to handle color and
mono c hrome images. In the case of Earth Observ ation image analysis and understand-
ing, where more than three sp ectral bands are in v olv ed, the default con guration of the
MPEG-7 descriptors could not b e used.
Figure 4.38: MPEG-7 Color and texture descriptors
In the quest of adapting MPEG-7 standard descriptors to handle m ultisp ectral Earth
Observ ation images w e prop ose p olar co ordinates transformation of the color space instead
of using the HSV color space. This will mak e p ossible to tak e in to accoun t the full sp ectral
resolution a v ailable in the EO data sets. Also, w e prop ose MPEG-7 complian t descriptors,
suc h as homogeneous texture descriptor and edge histogram descriptor for patc h-based
m ultisp ectral Earth Observ ation (EO) image classi cation and indexing able to extract
maxim um information from all the a v ailable sp ectral bands that last generation remote
sensing satellites pro vide.
Using the p olar co ordinates transformation of the re ectance v alues w e obtain illu-
mination in v arian t features whic h can b e used along with the scalable color descriptor
(SCD) presen t in MPEG-7 standard. Also, HTD and EHD features can b e extracted for
eac h band separately and then merged together to describ e the patc h con ten t.
4.6.6 Under-cloud EO image classi cation using p olar co ordi-
nates features
The prop osed metho d, p olar-SCD, pro vide an illumination in v arian t descriptor that
pro v es to enhance landco v er classi cation of the areas aected b y clouds and their shado ws
and pro vide similar classi cation results when compared with HTD, WLD, SH, SI and
BoW based descriptors suc h as BSI and BSV on cloud free areas.
Being motiv ated to adapt m ultimedia descriptors of the MPEG-7 standard to han-
dle m ulti-sp ectral EO images, w e assessed our p olar co ordinates scalable color descriptor
(pSCD) for EO landco v er classi cation case. In order to pro vide relev an t results, the sug-
gested metho d is tested on dieren t scenes acquired from the recen tly released Sen tinel-2
satellite. Also the results are compared with other image descriptors suc h as homo-
geneous texture descriptor (HTD), w eb er lo cal descriptor (WLD), Sp ectral Histogram
(SH), Sp ectral Indices (SI) and BoW-based descriptors lik e Bag-of-Sp ectral-Indices (BSI)
and Bag-of-Sp ectral-V alues (BSV).
78
Metho ds and algorithms for Earth Observation image information mining
Figure 4.39: Clear sky vs cloud aected image
W e p erformed p olar-SCD feature extraction on all 13 sp ectral bands of Sen tinel-2
image ha ving almost 39% cloud co v erage in order to pro v e the capabilities of the prop osed
descriptor to c haracterize the land co v er classes. The analyzed scene can b e observ ed in
Fig. 4.40.
Figure 4.40: R-G-B bands of Sen tinel-2 scene aected b y clouds and shado ws.
In 4.41 and 4.39 can b e analyzed the represen tation of the cloud aected area using
dieren t band com binations and also using the p olar co ordinates transformation. W e can
observ e that in the case of p olar co ordinates transformation, the area under the clouds
ha v e some information ab out the ground surface to o.
Figure 4.41: Clouds seen as Ir G B and p olar co ordinates 1 2
Figure 4.42: Cloud Classi cation Assessmen t. Left: SVM classi cation of HDT features; Righ t: SVM classi cation of
p olar-SCD features
In Fig. 4.42 is presen ted the visual represen tation of the classi cation under the
clouds. In the righ t part of the gure, where the p olar-SCD can b e observ ed, w e can
distinguish more structures than in the HTD feature classi cation case.
4.7 Conclusions
In this c hapter w e presen ted our con tributions regarding patc h-based feature extrac-
tion metho ds for EO image classi cation and indexing. The features w e presen t can b e
in tegrated in to con ten t based image retriev al platform and data mining systems.
79
Chapter 4. F e atur e extr action metho ds and algorithms for Earth Observation data understanding
Our main con tributions can b e quan ti ed in the dev elopmen t of a fast BRIEF de-
scriptor, feature descriptors for texture and sp ectral analysis, and a new p olar co ordinates
based descriptor that can b e in tegrated within the MPEG-7 standard, whic h giv es go o d
classi cation results for cloud aected areas.
80
Chapter 5
Data Mining Systems
The data mining concept is referring to the pro cess of analyzing large quan tities of
data b y extracting relev an t information trough mathematical and statistical means. Th us,
data mining is an in terdisciplinary branc h of computer science, b eing the computational
pro cess in detecting patterns in large data sets in v olving metho ds in areas suc h as arti cial
in telligence, mac hine learning, statistics and database systems. Th us, data mining is an
automatic or semiautomatic pro cess of disco v ering patterns in data. Is imp ortan t that
these disco v ered patterns to b e meaningful in that they lead to some adv an tage, usually
an economic one as it is stated in [98]. This is also the case of EO data mining in whic h
patterns are searc hed in large geospatial databases. It is kno wn that in the last decades
is recorded a con tin uous gro wth of the remote sensing data, acquired with a large v ariet y
of sensors, in dieren t acquisition mo des. This has lead to large collections of EO image
data that m ust b e understo o d and analyzed.
5.1 An o v erview of data mining systems
Data mining systems can b e classi ed dep ending on the tec hnologies in v olv ed in
the computational pro cess that result in patterns retriev al using criteria lik e Statistics,
Database T ec hnology , Mac hine Learning, Information Science, Visualization and Other
Disciplines.
Ev en t though there are a lot of data mining systems and tec hnologies a v ailable, w e are
fo cused on analysis and retriev al of EO data. Searc hing for relev an t kno wledge through
heterogeneous geospatial databases require additional seman tic kno wledge on the mean-
ing of images, a trained ey e for visual patterns and eectiv e strategies for collecting and
analyzing data with minimal h uman in terv en tion. F urthermore, in geospatial data mining
is essen tial to deal with the div ersit y and m ultitude of remote sensing data. T raditional
metadata suc h as geographical co v erage, time of acquisition, sensor parameters and man-
ual annotations are in most of the cases insu cien t to retriev e searc hed phenomenons or
images in scenes where the visual con ten t that is analyzed con tains only homogeneous
information.
The in tegration of new metho dologies and the dev elopmen t of new systems for auto-
matic features extraction, visual selection and ric h seman tic kno wledge managemen t for
e cien t query in image databases are needed to assist image analysts [55]. Suc h data
81
Chapter 5. Data Mining Systems
mining systems w ould optimize con ten t-based EO image analysis b y allo wing the user to
fo cus on truly relev an t p ortions of the image.
Figure 5.1: W ork o w for F eature Extraction and Earth Observ ation image classi cation
No matter the used metho ds, a generic con ten t based image retriev al (CBIR) system
should b e able to p erform feature extraction, to compute a similarit y measure or solv e
sp eci c classi cation tasks and to displa y the results. A simple w ork o w [33] is presen ted
in Fig. 5.1. Also, for a b etter understanding and e cien t retriev al of EO image data from
large database collections ha v e b een dev elop ed p o w erful to ols suc h as:
1.KEO=KIM (2002) – kno wledge-driv en information mining [25], whic h is based
on h uman-cen tered concepts (HCCs) and implemen ts new features and functions
allo wing impro v ed feature extraction, searc h on a seman tic lev el, the a v ailabilit y of
collected kno wledge and in teractiv e kno wledge disco v ery [26]
2.SemQuery (2002) -Seman tic Query is a seman tics-based clustering and indexing
approac h, used to supp ort visual queries on heterogeneous features of images [86]
3.VisiMine is another in teractiv e data mining and statistical analysis system for
large collections remotely sensed data. VisiMine is extracting texture features using
Gab or w a v elets and Haralic k's co-o ccurence and image momen ts for geometrical
prop erties ev aluation. [52]
4.GeoIRIS (2006) – a con ten t-based m ultimo dal Geospatial Information Retriev al
and Indexing System, whic h includes automatic feature extraction, visual con ten t
mining from large-scale image databases, and high-dimensional database indexing
for fast retriev al. This enables scalable pro cessing and retriev al of a large v olume
of data b y automatically prepro cessing and indexing satellite images. In [88], the
authors are explaining that the framew ork has b een dev elop ed using 0.6- and 1.0-m
resolution imagery , although with minor mo di cations, it could b e applied to higher
resolution imagery , but ma y encoun ter di culties for lo w er resolution images.
5.VIS STAMP is a geo-visual analytic soft w are pac k age that couples computa-
tional, visual, and cartographic metho ds for exploring and understanding spatio-
temp oral and m ultiv ariate data. It can help analysts in v estigate complex patterns
across m ultiv ariate, spatial, and temp oral dimensions via clustering, sorting, and
visualization. Sp eci cally , VIS-ST AMP builds on a self-organizing map, a paral-
lel co ordinate plot, sev eral forms of reorderable matrices , and a geographic small
m ultiple displa y . [39]
82
Metho ds and algorithms for Earth Observation image information mining
6.MOB (2007) – Middlew are for On tology Driv en Brok ering (MOB) is a system that
pro vides functionalities for on tology managemen t, storage, query and inference ser-
vices [49]. The framew ork that MOB implemen ts is designed for seman tics driv en
in tegration of resources in the con text of coastal areas in whic h image information
mining is crucial for optim um exploitation of h uge remote sensing data arc hiv es.
7.GeoDMA (2008) or Geographic Data Mining Analyst is in tegrating image pro cess-
ing tec hniques, feature extraction and data mining solution in a framew ork that
pro vides spatio-temp oral analysis to ols to in terpret image time series.
8.IKONA is a CBIR system that pro vides the abilit y of retrieving images b y visual
similarit y in resp onse to a query that satis es the in terest of a user as it is explained
in [74]. Moreo v er, IKONA is a system dev elop ed using a clien t-serv er arc hitecture,
and is indep enden t from the platform it runs.
9.TELEIOS is a database p o w ered virtual earth observ atory , dev elop ed in the frame
of an Europ ean pro ject that addresses the need of scalable access to p etab ytes of EO
data and also the disco v ery and exploitation of kno wledge that is hidden in them
[53].
10.ESA=DLR EOLib or Earth Observ ation image Librarian is a pro ject of DLR and
ESA that is in tended to implemen t no v el tec hniques for image con ten t exploitation.
In order to ac hiev e the goal of automatic or semiautomatic EO con ten t disco v ery ,
annotation and retriev al, w e can observ e that in the dev elopmen t of the data mining
system for remotely sensed data, the metho dologies used are concerning either rapid
mapping or arc hiv e na vigation. As is stated in [77], image pro cessing and analysis, data
mining, geographic data managemen t, ob ject detection and recognition metho dologies
are relev an t, y et need to b e adapted and com bined in order to allo w c haracterizing v ast
v olumes of unkno wn con ten t.
Due to the large quan tit y of information stored in to remotely sensed data, b efore the
dev elopmen t of an EO data mining system, w e should closely analyze the destination of
the image mining system. Also is imp ortan t to kno w the answ er to questions lik e: what
EO data do w e use, what image features are imp ortan t for our analysis, is the metadata
needed, what mac hine learning algorithms can w e use. A simple concept sc hema can b e
found in Fig. 5.2
Ha ving a w ell de ned concept of EO data mining systems, it is still need the presence
of a h uman op erator either for training or v alidation. Th us, considering the h uman
comp onen t on the top of a data mining system, w e should also dev elop a framew ork for
h uman mac hine comm unication.
5.2 Human Mac hine Comm unication
In the con text of big data from space, exploitation and exploring the data em b ed-
ded information, is an extremely complex and di cult task, that is surpassing curren t
tec hnological capabilities. V arious pro cedures and tec hniques for an e cien t analysis and
managemen t of large databases, fo cusing on con ten t-based searc h, data mining and kno wl-
edge disco v ery ha v e b een prop osed. The dev elopmen t of b est of these tec hniques has often
83
Chapter 5. Data Mining Systems
Figure 5.2: EO image information mining conceptual sc hema
b een made either for separate categories of data suc h as text, sound, digital recordings,
images, or the collections of heterogeneous data, suc h as video and sound, dep ending on
the eac h metho d particularities.
Systems for satellite imagery exploration requires adv anced database indexing pro ce-
dures and visualization tec hniques. Implemen tation of m ultidimensional data structures,
pattern recognition algorithms or mac hine learning metho ds (automatic or semiautomatic)
b ecomes mandatory . In addition, the systems ha v e to b e able to handle not only a wide
range of satellite images, but also GIS data, metadatas and digital cartograph y data.
In teraction with suc h a system can b e de ned as a dialog b et w een man and mac hine
that is p erformed in a language whic h uses discrete units appro ximating the seman tic
meaning of h uman language. Searc hes made on the basis of this language w ere applied
generally to databases that use a lo w lev el appro ximation of the seman tic meaning (eg
primitiv e features of images: color, texture, geometry). The most di cult asp ect in
implemen ting database query systems is ho w to in tegrate in teractiv e capabilities and user
in teraction in to the pro cessing cycle. Also, if the user is the one analyzing the results of
the query , this can help the system so that the nal result is in agreemen t with h uman
understanding.
Some functions suc h as query b y con ten t, high lev el feature extraction or seman tic
annotation can b e impro v ed with suc h h uman-mac hine dialog. Th us, for easy access and
also for in terpretation of large satellite images is imp ortan t to dev elop sp eci c algorithms
of feature extraction. The solution w ould in v olv e access to an en vironmen t/platform en-
abling in the rst step of the pro cessing c hain that the fo cus will b e automatically directed
to image feature extraction algorithms. Besides feature extraction and in terpretation, the
prop osed pro cedure ma y b e the foundation of a univ ersal pro cessing c hain that can b e
adapted to user requiremen ts and can b e applied on new data sets. Also, w e should tak e
in to consideration that the pro cess will b e dep enden t on the nature of the data review ed
and do es not guaran tee accuracy of the results when applied to large collections of images.
F urthermore, adding images in this pro cessing c hain w ould in v olv e the de nition of visual
co deb o oks, co des or signs that can appro ximate the seman tic con ten t.
T o dev elop suc h complex systems, is necessary to establish a h uman mac hine com-
m unication mec hanism and to design new algorithms able to compute a mapping of the
data from signal to sign/sym b ol/co de and bac k in order to ensure the o w of information
84
Metho ds and algorithms for Earth Observation image information mining
from data to user. The w a y information is p ercept will not necessarily dep end on the form
in whic h it is transmitted, but on ho w the represen tation of the con ten t and kno wledge
is made.
The relationship b et w een an image seen as a sign/sym b ol/co de and the seman tics
information can b e quan tized using similarities measuremen ts functions. Ho w ev er, the
resem blance b et w een collections of signs and sym b ols is v ery general and is meaningless
without the user in terpretation. Therefore,h uman in terv en tion is required to consisten tly
index a collection of images according to the needs of analyzed scenarios. This in teraction
b et w een man and mac hine is realized using data mining systems based on an in teractiv e
dialog in whic h the v o cabulary used is comp ound of w ell de ned sym b ols/signs/co des
that b oth man and mac hine can easily understand and op erate.
Figure 5.3: Hierarc hical information represen tation in a data mining system
Also, for a b etter understanding of h uman mac hine comm unication concept, w e ha v e
de ned sym b ols lik e:
P atc hes – W e consider the patc hes to b e the smallest areas or groups of pixels from
within an image, whic h w e can use to assign or extract seman tic information. The
patc h size is usually v ariable, b eing dep enden t on the spatial resolution of the images
b eing analyzed.
F eature descriptors – represen t the feature v ectors that are extracted from the
patc hes using sp eci c algorithms. These are compact n umeric represen tations of
the con ten ts of the patc h whic h are fo cused on prop erties suc h as color, geometry
or texture.
Classes of elemen ts – are a collection of tags whic h de ne groups of feature descriptors
that resem ble in a large p ercen tage according to certain criteria. By grouping similar
feature v ectors, are group ed actually the patc hes. The pro cess can ha v e a double
meaning: the computer will op erate with n um b ers, applying algorithms to group
the feature v ectors according to some similarities measuremen ts and the user will
get collections of patc hes that are more or less similar.
85
Chapter 5. Data Mining Systems
Seman tic tags – are sp eci c classes with seman tic meaning assigned b y the user as
a consequence of the analysis and in terpretation of results.
These categories of sym b ols are extracted from the analyzed data, inside the data
mining system. The h uman mac hine comm unication is computed hierarc hically . The
de nition of eac h pro cessing lev el is dep enden t on the user in teraction and parameter
setups. The theoretical concept is illustrated in Fig. 5.3 and is follo wing the mo del of a
classical comm unication c hannel, as presen ted in Fig. 5.4, in whic h the image is the source
of information and the seman tic annotation is the receiv er. A ccording to the prop osed
mo del, the image data is gradually enco ded. In this w a y only the relev an t information
will b e deliv ered to the receiv er.
Figure 5.4: Classical mo del of a comm unication c hannel
Considering the hierarc hical information represen tation scehma from Fig. 5.3, whic h
ev olv ed from the classical comm unication c hannel mo del, w e desire to in tegrate the h uman
comp onen t in to the data pro cessing c hain. By doing so, w e adapt the classical comm u-
nication c hannel mo del to a h uman-mac hine comm unication mo del, in whic h the dialog
b et w een h uman and mac hine is mediated b y a graphical user in terface (GUI). Inside the
GUI, the user can adjust pro cessing parameters, and can start and stop v arious pro cesses.
Also, the GUI is used for results visualization and v alidation.
Figure 5.5: Human Mac hine Comm unication data o w
The GUI is acting lik e a bridge b et w een the user, database and implemen ted algo-
rithms, as it can b e observ ed in Fig. 5.5. Moreo v er, the h uman comp onen t is pla ying an
imp ortan t role in the data mining metho dology . Th us, the user can access information
from the database, can visualize it, can start dieren t pro cessing c hains and also can tak e
decision in the v alidation pro cess.
86
Metho ds and algorithms for Earth Observation image information mining
5.3 Data Mining T o ol
Regarding the idea of nding a common ground b et w een syn thetic ap erture radar
(SAR), optical data and ev en data fusion pro ducts, w e prop osed metho dologies for feature
extraction and classi cation. F urthermore, the feature extraction algorithms w e presen ted
in the previous c hapter are part of a data mining system w e ha v e implemen ted.
Our system is complian t with data mining metho dologies and the mo dules dev elop ed
in the Data Mining T o ol application are follo wing the conceptual sc hema illustrated in
Fig. 5.2. The to ol w e prop ose, w as dev elop ed in Visual Studio, using .Net tec hnologies.
Moreo v er, to read EO data w e are using GD AL (Geospatial Data Abstract Library) library
and for the use of mac hine learning algorithms, w e are relying on Op enCV library .
Un til no w, w e ha v e presen ted our w ork in an comparativ e view b et w een theory and
practice, and b et w een state of the art metho ds and algorithms and our solutions. Th us,
to presen t the ev olution of data mining systems, from con ten t based image retriev al ap-
plications to more complex image information mining arc hitectures, w as just a logical
step. Also, in our quest of de ning EO data mining systems w e found to b e of vital
imp ortance the h uman mac hine comm unication principles, whic h are used in almost all
data mining systems. F urthermore, considering the wisdom of previous dev elop ed data
mining systems, w e built our o wn arc hitecture for EO image information mining.
In the follo wing, w e will motiv ate the system arc hitecture and soft w are design adopted
and it will b e presen ted a pro of of concept. Some p ertinen t examples on ho w the data
mining system lo oks lik e, ho w data is read and ho w information mining is p erformed in
our system, will also b e pro vided.
5.3.1 Soft w are arc hitecture
T o describ e the Data Mining T o ol w e dev elop ed from a soft w are arc hitecture p oin t
of view, a few concepts should b e de ned. Multiple authors presen t v arious de nitions in
whic h soft w are arc hitecture is de ned as a blueprin t of the application b eing dev elop ed.
Also, a more tec hnical de nition can b e found in [91] and is presen ting soft w are arc hitec-
ture as "an abstraction, or a high-lev el view of the system. It fo cuses on asp ects of the
system that are most helpful in accomplishing ma jor goals, suc h as reliabilit y , scalabil-
it y , and c hangeabilit y . The arc hitecture explains ho w y ou go ab out accomplishing those
goals".
In the case of an image information mining system, the reliabilit y ma y refer to the
abilit y of the system to p erform required functions, under stated conditions for a sp eci ed
time. The scalabilit y is referring to the capabilit y of a system or pro cess, to handle
a gro wing amoun t of w ork. In most of the cases, when sp eaking ab out a system, w e
consider it scalable if it can increase its total output under certain circumstances when
resources, lik e additional hardw are, are added. In terms of c hangeabilit y , a system ha v e
to b e capable to handle elemen ts of c hange. The c hange can b e de ned as a transition
o v er time of a system to an altered state, so if a system remains the same at time i and
i+ 1 , then it has not c hanged, as it is explained in [83].
Also, in the system arc hitecture w e ha v e to tak e in to accoun t the principles of h uman
mac hine comm unication. This is a direct consequence of the fact that w e dev elop a
87
Chapter 5. Data Mining Systems
computer application that is addressed to a h uman target group, ev en though in the
bac kground can run v arious mac hine to mac hine comm unication proto cols.
Ha ving in to accoun t the EO image information mining conceptual sc hema illustrated
in Fig. 5.2 and the classical comm unication c hannel mo del (Fig. 5.4) for handling the
data inside the application, t w o system arc hitectures can b e tak en in to accoun t, one
arc hitecture is a net w ork based mo del (Fig. 5.6) and the other one is based on a desktop
mo del, as it is illustrated in Fig. 5.7.
Figure 5.6: Net w ork based image information mining system arc hitecture mo del
The net w ork based system arc hitecture presen ted in Fig. 5.6 is oering adv an tages
in terms of scalabilit y , and c hangeabilit y . Dev eloping suc h a data mining system requires
serv er con gurations o v er a net w ork and a w ell de ned data ingestion pro cess. Also,
when sp eaking ab out the user exp erience, the computations should let the impression
of real time pro cessing. This means that all the feature extraction metho ds ha v e to b e
computed in the data ingestion step. The data pro cessing step is referring in this case to
database queries, sp eci c parameters setups and commands that are sen t to the serv er to
b e analyzed and pro cessed.
Figure 5.7: Desktop based image information mining system arc hitecture mo del
Ev en though the net w ork based arc hitecture is oering immediate adv an tages, the
desktop based arc hitecture is also reliable, scalable, and c hangeable. In a desktop based
88
Metho ds and algorithms for Earth Observation image information mining
approac h, lik e the one presen ted in Fig. 5.7, there is no need for real time illusion. In this
case, data ingestion step is done lo cally and the computed features can b e stored in to a
lo cal database. The p erformance of suc h system is dep enden t on the mac hine that runs
the application.
Considering the prop osed arc hitectures, for b oth desktop and net w ork, there are
adv an tages and disadv an tages on b oth sides. In the case of net w ork based mo del, our
application can b e more scalable, but can only p erform certain op erations and the data
ingestion step requires a sp ecialized user group that can handle the input data and ha v e
a go o d kno wledge ho w to setup pro cessing parameters. On the other hand, in a desk-
top based application, the scalabilit y is dep enden t on the maxim um hardw are upgrade
capabilities, and there is no need for prede ned ingestion steps.
Figure 5.8: Data Mining T o ol system arc hitecture
In the data mining to ol w e prop ose, a desktop based application is dev elop ed, in
whic h remotely sensed images, SAR or optical, can b e loaded. On the loaded scene,
the user can p erform patc h-based feature extraction and classi cation. Other features
for whic h our system is designed are qualit y and quan tit y ev aluation, external training
datasets, data visualization and data exp ort. A simpli ed system arc hitecture can b e
observ ed in Fig. 5.8.
5.3.2 Pro of of concept
Eac h mo dule of the Data Mining T o ol (DMT) application w as dev elop ed on the basis
of the system arc hitecture diagram presen ted in Fig. 5.8. In the dev elopmen t of DMT w e
fo cused on building an easy to use application with a simple asp ect, whic h includes w ell
de ned elemen ts. F or this purp ose, the data o w w e prop ose is strictly follo wing the sys-
tem arc hitecture diagram. F urthermore, the graphical user in terface (GUI) is in tegrating
mo dules for reading EO data, feature extraction, feature classi cation, database manage-
men t, data visualization, qualit y and quan tit y ev aluation and data exp ort. In Fig. 5.9 are
illustrated the mo dules of DMT and also the o w of information b et w een the mo dules.
DMT is a pro ject based application, whic h allo ws the user to return to previous
w ork, just b y loading the pro ject le. In DMT, the pro ject le is k eeping the history
of op erations previously made. As a feature, the pro ject le is automatically sa v ed eac h
time an up date inside of pro ject is triggered. Also, as it can b e seen in Fig. 5.9, the
pro ject handler is the main stream used to in terconnect all the other mo dules of the
89
Chapter 5. Data Mining Systems
image information mining to ol w e prop ose. F or example, when a new pro ject is created,
the pro ject le up dates its con ten t immediately an image is op ened, a feature extraction
or a classi cation. Also, in parallel, the database is p opulated with useful information.
The EO Data Reader mo dule is in tended to handle m ultisp ectral data, with sev eral
sp ectral bands, panc hromatic and SAR images. Also in this mo dule is read the spatial
reference system, if it is presen t in the analyzed image. Just after the data is read,
if only one sp ectral band is presen t in the image, the user is ask ed to c ho ose b et w een
panc hromatic image and SAR image. This is helpfull in the feature extraction step,
where certain algorithms are dedicated for m ultisp ectral or SAR images only .
Figure 5.9: Data Mining T o ol soft w are design diagram
In the F eature Extraction mo dule, are implemen ted the metho ds w e describ ed in the
previous c hapter. Also in this mo dule, the user can adjust the settings of the feature
extraction algorithms.
Database managemen t mo dule is dev elop ed to k eep trac k on the extracted features
and also is used to select features in tended to b e the training sets that sup ervised classi-
cation is needing.
F eature classi cation mo dule is in tegrating SVM, k-NN and k-Means algorithms. In
the frame of this mo dule, sp eci c parameter setups can b e made to impro v e the classi -
cation results.
Visualization mo dule is in tended for visualization of the results. In the frame of
this mo dule, external datasets can b e op ened, b oth raster and v ector formats. As a
requiremen t for data visualization, in case geographical data is op ened, the imp orted les
should ha v e the same reference system as the analyzed images. Also, in the visualization
mo dule is included the graphical represen tation of the c harts obtained as a pro duct of
Qualit y & Quan tit y Assessmen t mo dule.
In the Qualit y & Quan tit y Assessmen t mo dule, is computed the confusion matrix
b et w een a reference data set and the classi cation results. Also, another output of this
mo dule is the Precision-Recall c hart, obtained from the confusion matrix.
The graphical user in terface of Data Mining T o ol , illustrated in Fig. 5.10, is using
an user friendly in terface, similar to the one encoun tered in wizard applications. The
pro cessing steps and their settings can b e found group ed in tabs, in the upp er side of the
in terface, while in the b ottom are placed t w o progress bars: one for the curren t op eration
that is running (for example extracting image features from a patc h) and one for the
total progress of the curren t pro cessing. This is done due to the fact that sometimes
90
Metho ds and algorithms for Earth Observation image information mining
the feature extraction pro cess ma y b e time consuming and is necessary to mak e the user
a w are that the application is still running bac kground pro cess. Ab o v e the progress bars,
in the b ottom righ t part, there is a c hec kb o x, whic h will op en a new windo w if its c hec k ed.
This windo w is used for visualization purp oses and is one of the GIS comp onen ts of the
data mining to ol w e prop ose. In Fig. 5.11 can b e observ ed the EO data viewer windo w.
This comp onen ts are a v ailable, no matter the step user is p erforming.
Figure 5.10: Data Mining T o ol graphical user in terface – First screen
Considering the in terface sho wn in Fig. 5.10 and the soft w are design diagram from
Fig. 5.9, the data mining w ork o w is follo wing the metho dology prop osed in this thesis.
The data is read in to the EO Data Reader mo dule, whic h can b e observ ed in the rst
screen of the application. Firstly , a w orking directory need to b e set then the pro ject
handler, that is w orking b ehind the scenes, is creating the pro ject le, enabling this w a y
the p ossibilit y of op ening an EO image. If the image has one sp ectral band, a dialog will
app ear and ask the user if is using a panc hromatic or SAR image, otherwise the radio
buttons will b e set to m ultisp ectral if in the op ened image are at least three sp ectral
bands. T o kno w the t yp e of image data used and the n um b er of sp ectral bands is handful
b ecause some of the feature extraction metho ds are dev elop ed for certain t yp es of images.
Also, dep ending on the n um b er of sp ectral bands, the size of the feature v ector is v arying.
Figure 5.11: Earth Observ ation View er – Simpli ed GIS displa y
The visualization mo dule, as sho wn in the design diagram can b e accessed at an y
91
Chapter 5. Data Mining Systems
time, in an y pro cessing step. Th us, after a pro ject is created and an image is set, the
user can visualize it in to a simple GIS in terface, as sho wn in Fig. 5.11. Also in the
visualization mo dule, to ols for zo oming and panning are a v ailable, and the user has the
p ossibilit y to add new la y ers to the displa y ed map. As it can b e seen in Fig. 5.11, spatial
measuremen ts can b e p erformed, due to the fact that images are op ened with their spatial
reference information.
Figure 5.12: Image classi cation to ols windo w
The mo dules for feature extraction and feature classi cation are group ed together
in the next tab, named Image classi cation to ols. In this section, the user can set the
parameters used in image features extraction. Also, the analysis patc h dimensions can
b e c hanged using either the co v ered ground surface or the n um b er of pixels in the patc h
(Fig. 5.12 – section 1). After the feature extraction step is b eing completed, the metho d
will app ear in a list of a v ailable features placed in the Image Classi cation section (Fig.
5.12 – righ t). The classi cation algorithms used in this to ol are group ed in sup ervised (k-
NN, SVM) and unsup ervised (k-Means) and according to the selection, the classi cation
parameters ma y c hange. Also, the class manager is a v ailable trough the button Op en
Class Manager , whic h will op en a new windo w whic h is part of the Database mo dule.
Figure 5.13: P erformance Ev aluation windo w
In the Database mo dule, the user can imp ort training sets and visualize them. This
training sets are used in the classi cation section. F urthermore, a consisten t part of the
database mo dule is not in uencing user exp erience, due to the fact that most of the
92
Metho ds and algorithms for Earth Observation image information mining
pro cesses are running in bac kground. Some examples of bac kground pro cesses are loading
or sa ving image features, loading trainsets needed in the classi cation step. The Database
mo dule is commanded b y the pro ject handler, whic h is also running in bac kground of the
application, making the application easy to use.
Qualit y and Quan tit y Assessmen t mo dule is dev elop ed under the P erformance ev al-
uation section, whic h can b e observ ed in Fig. 5.13. In this mo dule, the inputs are the
results obtained after the classi cation step and a man ual annotation. The outputs that
this mo dule returns are a confusion matrix and a precision-recall c hart. F or the results
to b e accurate, it is desirable that the man ual annotation should b e done taking in to
accoun t all the sp ectral bands a v ailable in the m ultisp ectral image, considering this w a y
the maxim um of information that a h uman analyst can extract. Also, exp orts of the
p erformance ev aluation results can b e p erformed in to ascii le format for the confusion
matrix and bitmap image le format (*.BMP) for the precision-recall graphical result.
5.4 Systems emerged from Data Mining T o ol
As demonstrated earlier, our concerns w ere orien tated not only to theoretical asp ects,
but also to practical ones. Th us, from the data mining to ol concept w e presen ted, emerged
t w o researc h pro jects, one protot yp e and one fully functional online platform, funded b y
the Europ ean Space Agency (ESA).
In the frame of rst researc h pro ject, named CONCEDE, the goal w as to pro vide
a protot yp e based on a framew ork for image information mining that can handle b oth
medical images and EO images. Moreo v er, just lik e an ev olving organism, the protot yp e
w as extended to the next phase of dev elopmen t to a functional platform that can b e used
b y a large comm unit y of users, b oth adv anced and uninitiated in remote sensing imagery .
The acron ym of this platform is OSIRIDE and is a fully op erational EO data mining to ol,
that in tegrates Gab or, WLD, Sp ectral Histogram and lo cal binary pattern (LBP) feature
extraction metho ds, giving the opp ortunit y to the user to set the p ercen tages in whic h
dieren t features can aect image retriev al pro cess.
5.4.1 Con ten t based query concept
In the frame of "Con ten t Based Query Concept for Exploration and Disco v ery of
Information in Earth Observ ation and Medical Libraries", the goal w as to dev elop a
theoretical and practical framew ork in whic h the analysis of heterogeneous datasets w ould
rev eal the seman tic information hidden in to image libraries. Th us, kno wledge from v arious
domains lik e image pro cessing, database managemen t, information theory and mac hine
learning will b e used to de ne a new concept of con ten t based image retriev al, that will
w ork on medical image datasets and also on Earth Observ ation data. CONCEDE pro ject
is prop osing to dev elop a demo to ol whic h will b e used for ev aluation and v alidation of
the a theoretical concept.
The system prop osed, is in tended to b e orien ted on remote sensing applications and
also to nd its utilit y for users in the eld of medical imaging. The arc hitecture of the
demo to ol is based on a data managemen t system (DBMS) and a m ulti-la y ered concept
that connects the user (high lev el) to m ultiple sources/t yp es of data (lo w lev el).
93
Chapter 5. Data Mining Systems
F or pro cessing and query b y con ten t are used data t yp es suc h as satellite images
(optical, SAR), medical images, metadata (b oth satellite imagery and for medical in-
formation), GIS v ector data, text descriptions of the image con ten t, maps (images with
reference annotations – for example Corine Land Co v er, Urban A tlas). Image descriptors
extracted from these data (lo w lev el features) will b e com bined in seman tic classes and
sym b olic represen tations (high lev el features) to pro vide the user an in teractiv e metho d
of de ning the seman tic meaning of an image and an e cien t to ol for querying image
arc hiv es. By handling m ultiple sources of information, these queries will gain great im-
p ortance to a wide range of users.
A t its rst op erating lev el, the prop osed demo to ol allo ws data reading (images,
v ectors, metadata, text) from external sources and sa v es them in the storage mo dule
named "Data Rep ository". A t the second op erating lev el, sp eci c image features will
b e extracted from stored data. F urthermore, dep ending on the data b eing analyzed,
sp eci c pro cessing algorithms will b ecome a v ailable. Also the results of the pro cessing
are collected trough the mo dule "Auxiliary Data Generation", and the initial data will b e
mark ed with sp eci c annotations, con tributing this w a y to the seman tic indexing of the
database. A t this pro cessing lev el, the data is stored as feature v ectors and lab els with
dieren t lev els of seman tic meanings or other represen tations.
In the follo wing lev els, the features extracted in the rst op erating lev el, are group ed
to de ne a seman tic of high lev el features. A t this p oin t, the user will b e able to select a
classi cation metho d, according to the format of data b eing analyzed, in order to obtain
b est results. New lab els are generated as a result of the classi cation pro cess and sa v ed in
the "Data Rep ository" mo dule. These data will b e used in the indexing pro cess. F urther,
the user will de ne the desired query to the database.
The system arc hitecture is designed using stand-alone mo dules that can w ork inde-
p enden tly . In this w a y new metho ds and algorithms can b e added without aecting other
mo dules.
In Fig. 5.14 can b e observ ed the system arc hitecture of the demo to ol whic h presen t
the pro cessing w ork o w. As an observ ation, w e can easily remark stand-alone mo dules
link ed together as a system.
Figure 5.14: Con ten t Based Query Concept for Exploration and Disco v ery of Information in Earth Observ ation and Medical
Libraries system arc hitecture
94
Metho ds and algorithms for Earth Observation image information mining
Case study: Medical image data analysis
In this example, the goal is to v alidate the concept in the case of medical images
analysis. The data b eing tested is a histological image, represen ting tissue from the
digestiv e system of a herbiv ore animal, b eing illustrated in Fig. 5.15. T o mak e the
analysis more accurate, w e ask ed exp ertise from a sp ecialist in cellular biology , histology
and em bry ology from the Univ ersit y of Agronomic Sciences and V eterinary Medicine of
Buc harest.
The image illustrated in Fig. 5.15 w as acquired using a photo camera Olympus
attac hed to an optical microscop e Olympus. In the analyzed image can b e observ ed a
large div ersit y of cells, with dieren t t yp es of cellular n ucleus, in ter-cellular uids or other
p eculiarities of maxim um relev ance in the analysis pro cess. The goal is to extract the
maxim um n um b er of dieren t cells categories for an optimal image exploitation.
Figure 5.15: Tissue from the digestiv e system of a herbiv ore animal
In our case, can b e iden ti ed six dieren t cell categories that can b e observ ed in Fig.
5.16. This analyzed scenario can b e applied in the eld of optical microscop y for studying
tissues morphology .
Figure 5.16: Classes of tissue extracted from the analyzed medical image
Case study: Earth Observ ation image data analysis
In the second analyzed scenario, the demo to ol is used for EO image analysis. Ev en
though an y m ultisp ectral image is supp orted, for this demonstration w e will consider only
the case of a Landsat 8 scene, with 8 sp ectral bands and 30m spatial resolution, co v ering
a region from North of Buc harest, Romania.
95
Chapter 5. Data Mining Systems
The scene w e used for image classi cation can b e observ ed in the left side of Fig.
5.17 along with the man ual annotation and some classi cation results. In the righ t part
of the gure can b e found the classi cation legend for eac h of the thematic classes (C1 –
Dense Urban, C2 – V egetation, C3 – Agriculture1, C4 – Agriculture2, C5 – Lo w Urban).
Also the classi cation results are presen ted in T able 5.1.
Figure 5.17: Landsat-8 image, man ual annotation and classi cation results
T able 5.1: Precision-Recall for SVM classi cation of LandSA T-8 m ultisp ectral image
Gab or WLD SH
Precision Recall Precision Recall Precision Recall
C1 85.87% 40.74% 91.26% 60.03% 77.35% 61.39%
C2 28.83% 22.08% 53.36% 39.10% 42.05% 35.84%
C3 43.80% 75.82% 48.49% 59.97% 44.59% 50.16%
C4 43.65% 77.78% 58.28% 72.24% 32.43% 66.67%
C5 67.92% 30.97% 63.89% 51.31% 74.19% 33.36%
F or the feature extraction step w ere used patc hes of 2525 pixels, co v ering a surface
of pro ximately 0:56km2. Due to the lo w resolution of the m ultisp ectral image, it is
exp ected that lo w classi cation results ma y o ccur, ev en though the patc h size is optim um
for micro-texture ev aluation.
5.4.2 W eb platform for EO image information mining
Despite previous pro ject, in whic h the goal w as to dev elop a theoretical and practical
framew ork for heterogeneous datasets analysis, in OSIRIDE the goal w as to de ne a
fully op erational platform for EO image information mining and query b y image con ten t,
that can b e used b y users with dieren t lev els of exp erience with m ultisp ectral remotely
sensed images.
First, the platform for EO image information mining and query b y image con ten t
will include a set of functions to extract prop erties of EO pro ducts, b eing able to handle
images, metadata and other categories of information. The obtained features are to b e
96
Metho ds and algorithms for Earth Observation image information mining
further reduced, indexed through a grouping pro cess to pro vide computationally manage-
able data quan tities and recorded in to a catalogue inside the data rep ository . Seman tic is
assigned through a sup ervised classi cation and learning pro cess. New indexes will th us
con tribute to the structure of the database and allo w seman tic searc hing. The features
inside a query image selected b y the user are compared with the en tire c haracteristics
space. The protot yp e system returns to the user (on the displa y) the results, allo wing
him to re ne the searc h or annotate the images. The v alidation of the outcomes dep ends
on reference dataset and h uman bias, according to the use case scenarios. The describ ed
concept is c haracterized b y a mo dular functionalit y that is a exible analysis. A generic
functional concept for CBIR that will b e follo w ed in system dev elopmen t is represen ted
in Fig. 5.18.
Figure 5.18: System arc hitecture of the w eb-based platform for Earth observ ation image information mining
The approac h fo cuses on ac hieving the assumed ob jectiv e, to dev elop implemen t
and in tegrate to ols for EO CBIR in to a p o w erful and ready to use platform with an
op en source c haracter. An analysis and review of algorithms is required in order to
implemen t suitable metho ds for extracting information and ranking of relev an t results
for individual users. The mo dularit y of the protot yp e system pro v es its sustainabilit y
for op en implemen tation, pro viding exibilit y for selection, up dating and use of relev an t
additional sources of information (i.e. gazetteers, taxonomies pre-annotated maps etc.).
This approac h addresses mainly the needs of general individuals and scien ti c op erators
and not those of the users in the pa yload data ground segmen t.
In this platform for EO image information mining and query b y image con ten t, sp ecial
atten tion is giv en to feature extraction algorithms for EO data, image retriev al from SAR
data, utilization of additional sources of information for impro ving searc h results, the
automate classi cation mec hanisms, A ctiv e Learning and Relev ance mec hanisms, visual
data mining algorithms and to implemen tation of use cases: scripts for sp ecial pro cess
c hains (e.g. o o ding, forest mapping, etc.).
The graphical user in terface will facilitate the analysis and b enc hmarking for auto-
matic retriev al tec hnologies. The in teraction with the user helps to adapt the existing
tec hnologies to the p eculiarities of EO image con ten t and use. Its structure allo ws m ulti-
ple queries and allo ws in tegration of additional information from external sources in order
to impro v e retriev al capabilities. The in terface of the online application can b e observ ed
in Fig. 5.19.
97
Chapter 5. Data Mining Systems
Figure 5.19: OSIRIDE in terface
5.5 F uture directions
In this thesis w e prop osed a new metho dology for feature extraction, w e dev elop ed
new feature extraction metho ds and implemen ted an EO image information mining to ol.
Moreo v er, w e pro v ed our concept in the frame of ESA pro jects b y building a protot yp e
and a platform for image information mining.
The prop osed framew ork for feature extraction and data mining, is going to b e im-
plemen ted in new researc h pro jects that aim at in tegrating data mining (DM) to ols in to a
geographical information system (GIS) platform, dev eloping a m ulti-lev el indexing struc-
tures for big data analysis or dev eloping a m ultisp ectral data analysis to olb o x for ESA's
Sen tiNel Application Platform (SNAP).
5.5.1 In tegrated GIS-DM system
The role pla y ed b y GIS tec hnologies and Earth Observ ation resources in the con text
of Europ ean p olicies with global impact (en vironmen t, securit y , urbanization, climate
c hange, etc.) is constan tly gro wing, as these tec hnologies allo w for long term monitoring
of the Earth's surface and critical ob jectiv es. The dynamic of p oten tial applications
for EO data is unimaginable no w ada ys, and the div ersit y of sensors, image pro ducts,
spatial, sp ectral and temp oral resolutions are the pillars on whic h they dev elop. Critical
Infrastructure monitoring falls in to the category of applications whic h in terest not only at
lo cal scale but also at Europ ean and global lev el. This comes in the con text of p opulation
gro wth and dynamic dev elopmen t and migration, whic h lead to rapid c hanges in the usage
and distribution of natural resources and urban areas expansion patterns. Consequen tly ,
the c haracteristics of the en vironmen t are aected. In this con text, monitoring natural
and man-made assets (critical or non-critical) to determine their vulnerabilities and oer
98
Metho ds and algorithms for Earth Observation image information mining
rapid resp onse is crucial, and the imp ortance of Geographic Information Systems for
public safet y and related domains is b ecoming more and more recognized.
Ho w ev er, GIS en vironmen ts rely on the user in teraction and input (georeferenced
data) and furnish the instrumen ts for data analysis. F rom this p ersp ectiv e, GIS are pas-
siv e systems and are limited b y what the user can see. By adding imagery (eg. EO
imagery) analysis in tegrated with mac hine learning w e can go b ey ond these capabilities,
incorp orating means for kno wledge disco v ery . In the GIS-DM pro ject, the goal is to
dev elop a GIS platform with in tegrated Data mining to ols that can ha v e a big impact
in situation a w areness regarding critical infrastructure (CI) monitoring. The inno v ativ e
functionalities of this GIS platform with in tegrated DM to ols are to automatically dis-
co v er, lo cate and lab el t yp ologies that aren't customarily indicated in EO image data.
The immediate added v alue is the enhancemen t in the reaction time in case of a crisis
managemen t and CI monitoring scenario and the minimization of the amoun t of time
needed for an action resp onse. In the case of Situational A w areness (SA), an in tegrated
Data mining to ol in to a GIS system can b e of vital imp ortance. The main adv an tages of
the prop osed protot yp e are: the p ossibilit y to p erform GIS-driv en Data Mining or spatial
Data Mining, a data-driv en approac h to geographical analysis and mo deling, in tegra-
tion of seman tic searc hes for kno wledge disco v ery , enhanced represen tations o v er dynamic
spatio-temp oral dimensions, kno wledge disco v ery from v arious data t yp es (v ector and
raster, geo-referenced imagery), p ossibilit y to assess qualit y of results through confusion
matrices, while at the same time ha ving the capabilities to generate co v erage maps of
analyzed areas, and pro vide the to ols needed for geographical analysis.
The in tegrated DM mo dule will p erform similarit y searc hes and iden ti cation of pat-
tern sequences based on the input pro vided b y the GIS platform (in terms of parameters,
con gurations, constrain ts, training sets). One of the inno v ations will b e the selection
pro cedure whic h will allo w the in teraction with div erse t yp es of geospatial information.
The usage of Data Mining o v er c hange detection maps and/or m ulti-temp oral analysis is
another immediate adv an tage of the solution. But the k ey inno v ativ e asp ect is related to
constructing an e cien t Data Exc hange Mo del b et w een the t w o ma jor comp onen ts of the
system. This Data Exc hange Mo del shall constitute the bac kb one of the system and will
enable the addition of more functionalities and mo dules in further dev elopmen ts.
As the ESA Cop ernicus Programme con tin ues to gather resources from o v er 30 con-
tributing missions and is con tin uously w elcoming inno v ativ e dedicated missions lik e the
SENTINELS (S-1,2,3 already in orbit), the data v olume p oses new c hallenges to the user
comm unit y , whic h calls for en vironmen ts that facilitate user in teraction with the data and
kno wledge disco v ery . Romania is curren tly building capacities for EO data pro cessing,
kno wledge disco v ery and visualization. The pro ject targets the ESA Earth Observ ation
programme. The pro ject activities, researc h domain and applications are in line with
the Cop ernicus thematic streams: land monitoring, emergency managemen t and securit y .
The dev elopmen t of the GIS-DM protot yp e will on the one hand v alidate the concept
of GIS-Driv en Data Mining and will pro vide su cien t use case scenarios to sustain the
con tin uation of the dev elopmen t up to a higher TRL (curren tly start TRL 2 – end TRL
3) under future industrial partnerships. Th us, the pro ject shall generate new initiativ es
for ESA researc h pro jects, fostering co op eration, inno v ation and promoting the usage of
ESA EO resources to build solutions b oth for critical applications and ev eryda y citizen
needs.
Th us, GIS-DM in tends to dev elop a GIS platform with in tegrated Data mining to ols
99
Chapter 5. Data Mining Systems
Figure 5.20: GIS-DM system arc hitecture
that can ha v e a big impact in situation a w areness regarding critical infrastructure (CI)
monitoring. The inno v ativ e functionalities of this GIS platform with in tegrated DM to ols
are to automatically disco v er, lo cate and lab el t yp ologies that aren't customarily indicated
in EO image data. The immediate added v alue is the enhancemen t in the reaction time
in case of a crisis managemen t and CI monitoring scenario and the minimization of the
amoun t of time needed for an action resp onse. In the case of Situational A w areness (SA),
whic h refers to the p erception of en vironmen tal elemen ts with resp ect to time and space,
the comprehension of their meaning, and the pro jection of their status after some v ariable
has c hanged, an in tegrated Data mining to ol in to a GIS system can b e of vital imp ortance.
The system can also generate co v erage maps of analyzed areas, and can pro vide the to ols
needed for geographical analysis.
The pro ject sustainabilit y is ensured b y the existing researc h directions and activities
ongoing in b oth partnering institutions, and b y the Europ ean trends. The m ultitude of
thematic platforms (e.g. ESA Thematic Exploitation Platform) and collab orativ e en vi-
ronmen ts are pro v en in terests in the exploitation of EO resources that go b ey ond the
classical approac hes.
100
Metho ds and algorithms for Earth Observation image information mining
The prop osed approac h will ha v e an in terdisciplinary c haracter, due to the fact that
the prop osed solutions will in tegrate elemen ts of mathematical mo deling, statistical analy-
sis, computer science and mac hine learning, heterogeneous data storing and manipulating,
databases tec hnologies, in synergy with geographic analysis. This will increase the exp er-
tise of the established researc h teams and will generate collab orations b et w een exp erts
and with the target user groups.
5.5.2 Multi-lev el Indexing Structures for Big Data Analysis
In the frame of m ulti-lev el indexing structures for big data analysis, the goal is to
establish a complex net w ork of indexes describing the EO data and its con ten t, a space-
time-feature h yp ercub e designed to create a ma jor impact on the op erabilit y of EO data
for researc hers, end users and stak eholders. The prop osed concept is a unifying paradigm
to encourage data a v ailabilit y and enable v ery fast queries in the Big Data era. In order
to do so, new functionalities will b e em b edded to allo w fast com binations of data and
sp eci c analysis to mo del the data based on the user's in terest.
Figure 5.21: Multi-lev el Indexing Structures for Big Data Analysis system arc hitecture
The arc hitecture will not b ound the size of the index catalogue in the database, nor
will comp el the user to c ho ose for a feature or another, but will allo w him to create its
o wn, to transfer his kno wledge through activ e learning and store seman tic features in to
the database. This h uman data in teraction will comp ensate for the lac k of univ ersal
information mining pro cedures and try to cop e with the data dynamics and heterogeneit y .
F ortunately , the tec hnological dev elopmen t is supp orting the need for a h uge storage. Un-
fortunately , the amoun t of information that one can obtain from EO data is exp onen tially
increasing with the acquisition of new data. Databases require smart structuring to w ell
organize all that information. The prop osed concept in tro duces a m ulti-lev el approac h to
align and connect the metadata and v arious features in order to scale whilst the size of
the database increases without in tro ducing corruption or resulting in an o v er-cro wded in-
dexing structure.The prop osed scalable space-time-feature data framew ork, together with
query to ols, is a promising solution to dramatically reduce the pro cessing time o v erheads
for Big Data related queries.
The pro ject aims at de ning an indexing concept for impro ving the exploitation,
manipulation and in terop erabilit y of heterogeneous data sources. The EO-GRID concept
aims at transforming the EO data arc hiv es from `nice to ha v e' collections with limited use
in to a living and accessible arc hiv e. The stream of dev elopmen t impro v es the fundamen tal
pro cesses and algorithms that are used to transform the data in to useful pro ducts, as the
feature extraction mo dule is directly connected to the database. There is en visaged a
m ulti-lev el indexing structure, a space-time-feature h yp ercub e to increase the access to Big
101
Chapter 5. Data Mining Systems
EO Data arc hiv es while in tegrating heterogeneous data, fast pro cessing and visualization
paradigms to exploit the high p oten tial of past, curren t and future sensors for scien ti c
needs and stak eholders in terest. A protot yp e system will b e build to demonstrate the
concept. An application scenario in v olving the monitoring of Buc harest cit y is en visaged.
5.5.3 Multisp ectral Data Analysis T o olb o x for SNAP
In the frame of Multisp ectral Data Analysis T o olb o x for SNAP the ob jectiv es are
directed to w ards the elab oration of dedicated algorithms for con ten t description and se-
man tic analysis of Sen tinel 2 data. Sp ectral band disco v ery for exploratory visual analysis
will b e added in supp ort of MSI image understanding.
An unitary data mining framew ork is en visaged in order to enhance relev an t asp ects
and transp ose the Sen tinel 2 data in to actionable information. The goal is to design,
implemen t and in tegrate a to ol for m ultisp ectral data analysis to act as a SNAP to ol-
b o x. There will b e 3 separate soft w are mo dules including dieren t algorithms for feature
extraction, classi cation and exploratory data analysis. The arc hitecture of SNAP will
pro vide the general framew ork to connect all the individual mo dules.
The elab oration and implemen tation of eac h comp onen t, together with the nal in te-
gration will b e consisten t with the detailed design. Fig. 5.22 illustrates a generic concept
of the prop osed to olb o x, where the main soft w are comp onen ts for data pro cessing lik e
feature extraction, classi cation and exploratory visual analysis are link ed through the
Data preparation mo dule to the Sen tinel 2 data collection.
Figure 5.22: Multi-lev el Indexing Structures for Big Data Analysis system arc hitecture
Algorithms for con ten t seman tic description and exploratory visual analysis adapted
for Sen tinel 2 image c haracteristics will b e included in the prop osed framew ork. In the
b eginning, a review of the tec hnical sp eci cation of Sen tinel 2 sensor is required. Once the
data understanding is completed, one is able to de ne use cases and applications scenarios
and also to adapt and implemen t feature extraction tec hniques for the particularities of
Sen tinel 2 data con ten t. The image con ten t is comp osed of a m ultitude of primitiv e
features and structures.
The nal goal of the pro ject is a soft w are pac k age, a to olb o x including algorithms
sp eci cally designed to cop e with Sen tinel 2 data. Ho w ev er, the application w e are tar-
geting is the in tegration of the soft w are pac k age as a plugin to the Sen tinel 2 to olb o x for
the ESA Sen tinel Application Platform (SNAP).
102
Metho ds and algorithms for Earth Observation image information mining
5.6 Conclusions
This c hapter is fo cused on extending the prop osed feature extraction framew ork to
b e in tegrated in an EO data mining system. T aking in to accoun t h uman mac hine comm u-
nication principles w e established the data o w inside the prop osed data mining system.
F urthermore in this c hapter w e demonstrated the usabilit y of prop osed framew orks for
b oth feature extraction and data mining, b y dev eloping a con ten t based query concept for
exploration and disco v ery of information in earth observ ation and medical libraries and
of a w eb platform for EO image information mining. Also, the dev elop ed framew orks will
b e in tegrated in to future applications that will ha v e a great impact in the remote sensing
data understanding.
103
Chapter 6
Conclusions
This thesis concen trates on the enhancemen t and ev aluation of feature extraction
metho ds for Earth observ ation image data analysis. Ev en though our atten tion w as
fo cused on data in terpretation and understanding, in the dev elopmen t of the feature
extraction metho ds, the en tire lifecycle of remotely sensed data w as closely analyzed.
In our assessmen ts w e mo di ed state of the art feature extration metho ds for sp ec-
tral, texture and shap e analysis to b e complian t with earth observ ation image data. In
the con text of sp ectral analysis w e computed sp ectral histograms, sp ectral indices and
p olar co ordinates transformation of the sp ectral v alues. W e pro v ed that these features
based on the sp ectral v alues of the remotely sensed images are e cien t and easy to use
when compared with other feature extraction metho ds. Moreo v er, sp ectral features do es
not pro vide an y spatial information ab out the distribution of the sp ectral v alues in the
image, but can b e successfully used in con ten t based image retriev al applications due to
the simplicit y of extracting color information from images and represen ting visual con ten t.
F urthermore, w e used sp ectral features and p olar co ordinates transformation of the sp ec-
tral v alues to create new feature descriptors using the bag-of-w ords framew ork, increasing
this w a y the p erformances of the initial features.
Most of the remotely sensed images ha v e their con ten t c haracterized b y rep etitiv e
patterns that ha v e an uniform spatial distribution of the pixel in tensities. Th us, in this
con text, texture analysis metho ds can pro vide a b etter represen tation of the image con-
ten t. In our assessmen t of feature extraction metho ds for texture analysis w e computed
Gab or, WLD, and edge histogram descriptors in a patc h-based approac h. F urthermore,
in the frame of shap e analysis w e prop osed a mo di ed BRIEF(binary robust indep enden t
elemen tary features) descriptor that can b e used for patc h classi cation instead of its
original purp ose of image matc hing.
In the quest of dev eloping feature extraction metho ds that can b e in tegrated within a
standard, our in terest w as fo cused on visual descriptors pro vided b y MPEG-7 standard, for
b oth sp ectral and texture analysis. W e mo di ed image descriptors lik e Gab or (HTD), edge
histogram descriptor (EHD) and scalable color descriptor (SCD), to handle remote sensing
image data using a patc h-based approac h. Our descriptors are taking in to accoun t full
sp ectral resolution of the image b eing analyzed. Th us, in the case of m ultisp ectral images
with more than three sp ectral bands, the computation of SCD is done b y p erforming a
transformation of the sp ectral v alues space in to p olar co ordinates in order to obtain a
HSV lik e sp ectral space on whic h w e could compute scalable color descriptor.
104
Metho ds and algorithms for Earth Observation image information mining
The step of kno wledge extraction and data understanding is done using unsup ervised
and sup ervised learning metho ds suc h as k-Means, k-NN and SVM. Moreo v er, qualit y and
quan tit y ev aluation is done using confusion matrices, obtained from comparing a man ual
annotation of the image data with the results of the sup ervised classi cation.
The implemen ted metho ds are pro viding the necessary basis for the dev elopmen t
of an image information mining system. Ha ving in to accoun t the framew ork prop osed
for kno wledge extraction from EO image data, w e also dev elop ed a data mining system
capable to p erform op erations lik e feature extraction, classi cation, qualit y and quan tit y
ev aluation and data visualization on b oth optical and syn thetic ap erture radar data. Also
the system arc hitecture implemen ted in the data mining to ol and the feature extraction
metho ds as w ell w ere implemen ted in an image information mining protot yp e whic h w as
upgraded in to a fully functional data mining system that can b e accessed from a w eb
platform. As presen ted in a previous c hapter, further dev elopmen ts are also tak en in to
accoun t.
105
Copyright Notice
© Licențiada.org respectă drepturile de proprietate intelectuală și așteaptă ca toți utilizatorii să facă același lucru. Dacă consideri că un conținut de pe site încalcă drepturile tale de autor, te rugăm să trimiți o notificare DMCA.
Acest articol: In the last decades w e can observ e a con tin uous gro wth of the remote sensing data, [615320] (ID: 615320)
Dacă considerați că acest conținut vă încalcă drepturile de autor, vă rugăm să depuneți o cerere pe pagina noastră Copyright Takedown.