Dicuionutdisertatietapi [618364]

THE UNIVERSITY OF PITEȘTI
THE FACULTY OF SCIENCE, PHYSICAL EDUCATION AND
INFORMATICS

Dissertation

Scientific coordinator ,
Lecturer univ.dr.mat. Miroiu Maria
By
Dicu Florea -Ionuț

Pitești
2017

THE UNIVERSITY OF PITEȘTI
THE FACULTY OF SCIENCE, PHYSICAL EDUCATION AND
INFORMATICS
ADVANCED TECHNIQUES FOR INFORMATION
PROCESSING

Dissertation
Regression Analysis and Applications

Scientific coordinator ,
Lecturer univ.dr.mat. Miroiu Maria
By
Dicu Florea -Ionuț

Pitești
2017

Contents

Introduction ………………………….. ………………………….. ………………………….. ………………………….. ……………… 4
Linear Regression ………………………….. ………………………….. ………………………….. ………………………….. ………. 6
THE LINEAR REGRESSION MODEL ………………………….. ………………………….. ………………………….. …………. 6
The Least Squares Estimates ………………………….. ………………………….. ………………………….. …………….. 9
Likelihood Inference Results ………………………….. ………………………….. ………………………….. …………… 12
Bayesian Inference Results ………………………….. ………………………….. ………………………….. …………….. 13
Nonlinear Regression ………………………….. ………………………….. ………………………….. ………………………….. .. 16
THE NONL INEAR REGRESSION MODEL ………………………….. ………………………….. ………………………….. … 17
Transformably Linear Models ………………………….. ………………………….. ………………………….. …………. 18
The Geometry of the Expectation Surface ………………………….. ………………………….. …………………….. 19
DETERMINING THE LEAST SQUARES ESTIMATES ………………………….. ………………………….. ……………….. 21
The Gauss -Newton Method ………………………….. ………………………….. ………………………….. ……………. 21
The Geometry of Nonlinear Least Squares ………………………….. ………………………….. ……………………. 25

Introduction

Regression was used for the first time as the method of least squared presented by
Legendre in 1805, and followed by Gauss in 1809. The problem of determining the orbits of
bodies(comets and newly discovered minor planets). Gauss published the theory of least squares
in 1821 contains the Gauss -Markov theorem.
"Regression" term was given by Francis Galton in the 19th century . This term was used
to describe a biological phenomenon that observed t he heights data of descendants ancestors tend
to revert approaching a normal average known as regression toward the mean . Galton work was
extended by Udny Yule and Karl Peterson to the joint distribution .
Recently, new methods were developed for robust regression using time series and
expansion curves where the response v ariable s are complex data objects.
Regression analysis is used in statistical modeling as statistical process for estimating the
contact through variables wich includes advanced techniques used for modeling and analyzing
big variables data between a predictors dependent variable and one/ more independe nt variables .
Regression analysi s helps understand how a dependent variable value changes when a
single of the independent variable varied . Conditional anticipation of the dependent variable
dispose the average value of dependent variable in case of the independent variables are
established .
The estimation on regression analysis target represent the independent variables named
the regression function that is also of interest to enumerate the variation of the dependent
variable used around the regressi on function who was described by a probability distribution .
A related of regression function is necessary condition analysis(NCA) and estimates the
highest value of the dependent variable used for a given independent variable data.
Regression analysis is used in our days for prediction s and for a largest variety of
machine learning systems . Another use is to interpret which among the independent variables
data are linked to the dependent variable, and to find the forms of the relationships .
Regression analysis is also used to deduct causal relationship between the independent
and dependent variables data. This can guide to illusions or false relationships .

Some t echniques used for regression analysis were created linear regression s followed
by ordinary least squares regression, defined as a finite number used for unknown parameters
and estimated using an amount of data.
Another techniques that allow the regression function to stand in a set of finite –
dimensional or infinite -dimensional functi ons was named as nonparametric regression .
The data generating process and regression analysis use some extent on making
assumptions to make testable if data is available. Regression models used for prediction are
useful if the assumptions are quite offended and may not perform in parameter . In some
applications , regression methods can give baffling results.
Regression refers to the estimation of continuous response variables data, negate the
discrete feedback of the variables . The continuous output v ariable case may referred to the
metric regression to differentiate it from used related problems .

Linear Regression

Regression analysis is used to find a relation between the values of two or more
variables data, of which at least one use random vari ation, and to test a
relation. It is used for detecting relation between parameters data from different places, between
the parameters data of model between water paramete rs and soil parameters, between
productivity and weather , and so on.

THE LINEAR REGRESSION MODEL

Linear regression gives estimates and presumed results for the parameters
in the model:

represents the feedback for case n and has a deterministic part and a
stochastic part represented by the random variable .
For N cases the linear regression model can be written:

where represents the vector of random variables and is a matrix of regressive variables of
dimension:
[

]
and is the disturbances vector of random variables.
A nonzero represents for can be integrated into a function:

[ ] [ ]
We call the expectation function used for the regression model. is the derivative
matrix, where (n,p)th is derivative term of the expectation function with pth parameter respect.
Let be normally distributed with :
[ ] [ ] ,
where is identi ty matrix of dimension and the joint density function for Y with the
variance is:
| (
) ( || ||
)

Application :

% Simple linear regression.
mpg = [26.5; 28.2; 33.1; 36.1; 32.8; 23.0]; % Object's weight
wt = [2.660; 2.360; 1.275; 2.115; 1.920; 3.015];
% Calculate standard deviations and means:
mean(wt)
std(wt)
mean(mpg)
std(mpg)
n = length(mpg); % n = sample size
X = [ones(6,1) wt]; % Compute the design matrix X
bhat = inv(X'*X)*X'*mpg; % estimated regression coefficients
yhat = X*bhat; % fitted values
r = mpg – yhat; % residuals
plot(wt, mpg, 'o', wt, yhat)
title('Object Data' )
axis([1.5, 3, 20, 40])
ylabel('Miles per Gallon (mpg)' ) % 1 US gallon = 3.78541178
liters
xlabel('Weight of the Object' )
% Draw residuals versus fitted values using plot function :
plot(yhat, r, 'o', linspace(20,40,n), zeros(n,1))
xlabel('Fitted Values' )
ylabel('Residuals' )
title('Residual Plot' )
% fit a polynomial to the dat a and the last number is the degree
of the polynomial
polyfit(wt, mpg, 1)

Output:
mean(wt) = 2.2242
std(wt)= 0.6065
mean(mpg) = 29.9500
std(mpg) = 4.8813
polyfit(wt, mpg, polynomial_degree) =
-6.3973 44.1787

The Least Squares Estimates

The like lihood function | for and , is identical to the joint probability
frequency , with the conditional parameters. The li kehood is a function of the responses
conditional of the parameters, suppressing using the constant and we write:
| || ||

The like lihood function is maximized using respect to on the residual sum of squares is
a minimum :
|| || ∑[ ∑
]

The maximum like lihood evaluate ̂ which maximizes . ̂ is the least squared
evaluation and is written as:
̂ .
Using sampling t heory for least squares estimates where the least squares estimator i s the
minimum variance for and ̂ is the mode function of the marginal posterior frequency for .

Application :
The least squares fitting using linear algebra in matlab :
clc; % clear the console
% fit a line using given data to extract the gradient.
x = [0 0.3 1.2 1.3 2 4 6 7 8]; % time
y = [0 -0.123 -0.132 -0.432 -0.235 -0.764 -1.546 -1.444 -2.111];
% dimensionless

figure
plot(x,y, '*')

X = [(x.^0)' x'] % matrix form

% solve linear equations
b = (X'*X) \(X'*y')
b2 = b(2)

% the least squares
b = X\y'
b2 = b(2)

% b(1) is the intercept and b(2) is the gradient. We draw a
simple function of the fit on the data.
fit = @(z) b(1) + b(2)*z;
hold all
h = plot(x, X*b, 'r- ');

Output:
X =
1.0000 0
1.0000 0.3000
1.0000 1.2000
1.0000 1.3000
1.0000 2.0000
1.0000 4.0000
1.0000 6.0000
1.0000 7.0000
1.0000 8.0000

b =
0.0599
-0.2459

b2 =
-0.2459

b =
0.0599
-0.2459

b2 =
-0.2459

Likelihood Inference Results

The like lihood | depends on where || || and likelihood contours has the form:
|| ||
and = constant. The contour of a likelihood region for which c is identical joint
determination region :
( ̂)[
]
Application :
Maximum Likehood Estimation in matlab:
clc;
npop = 1; % Population parameter
n = 100; % Sample size
nsamp = 1000; % Number of samples

rng default % For reproducibility
samples = exprnd(npop,n,nsamp ); % Population samples
means = mean(samples); % Sample means

% The normfit function is used to find the normal distribution
which fits the means
[muhat,sigmahat,muci,sigmaci] = normfit(means)
% The function returns MLEs for:
% the mean and standard deviation
% their 95% confidence intervals.

numbins = 10;
hist(means,numbins)
hold on
[bincounts,binpositions] = hist(means,numbins);
binwidth = binpositions(2) – binpositions(1);
histarea = binwidth*sum(bincounts);
x = binpositions(1):0.001:binpositions (end);
y = normpdf(x,muhat,sigmahat);
% Visualize the distribution of sample means with the specific
fitted normal distribution
plot(x,histarea*y, 'r','LineWidth' ,2)

Bayesian Inference Results

The Bayesian marginal posterior frequency used for and has form :

̂ ̂

where -variate People has T density with ̂ location parameter, scaling matrix and
degrees of freedom are represented by .

Example :
We calculate parameter estimates and inference region using ̂ .
Let ̂
[
]

[
]
[
]

The inference region :

Application :
Bayesian Inference is the process of analyzing statistical models using the model or model
parameters.

The posterior distribution for theta is:

Bayesian Inference application in matlab:
rng(0,'twister' );

n = 25;
sigma = 55;
x = normrnd(10,sigma,n,1);
mu = 35;
tau = 25;
theta = linspace( -45, 105, 505);
y1 = normpdf(mean(x),theta,sigma/sqrt(n));
y2 = normpdf(theta,mu,tau);
postMean = tau^2*mean(x)/(tau^2+sigma^2/n) +
sigma^2*mu/n/(tau^2+sigma^2/n);
postSD = sqrt(tau^2*sigma^2/n/(tau^2+sigma^2/n));
y3 = normpdf(theta, postMean,postSD);
plot(theta,y1, '-', theta,y2, '–', theta,y3, '-.')
legend('Likelihood' ,'Prior','Posterior' )
xlabel('\theta')

% Truck Experiment Data
% A set of truck weights
weight = [2705 2805 2550 2700 2400 2900 3400 3600 3500 3600 3400
3900]';
weight = (weight – 2900)/1000; % centering and scaling
% Tested trucks for each weight
total = [47 90 31 34 31 21 23 23 21 16 19 23]';
% The number of trucks with poor mpg performances for each
weight
poor = [2 4 0 4 8 8 14 17 19 15 17 21]';

% Logistic regression is a special case of a generalized linear
model
% the response variable is bino mial: X = design matrix, b =
vector of model parameters
vlogitp = @(b,x) exp(b(1) + b(2) .* x) ./ (1 + exp(b(1) + b(2)
.* x));

% use normal priors
prior1 = @(b1) normpdf(b1,0,22); % prior for intercept

prior2 = @(b2) normpdf(b2,0,22); % prior for slope

% the joint posterior distribution(of the model parameters) ? to
the likelihood and priors
post = @(b) prod(binopdf(poor, total, vlogitp(b, weight))) *
prior1(b(1)) * prior2(b(2));

simpost = zeros(5,5);
b_1 = linspace( -3, -1.2, 55);
b_2 = linspace(3.5, 5, 55);
for i = 1:length(b_1)
for j = 1:length(b_2)
simpost(i,j) = post([b_1(i), b_2(j)]);
end;
end;
mesh(b_2,b_1,simpost)
xlabel('Slope axis' )
ylabel('Intercept axis' )
zlabel('Posterior density' )

..

Nonl inear Regression

Linear regression techniques can be extended and can have considerable complexity. A
mathematical expression that relates the regression response to the predictor variables data is
called nonlinear regression.

THE NONLINEAR REGRESSION MODEL

A nonlinear regression model has the form:

where represent s the expectation function and represents a vector of an correlated regressor
or a vector of independent variables data for the -th case. This module has the same form as the
linear regression model, except that the conventional responses are represented by nonlinear
functions of used parameters. A nonlinear model use a t least one of the regression derivatives of
the expectation function with deference to the parameters data lean on at least one of the data
parameter s.
Consider the vectors , , as fixed and attend on the dependence of the
conventional responses data on , where is used for the parameters data in a nonlinear model
and is used for the number of parameters,
The -vector with -th element has form:

where and the nonlinear regression model has form:

where have a conventional spherical distribution and meets the next condition s:
[ ]
[ ]

Transformable Linear Models

We use transformable linear models by indicate the reciprocal function of the velocity
function of the mutual substrate :

A transformation of the data means a transformation of the interruption term and affects
the expectation on it. Spherical normal disturbance term is a relevant representation of the
experimental case where the assumptions will not be closest for the transformed data. We use
nonlinear regression on the original regression data and on the weighted least squares
transformed data. When t ransforming a regression data set to generate const ant variance and
produce a linear expectation function used on the transformed regression data.

Application :
Laplace transform ation:
% the inverse Laplace transform with respect to y at the transformation x:
syms x y
F = 1/y^2;
ilaplace(F, y, x)

% the inverse Laplace transform calling the ilaplace function:
syms a s x
F = 1/(s – a)^2;
ilaplace(F, x)

Output:
ans =
x

ans =
x*exp(a*x)

The Geometry of the Expectation Surface

A spherical regular distribution assumption used for the disturbance term Z let us to grant
the N-dimensional response space of the Euclidean geometry for attracted in the least squares
estimates ̂ of the regression parameters. The -vectors determine a P-dimensional surface
named the expectation surface used in the regression response space and the least squares
estimates correlate to the point used on the regression expectation surface:
̂ ( ̂)
Where ̂ is used to minimize the residual regression sum of squares:
|| ||

Application:
Find a closer approximation in matlab:
x = [9 99 999 9999]
n = length(x)
C = zeros(n,n)
for i = 1:n
for j = 1:n
C(i,j) = mean(x(i)*x(j)); % x(i)*x(j) is scalar
end
end

% find a closer approximation
X = [10 100 1000 10000]
xC = zeros(n,n)
for i = 1:n
for j = 1:n
xC(i,j) = mean(X(:,i).*X(:,j));
end
end

Xn = X(1:end -n) % shifted signal
t = 0:0.1:3 % time vector
X = exp(t) % X(t)
n = 0.2 % time value used for delay
tn = t(t>n) % delayed time value
Xn = X(1:length(tn)) % delayed signal value
plot(t,X, 'r*-',tn,Xn,'g.-') % plot expectation

DETERMINING THE LEAST SQUARES ESTIMATES

The least squares estimates use a data vector y and an expectation function
. Note the expectation surface is closest to y with the point ̂ and determine ̂(parameter
vector) for the point ̂.
The explicit expression for ̂ which is closest to y is ̂ and for a nonlinear
model has P -dimensional plane , which maps to the expectation plane , we have ̂ ̂.

The Gauss -Newton Method

Gauss started to use an initial guess for , where represents a linear approximation applied
to the expectation function :
( ) ∑ ( )
|

For all N cases , we have:
( ) ( )
and is derivative matrix with elements.
Approximation the residuals we have:

( )

where:

( )

Next step is to calculate the Gauss increment to minimize the residual approximation
sum of squares:
|| ||

̂
and = , ̂ ( ) ( )
Next step is to move to the better parameter value and calculati ng new
residual s . We calculate using this process until the increment is very small for
change the vector parameter.
Apllication :
Gauss Newton method in matlab:
function [unknowns,steps,S] = GaussNewton()
% The Gauss -Newton method perform a non -linear least
% squares appr oximation for the origin of a circle of points
% We use as input a row vector of x and a row vector y values,
and an initial guesses vector
% In the df function must entered partial derivatives for the
jacobian
format long
tol = 100000; %set accuracy
maxstep = 55; %set maximum steps
z = 2*pi*rand(55,1); %create 55 random points
p = cos(z); %create a circle with given points
q = sin(z);
d = ones(55,1); %set distance(for all points) to origin as 1
a = [0.1;0.1]; %set initial guess for origin
m=length(p); %determine number of functions
n=length(a); %determine number of unkowns
aold = a;
for k=1:maxstep %iterate through process
S = 0;
for i=1:m
for j=1:n

J(i,j) = df(p(i),q(i),a(1,1),a(2,1),j); %calculate
Jacobian
JT(j,i) = J(i,j); %calculate trnaspose
end
end
Jz = -JT*J; %multiply Jacobian and negative transpose
for i=1:m
r(i,1) = d(i) – sqrt((a(1,1) -p(i))^2+(a(2,1) -q(i))^2);
%calculate r
S = S + r(i,1)^2; %calculate sum of squares of residuals
end
S
g = Jz\JT; %mulitply inverse of Jz by J transpose
a = aold -g*r; %search for new approximation
unknowns = a;
err(k) = a(1,1) -aold(1,1); %calculate error
if(abs(err(k)) <= tol); %break loop if error is less than
tolerance
break
end
aold = a;
end
steps = k;
hold all
plot(p,q, 'bx ') %plot the data points
plot(a(1,1),a(2,1), 'bo ') %plot the approximation points of the
origin
title('Gauss-Newton Approximation Origin of Circular Data
Points')
xlabel('X')
ylabel('Y')
legend('Data Points' ,'Gauss-Newton Approximation of Origin' )
hold off
end

function value = df(p,q,a1,a2,index) %calculate partials
switch index
case 1
value = (2*a1 – 2*p)*0.5*((a1 -p)^2+(a2 -q)^2)^(-0.5);
case 2
value = (2*a2 – 2*q)*0.5*((a1 -p)^2+(a2 -q)^2)^(-0.5);
end
end

Output:

S = 0.577024054502000
ans =
0.000510594811903
-0.001525253388172

The Geometry of Nonlinear Least Squares

The geometry of a Gauss -Newton iteration incorporate:
1. An approximation using a Taylor sequence expansion
2. The residual generator vector
3. The residual projected on the tangent plan with ̂ result ,
4. Mappin g ̂ over the linear coordinate system to output the increment
5. Approach .

We approximate the planar assumption(the expectation surface near ) and the uniform
coordinate assumption(a linear coordinate system V( ) on the relative to tangent plane)

Application :
How solve a nonlinear least squares problem in matlab:
Nonlinearleastsqfunc.m:
function [F,J] = nonlinearleastsqfunc(x)
k = 1:10;
F = 2 + 2*k -exp(k*x(1)) -exp(k*x(2));
if nargout > 1
J = zeros(10,2);
J(k,1) = -k.*exp(k*x(1));
J(k,2) = -k.*exp(k*x(2));
end

Computes the objective function values and draw those:
function F = nonlinearleastsqfunc(x)
clc
x=[0.2578 0.2578]
k = 1:10;
F = 2 + 2*k -exp(k*x(1)) -exp(k*x(2));
plot(F, '*-')

Output:
x =
0.2578 0.2578

ans =
Columns 1 through 5
1.4118 2.6507 3.6658 4.3911 4.7417
Columns 6 through 10
4.6072 3.8449 2.2704 -0.3554 -4.3415

Bibliography

1. Douglas M. Bates and Donald G. Watts, Nonlinear Regression Analysis and Its
Applications , second edition, A Wiley -Interscience Publication, JOHN WILEY &
SONS, INC.
2. R.J. Oosterbaan , Drainage Principles and Applications, Publication 16 , second
revised edition , International Institute for Land Reclamation and Improvement(ILRI),
H.P.R itzema (Ed.), Wageningen, The Netherlands , 1994
3. https://en.wikipedia.org/wiki/Regression_analysis
4. http://matlab.cheme.cmu.edu/2011/09/24/linear -least-squares -fitting -with-linear –
algebra/
5. https://www.mathworks.com/help/stats/maximum -likelihood -estimation.html
6. https://www.mathworks.com/help/stats/exampl es/bayesian -analysis -for-a-logistic –
regression -model.html
7. http://www.wright.edu/~thaddeus.tarpey/STT363chapter5.pdf
8. https://ccrma.stanford.edu/~wherman/tulane/gauss_newton.pdf
9. https://www.mathworks.com/help/optim/ug/nonlinear -least-squares -with-full-
jacobian .html
10. https://dsp.stackexchange.com/questions/19291/matlab -how-do-i-implement -the-
expectation -operator -for-time-series
11. https:// www.mathworks.com/help/symbolic/ilaplace.html