Dicuionutdisertatietapi [618361]

THE UNIVERSITY OF PITEȘTI
THE FACULTY OF SCIENCE, PHYSICAL EDUCATION AND
INFORMATICS

Dissertation

Scientific coordinator ,
Lecturer univ.dr.mat. Miroiu Maria
By
Dicu Florea -Ionuț

Pitești
2017

THE UNIVERSITY OF PITEȘTI
THE FACULTY OF SCIENCE, PHYSICAL EDUCATION AND
INFORMATICS
ADVANCED TECHNIQUES FOR INFORMATION
PROCESSING

Dissertation
Regression Analysis and Applications

Scientific coordinator ,
Lecturer univ.dr.mat. Miroiu Maria
By
Dicu Florea -Ionuț

Pitești
2017

Contents

Introduction ………………………….. ………………………….. ………………………….. ………………………….. ……………… 4
Linear Regression ………………………….. ………………………….. ………………………….. ………………………….. ………. 6
THE LINEAR REGRESSION MODEL ………………………….. ………………………….. ………………………….. …………. 6
The Least Squares Estimates ………………………….. ………………………….. ………………………….. …………….. 9
Likelihood Inference Results ………………………….. ………………………….. ………………………….. …………… 12
Bayesian Inference R esults ………………………….. ………………………….. ………………………….. …………….. 13
Nonlinear Regression ………………………….. ………………………….. ………………………….. ………………………….. .. 15
THE NONLINEAR REGRESSION MODEL ………………………….. ………………………….. ………………………….. … 16
Transformably Linear Models ………………………….. ………………………….. ………………………….. …………. 17
The Geometry of the Expectation Surface ………………………….. ………………………….. …………………….. 18
DETERMINING THE LEAST SQUARES ESTIMATES ………………………….. ………………………….. ……………….. 19
The Gauss -Newton Method ………………………….. ………………………….. ………………………….. ……………. 19
The Geometry of Nonlinear Least Squares ………………………….. ………………………….. ……………………. 23

Introduction

Regression was used for the first time as the method of least squared presented by
Legendre in 1805, and followed by Gauss in 1809. The problem of determining the orbits of
bodies(comets and newly discovered minor planets). Gauss published the theory of least squares
in 18 21 contains the Gauss -Markov theorem.
"Regression" term was given by Francis Galton in the 19th century . This term was used
to describe a biological phenomenon that observed t he heights data of descendants ancestors tend
to revert approaching a normal ave rage known as regression toward the mean . Galton work was
extended by Udny Yule and Karl Peterson to the joint distribution .
Recently, new methods were developed for robust regression using time series and
expansion curves where the response v ariable s are complex data objects.
Regression analysis is used in statistical modeling as statistical process for estimating the
contact through variables wich includes advanced techniques used for modeling and analyzing
big variables data between a predictors depen dent variable and one/ more independent variables .
Regression analysi s helps understand how a dependent variable value changes when a
single of the independent variable varied . Conditional anticipation of the dependent variable
dispose the average value o f dependent variable in case of the independent variables are
established .
The estimation on regression analysis target represent the independent variables named
the regression function that is also of interest to enumerate the variation of the dependen t
variable used around the regression function who was described by a probability distribution .
A related of regression function is necessary condition analysis(NCA) and estimates the
highest value of the dependent variable used for a given independent variable data.
Regression analysis is used in our days for prediction s and for a largest variety of
machine learning systems . Another use is to interpret which among the independent variables
data are linked to the dependent variable, and to find the form s of the relationships .
Regression analysis is also used to deduct causal relationship between the independent
and dependent variables data. This can guide to illusions or false relationships .

Some t echniques used for regression analysis were created linear regression s followed
by ordinary least squares regression, defined as a finite number used for unknown parameters
and estimated using an amount of data.
Another techniques that allow the regression function to stand in a set of finite –
dimensional or i nfinite -dimensional functions was named as nonparametric regression .
The data generating process and regression analysis use some extent on making
assumptions to make testable if data is available. Regression models used for prediction are
useful if the a ssumptions are quite offended and may not perform in parameter . In some
applications , regression methods can give baffling results.
Regression refers to the estimation of continuous response variables data, negate the
discrete feedback of the variables . The continuous output variable case may referred to the
metric regression to differentiate it from used related problems .

Linear Regression

Regression analysis is used to find a relation between the values of two or more
variables data, of which at leas t one use random variation, and to test a
relation. It is used for detecting relation between parameters data from different places, between
the parameters data of model between water paramete rs and soil parameters, between
productivity and weather , and so on.

THE LINEAR REGRESSION MODEL

Linear regression gives estimates and presumed results for the parameters
in the model:

represents the feedback for case n and has a deterministic part and a
stochastic part represented by the random variable .
For N cases the linear regression model can be written:

where represents the vector of random variables and is a matrix of regressive variables of
dimension:
[

]
and is the disturbances vector of random variables.
A nonzero represents for can be integrated into a function:

[ ] [ ]
We call the expectation function used for the regression m odel. is the derivative
matrix, where (n,p)th is derivative term of the expectation function with pth parameter respect.
Let be normally distributed with :
[ ] [ ] ,
where is identity matrix of dimension and the joint density function for Y with the
variance is:
| (
) ( || ||
)

Application :

% Simple linear regression.
mpg = [26.5; 28.2; 33.1; 3 6.1; 32.8; 23.0]; % Object's weight
wt = [2.660; 2.360; 1.275; 2.115; 1.920; 3.015];
% Compute means and standard deviations:
mean(wt)
std(wt)
mean(mpg)
std(mpg)
n = length(mpg); % n = sample size
X = [ones(6,1) wt]; % Compute the design matrix X
bhat = inv(X'*X)*X'*mpg; % bhat = estimated regression
coefficients
yhat = X*bhat; % yhat = fitted values
r = mpg – yhat; % r = residuals
plot(wt, mpg, 'o', wt, yhat)
title('Object Data' )
axis([1.5, 3, 20, 40])
ylabel('Miles per Gallon (mpg)' ) % 1 US gallon = 3. 78541178
liters
xlabel('Weight of the Object' )
% Make a plot of residuals versus fitted values:
plot(yhat, r, 'o', linspace(20,40,n), zeros(n,1))
xlabel('Fitted Values' )
ylabel('Residuals' )
title('Residual Plot' )
% function that will fit a polynomial to t he data
% the last number = the degree of the polynomial.
polyfit(wt, mpg, 1)

Output:
mean(wt) = 2.2242
std(wt)= 0.6065
mean(mpg) = 29.9500
std(mpg) = 4.8813
polyfit(wt, mpg, polynomial_degree) =
-6.3973 44.1787

The Least Squares Estimates

The like lihood function | for and , is identical to the joint probability
frequency , with the conditional parameters. The li kehood is a function of the responses
conditional of the parameters, suppressing using the constant and we write:
| || ||

The like lihood function is maximized using respect to on the residual sum of squares is
a minimum :
|| || ∑[ ∑
]

The maxim um like lihood evaluate ̂ which maximizes . ̂ is the least squared
evaluation and is written as:
̂ .
Using sampling t heory for least squares estimates where the least squares estimator i s the
minimum variance for and ̂ is the mode function of the marginal posterior frequency for .

Application :
The least squares fitting using linear algebra in matlab :
clc; % clear the console
% fit a line using given data to extract the gradient.
x = [0 0.4 1.1 1.2 1 2 3 4 6]; % time
y = [0 -0.123 -0.132 -0.432 -0.235 -0.764 -1.546 -1.444 -2.111];
% dimensionless

figure
plot(x,y, '*')

X = [(x.^0)' x'] % matrix form

% solve linear equations
b = (X'*X) \(X'*y')
b2 = b(2)

% the least squares
b = X\y'
b2 = b(2)

% b(1) is t he intercept and b(2) is the gradient. We draw a
simple function of the fit on the data.
fit = @(z) b(1) + b(2)*z;
hold all
h = plot(x, X*b, 'r- ');

Output:
X =
1.0000 0
1.0000 0.4000
1.0000 1.1000
1.0000 1.2000
1.0000 1.0000
1.0000 2.0000
1.0000 3.0000
1.0000 4.0000
1.0000 6.0000

b = 0.0404 -0.3824

b2 = -0.3824

b = 0.0404 -0.3824

b2 = -0.3824

Likelihood Inference Results

The like lihood | depends on where || || and likelihood contours has the form:
|| ||
and = constant. The contour of a likelihood region for which c is identical joint
determination region :
( ̂)[
]
Application :
Maximum Likehood Estimati on in matlab:
clc;
npop = 1; % Population parameter
n = 100; % Sample size
nsamp = 1000; % Number of samples

rng default % For reproducibility
samples = exprnd(npop,n,nsamp); % Population samples
means = mean(samples); % Sample means

% The normfit fun ction is used to find the normal distribution
which fits the means
[muhat,sigmahat,muci,sigmaci] = normfit(means)
% The function returns MLEs for:
% the mean and standard deviation
% their 95% confidence intervals.

numbins = 10;
hist(means,numbins)
hold on
[bincounts,binpositions] = hist(means,numbins);
binwidth = binpositions(2) – binpositions(1);
histarea = binwidth*sum(bincounts);
x = binpositions(1):0.001:binpositions(end);
y = normpdf(x,muhat,sigmahat);
% Visualize the distribution of sample means wi th the specific
fitted normal distribution
plot(x,histarea*y, 'r','LineWidth' ,2)

Bayesian Inference Results

The Bayesian marginal posterior frequency used for and has form :

̂ ̂

where -variate People has T density with ̂ location parameter, scaling matrix and
degrees of freedom are represented by .

Example :
We calculate parameter estimates and inference region using ̂ .
Let ̂
[
]

[
]
[
]

The inference region :

Application :
Bayesian Inference is the process of analyzing statistical models using the model or model
parameters.

The posterior distribution for theta is:

Bayesian Inference application in matlab:
rng(0,'twister' );

n = 25;
sigma = 55;
x = normrnd(10,sigma,n,1);
mu = 35;
tau = 25;
theta = linspace( -45, 105, 505);
y1 = normpdf(mean(x),theta,sigma/sqrt(n));
y2 = normpdf(theta,mu,tau);
postMean = tau^2*mean(x)/(tau^2+sigma^2/n) + sigma^2*mu/n/(ta u^2+sigma^2/n);
postSD = sqrt(tau^2*sigma^2/n/(tau^2+sigma^2/n));
y3 = normpdf(theta, postMean,postSD);
plot(theta,y1, '-', theta,y2, '–', theta,y3, '-.')
legend('Likelihood' ,'Prior','Posterior' )
xlabel('\theta')

% Truck Experiment Data
% A set of truck weights
weight = [2705 2805 2550 2700 2400 2900 3400 3600 3500 3600 3400 3900]';
weight = (weight – 2900)/1000; % centering and scaling
% Tested trucks for each weight
total = [47 90 31 34 31 21 23 23 21 16 19 23]';
% The number of tru cks that have poor mpg performances at each weight
poor = [2 4 0 4 8 8 14 17 19 15 17 21]';

% Logistic regression is a special case of a generalized linear model
% the response variable is binomial: X = design matrix, b = vector of model
parameters
logitp = @(b,x) exp(b(1)+b(2).*x)./(1+exp(b(1)+b(2).*x));

% use normal priors
prior1 = @(b1) normpdf(b1,0,22); % prior for intercept
prior2 = @(b2) normpdf(b2,0,22); % prior for slope

% the joint posterior distribution(of the model parameters) ? to th e
likelihood and priors
post = @(b) prod(binopdf(poor,total,logitp(b,weight))) * prior1(b(1)) *
prior2(b(2));

b1 = linspace( -3, -1.2, 55);
b2 = linspace(3.5, 5, 55);
simpost = zeros(5,5);
for i = 1:length(b1)
for j = 1:length(b2)

simpost(i,j) = post([b1(i), b2(j)]);
end;
end;
mesh(b2,b1,simpost)
zlabel('Posterior density' )
ylabel('Intercept' )
xlabel('Slope')

..

Nonl inear Regression

Linear regression techniques can be extended and can have considerable complexity. A
mathematical expression that relates the regression response to the predictor variables data is
called nonlinear regression.

THE NONLINEAR REGRESSION MODEL

A nonlinear regression model has the form:

where represent s the expectation function and represents a vector of an correlated regressor
or a vector of independent variables data for the -th case. This module has the same form as the
linear regression model, except that the conventional responses are represented by nonlinear
functions of used parameters. A nonlinear model use a t least one of the regression derivatives of
the expectation function with deference to the parameters data lean on at least one of the data
parameter s.
Consider the vectors , , as fixed and attend on the dependence of the
conventional responses data on , where is used for the parameters data in a nonlinear model
and is used for the number of parameters,
The -vector with -th element has form:

where and the nonlinear regression model has form:

where have a conventional spherical distribution and meets the next condition s:
[ ]
[ ]

Transformable Linear Models

We use transformable linear models by indicate the reciprocal function of the velocity
functio n of the mutual substrate :

A transformation of the data means a transformation of the interruption term and affects
the expectation on it. Spherical normal disturbance term is a relevant representation o f the
experimental case where the assumptions will not be closest for the transformed data. We use
nonlinear regression on the original regression data and on the weighted least squares
transformed data. When t ransforming a regression data set to generate const ant variance and
produce a linear expectation function used on the transformed regression data.

The Geometry of the Expectation Surface

A spherical regular distribution assumption used for the disturbance term Z let us to grant
the N-dimensional response space of the Euclidean geometry for attracted in the least squares
estimates ̂ of the regression parameters. The -vectors determine a P-dimensional surface
named the expectation surface used in the regression response space and the le ast squares
estimates correlate to the point used on the regression expectation surface:
̂ ( ̂)
Where ̂ is used to minimize the residual regression sum of squares:
|| ||

Application:

DETERMINING THE LEAST SQUARES ESTIMAT ES

The least squares estimates use a data vector y and an expectation function
. Note the expectation surface is closest to y with the point ̂ and determine ̂(parameter
vector) for the point ̂.
The explicit expression for ̂ which is closest to y is ̂ and for a nonlinear
model has P -dimensional plane , which maps to the expectation plane , we have ̂ ̂.

The Gauss -Newton Method

Gauss started to use an initial guess for , where represents a linear approximation applied
to the expectation function :
( ) ∑ ( )
|

For all N cases , we have:
( ) ( )
and is derivative matrix with elements.
Approximation the residuals we have:

( )

where:

( )

Next step is to calculate the Gauss increment to minimize the residual approximation
sum of squares:
|| ||

̂
and = , ̂ ( ) ( )
Next step is to move to the better parameter value and calculati ng new
residual s . We calculate using this process until the increment is very small for
change the vector parameter.
Apllication :
Gauss Newton method in matlab:
function [unknowns,steps,S] = GaussNewton()
% The Gauss -Newton method perform a non -linear least
% squares approximation for the origin of a circle of points
% We use as input a row vector of x and a row vector y values,
and an initial guesses vector
% In the df function must entered partial derivatives for the
jacobian
format long
tol = 100000; %set accuracy
maxstep = 55; %set maximum steps
z = 2*pi*rand(55,1); %create 55 random points
p = cos(z); %create a circle with give n points
q = sin(z);
d = ones(55,1); %set distance(for all points) to origin as 1
a = [0.1;0.1]; %set initial guess for origin
m=length(p); %determine number of functions
n=length(a); %determine number of unkowns
aold = a;
for k=1:maxstep %iterate through process
S = 0;
for i=1:m
for j=1:n

J(i,j) = df(p(i),q(i),a(1,1),a(2,1),j); %calculate
Jacobian
JT(j,i) = J(i,j); %calculate trnaspose
end
end
Jz = -JT*J; %multiply Jacobian and negati ve transpose
for i=1:m
r(i,1) = d(i) – sqrt((a(1,1) -p(i))^2+(a(2,1) -q(i))^2);
%calculate r
S = S + r(i,1)^2; %calculate sum of squares of residuals
end
S
g = Jz\JT; %mulitply inverse of Jz by J transpose
a = aold -g*r; %search for new approximation
unknowns = a;
err(k) = a(1,1) -aold(1,1); %calculate error
if(abs(err(k)) <= tol); %break loop if error is less than
tolerance
break
end
aold = a;
end
steps = k;
hold all
plot(p,q, 'bx ') %plot the data points
plot(a(1,1),a(2,1), 'bo ') %plot the approximation points of the
origin
title('Gauss-Newton Approximation Origin of Circular Data
Points')
xlabel('X')
ylabel('Y')
legend('Data Points' ,'Gauss-Newton Approximation of Origin' )
hold off
end

function value = df(p,q,a1,a2,index) %calculate partials
switch index
case 1
value = (2*a1 – 2*p)*0.5*((a1 -p)^2+(a2 -q)^2)^(-0.5);
case 2
value = (2*a2 – 2*q)*0.5*((a1 -p)^2+(a2 -q)^2)^(-0.5);
end
end

Output:

S = 0.57702 4054502000
ans =
0.000510594811903
-0.001525253388172

The Geometry of Nonlinear Least Squares

The geometry of a Gauss -Newton iteration incorporate:
1. An approximation using a Taylor sequence expansion
2. The residual gene rator vector
3. The residual projected on the tangent plan with ̂ result ,
4. Mappin g ̂ over the linear coordinate system to output the increment
5. Approach .

We approximate the planar assumption(the expectation surface near ) and the uniform
coordinate assumption(a linear coordinate system V( ) on the relative to tangent plane)

Application :
How solve a nonlinear least squares problem in matlab:
Nonlinearleastsqfunc.m:
function [F,J] = nonlinearleastsqfunc(x)
k = 1:10;
F = 2 + 2*k -exp(k*x(1)) -exp(k*x(2));
if nargout > 1
J = zeros(10,2);
J(k,1) = -k.*exp(k*x(1));
J(k,2) = -k.*exp(k*x(2));
end

Computes the objective function values and draw those:
function F = nonli nearleastsqfunc(x)
clc
x=[0.2578 0.2578]
k = 1:10;
F = 2 + 2*k -exp(k*x(1)) -exp(k*x(2));

Output:

x =
0.2578 0.2578
ans =
Columns 1 through 5
1.4118 2.6507 3.6658 4.3911 4.7417
Columns 6 through 10
4.6072 3.8449 2.2704 -0.3554 -4.3415

Bibliography

1. https://en.wikipedia.org/wiki/Regression_analysis
2. http://www.stat.wisc.edu /courses/st849 -bates/slides/nraia2.pdf
3. http://www.waterlog.info/pdf/regtxt.pdf
4. http://matlab.cheme.cmu.edu/2011/09/24/linear -least-squares -fitting -with-
linear -algebra/
5.