k-NN 分类器实验

k-NN 分类器实验1．实验目的

掌握 k-NN 最近邻算法的基本思想；认识影响算法性能的因素。编写对实际模式样本正确分类的算法程序。 2．实验内容（ 1 ）产生二维正态分布模式，并将产生的样本集随机地分为参照集和测试集；（ 2）统计错分概率，分析产生错分的原因；（ 3）（选做）针对视频数据集或对纸币数据集，重复上述实验。

3 ．拓展内容（选做，视频中的运动目标检测） 1of6

要求：运用统计模式识别原理，设计一个视频中的运动目标检测程序。

图 1 视频中的运动目标检测原理框图

标记运动目标目标判决

Bayes 分类器判决 s(x,y,t) 属于哪个 GMMi ，

i=1,2,...,c

GMM 生成、更新用矩估计法建立或更新

c 个 Gauss 分布模型 GMMi ， i=1,2,...,c

输入视频

GMM ： Gaussian Mixture Model, GMMi~N(i,i), i=1,2,...,c

逐帧、逐点输入像素值 s(x,y,

t)

s(x,y,t)

GMM


一、输入数据结构： Frames:hwn 三维数组

x

y

t

第 1帧

第 2 帧

第 n帧

h

w

s(x,y,1)

Frames(x,y,1:n), s(x,y,t) =Frames(x,y,t)(1xw, 1yh , 1

tn)

图 2 视频数据结构示意图

22 2

222

( ) ( )2( ) exp[ ]

22

P sGMM s


表 1 GMM(x,y) 参数

2

2

( ) ( )( ) exp[ ]

22i i

iii

P sGMM s

二、 GMM ： Gaussian Mixture Model, GMMi~N(i,i), i=1,2,...,c

i

1 2 3

P(i) 4/50 29/50 17/50

i 145 191.62 86.41

i 1 4.25 10.28

GMM 数据结构： GMM(x,y,,i),=(P(i),i, i)’ hw3c 五维数组


三、 GMM 初始化GMM 数据结构： GMM(x,y,,i),=(P(i),i, i)’ hw3c 五维数组c = 3; % 设有 3 个 Guass 模型GMM(:,:,:,2:3)=zeros(h,w); %GMM2,GMM3 为空

GMM(:,:,1,1)=ones(h,w); %GMM1 只含一点GMM(:,:,2,1)=Frames(:,:,1);%GMM1 一阶矩（均值）为第一帧GMM(:,:,3,1)=Frames(:,:,1)^2;%GMM1 二阶矩为第一帧


四、 GMM 更新for k=2:n;x=1:h;y=1:w

s=Frames(x,y,k)

for i=1:c

sGMM(x,y,:,i)?

更新 GMM(x,y,:,j),j=iGMM(x,y,1,j)++ ； GMM(x,y,2,j) += s ； GMM(x,y,3,j) += s^2 ；

Yes

合并最相似的两个 GMM(x,y,:,j1)

GMM(x,y,:, j2)j=arg min{GMM(x,y,1, j1), GMM(x,y,1,j2)}

sGMM(x,y,:,:)?Yes

新建立 GMM(x,y,:,j)GMM(x,y,1,j) = 1 ； GMM(x,y,2,j) = s ； GMM(x,y,3,j) = s^2 ；

No

No

End

判决 s 是否是背景


五、背景判决

(:) = GMM(x,y,2,:) / GMM(x,y,1,:)

2(:) = GMM(x,y,3,:) / GMM(x,y,1,:) - (:)2

g(:) = GMM(x,y,1,:)/ 2(:)

if j == arg max{g (:)}

then s is background.

% OneNN_Exe_6_3.m % OneNN_Exe_6_3.m 用用 1-NN 1-NN 解习题解习题 2.162.16clear all;close all;clc;clear all;close all;clc;X1 = [1 0; 0 1; 0 -1]; X1 = [1 0; 0 1; 0 -1]; % % 第一类样本集第一类样本集X2 = [0 0; 0 2; 0, -2; -2 0]; X2 = [0 0; 0 2; 0, -2; -2 0]; % % 第二类样本集第二类样本集sX{1} = X1;sX{2} = X2;sX{1} = X1;sX{2} = X2;figure;title('1-NN for Exe 6.3'); xlabel('x');ylabel('y');figure;title('1-NN for Exe 6.3'); xlabel('x');ylabel('y');hold on;hold on;plot(X1(:,1),X1(:,2),plot(X1(:,1),X1(:,2),'ro''ro');plot(X2(:,1),X2(:,2),);plot(X2(:,1),X2(:,2),'bs''bs'););forfor y = -2.5:0.1:2.5 y = -2.5:0.1:2.5 forfor x = -2.5:0.1:2.5 x = -2.5:0.1:2.5 z = OneNNFun(sX,[x y]);z = OneNNFun(sX,[x y]); ifif (z == 1) (z == 1) plot(x,y,'m*');plot(x,y,'m*'); elseifelseif z == 2 z == 2 plot(x,y,plot(x,y,'b''b');); endend endendendend

function ClassNo = OneNNFun(sX,x)function ClassNo = OneNNFun(sX,x)% 1-NN Nearest Neighbor classification function% 1-NN Nearest Neighbor classification function% Input: sample set sX is a cell array. % Input: sample set sX is a cell array. % ClassNum = length(sX) is the number of classes% ClassNum = length(sX) is the number of classes% % sX{i}sX{i}: Ni-by-d matrix whose rows are observations: Ni-by-d matrix whose rows are observations%% (sample vectors) (sample vectors)% and whose columns are variables (features), where,% and whose columns are variables (features), where,% Ni = length(X{i}(:,1)): % Ni = length(X{i}(:,1)): %% the number of samples in class i;the number of samples in class i;% d: the dimension of sample vectors;% d: the dimension of sample vectors;% i=1:ClassNum% i=1:ClassNum% % xx: 1-by-d sample vector need be classified;: 1-by-d sample vector need be classified;% Output: % Output: % % ClassNoClassNo: the class number which x belong to : the class number which x belong to %% by 1-NN algorithm.by 1-NN algorithm.

ClassNum = length(sX);ClassNum = length(sX);ClassNo = 0;ClassNo = 0;if (ClassNum < 2)if (ClassNum < 2) return;return;endend

Index = 0;Index = 0;for i = 1:ClassNumfor i = 1:ClassNum N = length(sX{i}(:,1));N = length(sX{i}(:,1)); for j = 1:Nfor j = 1:N Index = Index + 1;Index = Index + 1; D(Index) = norm(x - sX{i}(j,:));D(Index) = norm(x - sX{i}(j,:)); end end % next j% next j k(i) = Index;k(i) = Index;end end % next i% next i[Min IX] = min(D);[Min IX] = min(D);ClassNo = 1;ClassNo = 1;for i = 1:ClassNumfor i = 1:ClassNum if IX > k(i)if IX > k(i) ClassNo = ClassNo+1;ClassNo = ClassNo+1; elseelse break;break; endendend end % next i% next i

% EditKnn.m% EditKnn.m%--- Edit Knn %--- Edit Knn 算法算法clear clear allall;close ;close allall;clc;;clc;N = 100; N = 100; % the number of samples% the number of samplesBestK = 3;BestK = 3;mu1=[-1,0]';mu1=[-1,0]';mu2=[1,0]';mu2=[1,0]';Cov1=[1,0.5;0.5,1];Cov1=[1,0.5;0.5,1];Cov2=[1,-0.5;-0.5,1];Cov2=[1,-0.5;-0.5,1];%--- %--- 每类生成每类生成 NN 个服从上述参数的随机矢量个服从上述参数的随机矢量X1 = NormRandomFun(mu1,Cov1,N);X1 = NormRandomFun(mu1,Cov1,N);X2 = NormRandomFun(mu2,Cov2,N);X2 = NormRandomFun(mu2,Cov2,N);%--- %--- 建立样本集建立样本集 CellCelloX{1} = X1; oX{1} = X1; oX{2} = X2;oX{2} = X2;%--- %--- 剪辑样本剪辑样本sX = EditKnnFun(oX,BestK);sX = EditKnnFun(oX,BestK);

%--- 用第二组样本测试 1-NN 分类器的错误率X11 = NormRandomFun(mu1,Cov1,N);% 生成测试样本集X22 = NormRandomFun(mu2,Cov2,N);E12 = 0; % 错将第一类判为第二类的错分概率E21 = 0; % 错将第二类判为第一类的错分概率for i = 1:N y1 = KnnFun(X11(i,:),sX,1); if (y1 ~= 1) E12 = E12 + 1;% 错将第一类判为第二类 end y1 = KnnFun(X22(i,:),sX,1); if (y1 ~= 2) E21 = E21 + 1;% 错将第二类判为第一类 endendE12 = E12/NE21 = E21/NPe = (E21+E12)/2

%--- %--- 显示样本分布显示样本分布delta = 0.2;delta = 0.2;[X,Y] = meshgrid(-5:delta:5);[X,Y] = meshgrid(-5:delta:5);d = length(X(1,:));d = length(X(1,:));figure;figure;%--- %--- 显示原始样本集显示原始样本集plot(X1(:,1),X1(:,2),plot(X1(:,1),X1(:,2),'ro''ro'););hold hold onon;;plot(X2(:,1),X2(:,2),plot(X2(:,1),X2(:,2),'bo''bo'););

%--- %--- 标记剪辑样本集标记剪辑样本集plot(sX{1}(:,1),sX{1}(:,2),plot(sX{1}(:,1),sX{1}(:,2),'ro''ro', ,

'MarkerEdgeColor''MarkerEdgeColor',,'k''k',,'MarkerFaceColor''MarkerFaceColor',,'r''r'););

plot(sX{2}(:,1),sX{2}(:,2),plot(sX{2}(:,1),sX{2}(:,2),'bo''bo', , 'MarkerEdgeColor''MarkerEdgeColor',,'k''k',,'MarkerFaceColor''MarkerFaceColor',,'b''b'););

axis([X(1,1),X(1,d),Y(1,1),Y(d,1)]);axis([X(1,1),X(1,d),Y(1,1),Y(d,1)]);

title([num2str(N),title([num2str(N),' normal samples,Edit ' normal samples,Edit Knn,K='Knn,K=',num2str(BestK),,num2str(BestK),',ErrorProb='',ErrorProb=',n,num2str(Pe)]);um2str(Pe)]);

xlabel(xlabel('x1''x1'););

ylabel(ylabel('x2''x2'););

%--- %--- 用用 1-NN1-NN 分类分类 ,, 显示判决区域显示判决区域forfor i = 1:d i = 1:d

forfor j = 1:d j = 1:d

x = [X(i,j),Y(i,j)];x = [X(i,j),Y(i,j)];

y1 = KnnFun(x,sX,y1 = KnnFun(x,sX,11););

ifif (y1 == 1) (y1 == 1)

plot(X(i,j),Y(i,j),plot(X(i,j),Y(i,j),'m.''m.'););

elseelse

plot(X(i,j),Y(i,j),plot(X(i,j),Y(i,j),'g.''g.'););

endend

endend

endend

% END of EditKnn.m% END of EditKnn.m

function ClassNo = KnnFun(x,sX,k)% Knn nearest neighbor classification function% Input: sample set sX is a cell array. % ClassNum = length(X) is the number of classes% x: 1-by-d sample vector to be classified;% sX{i}: Ni-by-d matrix whose rows are sample vectors% and whose columns are features, where,% Ni = length(X{i}(:,1)): % the number of samples in class i;% d: the dimension of sample vectors;% i=1:ClassNum% k: the number of neighbors;%% Output: ClassNo: % the class number which x belong to % by Knn algorithm.

ClassNum = length(sX);ClassNum = length(sX);ifif ((ClassNum < 2)||(k < 1)) ((ClassNum < 2)||(k < 1)) returnreturn;;endendIndex = 0;Index = 0;forfor i = 1:ClassNum i = 1:ClassNum N = length(sX{i}(:,1));N = length(sX{i}(:,1)); forfor j = 1:N j = 1:N Index = Index + 1;Index = Index + 1; D(Index) = norm(x - sX{i}(j,:));D(Index) = norm(x - sX{i}(j,:)); Si(Index) = i; Si(Index) = i; % the class No. of % the class No. of IndexIndexTH TH

samplesample

endend % next j% next jendend % next i% next i

% the counter of nearest neigghbor% the counter of nearest neigghbor

Knum = zeros(1,ClassNum); Knum = zeros(1,ClassNum);

forfor j = 1:k j = 1:k

[V, Mi] = min(D); [V, Mi] = min(D); % find the min element in D% find the min element in D% increase neighbor counter of class Si(Mi)% increase neighbor counter of class Si(Mi)

Knum(Si(Mi)) = Knum(Si(Mi)) + 1; Knum(Si(Mi)) = Knum(Si(Mi)) + 1;

% delete this min element for finding next % delete this min element for finding next

% % nearest neighbornearest neighbor

D(Mi) = inf; D(Mi) = inf;

endend % next j% next j

[V, ClassNo] = max(Knum);[V, ClassNo] = max(Knum);

returnreturn;;

（ 1）如果已知类别样本集不含错误样本，还有必要进行样本剪辑吗？

思考题：

（ 3） 1-NN 最近邻法中，是否所有已知类别样本对正确分类的贡献是一样的？若不同，哪些样本贡献大？请用二维图示说明之。

（ 2）剪辑 k-NN 最近邻法中，第一步用 k-NN 法进行剪辑，为什么第二步却用 1-NN 法进行分类？

functionfunction ClassNo = KnnFun(x,sX,k) ClassNo = KnnFun(x,sX,k)% % KnnKnn nearest neighbor classification function nearest neighbor classification function%% using using sortsort function, it’s fit for the case: function, it’s fit for the case: k >= logN k >= logN% Input: sample set sX is a cell array. % Input: sample set sX is a cell array. % % ClassNum = length(X) is the number of ClassNum = length(X) is the number of

classesclasses% % xx: 1-by-d sample vector : 1-by-d sample vector toto be classifi be classifieded;;% % sX{i}:sX{i}: Ni-by-d matrix whose rows are sample Ni-by-d matrix whose rows are sample

vectorsvectors% and whose columns are features, where,% and whose columns are features, where,% Ni = length(X{i}(:,1)): % Ni = length(X{i}(:,1)): % % the number of samples in class i;the number of samples in class i;% d: the dimension of sample vectors;% d: the dimension of sample vectors;% i=1:ClassNum% i=1:ClassNum% % k:k: the number of neighbors; the number of neighbors;% Output: % Output: ClassNoClassNo: : % % the class number the class number whichwhich xx belong to belong to %% by Knn algorithmby Knn algorithm..

（使用 sort 函数的 KnnFun ）

ClassNum = length(sX);ClassNum = length(sX);ClassNo = 0;ClassNo = 0;ifif ((ClassNum < 2)||(k < 1)) ((ClassNum < 2)||(k < 1)) returnreturn;;endendIndex = 0;Index = 0;forfor i = 1:ClassNum i = 1:ClassNum N = length(sX{i}(:,1));N = length(sX{i}(:,1)); forfor j = 1:N j = 1:N Index = Index + 1;Index = Index + 1; D(Index) = norm(x - sX{i}(j,:)); D(Index) = norm(x - sX{i}(j,:)); %% 计算计算 xx 到各样到各样

本的距离本的距离 endend % next j% next j ClassSegment(i) = Index; ClassSegment(i) = Index; %% 记录记录类分段标类分段标

记记 NNo(i) = 0;NNo(i) = 0;endend % next i% next i

[D IX] = sort(D);[D IX] = sort(D); %% 按升序排序按升序排序forfor j = 1:k j = 1:k cNo = 1;cNo = 1; %% 所属类号所属类号 forfor i = 1:ClassNum i = 1:ClassNum ifif IX(j) > ClassSegment(i) IX(j) > ClassSegment(i) cNo = cNo+1; cNo = cNo+1; %% 大于类分段标记则是下大于类分段标记则是下

一类一类 elseelse breakbreak; ; %% 已找到所属类号已找到所属类号 endend endend % next i% next i NNo(cNo) = NNo(cNo)+1; NNo(cNo) = NNo(cNo)+1; %% 最近邻元个数最近邻元个数 ++

11endend % next j% next j

MaxNNum = 0; MaxNNum = 0; %% 最大最近邻元个数最大最近邻元个数forfor i=1:ClassNum i=1:ClassNum ifif MaxNNum < NNo(i) MaxNNum < NNo(i) MaxNNum = NNo(i);MaxNNum = NNo(i); ClassNo = i; ClassNo = i; % % xx 归属该类归属该类 endendendend % next i% next i% END of % END of functionfunction ClassNo = KnnFun(x,sX,k) ClassNo = KnnFun(x,sX,k)

Documents

k-NN 分类器实验