PDF《- 以分类问题为例》

基于信息理论的机器学习准则 - 以分类问题为例 Information-based Criteria in Machine Learning C Study on Classifications 中国科学技术大学自动化系学术讲座 Oct.

20,

2008 Bao?Gang?Hu?( 胡包钢 ) NLPR/LIAMA Institute?of?Automation Chinese?Academy?of?Sciences email:?hubg@nlpr.ia.ac.cn

2 Outline ( 大纲 ) 1. 问题的提出 2. 相关工作 3. 互信息准则在分类问题中的基本 表达公式 4. 互信息与传统性能指标关系 5. 分类应用实例 6. 分类问题扩展:相似性研究 7. 小结

3 1. Introduction Learning Machines: Models that allow computers to learn from data. Types on methods - Decision Trees (DTs) - Artificial Neural Networks (ANNs) - Fuzzy Systems, Knowledge-base Models - Bayesian Models, Hidden Markov Models - Supported Vector Machines (SVMs) - Others

4 Types of Learning Supervised Learning: given training examples of inputs and corresponding outputs, produce the correct outputs for new inputs. (character recognition). Unsupervised Learning: given only inputs as training, find structure in the world: discover clusters, manifolds, (clustering, probability density estimation,novelty detection, compression, embedding). From Yann LeCun'

s web,

2007 5 Types of Learning Reinforcement Learning (Or animal learning): an agent takes inputs from the environment, and takes actions that affect the environment. Occasionally, the agent gets a reward or punishment. The goal is to learn to produce action sequences that maximize the expected reward (driving a robot without bumping into obstacles). Semi-supervised Learning: given only a few samples with labels. (web searching). From Yann LeCun'

s web,

2007 6 Types on Tasks Learning machines - Classification: classifiers - Regression: filters or approximators - PDF estimations: estimators - Prediction: predictors - Control: controllers - etc Classifiers: Supervised learning Data driven inference Nonlinear mapping From Vapnik,

1998 7 二值分类实例 分类器 f (X,θ) T =

1 ,苹果 T =

2 ,梨X=(x1,x2,x3,x4,x5,x6) 颜色特征 T=(1,2,2,1,2,1) C= [2, 1, A=4/6 1, 2] Y=(1,2,2,1,1,2)

8 Criteria Selection - 机器学习 , 模式识别 Feature selection Criteria selction? - 学习准则(或学习指标)选定是机 器学习中的首要任务. 应用规定方式:简单 理论研究 : 难点问题

9 例题:分类 (Duda, et al, 2001) - 线性可分问题 - 最小误差平方 : min E= Σ(yi -ti )2 - 无论应用何种 分类器都将产生错 误分类

10 学习准则的类别 - 功能指标 : 分类,回归,控制, ... - 性能指标(以分类为例) : 局部类:正类样本的分类准确率 整体类: ROC 曲线 直接型:分类误差、计算消费 间接型:误差界、最大分类边缘、 互信息 { {

11 Question One 基本问题

1 Q1: Can we apply entropy criterion as a generic measure for dealing with uncertainty of data in machine learning ? 机器学习是以处理信息为基本任务, 那么信息准则是否具有更强的普适 性?

12 Feature Selection 特征选择 二值分类:苹果-梨-特征: 重量 颜色 形状 - 准则: 分类误差 相关系数 互信息( Kwak,

2002 ) From: http://www.nipic.com/

13 Image Registration 图像配准 - 遥感图像 不同时空 - 医学图像 不同模式 - 准则: 配准误差 归一化互相关 互信息 (Maes, 1997) MR CT Studholme,

1999 14 Question Two 基本问题

2 Q2: What are the relations between mutual information and classification accuracy? 基于信息理论的各种学习准则(如互 信息)与传统性能指标(如分类精 度)的关联,独特性以及局限性是什 么?