, 0,1

, ,

N

X Y N real valued

NY = snrX +N

X – arbitrary fixed input distribution holds for scalar and vector signals

2

0

d

d

result known before

Xsnr

I snrsnr

Ä

No

tes

No

tes

d 1mmse

d 2I snr snr

snr

0

cmmse dsnr

I snr cmmse mmsesnr E

Mean of Filtering (causal) MMSE = Smoothing (noncausal) MMSE

‚

Γ uniformly distributed in [0,snr]

Continuous

Continuous

time

time

ˆ ; | ;f Y X Y snr E X Y snr

2ˆ;mse X X E X f Y

|mmse snr mmse X X snr N

| ;

;

| ;( ; ) log

;Y X snr

Y snr

p Y X snrI X Y E

p Y snr

; | ;; | ;Y snr Y X snrp y snr E p y X snr

2| ;

1 1| ; exp

22Y X snrp y x snr y x snr

; ;I X Y I X X snr N

Y = snrX +N

1log 1

2I snr snr ˆ ;

1

snrf Y X Y snr Y

snr

1

1mmse snr

snr

d 1mmse

d 2I snr snr

snr loge[ ]nats

Let N~(0,1) indep. of X.For every distr. pX with EX2<: d 1

mmsed 2

I snr snrsnr

ˆ ; | ;f Y H X Y snr E H X Y snr

2ˆ;mse H X H X E H X f Y

2ˆ ;mmse snr E H X H X Y snr

|mmse snr mmse X snr H X N

| ;

;

| ;( ; ) log

;Y X snr

Y snr

p Y X snrI X Y E

p Y snr

; | ;; | ;Y snr Y X snrp Y snr E p y X snr

2

2| ;

1| ; 2 exp

2

L

Y X snrp y x snr y snr H x

; ;I X Y I X snr H X N

L,KY = snr +H X N

Let N~(0,I) indep. of X.For every distr. pX with EX2<:

d 1; mmse

d 2,

in [ ]

I snrsnr

X snr H X N

I nats

2mmse snr

1 1 1

2 1 2 2

Y X N

Y Y N

1 2, indep. 0,1N N N~

21

2 21 2

1

1snr

snr

1

2

2 22; | |EI X Y Y X E X Y

1 2; ;I X Y I X Y 1 2 2; , ;I X Y Y I X YÏ I snr I snr

Known

for 0 :

1 2 Markov, chainX Y Y

1 2 2; |I X Y Y I snr I mmse sns r rn

1 11

; ,... ; | ,...,,n

in i ni

I X Y Y I X Y Y Y

Chain rule:

1

1 1; |;n

i ii

I X Y YI X Y

1 Ma... , rkov chain:nX Y Y

10

1

li2

;1

min

n

mmse rY nI X s

0

1

2

snrmmss er dI n 1d

2dI snr

smms

nre snr

d 1mmse

d 2I snr snr

snr

2

mmse ˆ ,

ˆ ; | ;

snr E X X

X f Y snr E X Y snr

Y = snrX +N lim ;snr

H X I X Y

20

1 ˆ d2E X X snX rH

2' E , XX X N~ ' '|| lim lim '; ' ;||X X Y Ysn snrrD p p D p I X Y I Xp Y

= =

2

20

2|| | d1

E1

,2

X

XX X XD D mmse X snrX Np X snr

snr

N =

0 0

1 1 d; log d

d dT T XY

XYX Y

I snrT T

pI X Y p

p p

0

2

00

cmmse cmmse ,1

d

1| ; d

T

T

t tt

snr snr t tT

E X E X Y snr tT

0

2

00

mmse mmse ,1

d

1| ; d

T

t tTT

snr snr t tT

E X E X Y snr tT

{Nt} – Gaussian channel; t[0,T]

{Xt} – r.p. lim. power 0

TEXt2dt<

Xab – {Xt} in [a,b]

pX – probab. measure of {Xt} in [0,T]

t t tY = snrX + N

1

2dlog 1 2XI snr snr S

d

2mmse

1X

Xsr

Ssn

nr S

1c1 d

loge2

mms Xsnr Sn

rr

sns

:XX PSD = Sstationary, ~Gaussian,

1dmmse

2dI snr

snrsnr

0

1cmmse mmse d

snrsnr

snr ‚

cmmse2

I snrsnr

snr

2i ini,snr = E X -E X |Y ;snrmmse

2ii-1

ii,snr = E X -E X |Y ;snrmmsep

2i iii,snr = E X -E X |Y ;snrmcm se

{Ni} – seq. of indep. Gaussian i=1,2,…; {Xi} – limited power r.p.

Xn = [X1,… Xn]T

i i iY = snrX + N

1

1mmse ,

2

d;

dn n

n

i

nI X snr X Yr

in

snrs

1 1mmse ,c pmmse ,2 2

; | ii i

snr snri sn I X Y Yr i snr ‚

1. Some direct implications on other results:Equivalency with De Bruijn’s identity which connects the differential entropy h() to the Fisher’s

information matrix J() (connected to the CRLB)

Derivative of the divergence gets an interesting form can be used also in multiuser systems:

2. Duality Information Theory Information Theory Estimation TheoryEstimation Theory:New lower/upper bounds on MMSE and mutual info. I(;)Results from one domain can be applied to the otherResults from one domain can be obtained/proven by the other e.g., linking the filtering and smoothing by a direct expression in

continuous time domain and sandwiching relation in discrete time

1 KY = H X +N, = snr ,…, snrdiag