A Non-Local Cost Aggregation Method for Stereo Matching

Yang, QingXiong (杨庆雄 )City University of Hong Kong

3 6 9

2 5 8

1 4 7

A 2D image (3x3) => a planar graph

3 6 9

2 5 8

1 4 7

Computing minimum spanning tree (MST)

3

6

9

2

5

8

1

4

7

Obtained MST

𝑣1

𝑣2

𝑣3

𝑣4

𝑣5

𝑣6

𝑣7

𝑣8

𝑣9

𝑒1=¿2−1∨¿1

𝑒2=¿ 3−2∨¿1

𝑒3=¿6−3∨¿ 3

𝑒4=¿5−4∨¿1

𝑒5=¿ 6−5∨¿1

𝑒9=¿6−9∨¿3

1

1

3

6

9

2

5

8

1

4

7𝑣1

𝑣2

𝑣3

𝑣4

𝑣5

𝑣6

𝑣7

𝑣8

𝑣9

𝑒1=1

𝑒2=1

𝑒3=3

𝑒4=1

𝑒5=1

𝑒9=3

1

1Distance

𝐷 (𝑣1 ,𝑣2 )=?

3

6

9

2

5

8

1

4

7𝑣1

𝑣2

𝑣3

𝑣4

𝑣5

𝑣6

𝑣7

𝑣8

𝑣9

𝑒1=1

𝑒2=1

𝑒3=3

𝑒4=1

𝑒5=1

𝑒9=3

1

1Distance

𝐷 (𝑣1 ,𝑣2 )=¿𝑒1

3

6

9

2

5

8

1

4

7𝑣1

𝑣2

𝑣3

𝑣4

𝑣5

𝑣6

𝑣7

𝑣8

𝑣9

𝑒1=1

𝑒2=1

𝑒3=3

𝑒4=1

𝑒5=1

𝑒9=3

1

1Distance𝐷 (𝑣1 ,𝑣4 )=?

3

6

9

2

5

8

1

4

7𝑣1

𝑣2

𝑣3

𝑣4

𝑣5

𝑣6

𝑣7

𝑣8

𝑣9

𝑒1=1

𝑒2=1

𝑒3=3

𝑒4=1

𝑒5=1

𝑒9=3

1

1𝐷 (𝑣1 ,𝑣4 )=𝐷 (𝑣4 ,𝑣1)

Distance

¿𝑒1+¿𝑒2+¿𝑒3+¿𝑒5+¿𝑒4¿7

shortest distance of traveling from one node to another¿

3

6

9

2

5

8

1

4

7𝑣1

𝑣2

𝑣3

𝑣4

𝑣5

𝑣6

𝑣7

𝑣8

𝑣9

𝑒1=1

𝑒2=1

𝑒3=3

𝑒4=1

𝑒5=1

𝑒9=3

1

1𝐷 (𝑣1 ,𝑣4 )=𝐷 (𝑣4 ,𝑣1)

Distance

¿𝑒1+¿𝑒2+¿𝑒3+¿𝑒5+¿𝑒4¿7

Similarity:𝑆 (𝑣4 ,𝑣1 ) ¿exp (−𝐷 (𝑣1 ,𝑣 4)  

𝜎)

¿𝑆 (𝑣1 ,𝑣2 ) ∙𝑆 (𝑣2 ,𝑣3 )∙𝑆 (𝑣3 ,𝑣6 ) ∙𝑆 (𝑣6 ,𝑣5 ) ∙𝑆 (𝑣5 ,𝑣4 )¿𝑆 (𝑣1 ,𝑣 4 )

¿𝑆 (𝑒1) ∙ 𝑆 (𝑒2) ∙𝑆 (𝑒3 ) ∙ 𝑆 (𝑒5 )∙ 𝑆 (𝑣4 )

3

6

9

2

5

8

1

4

7𝑣1

𝑣2

𝑣3

𝑣4

𝑣5

𝑣6

𝑣7

𝑣8

𝑣9

𝑒1=1

𝑒2=1

𝑒3=3

𝑒4=1

𝑒5=1

𝑒9=3

1

1

∑𝑖

𝑆 (𝑣 4 ,𝑣𝑖 ) ∙𝑣 𝑖

Supports received from other nodes:𝑣4

That is: after cost aggregation,

𝑣4(𝑎𝑔𝑔𝑟𝑒𝑔𝑎𝑡𝑒𝑑 )¿∑

𝑖

𝑆 (𝑣 4 ,𝑣𝑖 ) ∙𝑣 𝑖

A Linear Time Algorithm

3

6

9

2

5

8

1

4

7𝑣1=𝑣1↑

𝑣3

𝑣4=𝑣4↑

𝑣5

𝑣6

𝑣7=𝑣7↑

𝑣8

𝑣9

𝑒1

𝑒2

𝑒3

𝑒4

𝑒5

𝑒8

𝑒7

𝑒9

1. Aggregating from leaf nodes to root node:

𝑣2↑=𝑣2+¿¿𝑣2+𝑆 (𝑒1 )∙𝑣1

𝑣 𝑖↑= 𝑓 (𝑣 𝑖)

𝑣2

𝑆 (𝑒1 ) ∙

3

6

9

2

5

8

1

4

7

𝑣3 𝑣5

𝑣6

𝑣9

𝑒1

𝑒2

𝑒3

𝑒4

𝑒5

𝑒8

𝑒7

𝑒9

𝑣 𝑖↑= 𝑓 (𝑣 𝑖)

𝑣8𝑣2↑

𝑣1=𝑣1↑

𝑣7=𝑣7↑

𝑣4=𝑣4↑

1. Aggregating from leaf nodes to root node:

𝑣8↑=𝑣8+¿¿𝑣8+𝑆 (𝑒7 ) ∙𝑣7

𝑆 (𝑒7 ) ∙

3

6

9

2

5

8

1

4

7

𝑣3 𝑣5

𝑣6

𝑣9

𝑒1

𝑒2

𝑒3

𝑒4

𝑒5

𝑒8

𝑒7

𝑒9

𝑣 𝑖↑= 𝑓 (𝑣 𝑖)

𝑣2↑ 𝑣8

↑

𝑣1=𝑣1↑

𝑣7=𝑣7↑

𝑣4=𝑣4↑

1. Aggregating from leaf nodes to root node:

𝑣3↑=𝑣3+¿𝑆 (𝑒2) ∙

3

6

9

2

5

8

1

4

7

𝑣5

𝑣6

𝑣9

𝑒1

𝑒4

𝑒3 𝑒5

𝑒8

𝑒7

𝑒9

𝑣 𝑖↑= 𝑓 (𝑣 𝑖)

𝑣4=𝑣4↑

𝑣8↑

𝑣1=𝑣1↑

𝑣7=𝑣7↑

1. Aggregating from leaf nodes to root node:

𝑣5↑=𝑣5+¿𝑆 (𝑒4 ) ∙

𝑒2

𝑣2↑

𝑣3↑

3

6

9

2 8

1 7

𝑣9

𝑣6

𝑒1

𝑒8

𝑒3 𝑒5

𝑒7

𝑒9

𝑣 𝑖↑= 𝑓 (𝑣 𝑖)

𝑣8↑

𝑣1=𝑣1↑

𝑣7=𝑣7↑

1. Aggregating from leaf nodes to root node:

𝑣9↑=𝑣9+¿𝑆 (𝑒8 ) ∙

𝑒2

𝑣2↑

5

4

𝑒4

𝑣4=𝑣4↑

𝑣5↑𝑣3

↑

3

6

2

1 7

𝑣6

𝑒1

𝑒3 𝑒5

𝑒7

𝑒9

𝑣 𝑖↑= 𝑓 (𝑣 𝑖)

𝑣3↑

𝑣1=𝑣1↑

𝑣7=𝑣7↑

1. Aggregating from leaf nodes to root node:

𝑣6↑=𝑣6+¿𝑆 (𝑒3 ) ∙

𝑒2

𝑣2↑

5

4

𝑒4

𝑣4=𝑣4↑

𝑣5↑ 9

8

𝑒8

𝑣8↑

𝑣9↑

3

6

2

1 7

𝑣6

𝑒1

𝑒3 𝑒5

𝑒7

𝑒9

𝑣 𝑖↑= 𝑓 (𝑣 𝑖)

𝑣3↑

𝑣1=𝑣1↑

𝑣7=𝑣7↑

1. Aggregating from leaf nodes to root node:

𝑣6↑=𝑣6+¿𝑆 (𝑒3 ) ∙

𝑒2

𝑣2↑

5

4

𝑒4

𝑣4=𝑣4↑

𝑣5↑ 9

8

𝑒8

𝑣8↑

𝑣9↑

𝑆 (𝑒5 )∙ +¿

3

6

2

1 7

𝑣6

𝑒1

𝑒3 𝑒5

𝑒7

𝑒9

𝑣 𝑖↑= 𝑓 (𝑣 𝑖)

𝑣3↑

𝑣1=𝑣1↑

𝑣7=𝑣7↑

1. Aggregating from leaf nodes to root node:

𝑣6↑=𝑣6+¿𝑆 (𝑒3 ) ∙

𝑒2

𝑣2↑

5

4

𝑒4

𝑣4=𝑣4↑

𝑣5↑ 9

8

𝑒8

𝑣8↑

𝑆 (𝑒5 )∙ +¿𝑆 (𝑒9 ) ∙ +¿

𝑣9↑

3

6

9

2

5

8

1

4

7

𝑣2↑

𝑣6↑=𝑣6

↓

𝑣5↑

𝑒1

𝑒2

𝑒3

𝑒4

𝑒5

𝑒8

𝑒7

𝑒9

2. Aggregating from root node to leaf nodes:𝑣 𝑖↓=𝑔 (𝑣 𝑖

↑)

𝑣8↑

𝑣3↑

𝑣9↓

𝑣1↑

𝑣7↑

𝑣4↑

𝑣9↓=𝑣9

↑+¿ ( )𝑣9↑𝑆 (𝑒9)∙𝑣6

↓−

¿

3

6

9

2

5

8

1

4

7

𝑣2↑

𝑣6↑=𝑣6

↓

𝑣9↓

𝑒1

𝑒2

𝑒3

𝑒4

𝑒9

𝑒8

𝑒7

𝑒5

2. Aggregating from root node to leaf nodes:𝑣 𝑖↓=𝑔 (𝑣 𝑖

↑)

𝑣8↑

𝑣3↑

𝑣5↓

𝑣5↓=𝑣5

↑+¿ ( )𝑣5↑𝑆 (𝑒5) ∙𝑣6

↓−

¿𝑣1

↑𝑣7

↑

𝑣4↑

3

6

9

2

5

8

1

4

7

𝑣2↑

𝑣6↑=𝑣6

↓

𝑣9↓

𝑒1

𝑒2

𝑒5

𝑒4

𝑒9

𝑒8

𝑒7

𝑒3

2. Aggregating from root node to leaf nodes:𝑣 𝑖↓=𝑔 (𝑣 𝑖

↑)

𝑣8↑

𝑣3↓

𝑣3↓=𝑣3

↑+¿ ( )𝑣3↑𝑆 (𝑒3)∙𝑣6

↓−

¿

𝑣5↓

𝑣1↑

𝑣7↑

𝑣4↑

3

6

9

2

5

8

1

4

7

𝑣2↑

𝑣9↓

𝑣6↑=𝑣6

↓

𝑒1

𝑒2

𝑒5

𝑒4

𝑒9𝑒3

𝑒7

𝑒8

2. Aggregating from root node to leaf nodes:𝑣 𝑖↓=𝑔 (𝑣 𝑖

↑)

𝑣8↓

𝑣1↑

𝑣7↑

𝑣4↑

𝑣8↓=𝑣8

↑+¿ ( )𝑣8↑𝑆 (𝑒8)∙𝑣9

↓−

¿

𝑣5↓𝑣3

↓

3

6

9

2

5

8

1

4

7

𝑣2↑

𝑣5↓

𝑣6↑=𝑣6

↓

𝑒1

𝑒2

𝑒5

𝑒8

𝑒9𝑒3

𝑒7

𝑒4

2. Aggregating from root node to leaf nodes:𝑣 𝑖↓=𝑔 (𝑣 𝑖

↑)

𝑣4↓

𝑣1↑=𝑣1

↓𝑣7

↑

𝑣8↓

𝑣4↓=𝑣4

↑+¿ ( )𝑣4↑𝑆 (𝑒4) ∙𝑣5

↓−

¿

𝑣9↓𝑣3

↓

3

6

9

2

5

8

1

4

7

𝑣8↓

𝑣3↓

𝑣6↑=𝑣6

↓

𝑒1

𝑒8

𝑒5

𝑒4

𝑒9𝑒3

𝑒7

𝑒2

2. Aggregating from root node to leaf nodes:𝑣 𝑖↓=𝑔 (𝑣 𝑖

↑)

𝑣2↓

𝑣1↑

𝑣7↑

𝑣4↓

𝑣2↓=𝑣2

↑+¿ ( )𝑣2↑𝑆 (𝑒2) ∙𝑣3

↓−

¿

𝑣5↓ 𝑣9

↓

3

6

9

2

5

8

1

4

7

𝑣3↓

𝑣8↓

𝑣6↑=𝑣6

↓

𝑒1

𝑒8

𝑒5

𝑒4

𝑒9𝑒3

𝑒2

𝑒7

2. Aggregating from root node to leaf nodes:𝑣 𝑖↓=𝑔 (𝑣 𝑖

↑)

𝑣7↓

𝑣1↑

𝑣2↓

𝑣4↓

𝑣7↓=𝑣7

↑+¿ ( )𝑣7↑𝑆 (𝑒7)∙𝑣8

↓−¿

𝑣5↓ 𝑣9

↓

3

6

9

2

5

8

1

4

7

𝑣3↓

𝑣2↓

𝑣6↑=𝑣6

↓

𝑒7

𝑒8

𝑒5

𝑒4

𝑒9𝑒3

𝑒2

𝑒1

2. Aggregating from root node to leaf nodes:𝑣 𝑖↓=𝑔 (𝑣 𝑖

↑)

𝑣1↓

𝑣7↓

𝑣8↓

𝑣4↓

𝑣1↓=𝑣1

↑+¿ ( )𝑣1↑𝑆 (𝑒1) ∙𝑣2

↓−¿

𝑣5↓ 𝑣9

↓