Interactive Computer Graphics

Ming Ouhyoung, 歐陽明Professor,

Dept. of CSIE, and GINMNTU

1

Introduction• 課號: (50402) 526 U0340 • 課名: 計算機圖學• 英文課名: Interactive Computer Graphics • 授課: 歐陽明• Textbook

– Interactive Computer Graphics: A Top-Down Approach with WebGL, 作者 Edward Angel, Dave Shreiner, 7th edition, Pearson出版, (開發代理)

• Suggested readings– Proceedings of ACM SIGGRAPH Computer Graphics/ACM Trans.

On Graphics – IEEE Computer Graphics & Applications (CG&A) – IEEE Trans. on Visualization and Computer Graphics

2

ICG Syllabus

1. Introduction1. What is interactive computer graphics, GUI?2. What is virtual reality, and augmented reality?3. What is OpenGL/WebGL and shading language?

2. Viewing in 3D and Graphics Programming1. Parallel projection, Perspective projection etc.2. First program: Sample codes in WebGL

3. Basic raster graphics algorithms for drawing 3D primitives 1. 3D primitives (Points and Vectors)2. Geometrics Transformations, Viewing in 3D, 3D clipping

4. Visible-surface determination1. Z-buffer algorithm 2. Binary space partition trees (BSP) 3. Visible-line determination

5. Illumination and Shading 1. The Phong Reflection Model2.. Flat shading , Gouraud shading , Phong shading 3. Bilinear interpolation for color and vertex normal interpolation 3

Syllabus6. Implementation of a Renderer

Graphics Pipeline: from modeling to displayImplementation by WebGL, Unity3D (game engine)

7. Global rendering--Photo-realistic graphics 1. Recursive ray tracing 2. Volume rendering: marching cubes method etc. 3. Radiosity method: progressive refinement approach

8. Curves and surfaces: Bezier curves, B-Splines, Spline surfaces9. Graphics Hardware and Graphics Processing Unit (GPU), GPGPU, APU,

Deep Learning Acceleration10. Implementation and practice

1. Graphics data set 2. How to quickly construct the 3D model data of the world for 3D

graphics/AR/VR? 3. Modeling methods and tools

11. Demonstration and Term Projects4

Learning experience• Engage Students Immediately with 3D Material: A top-

down, programming-oriented approach allows for coverage of engaging 3D material early in the course so students immediately begin to create their own graphics.

• Introduce Computer Graphics Programming with WebGLand JavaScript: WebGL is not only fully shader-based– each application must provide at least a vertex shader and a fragment shader–but also a version that works within the latest web browsers.

• Create games by example: to modify an existing program in Unity3D, and then create an interesting game/animation of your own.

Interactive Computer Graphics 2017TERM PROJECT LISTING

1. Animation of articulated figures (linked)2. Rigid body animation (Newton’s laws) 3. A viewing system for curved surfaces with textures (curves

and patches)4. *Two pass or Radiosity method (source codes available)5. Ray tracing method for a room / many teapots with

different materials (source codes available)6. *Volume rendering for a set of tomography slides

(台大醫院資料 etc.) 7. Image morphing (2D)

6

Interactive Computer Graphics 2017TERM PROJECT LISTING

8. Sketch system for animation (Teddy system)9. Oil painting and water color effects for images 10. 3D morphing and animation with skeleton mapping 11. Motion retargeting (motion of cats likes that of a human)12. Hardware Cg acceleration research and applications 13. Beautifying Images (Color harmonization, face

beautification) 14. Others—Human Computer Interface, VR/AR Apps,

Installation Arts, Water Rendering etc.15.360 Video, Omni-directional stereo video (360 VR

video)Reference papers available. *:considered difficult

7

Textbook Version Difference• Old version (version 4.0 to 6.0 etc) vs. 7th version

(WebGL)• 7th version (WebGL): written in JavaScript, Compatible

with OpenGL ES2.0 and HTML 5.0• JavaScript Syntax:

http://www.w3schools.com/js/js_syntax.aspExample codes: Index of /~angel/WebGL/7E/026th Version: • Use of C++ instead of C• Introduction to Volume Rendering• Compatible with OpenGL3.1

8

Computer Graphics: vs. Image Processing

Computer Vision, Pattern Recognition

9

Linear vs. Spiral model in teaching

Chap 1Chap 2…Chap N

10

The Advantage of Interactive Graphics

• One of the most natural means of communicating with a computer.

• A picture is worth ten thousand words. • A MOVING picture is worth ten thousand

STATIC ones.eg. movie, motion dynamics (flight simulators)

• Graphics User Interfaces

11

What's wrong with this "real" teapot?

12

A textbook?

What's wrong with this teapot?

• 曲面與塑形 (curved surfaces & modeling) • 隱藏面/線 (hidden surfaces / lines) • 著色與打光 (shading & lighting)• 透視 (perspectives )• 材質 (texture )• 陶壺 (pottery)，磁壺 (ceramic)，塑膠壺

(plastic)，銅壺 (copper)，紙壺 (paper)• What's the solution? --> research areas in

computer graphics

13

Expected Capabilities• Transformation & Clipping

• User Interface design & real time manipulation • A New Car Design• Shading & Lighting/ Curved surfaces • Wire frame, depth cueing,

Raster graphics, hidden surface removal• Advanced Illumination:

Ray Tracing, Radiosity, Volume Rendering• GPU programming (OpenGL shading language),

and WebGL• Can do graphics animation, and design for Virtual Reality

and Augmented Reality applicatiopns

14

Unity3D: game engine with editors

15

Father of Computer Graphics: Ivan E. Sutherland

16

2015/5 speech, at UNC CS 50th

celebration

17

Dr. Ivan E. SutherlandIvan Sutherland is considered by many to be the creator of

Computer Graphics. Starting with his Ph.D. thesis, Sketchpad, Sutherland has contributed numerous ideas to the study of Computer Graphics and Computer Interaction. Ivan introduced concepts such as 3-D computer modeling, visual simulations, computer aided design (CAD) and virtual reality.

EducationPh.D., Massachusetts Institute of Technology, 1963; M.S. EE, California Institute of Technology, 1960; B.S. EE, Carnegie Institute of Technology (now Carnegie Mellon University), 1959; Honorary MA, Harvard University, 1966;

18

Ivan Sutherland, IIAcademia Experience

Professor, Harvard University, 1966; Professor, University of Utah, 1968; Chairman of Computer Science Department, California Institute of Technology, 1974;

Professional ExperienceElectrical Engineer and First Lieutenant, National Security Agency, 1963; Researcher, Defense Department's Advanced Research Projects Agency, 1964; Co-Founder, Evans and Sutherland, 1968; Researcher, RAND Corporation, 1974 Vice-President and Technical Director, Sutherland, Sproulland Associates, 1980; Vice-President and Sun Fellow, Sun Microsystems, 1990;

19

James Foley (lower right, first textbook in CG),

Andries Van Dam (upper right, first textbook in CG),

Brian Barsky (the rest, line clipping)

20

Foley and Van Dam: first popular CG textbook

21

Turner Whitted (upper middle, ray tracing), Donald Greenberg( lower right, radiosity)

22

Frederick P. Brooks, Jr. 2006/8 Siggraph(Turing Award 1999, 工程院士, 科學院士, IBM 360/370 chief architect, molecular

visualization, 照片右1, 馬萬鈞, Google/Weta Digital, 右二, 林 奕 成, 交大,左一,葉正聖, 銘傳 )

23

Fred Brooks Jr., 2015/5

24

Henry Fuchs, Pixel-Planes 2015/5Computer Graphics Achievement Award, 1992

member of National Academy of Engineering (美國工程院士)

25

Tomoyuki Nishita, Rendering, 2011/11Steven Coons Award 2005, major Asian CG researcher

26

CG Electronic Theater/Animation DEMO

1. Luxo Jr. (1986 Siggraph, Pixar, The first Oscar 3D Animated short film Academy Award) : demo

2. The Centrifuge Brain Project (2012): demo3. Siggraph Trailer 2017: demo4. Siggraph 2016 (Accidents, Blunders and Calamities) : demo

Selected Best of Taiwan1. Early 2D animation: 宏廣 demo (Oscar Academy award)

曾經是全世界最大的單一動畫影片代工廠

2. Cubic Tragedy : demo(台科大)

3. Heavy Duty : demo (太極影音DigiMax Inc.)

27

Technology DEMO1. Mirror system: demo2. Da Vinci Surgical System: demo3. Realistic Human (Turing test): Emily_Final

4. Flight simulator : demo5. Facial animation: Face_Shift_Ming, 6. Tracking: ANIM1_LIN_YI_C, TRACK_LIN_YI_C,

7. Motion capture: Mocap_skating 太極拳

8. Cloning (Retargeting): demo

9. Eye surgery simulator: CatAR system: demo

28

GPU• In the fourth quarter of 2008, 38.5 million desktops shipped, and

15.2 million discrete GPUs were sold, meaning that almost 40 percent of desktops shipped with discrete GPUs.

• The GPU market has been doing very well despite the economic turmoil. “an annual increase of 22.5 percent and a quarter to-quarter increase of almost 18 percent.” (2008)

• According to the report, this quarter, the three giant chipmakers are holding 97.8 percent of the market share, with Intel at 49.4 percent, Nvidia at 27.8 percent, and AMD at 20.6 percent.

• Finally, both major CPU vendors plan to offer, well in advance of 2012, CPUs with onboard GPUs.

Reference sites: http://3dshaders.com

29

OpenGL history

• OpenGL 1.0– (1993) fixed-function configurable pipeline

• OpenGL 2.0– (9/2004) open up the processing pipeline for user

control by providing programmability for both vertex processing and fragment processing.

• OpenGL 3.0– (8/2008)portions of OpenGL were marked

“deprecated”, setting the stage for further streamlined specifications.

30

RenderMan Shading Language and OpenGL Shading Language

Pixar published RenderMan in 1988, 1. RenderMan shading language is similar to OpenGL

Shading Language2. RenderMan is similar to OpenGL3. OpenGL shading language closely maps to today’s

commercial graphics hardware4. RenderMan have been typically software based.5. OpenGL Shading Language can be thought of as a

descendant of the RenderMan shading language

31

Interface variables to a shader

• Uniform qualifiers: – Uniform: changes less frequently, – Specified outside of a shader

• In qualifiers:– In: input interface for a shader, – Frequently modified data

• Out qualifiers:– Out: The vertex shader writes the per-vertex

values to be interpolated to an OUT variable.

32

GPU introduction

• See my document page103_UBC_GPU

33

DEMO: GPU shader and SIGGRAPH animation from Taiwan

• See my speech slide

• IPO of Alibaba Inc., and its impact on TaiwanIT companies.

34

DefinitionPixel:

– a screen consists of N x M pixelsImage Frame Buffer:

– an array of data in memory mapped to screenColor:

– RGB model:– 16 bit/pixel: R 5 bits, G 5 bits, B 5 bits, 1 bit overlay – 24 bit/pixel: R,G,B each 8 bit – 8 bit/pixel: 28=256 colors, color map, indexing

Dithering (抖動著色)– (using 28 colors to approximate 216 colors)– halftone approximation: 網點著色– also called clustered-dot Ordered dither

35

Dithering

36

(Floyd-Steinberg) Error Diffusion Dithering

37

• Error diffusion pattern:

Error Diffusion Dithering: example

• How to approximate such an array if we have elements of the form 2xN only? (i.e. 0,2,4,6,8,10, etc)

38

++

++

−

+←

xxxx

xxxx 64/8218/33

64/3358/37611

11122

466

785313577

error intermediate calculation

Impressionism

• Impressionist painting characteristics include relatively small, thin, yet visible brush strokes,

39

Geometrical TransformationWhy homogeneous coordinate system?

Translation– P'=T + P ...... (I)

Scaling– P'=S x P ...... (II)

Rotation– P'=R x P ......(III)

T : translation matrix S : scaling matrix R : rotation matrix (II) & (III) are multiplications

40

Geometrical Transformation

• To be able to compose transformations, each point (x,y,z) is represented by (x,y,z,w) where w is usually 1. So, translation becomes

41

=

11000100010001

1'''

zyx

dzdydx

zyx

dxxx +='

3D rotation

42

−

=

1000010000cossin00sincos

)(TTTT

TRz

−

=

10000cossin00sincos00001

)(TTTT

TRX

−=

10000cos0sin00100sin0cos

)(TT

TT

TRY

3D Translation & Scaling

• Translation

43

=

11000100010001

),,(zyx

ddd

dddTz

y

x

zyx

• Scaling

=

11000000000000

),,(zyx

SS

S

SSSSz

y

x

zyx

Transformation Matrix

Any number of rotating, scaling, and transition can be multiplied together!

Post-multiplication vs. pre-multiplication

44

TAzyxzyx

A ]1[

1

=

Concatenation of transformations

Let

45

pCBAq =)))((( pABCq =

MPqCBAM == ,• In OpenGL, the syntax are

• The Rule in OpenGL: • The transformation specified most recently is the one applied first

Transformation OpenGL Syntax Examples

Rotation glRotatef(angle, vx, vy, vz) glRotatef(45.0, 1.0, 0.0, 0.0)

Translation glTranslatef(dx, dy, dz) glTranslatef(-4.0, -5.0, -6.0)

Scaling glScalef(sx, sy, sz) glScalef(2.0, 1.0, 1.0)

The Window-to-Viewport transformation

• World coordinate v.s. Screen coordinate

46

Viewing in 3D

• CW:– Center of the window

• VRP:– View Reference Point

• VPN:– View-Plane Normal

• DOP:– Direction of Projection

47

Viewing in 3D (eye at 0,0,-d)

48

無無無無無無無

0,1

,1

,

=+

=+

=

+=

+=

ppp

pp

Z

dz

yY

dz

xX

dzy

dY

dzx

dX

Viewing in 3D: in matrix form

49

無無無無無無無

[ ] [ ].

11

),0,,(),,(

111100000000100001

''''''

'''

'''''''

'

'

'

elyperspectivZYXtodtransformeiszyxSo

ww

wy

wxZYXLet

ZYX

zyx

dwzyx

ppp

ppp

p

p

p

=

⇒

=

More on Geometrical Transformations

• Affine transformation– preserving parallelism of lines, but not lengths and angles. – Rotation, translation, scaling and shear transformations

are affine.

• Shear (剪力)

50

+=

1110001001

yayx

yxa

3D shear

51

無無無無無無無

=

10000100010001

),( y

x

yxxy

SHSH

SHSHSH

Clipping Lines(see building walkthrough video)

Why clipping? inside building, closer viewCohen-Sutherland Line - Clipping algorithm

52

• Trivial rejection: if logical AND of the codes of

the endpoints is not zero• New segments created: replacement by intersection

point

Clipping Lines,line U = (A, B) to (C,D)

• Conditions to set a bit to 1,when input value A is:Larger than XmaxSmaller than Xmin,

when input value B is Larger than YmaxSmaller than Ymin,

53

Clipping line: calculate intersection points

54

Clipping of 3D polygons(see 3D clipping powerpoint)

• Scissoring ----> draw only those lying within the rasterized clip region

• Practical: If we are within a building with many rooms, how to reduce the polygons to be processed? (video)

55

Animation: double buffer

• If left ready, then show left, draw right buffer now therefore, no flickering

56

One Point Perspective VS. Two Point Perspective VS. Three Point

PerspectivePerspective projections are categorized by their number of

principal vanishing points.--> Number of axes the projection plane cuts

57

Two Point Perspective

58

Two-point perspective

59

Moving Point Perspective(北宋, 張擇端, 見DEMO)

60

• Aseem Agarwala. Maneesh Agrawala, Michael Cohen, David Salesin, Richard Szeliski,“Photographing Long Scenes with Multi-Viewpoint Panoramas”, SIGGRAPH 2006

61

One example

62

• Ordinary pictures are 3 point perspectives.

63

New graphics programming: WebGL self-learning

• How?

64

WebGL demo and codes• Using Google to search for the key words: “WebGL

experiments”• Chrome WebGL experiments

• Run on your Android/iOS phones, iPad, Windows NB etc. using native browser, or Chrome.

• Try “newest” selections Jelly fish• Try “oldest” selection -> Field (1000 grass blades

blowing by wind). Speed measurements: iPad Air plus, is running at 30 frames while my Android smartphone HTC M8x is running at 20 frames per second.

65

Source codes

• Learning WebGL (WebGL lessons, Tony Parisieditor in chief )

• It also has Chinese version!

66

Starting point: Simple triangles

• Trianglevertex1_X vertex1_Y vertex1_Z normal1_X normal1_Y normal1_Zvertex2_X vertex2_Y vertex2_Z normal2_X normal2_Y normal2_Zvertex3_X vertex3_Y vertex3_Z normal3_X normal3_Y normal3_Z

67

• COLOR• Triangle

frontcolor_R frontcolor_G frontcolor_B backcolor_R backcolor_G backcolor_Bvertex1_X vertex1_Y vertex1_Z normal1_X normal1_Y normal1_Zvertex2_X vertex2_Y vertex2_Z normal2_X normal2_Y normal2_Zvertex3_X vertex3_Y vertex3_Z normal3_X normal3_Y normal3_Z

68

One real example

Triangle255 255 255 250 0 0595.000000 453.000000 40.000000 0.000000 1.000000 0.000000482.000000 453.000000 0.000000 0.000000 1.000000 0.000000482.000000 453.000000 40.000000 0.000000 1.000000 0.000000

69

COLOR

How to draw (render) triangles

70

71

72

73

74

75

Standard Graphics Pipeline

Advanced rendering methods that usually cannpt be easily implemented by standard

graphics pipeline• Ra Tracing (光線追蹤法)• Volume Rendering• Radiosity (熱輻射法)

• Photon Mapping

Volume Rendering: result images

What is still missing in ray-traced images?

• Diffuse to diffuse reflection?

END part 1

81