Graduate Programming Examination (GPE)

- Experiences and Proposal of GPE rules

黃世昆交通大學資訊工程系

GPE EXPERIENCES

到底 GPE 完成了什麼事情？• 大家都會很懷疑到底 GPE 完成了什麼事情

？• 不就是 PC^2 架一架、電腦教室一開，就

開始了？• 中山與師大早就辦過多次的 two-site

contest ？ 這有什麼特別。

程式「賽場」與「考場」的差別• 賽場

– 比賽用的（如 ACM 亞洲區賽、大專杯程式競賽） – 環境專屬– 正式比賽封閉網路

• 比賽機器只能連 server ，不能連外、 不能連內部其他比賽機器

• 考場（如程式設計上機考試）– 考試用的（如各學科的上機考試）– 環境不是你的、但要清空與考試無關的設定與資料– 封閉網路（斷網、也不能連其他人的機器）

賽場、與考場• 大賽場頂多五年布置一次（但以前師大、中山例

外，每年一兩次），小賽場頂多一年一、兩次。– 我只有主持四次小賽場經驗（三次 PC^2 ，一次

GPEDomjudge) 、一次大賽場經驗 ( 使用 PC^2)– 經費：幾萬到幾百萬– 人力：數十人

• 考場每年多次– 我有主持小考場近 50 次的經驗 (20 多次 PC^2, 26 次

GPEDomjudge), – 二次大考場（二次 GPE ，都使用 GPEDomjudge)– 經費：幾千到幾萬– 人力： 2 人（小考場）到 10 人（大考場）

GPE 要完成什麼？• 同時開 100 個程式考場（每個考場容納 20-

200 人）• 有限的人力、經費• 每年可以多次舉辦

GPE Systems

1. GPE Exam Client2. GPE Judge System (On-line and On-site integration)

Judge Server: GPEDomjudge

• on-site judge 方面– 放棄著名的 PC^2 系統 ( 最近兩屆 world final 都未採用 PC^2 )– 換成另外一個 on-site judge: domjudge 。– 我們已進行大部分的翻修，可適應未來頻繁的 GPE 活動與管理，– 具備 on-line(not on-site)/on-site 雙重功能的 judge server 。

• 可連至以下網站，申請帳號或以 guest 訪客登入測試：• http://progexam.cs.nctu.edu.tw/

• 以往 PC^2 賽前預備費時、狀況頻繁，但採用此 judge server ，幾乎隨時可立即比賽，沒有什麼狀況。– Account 管理（報名、資料轉換）、密碼製作– PC^2 舊版無法跨 NAT 、測資 bug– 沒有 source ，無法有效修改

Exam Client Environment

• 採用 virtual machine: virtualbox,• VM image: CentOS linux (ICPC world final contest vm image)

and FreeBSD– 各地區中心的電腦環境互異，但利用 virtual machine 可快速建立相同的

比賽環境。• 比賽者 login 當地 PC 電腦之後， login script 將自動切換

至 remote desktop ，進入 local virtualbox 所 boot 的 centos 比賽環境。– 此 login 帳號將 disable ctl-alt-delete 。– 此環境將經由 VPN 或 IP firewall 限制與 on-site judge 進行比賽連線。

• 未來目標是在半小時內，切換超過 1000 台的 PC ，進入完全相同的比賽環境。

• Oct 11/2010 交大考場，半小時內切換 100 台 PC 為考場

Microsoft Azure Cloud Container and GPE

Microsoft will place in Chicago 112 containers holding 224,000 servers, each container with 2,000 servers

GPE: distribute 2,000 exam clients into more than 20 sites

Same issues: ease of maintenance, ease of resource schedulingDeployment of large number of VMs

Outline

• Problem Description• Potential Solutions• Proposed Solution• Experiences of two GPEs• Issues in this GPE• Topics to be Discussed• GPE Operation Procedure

Problem Description

• To evaluate the readiness of basic programming capability for graduate level work. – on-site on-line programming exam （現場上機程

式考試）– 1500 students for graduate school entrance exam– 1500 computers （必須準備 1500 台電腦）– Mission Impossible （不可能的任務）

Potential Solutions• Program Understanding Exam

– Trace Source Code with Specified Running Results– Feasible in choice-type questions– Only evaluate the “Reading Capability” :

read/modify/create capabilities are quite different• Programming in paper （程式寫在考卷上）

– In pseudo code (already in algorithm exam)– Impossible to grade manually and automatically

• Multiple times, 200 examines at a time– 8 exams with 8 different problem sets– Difficult to grade in the same ranking/scoring

standard

Proposed Solutions

• Multi-sites on-site online exam – Each site with 50-200 computers (feasible in

NCTU, and many other universities)– Jointly with 10-30 sites

• Difficulties– A super-scale programming contest ( 120 on-site

teams, claimed to be largest contest in the world)– Yes, zero-judge claimed to hold a 2000 teams on-

line contest (not sure if they were on-site)

More Issues• On-site exam

– Restricted network connections– Restricted programming environment including

software and hardware• The major Problem

– Each site with different operating environment– You cannot impact the normal operations of

computer rooms in more than 30 sites– You cannot expect service staff of 30 sites to hold a

“programming contest” yearly– Where are the programming problem sets ?

The Results of GPE trials on June 9 and Oct 11

• GPE on June 9• 9 problems

– 3 UVA level-1, 2 level-2, 3 level-3,1 level-4

– level-1 (solved in 10 minutes), level-2 in 15 minutes, level-3 in 30-100 minutes, level-4 in 120 minutes

• 102 Registered (51 in NSYSU, 51 in NCTU)– 86 attend the on-site exam (84%)

– Solved 7 problems (2 students), 6 (7), 5(12), 4(13), 3(17), 2(17), 1(10),

0(8)

• GPE on Oct 11• 9 problems

– 1 UVA level-1, 2 level-2, 4 level-3,2 level-4

– level-1 (solved in 10 minutes), level-2 in 15 minutes, level-3 in 30-100 minutes, level-4 more than 100 minutes

• 159 Registered (67 in NSYSU, 44 in NCTU, 18 in NTCU, 9 in NCU, FJU 19, 2 in CGU)– 124 attend the on-site exam (84%)

– Solved 8 problems (2 students), 6 (3),

5(3), 4(5), 3(4), 2(15), 1(20), 0(79)

Grading• June 9• 2.3% solved 7 problems (A+)• 10.4% solved > = 6 problems (A)• 24.4% solved >= 5 problems

(B+)• 39.5% solved >= 4 problems (B)• 59.3% solved >= 3 problems (C)------------------- Graduate Programming Level------

• 79% solved >=2 problems (C-)

• 90.6% solved >= 1 problems• 9.3% solved 0 problem

• Oct 11• 1.5% solved 8 problems

(A+)• 3.8% solved > = 6 problems (A)• 6.1% solved >= 5 problems

(B+)• 9.9% solved >= 4 problems (B)• 13% solved >= 3 problems (C)------------------- Graduate Programming Level------

• 24.4% solved >=2 problems (C-)

• 40% solved >= 1 problems• 60% solved 0 problem

(2010)GPE第 000003號

林煥博 係交通大學資訊工程學系學生

參與 2010/10/11程式能力檢定，成績如下：

考試性質 3小時，現場考試、封閉網路、不攜帶任何文件資料

解題數 6題（共 9題，全部平均解題數 1.06題，最高解 8題）

排名 第 3名（共 124位考生應考）, 排名比例：3/124=2.4%

詳細資料 http://gpe.acm-icpc.tw/u/foxs91092

考場資訊 交通大學 資訊技術服務中心 1F 電腦教室 4

2010 / 10 / 11

ACM-ICPC Contest Council Taiwan

研究生程式能力檢定成績單

Resolved Issues

• Unified Software Environment• Restricted Network Connections• Fast Deployment of the Unified Environment

– With limited computing resource– Pre-forged Virtual Machines Images– Auto-update VM images

• A Revised judge System– Why another judge system ?

A Revised Judge System• Why not contest system in On-line Judge

– UVA, TopCoder, Sphere, CodeChef– PKU judge (Binary release)– ZeroJudge (Open Source)

• For on-line contest, but not for on-site contest• unpublished problem set

• HDOJ : Programming Contest DIY• Why not On-site Judge

– PC^2 (binary)• Ad hoc contesting (easy to install server and client)• Hard to manage account/problem set/scoring data• No restrictions on login source (IP)

– Domjudge (Open Source) • Web-based• IP-based authentication

Specific Requirements of GPE Judge

• Published Problem Set with known solutions and reference implementations

• Individual postpone of submission time– Due to instable machines within large number of

hosts

Issues in former GPEs• SSL client certificate

(generate during auto-update)

• screendump/command log regularly (monitor strategy for anti-clone, anti-cheating,...)

• VM image partitions (core, fragile, local configuration)

• machine crash SOP (adding delay for individual examine)

• Too many Ajax updates, server jammed and slow response

• Incomplete JDK installations of several judge servers