Download pdf - Tse Featured

How Software Designers Interact with Sketchesat the Whiteboard

Nicolas Mangano, Thomas D. LaToza, Marian Petre, and Andre van der Hoek,Member, IEEE

AbstractWhiteboard sketches play a crucial role in software development, helping to support groups of designers in reasoning about

a software design problem at hand. However, little is known about these sketches and how they support design in the moment,

particularly in terms of the relationships among sketches, visual syntactic elements within sketches, and reasoning activities. To

address this gap, we analyzed 14 hours of design activity by eight pairs of professional software designers, manually coding over

4000 events capturing the introduction of visual syntactic elements into sketches, focus transitions between sketches, and reasoning

activities. Our findings indicate that sketches serve as a rich medium for supporting design conversations. Designers often use

general-purpose notations. Designers introduce new syntactic elements to record aspects of the design, or re-purpose sketches as the

design develops. Designers constantly shift focus between sketches, using groups of sketches together that contain complementary

information. Finally, sketches play an important role in supporting several types of reasoning activities (mental simulation, review of

progress, consideration of alternatives). But these activities often leave no trace and rarely lead to sketch creation. We discuss the

implications of these and other findings for the practice of software design at the whiteboard and for the creation of new electronic

software design sketching tools.

Index TermsInteraction styles, systems analysis and design, user-centered design

1 INTRODUCTION

SKETCHES play an important role in any design process.They serve as an extension of a designers own memory[37], help them reason through complex tasks [54], and sup-port them in picturing and evolving hypothetical ideas andabstract concepts [17]. Since sketches are explicit and exter-nalized representations [28], they assist designers in suchtasks as brainstorming [51], evaluating ideas [7], and collab-orating with others [8], [58].

Unsurprisingly, sketches play an important role in thesoftware design process as well. A typical software projectresults in numerous, diverse sketches [8], [41]. Fig. 1 presentstwo representative images taken from two software compa-nies: a whiteboard containing a sketch of a potential solutionto a design problem (left) and a whiteboard wall containingthe collective sketches frommany designmeetings (right).

These kinds of whiteboard sketches, and the design pro-cesses that they support, have seen an increase in interestfrom the software engineering research community in thepast several years. Building upon work that looks at soft-ware designers in action more generally (e.g., [13], [19],[21], [41], [60]), the design work of software designers at thewhiteboard has been examined from a range of perspectives,

including idea generation [2], design notations [6], [33], [42],[56], decision making [59], [60], epistemic uncertainty [4],and collaboration [32].

While these previous studies make important contribu-tions, our understanding of how software design sketchesare actually produced, and used, at the whiteboard is stilllimited. We have a sense of the kinds of sketches softwaredesigners produce and, at a high level, the types of activitiesthe sketches support (e.g., understanding code, idea exter-nalization, design review, comparison of alternatives). Wedo not know, however, precisely how the sketches comeabout and evolve, nor how they support designers in navi-gating their design problems.

To address this gap, we conducted the first study toexamine how sketches evolve and support reasoning activi-ties on a moment-to-moment basis. Drawing on data fromdesign sessions from eight pairs of professional softwaredesigners, we coded 14 hours of videos for 4238 eventsacross 155 sketches, capturing moment-to-moment data ofhow designers introduced types of visual syntactic ele-ments, shifted their focus between sketches, and usedsketches to support reasoning activities. We then analyzedthis data to answer several research questions:

Types of sketches. What types of sketches do designerscreate? Do the sketches created vary in relation tothe approach to design taken? How syntacticallycomplex are sketches? How do sketches evolve?

Focus. How long do designers focus on individualsketches? How do designers make reference tosketches? How do designers shift their attentionbetween sketches?

Reasoning. How do designers use sketches to under-stand and advance the state of the design at hand?Which types of reasoning activities do the sketches

N. Mangano is with Molimur, Mission Viejo, CA 92692.E-mail: [email protected].

T. D. LaToza and A. van der Hoek are with the Department of Informatics,Donald Bren School Information and Computer Sciences, University ofCalifornia, Irvine, CA 92697. E-mail: {tlatoza, andre}@ ics.uci.edu.

M. Petre is with the Faculty of Mathematics and Computing, The OpenUniversity, Milton Keynes, MK7 6AA, United Kingdom.E-mail: [email protected].

Manuscript received 28 Mar. 2014; revised 16 Sept. 2014; accepted 5 Oct.2014. Date of publication 13 Oct. 2014; date of current version 4 Feb. 2015.Recommended for acceptance by M. Dwyer.For information on obtaining reprints of this article, please send e-mail to:reprints.org, and reference the Digital Object Identifier below.Digital Object Identifier no. 10.1109/TSE.2014.2362924

IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. 41, NO. 2, FEBRUARY 2015 135

0098-5589 2014 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

support? Are the outcomes of design discussionsalways recorded in the sketches supporting thesediscussions?

The result of our analysis is a rich account of the natureof sketching at the whiteboard, revealing the relationshipbetween sketching and the design process it supports.While we report on many results that help shed light onhow software designers work, several findings stand out.First, we found that, while all designers created domain,requirements, and systems sketches, the number of sketchesthey created depended strongly on the approach to designtaken. Second, we found that designers rapidly shifted theirfocus over small groups of sketches, with each sketch play-ing an important and distinct role within the design conver-sation. As a result, designers rarely worked with only asingle sketch for more than 30 seconds. Third, we foundthat sketches served an important role in helping designersdiscuss alternatives, review their progress, and performmental simulations. Over 80 percent of the reasoning activi-ties we identified made use of sketches.

The remainder of this paper is organized as follows:Section 2 presents background material on sketching, bothgenerally and as studied in software development. Section 3introduces our study design and Section 4 the codingscheme used in our data anlysis. Section 5 presents ourobservations and results. Sections 6 and 7 discuss ourthreats to validity and findings, respectively, and section 8provides a brief conclusion.

2 BACKGROUND

Designers at work, with their tendency to sketch, havereceived a great amount of attention over the years. Earlywork focused on characterizing the activity of design. Fornon-routine, ill-structured problems, Simon and Newellcharacterized design as a process leading to a good enoughsolution to the problem in the context at hand [37], [48].Some models of design attempted to make design repeat-able by delineating the solution space in which alternativesolutions are weighed and explored [49]. Other work dem-onstrated that software design is a learning activity, andthat exploring the problem is as important as exploring thesolution [11]. Later research found that software designersapproach design using an opportunistic strategy, in whichdesigners work with designs different levels of abstractionsimultaneously, and dive selectively into detailed design

when it suits them [21], while showing a relative absence ofalternatives. Another study found that designers exploresolutions both vertically, by moving between levels ofabstraction, and laterally, by utilizing different strategies tosolve the same problem [17].

Other work has examined the role of sketching in thedesign process and the types of sketches that designers pro-duce. Ferguson categorized the sketches that designers pro-duce as thinking, talking, and communication sketches [15].Suwa found that sketches serve to help thinking activities byrapidly externalizing thoughts onto paper [52]. Schondescribed the process of reviewing and thinking over thesketches that a person has created as having a reflective con-versation with the material [45]. Suwa et al. found that suchreflective conversations over the visual details in onessketch led to unexpected discoveries which stemmed fromunintended details when drawn [51]. Goldschmidt calledthe process of reinterpreting sketches beyond their originalintended depiction as seeing as, and noted that professio-nals aremore adept at the act of reinterpretation [18].

Software design, in particular, has seen a similar explo-ration of design and sketching. Several studies have foundthat informal sketched prototypes receive more feed-back than formal prototypes [23], [46], [57]. Mynatt demon-strated that office workers can identify chunks of categorizedcontents in their own writing on the whiteboard [35]. Dammet al. found that software developers begin with informalsketches and refine them into more formal notations [12].Newman et al. observed that website developers switchbetween levels of abstraction in designingwebsites [38].

Recently, there have been an increasing number of stud-ies investigating software design. LaToza et al. found thatprofessional software developers consider whiteboards andpaper the most effective means of designing [29]. Cherubiniet al. found that whiteboards frequently serve as themedium of choice for working through important designdecisions, but also that most drawn sketches are transientand often discarded [8]. Dekel and Herbsleb found that theknowledge contained in sketches is highly dependent onthe designers ability to reconstruct the meaning of sketches,and that notations such as UML serve as idioms and are notfollowed strictly [14]. Petre found that designers juxtaposesketches and deliberately change formalisms to highlightdifferent aspects of a problem, and use provisionality toallow a dialog with incomplete ideas [41]. Yatani et al.found that, while there were important barriers to using

Fig. 1. Two examples of software design sketches on whiteboards.

136 IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. 41, NO. 2, FEBRUARY 2015

diagrams in a distributed development team, diagrams stillplayed an important role in ad-hoc meetings, designing,onboarding, project awareness, and documentation [58].

Recent work examined reasoning and decision-makingqualities used over sketches. Ball and Christensen demon-strated that mental simulations allow designers to discussincomplete ideas and reduce uncertainty [3]. Baker and vander Hoek showed that designers often examine two ideassimultaneously, and rotate between pairs of ideas [1].Zannier and Maurer found that designers are more criticalof designs when evaluating them all at once, and less criticalwhen evaluating ideas serially [60]. Tang et al. observed thatsoftware designers often make a decision without justifyingit and move forward [53]. Christiaans and Almendra notedthat software design is more complex than product design,because decisions in software must involve knowledgeof the user interface and structure, and thus decisions arechained [9].

Using data from three of the eight design sessionsreported in this paper, attendees of the Studying Profes-sional Software Design workshop derived a variety of mod-els of the design process [55]. Petre analyzed sketching atthe whiteboard from the perspective of cognitive dimen-sions of notations and highlighted both the importance ofgesture in whiteboard sessions and how, as the role of asketch changed over the course of a design discussion, thetype of representation evolved [43]. Ossher et al. found thatdesigners used multiple representations of concerns whichbecame increasingly detailed and formal through the designconversations [40]. Nakakoji and Yamamoto found thatdesigners used designing and drawing as synonyms,used drawings to establish shared understanding, used avariety of notations, reappropriated sketches for new pur-poses, redrew sketches, and sometimes did not write downimportant concepts [36].

Based on these studies of design, many tools have beenproduced for supporting design. These tools range fromgeneric whiteboard sketching tools, such as Flatland [34],DENIM [38], Post-Brainstorm [16], Designers OutPost [26],and Range [25], to tools more closely tied to softwaredesign, such as SILK [27], Knight [12], SUMLOW [20],InkKit [10], and Calico [30]. A comprehensive summary ispresented by Johnson et al. [24]. Based on an extensivereview of the literature, Moody proposes a set of principlesfor designing software design notations for cognitive effec-tiveness [33].

Our study builds on this workfocusing specifically onhow sketching at the whiteboard supports informal soft-ware designby collecting and analyzing moment-to-moment data on the introduction of new types of visualsyntactic elements, focus transitions between sketches, andreasoning activities.

3 METHOD

To investigate software design sketching at the whiteboard,we conducted an observational study of professional soft-ware designers. To recruit participants, we approachedalumni of UC Irvine with a request for access to professionalsoftware designers who excelled at software design. Thisrequest led to contacts at a number of different organizations,

which, through discussions and follow-up, resulted inrecruitment of the eighteen professional software design-ers who participated in the study. The professionals camefrom seven different organizations. As design at the white-board often occurs in collaborative settings involving mul-tiple designers [8], [30], such as ad-hoc meetings, weorganized participants into pairs. Each pair belonged tothe same organization (two organizations provided twopairs of designers).

We administered a uniform design prompt to all pairs,videotaped each pair at work during their design session,interviewed the participants afterward, and, once all videoswere collected, analyzed the whole set of videos. Weexcluded one video from the analysis due to periodicallyobstructed views and poor audio quality, yielding a total ofeight pairs that formed the basis of the results presented here.

The task instructions, coding guide, and raw data fromour study are publicly available.1

Participants. All those recruited were viewed as expertdesigners by their peers. Designers expertise ranged acrossa variety of domains (including health care, documentworkflow, and photo imaging software) and professionalspecializations (including system analyst, software architec-ture, UI development, and software development). Allworked directly with software and reported that theyemploy whiteboards in their work. Members of each pairwere familiar with each other, although they did not neces-sarily work together in their organizations at the time of thestudy. The professional experience of the participantsranged from 2 years to 26 years. Most designers had a num-ber of years of experience, with an average of 16.5 years.While we were initially surprised that a professional withonly 2 years of experience would be viewed as an expert, asubsequent conversation with his peers confirmed that hewas. 15 of the participants were male and one was female.Table 1 lists each designers pair, professional experience,and specialization.

Task. Each pair was provided with a written prompt ask-ing them to design an educational traffic flow simulationprogram to be used by a professor in a civil engineeringcourse to teach students traffic light patterns. The prompt,two pages in length, described a set of open-ended goalsand requirements, including offering students the ability to:(1) create visual maps of roads, (2) specify the behaviorof lights at intersections, (3) simulate traffic flow, and (4)change parameters of the simulation (e.g., traffic density).Pairs were asked to produce a design at the level of detailnecessary to present to a group of software developerswho will be tasked with implementing it.

Setting. The study was conducted at participants placesof work, with a constrained setup that was replicated asclosely as possible at each site. We asked each pair to workat a whiteboard approximately 8ft wide by 4ft tall. Partici-pants were asked not to write on the prompt or other paperin order to capture as much of their work as possible withthe particular camera setup we used.

All sessions lasted approximately two and a half hours.This time was divided into four segments:

1. Available online at: http://sdcl.ics.uci.edu/study-materials-and-data/

MANGANO ET AL.: HOW SOFTWARE DESIGNERS INTERACTWITH SKETCHES AT THEWHITEBOARD 137

1) approximately one hour and fifty minutes for thedesign activity itself (mean 1 hour and 47 minutes,min 1 hour, max 2 hours and 4 minutes),

2) a ten minute break,3) ten minutes for a summary, and4) a twenty minute semi-structured exit interview.At the start of the first segment, participants were given

the design prompt and asked to work on a design thataddressed this prompt. After the break, they had tenminutes to briefly recap, explain, and motivate theirdesign. In the exit interview, we asked participants to sum-marize how they went about their design, what their pro-fessional background was, and whether they felt thedesign process they followed was different from how theynormally proceeded and, if so, how (see Section 6 for a dis-cussion of these differences and how they affect the valid-ity of our results).

All sessions were video recorded, with one cameradirected at the whiteboard and one positioned to capture abroader view of the participants. Three of the videos areavailable by request.2 Our analysis focused primarily on thedesign activity portion of the videos. A total of 14.26 hoursof design at the whiteboard was studied and dissected.

4 CODING SCHEME

We created three coding schemes focusing on three aspectsof design at the whiteboard:

1) What types of sketches and visual syntactic elementsdo software designers use during design at thewhiteboard?

2) How do software designers focus on and transitionamong the sketches they produce?

3) What reasoning activities do software designers usetheir sketches to support?

Each coding scheme captured events that happened andidentified them as actions or activities. Some events are self-contained actions that happen within a moment (e.g., pro-posing a design alternative). Others describe activities thatoccur over a period of time (e.g., a review of progress).Hence activities are delineated by pairs of events, identify-ing the beginning and end of the activity. All events arelabeled with information characterizing the event and withappropriate timestamps.

The coding schemes were developed using an iterative,inductive approach, comparable to open coding [50]. First,we chose three sessions that we viewed as representative ofthe eight sessions. We then examined portions of these ses-sions to determine how to segment the sketches and activi-ties meaningfully, as well as to identify the relevantconcepts and categories that emerged. We operationalizedthis into coding schemes, articulating the coding criteria indetail. After several iterations, we finalized the codingschemes. Two authors then independently coded a fourthsession, yielding a Cohens kappa inter-rater agreement of.7 and percentage agreements of 80.0 percent, 82.4 percent,and 78.6 percent for the sketches, focus fixations and transi-tions, and reasoning activity categories, respectively. Thisindicates an acceptable inter-rater reliability in an analysisof 13 percent (1) of the design sessions. The remaining sevensessions were then coded by a single author.

Fig. 2 summarizes the coding schemes. In the remainderof this section, we describe each coding scheme in turn.

4.1 Sketches

Observations of designers have found that, rather thanviewing whiteboard content as a single entity, people seg-ment content on the whiteboard into distinct sketches [34].We define a sketch, for our purposes, as a segment of thewhiteboard containing content:

1) related to a single, central topic (an invariate [5]),with

2) similar visual features,3) proximity,4) continuity, and5) a similar initial creation time.Sketches may be primarily textual (e.g., a list), primarily

pictorial (e.g., a drawing), or contain elements of both (e.g.,a diagram).

The first coding scheme examined the sketches designerscreated, identifying events in which sketches were created orerased or new visual syntactic elements (see below for a pre-cise definition) were added. To segment whiteboard contentinto sketches, we performed two passes. In the first pass, weexamined snapshots of the video at 5-minute intervals,examining if new sketches had been introduced. Each newsketch that appeared in the snapshots was numbered fromleft-to-right, top-to-bottom. In the second pass, we added toand revised our set of sketches for each session by watchingthe videos, and identifying sketches that shared a centraltopic in conversation and visual focus by the designers. Forexample, a sketch consisting of a user interface for creatingmaps was revised into two different sketches when weobserved that the designers divided their attention betweenamap and its input panel (center left of Fig. 3).

TABLE 1Participants

Designer Pair Exp. (years) Professional specialization

1 A 26 UI development2 A 20 Interaction design3 B 15 Software architecture4 B 11 Software architecture and

UI development5 C 2 Software architecture6 C 24 Software architecture7 D 21 Project manager, software

architect8 D 26 Software architecture9 E 12 Software developer10 E 10 System analyst11 F 8 Software developer12 F 6 Software developer13 G 25 Software developer14 G 25 Software developer15 H 23 Software developer16 H 10 Software developer

Pairs C and D belonged to the same organization, as did pairs B and G.

2. Available online at: http://www.ics.uci.edu/design-workshop/videos.html


Sketch type. We labeled each sketch with a type based ontwo factors:

1) the set of graphical symbols and compositional rulescharacterizing the sketch, and

2) designers verbal references to the sketch.The first factor draws heavily on Moodys definition of

visual syntax [33], which he defines as the combination ofvisual vocabulary (graphical symbols) and visual grammar(compositional rules). Our usage differs in that our sketchtypes emerged from the data and from existing sketch typessuch as UML, rather than being taken from a fixed andabstract set of graphical symbols and compositional rules. Insituations where the first factor alone left the type of sketchambiguous, we also considered how the designers referred tothe sketch. For example, to differentiate sketches with bothER diagram and class diagram visual syntactic elements, werelied on how designers themselves referred to the sketch. Insome situations, sketches evolved in place from one sketchtype to another. In these cases, we used a combination of key

Fig. 2. An overview of the coding schemes, listing the three aspects of the design process analyzed and the dimensions (italics) and possible valuesfor each dimension.

Fig. 3. Whiteboard content was segmented into distinct sketches.


visual syntax elements and how designers referred to thesketch to determinewhen the sketch type changed.

Sketch domain. Sketches also varied in terms ofwhich aspectof design eachmodeled (e.g., a list of requirements vs. a list ofsoftware components). Through our iterative analysis ofsketch content, we arrived at four domains and used thesedomains to categorize each sketch. These four domainswere:

1) requirements repeating or extending the requirementsprovided in the prompt,

2) the architecture or implementation of the softwaresystem,

3) the user interface of the system as the user sees andinteracts with it, and

4) the problem domain modeling the problem describedin the prompt.

Visual syntactic element types. We identified each instancein which a new type of visual syntactic element (VSE) wasfirst added to a sketch. A visual syntactic element refers toa component of a sketch that is uniquely identifiedthrough a combination of its visual elements and itsarrangement in relationship to other components (i.e.,compositional rules). VSEs included text, graphical sym-bols (such as boxes or arrows), drawing elements (such asshading), and drawn objects (such as cars). To categorizevisual elements of VSEs, we used Bertins six retinal varia-bles, a set of atomic building blocks for any visual repre-sentation: shape, size, color, brightness, orientation, texture [5].For example, a box-and-arrow sketch contains two VSEsboxes and arrowsdiffering in the shape retinal variable,and a list title is distinct from a list item because of differ-ences in arrangement. In coding the types of VSEs, we iter-atively built a set of VSE types, recording each new VSEtype we identified during coding. Each VSE type wasrecorded with a picture used to identify subsequent occur-rences. To reduce the coding required, we did not recordadditional instances of the use of a VSE within a sketchafter recording its initial introduction.

4.2 Focus Fixations and Transitions

The second coding scheme examined software designersfocus and transitions between sketches. At each moment ofthe design sessions, we tracked each pairs focus point, whichwe coded as a specific whiteboard sketch, the prompt, ornone. We determined the focus point based on factors suchas the person speaking, where their gaze was directed(based on the two recorded camera angles), the location oftheir pen, and the topic of the active discussion. While bothdesigners nearly always had the same focus point, in situa-tions in which they differed, we coded the pairs focus pointas belonging to the designer who was speaking, or, if nei-ther was speaking, to the designer who was writing. Weignored all activity conducted off-camera (which was usu-ally re-reading portions of the prompt).

In our open coding, we observed that designers oftenreferred to adjacent sketches in short bursts, but did notmake any changes to that adjacent sketch. We define amomentary reference as an instance inwhich a sketch has a con-tinuous focus duration shorter than 3 seconds, during whichno changes were made. We distinguished three behaviourssignalling designers reference to adjacent sketches:

1) quick glance,2) pointwith a finger, or3) split focus (pointing and glancing at different

sketches).Intervals with no focus pointwhen the designers did

not gaze at the whiteboard or prompt for a period of 15 sec-onds or longerwere identified as inactive. Inactive inter-vals could contain momentary references; only establishinga new focus point with a 3 second gaze or edit ended inac-tive intervals. For example, if the designers spent twominutes talking without using the whiteboard, pointed at asketch for a brief second, and then returned to talking, thiswas coded as a single inactive interval with a momentaryreference to a sketch.

Whenever designers focus shifted between sketches onthe whiteboard, we noted the origin and destinationsketches, including their sketch type and domain (focustransitions to or from focus points other than whiteboardsketches were not labeled). In addition, we identified andcoded three additional dimensions of the relationshipbetween the source and destination sketches:

Abstraction. Sketches with an abstraction relationship rep-resented a concept at differing levels of abstraction, eitherby providing a more detailed view of the entire sketch (e.g.,a class diagram with only class names vs. a class diagramwith class names, methods names, relationships, cardinal-ity) or by providing a more detailed view of a portion of thesketch (e.g., a list and map, where the map depicts a con-crete realization of a list element).

Alternative. Sketches with an alternative relationshipdescribed different potential solutions. We coded sketchesas alternatives if the designers explicitly referred to the con-tent in the sketches as competing solutions and used thesketches to discuss and compare the solutions at least once.

Point-in-time. Sketches with a point-in-time relationshipwere visually similar sketches depicting changes over time,phases, or conditions.

4.3 Reasoning activities

The third coding scheme examined the reasoning activitiessupported by sketches. We define reasoning activities asevident cognitive processes (as reflected by speech) with aspecific purpose or goal. To code reasoning activities, weprimarily used speech, but we also used gaze and gesturewhen it helped to confirm or clarify designers intent. Wechose to focus on coding reasoning activities that:

1) involved lower-level goals over short periods oftime,

2) had well-defined boundaries, and3) could be consistently distinguished by both coders.Thus, we did not include designing as a reasoning activ-

ity, as it spanned much of the session and did not havewell-defined boundaries. This yielded three reasoningactivities:

Mental simulation. A mental simulation refers to instances inwhich the designers talk through an execution of a mentalmodel for the purpose of evaluating a design idea, allowingdesigners to turn uncertainly into approximate answers byperforming thought experiments [3]. We build on the defini-tion used by Ball and Christensen [3] in which the key feature


is that the talk-through involves a simulation run that altersthe designers mental representation to a new state. Similar toOlson et al.s definition of walkthrough [39], we includeinstances such as simulating the intended user interactingwith the design, following the movement of data within thesystem, or tracing state changeswithin the system.

Review of progress. A review of progress includes allmomentswhen the designers review their design for the pur-pose of taking stock of what they have done. Our definitionbuilds onwhat Olson et al. call summary [39], in which thedesigners momentarily take a step back from the design toconsider the progress they have made, the goals they haveyet tomeet, or the tasks they have yet to complete [27].

Suggested design alternative. A suggested design alternativeincludes instances in which a designer presents a solutionthat competes with or replaces an existing solution. Unlikeprior work that examined the structure of design rationalein detail [39], we did not build a comprehensive designrationale graph, but rather recorded instances wheredesigners verbally presented a new design alternative thatcontrasted an existing design idea. We recorded an eventwhenever an alternative was first mentioned; when design-ers quickly mentioned alternatives in rapid succession,these were coded as separate alternatives.

Mental simulation and review of progress were coded asa pair of events, signifying the beginning and end of theactivity. As we found that the discussion of distinct alterna-tives often overlapped and was difficult to distinguish, wetracked only the moment in which a design alternative wasintroduced, and we coded these as a single event. For eachinstantaneous event or event signifying the beginning of anactivity, we noted both the type of reasoning activity andthe role sketches served in supporting the activity:

1) No sketches were useddesigners did not gaze at orphysically reference the whiteboard.

2) Designers gestured over existing sketches withoutdrawing.

3) Designers drew on, annotated, or edited existingsketches.

4) Designers created a new sketch.

5 RESULTS

We first report our results on the sketches that designers pro-duced.We then report how designers focused on andmovedtheir attention among sketches. Finally, we report on thereasoning activities of the designers and how sketchessupported these activities. Throughout, we report bothquantitiatve data from our coding scheme and qualitativeobservations explaining this data. Numbers characterizingthe behavior of designers are reported as an average acrossall pairs with a standard deviation (e.g., 23 7 percent).

5.1 Sketches

In the following sections, we first examine how designerschoice of approach to design influenced the types ofsketches they drew. We then examine the types of sketchesdesigners created and the types of visual syntactic elementsused within these sketches. Finally, we examine howdesigners annotated sketches and how sketches evolvedthrough the design sessions.

5.1.1 Approaches to Design

While most pairs addressed the requirements, system, anduser interface of the design, pairs tended to focus theirefforts on a single aspect, using it to drive the design of theother aspects. This choice of focus led to three approachesto design that we observed, each of which was reflected inthe time designers spent with sketches of the correspondingdomains (Fig. 4, Fig. 9):

user interface-driven: pairs A, C, G, and H (user inter-face sketches: 42 13%),

system-driven: pairs B, D, and F (system sketches: 42 8%), and

requirements-driven: pair E (requirements sketches:33%).

These patterns were also evident in the quantity ofsketches designers produced (Table 2). Each pairsapproach to design was partially influenced by thedesigners job descriptions: the only pair with a softwareanalyst (E) was the only pair to focus on requirements,and the only pair with an interaction designer (A) focusedon user interface design. Pairs with software architectsand software developers were split between focusing onthe user interface and the system.

User-interface-driven. User-interface-driven pairs (A, C, G,H) generally asked themselves: How can users use our sys-tem to accomplish their goals?, focusing on front-endsketches and discerning how to get input from the user anddisplay a result. These pairs relied heavily on user interfacesketches to structure and organize their work (Fig. 3, Fig. 5a).

Reflecting the traffic simulator problem domain, theuser-interface-driven pairs made frequent use of maps(30 6% of session time) to brainstorm user interactionsand support discussion of the problem. Maps also served asa hub: pairs frequently shifted attention to a map beforeshifting to a sketch of a different sketch type.

Lists were the second most common sketch type for user-interface-driven pairs (19 9% of session time). Lists helpeddesigners to record potential classes they discovered whiledesigning the user interface. Lists also served an importantrole in recording requirements, which the designers period-ically referenced to verify that they were being met. Towardthe end of the sessions, three of the UI pairs finally consid-ered the software architecture in detail, either by creatingan ER diagram (A and H) or by making lists (C) describingthe structure of a map class. They stated that they did thisto satisfy the prompts requirements that they describe the

Fig. 4. The fraction of session time each pair spent examining the designprompt, working with sketches of each domain, or not working at thewhiteboard.


system in sufficient detail for software developers to imple-ment it.

System-driven. In contrast to user-interface-driven pairs,system-driven pairs (B, D, F) focused first on the systemdesign, designing classes and their interactions. In general,these pairs asked themselves: How does the systemwork? As seen in Fig. 5b, these pairs split their timebetween using software sketches to document and discussthe classes in their system design (as in the left of Fig. 5b)and using problem sketches to discuss and understand howtheir system worked (e.g., the map in the center of Fig. 5b).

All three system-driven pairs created sketches listingnouns and sketches representing a map with an intersectionsymbol. The left side of Fig. 5b presents an evolved repre-sentation of two examples of these sketches, where the listof nouns became the software architecture alongside theintersection sketch. Beyond this basic setup, system-drivenpairs proceeded with their own variations. One pair (B)picked a class from their design, defining it in detail. Late inthe session, two of the pairs (B, D) drew a second class dia-gram in order to define classes and their relationships inmuch greater detail.

While working with system sketches, designers pointedto classes, marked up elements with annotations andarrows, and explained classes verbally to their partner. Listswere sometimes used to document design decisions, suchas explicit instructions on how a particular componentbehaved (far right of Fig. 5b), or to record user stories (cen-ter of Fig. 5b). As these designers dug more deeply into thesystem design, they created sketches allowing them toexplore the system and problem in more detail, such as linegraphs and code. These sketches helped them exploreproofs of concept of their proposed solutions.

Requirements-driven. The requirements-driven pair (E)focused first on listing requirements, posing the question:What is it that we need the system to do? They inten-tionally avoided designing the system through much of the

session, later reporting that we wanted to be clear about whatwere building, not how were going to build it, only consider-ing the system and user interface late in the session.

Lists played a central role, helping the pair to directtheir thinking within the design session, reason aboutparts of the system, and document design decisions. Afterlooking at the prompt, they began with a list of require-ments (upper left corner of Fig. 5c), and then broke eachout into separate, more detailed lists of requirements,maintaining a one-to-one relationship between items inthe initial list and the title of each new list. They then laidout a set of use cases (right side of Fig. 5c), creating a mapto help them generate use cases by enumerating ways inwhich the user would interact with the map. As they wentthrough the use cases, they annotated each requirement,identifying how it mapped to use cases. As they addressedrequirements, they checked off the corresponding require-ments. At the end of the session, they erased a large sec-tion of the board to create an ER diagram.

5.1.2 Types of Sketches and Visual Syntactic Elements

Across all sessions, designers used a total of 11 sketch types(listed in Table 2). Designers often created ad-hoc sketchtypes reflecting the domain, including the most frequentsketch type-maps. Designers also created system diagramswith defined notations (e.g., class diagram, ER diagram),sketches with ad-hoc notations (e.g., array structure, GUI),and general-purpose sketches (lists, tables, line graphs,drawings). Designers created an average of 19.4 ( 8.9)sketches per session, each with an average of 4.8 types ofvisual syntactic elements. Table 3 lists the most frequenttypes of visual syntactic elements for the most frequenttypes of sketches.

Despite the traditional focus on software design nota-tions for describing system structure, most of the sketchesdid not concern the design of the system. Categorisingsketches by domain, pairs created an average of 6.5 ( 6.9)

TABLE 2The Sketches Designers Drew

Number of sketches (by pair) Avg.#VSE typesA B C D E F G H Avg.

(by sketch type)map 3 9 6 3 7 19 5 6 7.3 5.3list 5 3 6 3 15 7 2 5 5.8 4.0table 11 - - - - - 4 1 2.0 3.7GUI 2 1 1 - 2 5 - 2 1.6 4.5ER diagram 1 - - 1 1 - - 3 0.8 6.7class diagram - 2 1 - - - - - 0.4 11.8code - 1 - - - 2 - - 0.4 5.0drawing 1 - - - - - - - 0.1 1.0traffic signal - - - - - - 2 1 0.4 4.8line graph - 1 - - 1 - - - 0.3 4.5array structure - 1 - - - - - - 0.1 3.0

(by domain)problem 3 6 1 3 5 23 5 6 6.6user interface 16 5 6 0 5 3 6 4 5.5system 2 6 5 3 1 9 2 5 4.3requirements 2 1 2 1 15 1 0 3 3.3

Total sketches 23 18 14 7 26 36 13 18 19.4


problem sketches, 6.4 ( 4.4) user interface sketches, 4.1 (2.6) system sketches, and 3.6 ( 5.1) requirements sketches.Even when using system sketches, designers more oftenchose to sketch using the syntactically simple lists overmore elaborate and complete notations such as ER diagrams(0.8 1.0 per pair) or class diagrams (0.4 0.7 per pair). Forexample, designers often simply made lists of classes andmethods rather than use boxes with a title to denote classesas in class diagrams.

Reflecting the traffic problem domain, maps were themost common type of sketch created (7.3 5.4 per pair) andwere used by all pairs. Designers used maps to reason abouttraffic in two different ways: understanding how intersec-tions work, and designing user interfaces. Software-drivenpairs often used problem domain maps to figure out howcars moved through intersections. They asked questionssuch as how does a car decide to make a left turn, do carsinstantaneously move from one intersection to the next, andwhat component of the system controls a behavior. Toanswer these questions, designers sometimes went to a verylow level of abstraction, drawing individual cars in theirdepictions of maps (Fig. 5b).

In contrast, user-interface-driven pairs instead usedmaps depicting the user interface. These pairs generally rea-soned about traffic at the level of traffic flows, consideringscenarios such as that on Mondays and Wednesdays peoplewent out to lunch, changing the traffic density. As a result,the maps these designers drew were often at a higher level(Fig. 3, Fig. 5a).

In supporting these uses, the VSE types designers usedvaried between road-level views andmap-level views. Road-level views often used either lines demarcating the edges ofroads (parallel lines33 percent of maps, seven pairs) orintersecting roads (intersection30 percent, five pairs), andrepresented traffic with indicated individual cars, intersec-tions, and traffic patterns with rectangles (37 percent, sevenpairs), circles (28 percent, six pairs), or dotted lines (17 per-cent, five pairs). Map-level views used lines to represent indi-vidual roads (single line30 percent, six pairs) or sets ofroads (grid14 percent, five pairs), indicating flows of trafficwith arrowswith lines (47 percent, seven pairs).

Lists were the second most common sketch type (5.9 4.1 sketches per pair) and were used by all pairs. Designersused lists to record requirements, components, use cases,and component behavior. Most lists were syntacticallysimple and contained few types of VSEs (average 3.7 1.6),most commonly first-order elements (92 percent, eightpairs) and titles (76 percent, eight pairs). Designers oftendid not expand items in a list: only 36 percent containedsecond-order elements. Instead, designers created new lists.Parentheses were sometimes used to express provisionalinformation, often next to titles or first order elements.Designers very rarely reordered or rearranged lists. Mostwere not numerically numbered, and for those that were,the order served only to make items quickly referenceableor to provide a count of items. This suggests that designersrarely worked with lists that encoded meaning in theirorder. Lists often depended implicitly on other sketches,but designers rarely made these dependencies explicit; only20 percent contained arrows explicitly showing relation-ships from elements in the list to other sketches. Few listsreferenced sketches by name.

Tables were most often used to represent aspects of theuser interface, typically as a mechanism for inputting data(e.g., Fig. 3). Tables varied in the ways in which designerscreated them. Some tables were created bottom-up, begin-ning with values in the table, and only later (if ever) anno-tated with row headers. For example, when designing trafficintersections, designers wrote down row values into thetable, and later added the headers for these values. Other

Fig. 5. Each pair focused their efforts on a specific aspect of the design.


tables were created top-down; designers drew the row head-ers first and then typically included additional syntactic ele-ments such as column headers and grid lines.

GUIs were often created after interface maps, helpingdesigners to determine how the map content might bemanipulated by a user. For example, designers wouldmimic moving sliders, then imagine how the map interfacewould update as a result. GUIs most often contained abounding box (53 percent, five pairs), check box (46 percent,three pairs), and title (39 percent, three pairs).

Entity-relationship (ER) and class diagrams contained alarger set of core notations shared by most sketches(Table 3); this likely reflects ER and class diagrams statusas well-known notations with conventions on how to drawthem. Relationships between entities were ubiquitous, anddesigners often explicitly differentiated the type of relation-ship they denoted. For example, in ER diagrams, designersused relationships with no head or tail (83 percent, threepairs), relationships with triangle heads (67 percent, fourpairs), and solid dots (50 percent, three pairs). Once made,decisions on relationship type were never changed. Design-ers also tended to use ER notations (e.g., dots and hollowcircles to represent containment and referencing), evenwhen working with class diagrams, which have a different

formal representation defined in the UML (e.g., dashedarrows and diamond heads).

Designers sometimes explicitly avoided notations theyconsidered to be too detailed for their purposes. For example,upon noticing that their partner had created a second-orderitem in an ER diagram, one designer (E) interrupted, notingthat: we dont have time to go into that level of detail. Classdiagrams were used by only two pairs (B, C). To explain therun-time behavior of components, designers used class dia-grams rather than sequence diagrams. It is interesting toobserve that this allowed them to explore many distinct sce-narios instead of just one. However, in the exit interviews ofthree of the software pairs (B, D, F), they stated that theywould normally create sequencediagrams as the next steps.

5.1.3 Annotating Sketches

Designers often annotated their sketches, using visual syn-tactic elements to guide attention, support decision-making,record provisional information, or reference other sketches.As designers moved between sketches, they sometimesannotated sketches for emphasis or to guide attention. Forexample, one designer (H) walking through an explanationsaid especially this and underlined an element. Emphasismarks did not reflect any observable pattern: designers used

TABLE 3The Most Frequent Sketch Types and Their Most Frequent VSEs

Each sketch type is listed with an example image (above), the average number of sketches per pair (n), and the number of pairs that used the sketch type. Each VSEis listed with the percentage of sketches that used it and the number of pairs which used it.


a broad range of VSE types, including an underline, a circle, astar, a large X, emboldened dots, and vertical bars. Oncemade, thesemarkswere not subsequently used or referencedbut persisted for the life of the sketch. Designers sometimesused VSEs to explicitly refer to other sketches, typically withdeclarative labels referencing other sketches or with arrows.

Annotations helped to support decision-making. Severalpairs used check marks on lists, maps, class diagrams and ERdiagrams. One pair (H) discussed a requirement that involvedmany parts of the system. To address this issue, they drewquestion marks next to all event objects, and systematicallyvisited the question marks and placed either a check mark orlarge X to signify whether the object did or did not addressa requirement. Another pair (E) checked off requirements in alist as they verified that theywere addressed.

Designers also annotated sketches to record provisionalinformation. For example, designers who could not settleon a name for a list wrote a second name in parenthesesnext to the official title. This behavior occurred across manypairs (A, B, C, E, F) and several representation types, espe-cially ER and class diagrams. Other pairs used slashes inlists to indicate alternative names.

5.1.4 Evolving Sketches

Designers continually invented new types of visual syntac-tic elements to capture and externalize concepts they dis-cussed while designing. Fig. 6 plots the growth of VSEtypes in sketches over time. For example, one pair (B)wished to determine how users were interacting with mapsand drew a UI map sketch (right side of Fig. 5b). They firstdrew a box representing the whole area and then drew linesto represent the user adding streets to the map. They thenproposed: what if every time two streets intersected wouldcreate an intersection?, and drew an intersection VSEwhere two lines met (mark where lines meet in Fig. 3b).Next, they decided that cars would be generated and enterfrom off the maprepresented as an arrow VSEand beterminated when they exited the maprepresented as asolid dot VSE. Overall, the longer designers spent at asketch, the more VSE types they created (r .62 .36).

As designs progressed and designers had more conceptsto describe, designers sometimes externalized concepts onthe whiteboard by borrowing syntactic elements from othernotations. For example, designers often began sketches as alist, perhaps reflecting their simplicity. As the discussioncontinued, designers sometimes borrowed syntactic ele-ments from other types of sketches. For example, lists thatrepresented software entities sometimes began to borrowsyntax from class and ER diagrams such as arrows, cardi-nality, and bounding boxes (Fig. 7a and 7b; Fig. 8a and 8b).Lists representing requirements documents gained labelson first order elements or gained arrows to represent depen-dencies between requirements. As a result, some lists beganto evolve into sketches of another type (e.g. Figs. 7 and 8).

In some situations, designers chose to evolve theirsketches, often in situations where aesthetic presentation wasnot a priority. In other cases, sketches began their transition inplace before being redrawn (Figs. 7b and 7c; Figs. 8b and 8c).Redrawing sketches was often triggered by a shift from usinga sketch to help think through the design to using a sketch todocument the design. In some cases designers also left the

Fig. 6. The number of VSE types in a sketch as a function of time. Eachdata point depicts the number of VSE types in an individual sketch whenit has had focus for the corresponding time.


prior sketch. For example, toward the end of the session,when designers wished to record more of the design knowl-edge in their heads, pairs B and H redrew the sketch fromscratch in much greater detail. As they drew the new sketch,they left the old sketch intact, allowing them to refer to the oldsketch as they drew the new one.

Pairs varied greatly in how often they erased existingsketches, ranging from erasing 0 percent to 81 percent oftheir sketches. Overall, pairs erased 41 31% of theirsketches. Pairs that created more sketches erased more, asthey needed the space to create the additional setkches.

5.2 Focus and Transitions

Throughout the design sessions, pairs shifted their focusamong sketches on the whiteboard. Fig. 9 depicts the focusperiods of all sketches. Designers sometimes had a single,core sketch that dominated their design session. For exam-ple, pair B spent much of their session interacting with aclass diagram (top red line in Fig. 9b) and pair H interactedextensively with a map (top blue line in Fig. 9h). But,designers more often worked with groups of relatedsketches; and even pairs B and H used a wide range of addi-tional sketches in support of their core sketch.

Designers often rapidly shifted between related sketchesthat each served a separate purpose in supporting thecurrent design activity. For example, when enumeratingitems in a list, pairs shifted their focus to a map to discusshow cars moved, to a table to understand how different cat-egories of elements in the map interacted with each other,and finally to a class diagram to record new aspects of thedesign. Overall, transitions between sketches were very fre-quent, leading to short focus periods in which pairs workedwith a single sketch (Figs. 9 and 10). Designers rarely (12 5% of total focus periods) focused for more than 30 secondsat a single sketch, and they often spent less than 10 secondsat a sketch (66 9%).

As designers worked with a group of sketches, they oftenshifted back to a sketch with which they had recentlyworked (Fig. 11). In some cases, pairs ping-ponged backand forth between a single pair of sketches (e.g., top twosketches of pairs B and D in Fig. 9). Average focus periodswere short across all sketch typesthere were no types ofsketches that designers did not use in close collaborationwith other sketches (Fig. 9).

To examine the typical size of the groups of sketcheswith which designers worked, we counted the number ofsketches which received the designers focus within win-dows of 30 seconds, 1 minute, and 5 minutes (averagedacross all windows in 1 second increments; e.g., windows of

0-30, 1-31, etc.). Overall, pairs used an average of 1.5 ( .9)sketches every 30 seconds, 2.0 ( 1.2) sketches every minute,and 3.9 ( 2.0) sketches every 5 minutes (Fig. 12).

Focus transitions were often to sketches of different types(53 17%) or domains (43 16%) and sometimes crossedlevels of abstraction (27 13%). For example, system-drivenpairs (B, C, E) relied heavily on problem domain sketches tosupport reasoning about components within their systemdesign sketches.

All system-driven pairs frequently moved between amap that they marked up with circles to represent cars tothe equivalent entity in a system sketch. Often, the design-ers did not write on either sketch, but instead internalizedthe insights of their discussion. In other cases, designers didedit sketches. For example, several pairs (B, C, G) mentallysimulated the movement of cars through intersections usinga map. They rapidly switched back and forth between themap and the class diagram to design the data model, draw-ing relationships between different classes to capture neces-sary control and data flow. As designers very rarely drewseparate sketches to represent alternatives or separatepoints in time (see Sections 5.3.3 and 5.3.1, respectively),designers very rarely shifted between sketches representingalternatives (1 1%) or points in time (1 3%).

In 25 percent ( 6%) of the total focus periods, designersquickly referred to a sketch for less than 3 seconds withoutediting it as a momentary reference. We categorized eachmomentary reference as a quick glance, point, or split focus.Quick glances (43 14% of momentary references) alloweddesigners to gather information or seek confirmation. Point-ing (37 14% of momentary references) helped guide theattention of the other designer in the pair to explain, reviewprogress, or mentally walk through the design. Split focus(20 10% of momentary references) allowed designers toreason about how a design might work, using the knowl-edge gained from one sketch to help identify omissions,mistakes, and inconsistencies. For example, the designer onthe left in Fig. 13 simultaneously pointed at a data modeland a map to understand how a car object is passed betweencomponents as it travels through an intersection. Mentalsimulations were often mentally intensive, requiring exten-sive attention. Split focus enabled designers to externalizethe sketches relevant to the discussion, reducing the cogni-tive overhead of recalling where sketches were located andallowing more cognitive resources to be used on the reason-ing activity at hand. This is consistent with distributed cog-nition and the concept of computation offloading [44].

5.2.1 Transitions between Levels of Abstraction

Shifting to a sketch at a different level of abstraction enableddesigners to focus on a particular aspect of the design by

Fig. 7. Content (a) initially created as a list (b) evolved in place to a classdiagram, and (c) later was redrawn as a new class diagram (pair B).

Fig. 8. Content (a) initially created as a list (b) evolved by borrowing syn-tactic elements, and (c) later was redrawn as a new ER diagram (pair D).


Fig. 9. Sketch focus during the design sessions, depicted for each 10 second interval (focus periods shorter than 10 seconds are rounded up to 10seconds). The time shown begins when pairs were first given the design prompt and ends with the end of the design task. Each horizontal row corre-sponds to a single sketch or reasoning activity (bottom three rows). Sketches are listed in the order in which they were created.


omitting non-relevant details. For example, six of the eightpairs shifted at least once from a map of intersecting roadsto a sketch of a single road, which helped them focus on themechanics of individual cars while ignoring details of therest of the map. Designers did not shift between sketches atdifferent levels of abstraction until their design had becomesufficiently complex. Designers began shifting betweensketches at different levels of abstraction, on average, 33 (17) minutes into the session. After this point, abstractionshifts were frequent, occurring once every 49 ( 25) sec-onds. Overall, 26% ( 13%) of each pairs transitions shiftedbetween abstraction levels.

The designers frequently shifted to lists in order to dis-cuss components at a high level of abstraction; lists were thetarget sketch type in 39% ( 15%) of abstraction transitions.These lists were often lists of system components or require-ments. All designers created a list of software classes, allow-ing them to defer discussion of details until later in thesession and carry on conversations at a high level of abstrac-tion. As the design session progressed, some pairs (B, D, F)fleshed out classes with more detailed designs, oftenexpressed in ER diagrams. But the more abstract lists con-tinued to play an important role, helping designers to keeptrack of where they were in the design process by checkingoff, erasing, and pointing at the lists.

Maps were a frequent target of abstraction transitions aswell (35 19%), helping to support less abstract discussions.Designers shifted to map sketches from software and listssketch types in order to better understand the problemdomain. For example, designers moved between sketches ofan entire map to a single intersection, allowing them toexplore traffic configurations at an intersection without theclutter of a larger map. In turn, focusing at a lower level of

abstraction led to a large variety of visual syntactic elements,such as elements for single lanes, many lanes, lights, trafficconfigurations, and car queues.

5.3 Reasoning Activities

Sketches played an important role in supporting at leastthree distinct types of reasoning activities: mental simulation,review of progress, and suggesting design alternatives (Table 4).Pairs averaged 17.9 ( 8.4) mental simulations, 11.8 ( 4.6)reviews of progress, and 14.4 ( 7.4) discussions of alterna-tives per session (Table 5). Pairs varied greatly in the fre-quency of these types of reasoning activities: mentalsimulation (5 to 30), review of progress (5 to 17), and discus-sion of alternatives (5 to 27). Sketches played an importantrole in supporting reasoning activities: 81% ( 13%) ofreasoning activities used sketches, with 53% ( 8%) of rea-soning activities gesturing over existing sketches withoutediting, 22% ( 11%) editing an existing sketch, and 6% (5%) creating a new sketch. In the following sections, weexamine each reasoning activity in turn.

5.3.1 Mental Simulations

Mental simulations allowed designers to walk through thebehavior of their design. As in other reasoning activities,mental simulation was often supported by sketches (83

Fig. 10. Mean focus period length by sketch type (averaged acrosspairs).

Fig. 11. The number of focus transitions since the sketch last had focus,as a percentage of total focus transitions (reported by pair).

Fig. 12. The number of sketches receiving the focus within windows of30 sec, 1 min, and 5 min.


17%). Mental simulation sometimes involved edits (30 30%) or a creating a new sketch specifically to support themental simulation (9 11%). When designing user interfa-ces, designers sometimes mimicked using their design bytalking and gesturing over sketches, using the experience todemonstrate a nonfunctional requirement that the systemshould satisfy. For example, one designer (A) simulated auser creating roads by dragging the mouse across the mapand stated that in her proposed interaction you can createsomething very quickly without fussing with it. Mentalsimulation helped designers not only to imagine using theirdesign, but to put themselves in themindset of the end-user.

When designing with system and problem sketches,mental simulation enabled designers to expose the gutsof their design in order for it to be scrutinized. For example,one pair (D) declared that they were putting [their] footinto the dirty details before initiating a mental simulation.All system-driven pairs used mental simulations to explainhow data and control was passed between software classesor how the state of the model was changed over time. Men-tal simulation was typically not strictly sequential from startto finish: designers instead backed up in their explanations,paused to explain pieces, and abandoned simulations whenthey became distracted.

Surprisingly, designers almost never supported mentalsimulation by creating separate sketches representing dif-ferent conditions or points in time. In the single instancein which this did occur, pair B reasoned about the systemby drawing two adjacent intersection sketches labeled Tand T 1. They then simulated the movement of carsthrough an intersection by shifting between the beforeand after diagram.

Overall, mental simulations served several purposes:Propose new design ideas. As designers simulated and

walked through the behavior of aspects of the design with atemporal component, such as the behavior of end-users orcode, they sometimes generated design ideas. For instance,while walking through the end-user experience, designers(A, G, H) created new interface components on the fly tocontinue the simulation. As another example, while walkingthrough control or data flow, some designers (B, D, F) cre-ated the needed class diagram elements in a similar manner.

Clarify and refine. Mental simulations helped to bringfragmented assumptions about the system together into acohesive story. For example, after one designer (G) becamelost in the details of their design, the other designer clarifiedthe design by mentally simulating its steps.

Reflect and evaluate. Mental simulations brought outdetails of the design, allowing them to be evaluated. Forexample, after stepping through data flow between softwarecomponents, one pair (B) discovered a potential problem intheir design: well that may end upwith spaghetti code. . . sowhat about a traverser pattern. In another case, a designerin pair F changed a design decision and retreated to a previ-ously-rejected design idea. After performing a mental simu-lation, he stated: I didnt before, but now I think [showingindividual cars] is important. When a designer walkedthrough a simulation, the partner would sometimes jump inand ask: What about X happening? Mental simulationsoften spurred partners to propose situations and corner casesthat could lead to problems in the design.

Reveal implications. Mental simulations helped to revealimplications of the design that were not immediately obvi-ous. For example, one pair (D) had determined that carsmust go through an intersection. But after one designer sim-ulated this behavior with an explanation of how this occurs,the other designer asked: But how does the intersectionknow where to send it?, prompting a new discussion.

5.3.2 Review of Progress

Reviewing progress was a central part of designers activi-ties at the whiteboard, allowing designers to take stock ofwhere they were and evaluate the design as it existed. Onaverage, designers stopped to review progress every 9.3 (2.5) minutes for 49 ( 11) seconds, and all eight pairsreviewed their progress. When reviewing progress, design-ers often consulted whiteboard sketches they had drawn (88 13%). These consultations only occasionally led to edits(24 19% of instances reviewing progress). Reviews ofprogress almost always involved reviewing requirementsketches (86 13%). Designers reviewed lists of

Fig. 13. An example of a split focus between two sketches.

TABLE 4Reasoning Activities

Reasoning Activity Example

Mental simulation So the cop looks at the intersections [and] says, okay its a green light on these roads. This road has somenumber of cars in it. If the lights green one tick, [it] means we pop, and we do that queue and second pic-ture. So from T0 to T1, thats how weve gone through two.

Review of progress Okay. Maybe we should start talking about the view. Weve got kind of a rough sketch of the base class ofthe model and kind of a rough idea of the controller, which is a combination of this cop with the visitor. . .

Suggestion of designalternative

Or you could say, well the queuelets go back to the model for a secondso instead of just a straightqueue you could have like a. . .Well, we have for interior road[s], we have a maximum number of slots, solets say this is the head.


requirements, which were then cross-referenced with othersketches to determine if the current design satisfied therequirements as the designers understood them.

Pairs varied in how they tracked requirements, which inturn influenced how they reviewed their progress. Somepairs (D, F, G) did not list requirements. Pairs D and F weresystem-driven pairs that visited mainly system and problemdomain sketches. Pair D performed their reviews of prog-ress to check their work over software components. Pair F,in contrast, performed their reviews of progress by role-playing the designer and implementer roles; one designertold the other to: pretend Im the developer, describe thesystem to me. Pair G, a user-interface-driven pair, usedsketches primarily to summarize their progress.

Other pairs (A, C, and Hall user-interface-driven pairs)maintained one or two lists of high-level requirementsthroughout the design session (e.g., Fig. 5a). Designersvaried in terms of when the requirements were authored:two pairs (A, H) created their entire list of requirements ina short period of time, later using the list to check thatrequirements were satisfied but making very few edits; theother pair (C) made many edits and additions to theirrequirements throughout the design session.

Reviewing progress served several functions:Verify coverage of requirements. Designers reviewed

requirements one at a time, ensuring that each was satisfiedby the design by cross-referencing sketches. This sometimesled pairs to return to deferred issues, spawning new discus-sions to address them. Designers used check marks andother symbols to check off completed requirements.

Identify assumptions. Designers reviewed their sketches tocheck that the design was logically sound, helping touncover assumptions and portions of the design that wereincomplete. For example, some pairs realized that carscould travel between multiple intersections, spawning a dis-cussion to expand the design for these cases.

Plan what to address next. After reviewing a set ofsketches, designers sometimes discussed the sketches theyneeded next, pointing at existing sketches to establish whatthey had missed. For example, pair B looked over what theyhad created, checked it against the requirements, remarkedthat they had 30 minutes, and agreed that they shouldredraw everything to communicate the design to the devel-opers who would implement it.

5.3.3 Discussion of Alternatives

Designers often discussed alternatives; overall, pairs dis-cussed an alternative once every 8.7 ( 4.4) minutes. Discus-sing alternatives included both situations where alternativeswere considered seriously, and those where straw manalternatives were used to clarify an existing design idea with-out any intention of following them through.When unable tochoose definitively between alternatives, designers often

made a provisional decision, resolving to re-evaluate thedecision later, as: its more expensive to get stuck on an idea;[we] need tomove on! (A). For example, six of the eight pairsconsidered alternative designs for left turn signals. Severalchose one approach early on or deferred the decision, revisit-ing the decision later with in-depth discussion when review-ing progress ormentally simulating the situation.

When considering alternatives, designers often (74 17%)made use of whiteboard sketches, either by pointing, editing,or creating new sketches. Discussing alternatives was mostoften supported by simply pointing or glancing at existingsketches. In 76 percent ( 15%) of cases, no trace of discussedalternatives remained on the whiteboard. Only 18 percent (17%) of instances of discussing alternatives involved editingan existing sketch and only 6 percent ( 5%) involved creat-ing new sketches. Even when creating new sketches, design-ers rarely made comparisons between them by switchingbetween alternatives in competing sketches. Instead, theydiscussed the alternatives. Designers almost never (twosketches) discussed competing alternatives by explicitly rep-resenting each alternative in a separate sketch.

While sketches played a crucial role in helping designersto work through the ramifications of their choices, sketchesoften did not accurately reflect the current alternative underdiscussion. Nonetheless, designers discussed them as if theyrepresented the most up-to-date state of the design. Forexample, one pair (C) discussed user interactions for creatingroads, sketching alternatives in which roads were repre-sented by either connected square blocks or by straight lineswith notches. After deciding on the latter, they subsequentlygestured at the first sketch, speaking as if it had lines withnotches representing roads. In some cases, designers evenpointed and gestured at imaginary sketches. For example,one designer (H) pointed at areas in a blank square andexplained a design as if therewere sketched content.

6 THREATS TO VALIDITY

There are three major threats to the validity of our results:the selection of participants, the choice of the design task,and the context in which the design task occurred.

First, our selection of participants may threaten the gen-eralizabiliy of the results if our eight pairs are not represen-tative of the general population of software designers. It ispossible that the results might differ for designers with dif-ferent backgrounds or specializations or who are familiarwith other design approaches such as model-driven designor agile design. However, we mitigated this issue by recruit-ing participants with extensive experience in design froma variety of backgrounds. We recruited 16 designers fromseven different companies, each with its own cultureand design methodologies. Participants had a median of17.5 years of experience as professional designers. Ofcourse, as the designers were experienced, the results mightnot generalize to less experienced designers.

The second threat to validity is the choice of design task.Designers approach tasks based on their existing experi-ence, including their knowledge in the domain. Individualswith extensive experience in a domain may already knowwhich design alternatives are optimal or should be avoided,or use their intuition based on their experience to design

TABLE 5Number of Reasoning Activities (By Pair)

A B C D E F G H Avg

Men. Sim. 30 20 19 14 5 24 8 23 17.9Rev. Prog. 17 7 8 5 15 12 13 17 11.8Disc. Alt. 22 12 17 5 7 12 13 27 14.4


more efficiently. In order to place designers on an evenplaying field, we chose a design task in a domain that mostparticipants were likely to have encountered in their livesbut were unlikely to have worked on professionally. Thismay have influenced how the designers worked. However,a previous study which used the same task [31] comparedthe overall design process to that of Olson et al. [37], who inturn compared the process to that used in practice, andfound a great amount of similarity in practices.

The scope of the design task is also a factor. An overly-specific task might not allow designers sufficient creativity,artificially constraining the ways in which designers mightnaturally engage in a design process. Conversely, an overly-generic task might lead designers to approaches and solu-tions so divergent as to be incomparable. To mitigate thisissue, we asked for a specific set of deliverablesa verbalsummary of the design and an architecture to provide to ateam of developersbut allowed the designers completefreedom through the design session in how they arrived atthose deliverables. At the same time, we recognize thatour task was still relatively small, compared to real-worlddesign projects.

The third threat to validity is the context in which thedesign task was performed, including the time available forthe design session and the environment in which the designwork occurred. Design took place amongst pairs of design-ers, with equal responsibilities and knowledge of the task.In practice, design may involve groups larger than pairsand may sometimes involve designers with unequal knowl-edge, leading to participant behavior within sessions that isless balanced. Participants were instructed to write exclu-sively on the whiteboard rather than use other media suchas paper; while designers in practice may design primarilyusing the whiteboard, this may be supplemented by use ofmediums such as paper to record and make note of thedesign. Participants were limited to a single two-hour timespan and had no access to other people or resources. Inpractice, design typically occurs over longer periods of timeand involves many different stakeholders, with whom thedesigners may work and from whom they may obtain feed-back during the design process. Design in practice may alsooccur during a wide variety of software lifecycle phases,often forcing designers to consider the existing design andthe constraints that it imposes on the design task at hand. Incontrast, our participants approached a new design task.

Despite these potential issues, the designers generallyfound their experiences during the study to be represen-tative of their experiences in real world design. Duringthe interview at the conclusion of the design session,some designers stated that the design discussions withinthe session matched those that they have in their ownmeetings. This sort of discussion is very much what theengineers do . . . within the team working with projectmanagement and UI design cross-functionally. However,a few designers also reported that they adjusted the waythey approached the design problem due to the limitedamount of time. For example, one designer stated: itwas, I think, predictable and expected that we spend a lotof time on early discussion and progressively get quickerat making decisions and moving along and puttingtogether the data model at the end.

7 DISCUSSION

Our results reveal that sketches play a crucial role in sup-porting design conversations. In turn, variations in theseconversations, both from designers approach to the designproblem and the evolution of their conversations them-selves, influence the types of sketches designers draw andthe visual syntactic elements within these sketches. Wefound that design conversations span small groups ofsketches, with each sketch playing a distinct role in support-ing the design conversation.

In the remainder of this section, we first list our mainfindings and then discuss these findings and their implica-tions for tool design. Where our findings corroborate otherreported studies, we provide a citation.

7.1 Main Findings

7.1.1 Types of Sketches

What types of sketches do designers create? Designers createlists, tables, GUIs, ER diagrams, class diagrams, code, draw-ings, and domain specific sketches [8], [55]. Designers prefergeneral-purpose sketches that support the discussion ofmany scenarios over sketches specialized for an individualscenario. While designers sometimes use ER and class dia-grams to design the system, domain sketches are crucial tosystem design [40].

Do the sketches created vary with respect to the approach todesign taken? Designers may approach a design problemfrom the perspective of the user interface, the system, orrequirements, leading them to work with different types ofsketches. Nevertheless, all designers made use of a commoncore of sketch types, as their design process moved beyondtheir initial focus to other aspects of their design.

How syntactically complex are sketches? Overall, the aver-age number of distinct visual syntactic elements in sketchesranged from 4.0 for lists to 11.8 for class diagrams. Design-ers very rarely reordered or rearranged lists, suggestingthat order was either unimportant or was clear withoutexploration or external representation.

How do sketches evolve? Sketches increase in syntacticcomplexity as design conversations progress and designersintroduce new syntactic elements to record aspects of thedesign [39]. Designers sometimes borrowed syntactic ele-ments from other notations [14], [40]. As lists accumulatedadditional syntactic elements, they sometimes evolved intoclass and ER diagrams [14], [39].

7.1.2 Focus and Transitions

How do designers shift their attention between sketches? Duringdesign dialogues and periods of review, designers oftentransfer their focus repeatedly among a small group ofsketches. These groups of sketches typically provide alter-native views on some aspect of the design, for examplesketches of different types or domains, sometimes acrosslevels of abstraction (cf. [40]). Focus transitions most oftento a sketch with which a designer has worked recently.They are rarely between sketches representing distinct alter-natives or points in time.

How long do designers focus on individual sketches? Design-ers move rapidly between sketches. Designers rarely (12


5% of total focus periods) focused for more than 30 sec-onds on a single sketch, and often spent less than 10 sec-onds on a sketch (66 9%).

How do designers make reference to sketches? We observedthat designers often referred to adjacent sketches in shortbursts, even when they do not edit or add to them. Design-ers use momentary references to gather information, guidethe attention of other designers by pointing, and combineinformation from sketches by dividing their focus.

7.1.3 Reasoning Activities

How do designers use sketches to understand and advance thestate of the design at hand? Sketches play an important role insupporting reasoning activities: 81 percent of reasoningactivities used sketches. Sketches most often support rea-soning activities simply by providing information andsomething to gesture at. Only 22 percent of reasoning activi-ties resulted in an edit to a sketch, and only 6 percentresulted in the creation of a new sketch.

Which types of reasoning activities do the sketches support?Sketches support mental simulations, helping designers topropose new ideas, clarify and refine, and reveal implcia-tions. Sketches support designers in reviewing theirprogress, helping designers to check the coverage of require-ments, identify assumptions, and plan what to address next.Finally, sketches support the discussion of alternatives, mostoften through the use of existing sketches rather thanthrough editing (18 percent) or creating sketches (5 percent).

Are the outcomes of design discussions always recorded in thesketches supporting these discussions? Sketches did not repre-sent the entirety of the design or even, for the partsdescribed, its current state. Designers decided on solutionswithout writing them down and reused sketches fromrejected alternatives to support reasoning about differentdesign alternatives.

7.2 Sketches Support Design Conversations

Designers made constant use of sketches throughout theirdesign process, using the whiteboard to discuss designalternatives, tracing paths across the whiteboard to mentallysimulate the behavior of the systems they designed, andexamining sketches to review their progress. Yet, whilesketches were central to designers work, they did not repre-sent the entirety of the design or even, for the partsdescribed, the current state of the design. Instead, thedesign existed only in the minds of the designers and in theconversations in which they engaged. Designers decided onalternatives without writing them down, referred tosketches as if they were up to date, and gestured in space atimaginary sketches. As the design evolved through design-ers conversations, sketches helped support these conversa-tions, even when the evolution of the sketches did not keeppace with the evolution of the design.

Designers often worked with sketches that were general-purpose, enabling them to quickly simulate a wide range ofscenarios by pointing to and annotating a general-purposesketch, rather than drawing each scenario in a separatesketch. Much as a schematic of a car design might support adiscussion of a maintenance issue, or a timeline might sup-port a discussion of scheduling, general-purpose sketches at

a higher level of abstraction supported the designers in div-ing into the discussion of many alternatives and manyissues. In contrast to other normative and descriptiveaccounts of design at the whiteboard which emphasize theuse of separate sketches of alternatives to compare, weightradeoffs, and synthesize [7], [58], designers almost neverexplored alternatives by creating separate sketches for eachalternative. Rather, they pointed at existing sketches, evenpretending that they matched the design that existed onlyin their minds. One potential explanation comes from thecontext of our study: designers performed early explorationof a design. This kind of design task may allow designers tobe more creative and less constrained.

Designers preference for general-purpose sketches alsoshaped the types of sketches designers used. Rather thansketching individual scenarios separately using sequencediagrams, designers used class and ER diagrams whichafforded the flexibility to point at and rapidly flesh out mul-tiple scenarios within a single sketch. Even when simulatingthe behavior of a design over time, designers evidentlyfound it easier to point at and reason about elements withina class diagram than to repeatedly draw sequence diagrams.Similarly, designers working with user interfaces built gen-eral-purpose mockups, simulating scenarios through point-ing and annotating rather than sketching each state in astoryboard. This preference may reflect the demands ofearly, informal design in which many designs are consid-ered in quick succession; designers did, for example, reportthat they would typically create sequence diagrams later inthe design process. And designers did choose to externalizemore in situations in which the design conversation becamecomplex. For example, in designing a class as part of alarger class diagram, one pair (B) provided more detail bymaking a list.

Implications. These findings have important implicationsfor the design of both fully interactive whiteboard systemsand capture technologies such as electronic pens for physi-cal whiteboards. First and foremost, such tools must dealwith the discrepancy between design as it exists in design-ers minds and as designers record it on a whiteboard: infor-mal sketches provide support for discussing a design, ratherthan a record of the design. This has fundemental implica-tions for tools intended to capture an accurate record of adesign. Rather than simply record a static view of a sketch,finding ways to record the design process itself, includingthe reasoning activities enabled by sketching, is cruciallyimportant, as the design does not exist in the sketches but inthe conversations and interactions they enable. For example,to capture a sketch used in a mental simulation, it may bemore helpful to record the strokes designers draw, momentby moment, and designers corresponding discussion,allowing designers to later play back a video capturing theentire walkthrough of an aspect of a system that took placein the mental simulation.

Second, tools automatically capturing sketches for long-term usagesuch as in design documents, wikis, and issuetrackersmust take into account that a single sketch may beused to discuss a range of topics. Given sufficient context, itmay be possible for the original designer to identify the ratio-nale of a decision from a sketch. But, as sketches are used formultiple purposes, it may be insufficient to simply record a


sketch in one place. Thus, it again may be more effective forsuch tools to consider recording not simply sketches but thereasoning activities that occur with sketches, including thescenarios and alternatives annotated on top of sketches. Andeven out-of-date sketches may serve an important role incapturing the discussion of newer scenarios.

7.3 The Approach to Design Influences theSketches Designers Draw

While all designers were provided with the same materialsand were given the same prompt, three distinct approachesto design emerged, which in turn influenced the sketchesdesigners drew. Some designers chose to focus on thebehavior of the system, leading them to create systemsketches. Others focused on the user interface, creatingsketches describing user interactions. Others still designedfrom requirements, creating list sketches to work towardtheir designs. Yet all designers made use of a common coreof sketch types, as their design process moved beyond theirinitial focus to other aspects of their design.

Previous work has identified the types of sketches pro-duced during design activities [22], [47], [56]. Our findingsextend these results, identifying the frequency of individualvisual syntactic elements within several types of sketches.

Designers only infrequently made use of traditional nota-tions for software design; of the 155 sketches designers cre-ated, nine of them used a traditional notation, either a classor ER diagram. Instead, much of the design process tookplace in lists, used domain-specific sketch types to modelthe problem domain, or worked with user interface mock-ups. Even when designing the system, designers often didso while working with domain sketches, using them towork through scenarios and generate issues that the designshould address. Design sketches are simply not limite