InkSeine: i n situ search for active note taking

  • View

  • Download

Embed Size (px)


Ken Hinckley, Shengdong Zhao, Raman Sarin, Patrick Baudisch, Edward Cutrell, Michael Shilman & Desney Tan. InkSeine: i n situ search for active note taking. r. oronto. In situ s earch for digital ink. InkSeine : Ink + Seine -n: a fishing net -v: to fish with a seine - PowerPoint PPT Presentation


  • Ken Hinckley, Shengdong Zhao, Raman Sarin, Patrick Baudisch, Edward Cutrell, Michael Shilman & Desney Tanorontor

  • InkSeine : Ink +Seine -n: a fishing net-v: to fish with a seine= insanely cool ink search!

    InkSeine lets the user fish for useful information directly from ink notes.

    Fishermen catching salmon on the Columbia River using a seine. (courtesy Wikipedia)

  • XLibrispen-based system for active reading (reading + critical thinking + learning)

    InkSeine combines pen-based note taking with searching, linking, collecting & sensemaking: Sketch designsReflect on a topicCapture & extend creative ideas

  • Ink excels at capturing users thoughts & ideas with minimum distraction from formatting issues

  • Search digging through the file system on a tablet is really a painActive note taking benefits from low overhead access to supporting materials & related docsFocus on personal search, but support Web as well

  • Gather Make it easy to use what you find by pulling content directly into unstructured notesThumbnail hyperlinks to documentsClippings from documentsQueries that can be revisited later

  • InkSeine unifies these three elements in a fluid user experience that is integrated directly with inking

  • Leverage preexisting ink to initiate searchNo context switch to search app stay in the flowNo tedious transcription of text to a search boxQueries as first class objects, commingled with ink notesFlag some ink for search for laterVisible and salient search history as part of notesQueries can be copied, pasted, moved, etc.Interleave inking, searching & gatheringNo barrier between inking / searching note ideas while searching, side-by-side searching, span app boundaries for content gatheringTightly couple queries with application contentQueries naturally persist in originating contextEasy re-visitation of prior queriesResults of queries become new content

  • Continuum of time cost can defer at any point: 1 second: Capture thought I should search on this 5 seconds: Trigger search & see initial results list10-20 s: Open a doc from result list or drag result into notes20 s-2 minutes: Scroll through list and inspect details of resultsApply filters, revise queryGrab a snapshot from a short documentCorrect occasional ink reco errors Tablets reco is amazingly good!Longer: sufficient depth for common info needs; not mired by last 10% of complex queries that require full featured search toolsInspect result documents to see if they meet information needsSideways searches based on Title, Date, or Author of a result

  • Existing desktop search tool difficult to useDifficult text entryEnter and correct text

  • Existing desktop search tool difficult to useDifficult text entryEnter and correct textAnnoying context switch and flow interruptionSwitch among apps and context

  • Existing desktop search tool difficult to useDifficult text entryEnter and correct textAnnoying context switch and flow interruptionSwitch among apps and contextTedious information gatheringDifficult to link to documents and gather pieces of information

  • Breadcrumb

  • Query Area(CueTIP)

  • Result List

  • Result ListFocus Result

  • Filters

  • Tap - SelectCross Multi-selectDisjunctive cross Multi-selectCross & Turn Right Exclude

  • SelectionCorrectionSearch

  • Gesture discovery problem

    Highlighter hints

  • Context preservation & occlusion problem

    Smart positioning of search panel

  • Recognition problem for out-of-vocabulary words

    Personalized vocabulary from desktop search index

    UISTList?, Dist?, Gist?, Mist?, Hist? ?

  • In situ search experience

    Tailored to pen input

    Optimum workflow / maximum flexibility

    Multiple ways to gather content

    Minimize search screen real estate

    Span application boundaries

  • Pull in materials for searches from notes

  • Pull in materials for searches from notesSupport vector-based searches

  • Now we have a basic idea of the system, let me tell you the story behind its design

    In the early design stage, we wanted to convince ourselves whether users see values for ink-based search, as well as how they feel about the existing approaches. We conducted a paper prototype studies with 5 researchers.

    This is a screenshot a study in progress. The process goes as follows: they are first asked to write down a page of notes, then comments on if there are terms or phrases they like to search for. The feedback is consistently positive. They all found search for terms and phrase within their notes valuable. We then ask them to perform the search on these terms using an existing desktop search tool called Phlat (which is based on windows desktop search). Once they have done that, we ask them to propose improvements on the current search interfaces and suggest wanted features. Here are the screen shot of various interfaces proposed during the study. What we have learned from this study is that, first, existing desktop search tool (which literally designed for desktop computers with keyboard and mice) are very difficult to use. One major problem is that text entry through the tip interface is difficult and tedious. And its annoying to switch between the search application and the active note taking application. Both the tedious text entry process, and the application switches disrupt the train of thoughts, which is very important for active note taking. Finally, the lack of support of easy gathering and linking the relevant information together in the existing application making the active note taking process inconvenient to deal with.

    Based on these findings, we developed a prototype to overcome the forth mentioned limitations, and try to provide an optimal experience for searching while active note taking.

    Earlier, we saw the video of how the system works as whole, now I will go through the system step-by-step with you, and discuss the individual features in more details. The search tool live within an active notebook application, we call it the notebook control (example application are windows journal or Scriblet) The application can have a number of objects, and support a number of gestural commands. Now we have a basic idea of the container application, lets look at how we can perform the search tasks.

    As you can see, we are active note taking on the subject of InkSeine, and trying to gather some related work. We remember there is a paper called Bumptop last year thats so what interesting, so we want to look it up. To do that, we draw an open ended lasso around the word: BumpTopImmediately, a small icon appears at the end of the lasso. This icon is called the breadcrumb and its an indication that I am interested to perform a search on the selected word. However, I dont have to dive into the search right away, for example, if my train of thoughts has a number of related work I am thinking about, I can continue write down all the other papers I can think of. The breadcrumb serves as a reminder, information storage, and a placeholder for further actions. Breadcrumbs provides information about the searches they represent. By drawing a mark start on the breadcrumb and toward the south, we trigger the personal search, and brings up the search panelThe search panel is composed of different functional units. On the top, right under the breadcrumb is the query editing and revising area. Since ink can sometimes be mis-interpreted, we need a place to allow the user to correct the ink. Also, user could change their mind and revise the query, the query area also allow the user to add, delete, and edit the query terms.

    Once user are sure about their search query, they want to look at the result, which is located right below the query box. It contains multiple columns of information on the research results. Point to any of the record will make it the focus result with additional information, such as text snippets, thumbnail, and hyperlinks to offer actions for that document. Based on the lessons we learned from earlier search interfaces such as phlat, two types of filters are the most useful for people to narrow down their search result for personal items. The two types are the file type filter (which is located vertically on the right), and the timeline filter, which is located at the bottom.

    Both filters support a number of convenient gestures for pen based interface. The search panel is composed of different functional units. 1) Can drag link to document into notes OR can open document.*When a document opens, we provide the user with a tracking menu to afford pen interaction with legacy mouse-and-keyboard applications. It supports scrolling documents with a scroll ring gesture, and a capture tool for grabbing clippings from a document. The earlier video shows how it works.