Passive Capture & Ensuing Issue for a Personal

Lifetime Store

Jim Gemmell, Lyndsay Williams, Ken Wood, Jim Gemmell, Lyndsay Williams, Ken Wood, Roger Lueder & Gordon BellRoger Lueder & Gordon Bell

CARPE Workshop Oct 15, 2004CARPE Workshop Oct 15, 2004

The Price of Active Capture

“ “When I had my first child, I When I had my first child, I bought a camera and took many bought a camera and took many pictures. But eventually I pictures. But eventually I realized I was living behind the realized I was living behind the camera and no longer taking camera and no longer taking part in special events. part in special events. I gave that up – now I don’t have I gave that up – now I don’t have nearly as many pictures of my nearly as many pictures of my second child.” second child.”

Passive Capture

Make capture passiveMake capture passiveLet the user experience!Let the user experience!

Save a rich record of personal experienceSave a rich record of personal experiencePossible application as memory aidPossible application as memory aidThe challenge: we were already swamped The challenge: we were already swamped

in media… won’t this just make it worse?in media… won’t this just make it worse?

Outline

What is SenseCamWhat is SenseCam What is MyLifeBitsWhat is MyLifeBits DemoDemo ConclusionConclusion

SenseCam

sensecama photographic memory for

everyone

Ken Wood & Lyndsay WilliamsKen Wood & Lyndsay Williams

Microsoft Research CambridgeMicrosoft Research Cambridge

15 Oct 200415 Oct 2004

sensecam Essentially a “Black Box” data and image recorder for the Essentially a “Black Box” data and image recorder for the

human body human body Current wearable prototype can be worn for a day and captures Current wearable prototype can be worn for a day and captures

up to 2000 images in 12 hoursup to 2000 images in 12 hours

Image capture triggered by sensors, e.g. motion, light, Image capture triggered by sensors, e.g. motion, light, temperature, people in field of view, ...temperature, people in field of view, ...

Sensor data is also recorded for later presentation, Sensor data is also recorded for later presentation, analysis, and correlationanalysis, and correlation

a short walk through Cambridge

Accelerometer measures motion of the deviceAccelerometer measures motion of the device When any sensor indicates image capture, we wait a few When any sensor indicates image capture, we wait a few

tens of milliseconds to capture image when movement is tens of milliseconds to capture image when movement is less than 10 degrees/second if possibleless than 10 degrees/second if possible

Greatly reduces the number of blurred imagesGreatly reduces the number of blurred images

Before After

image stabilisation

Captures everything regardless of exact camera orientationCaptures everything regardless of exact camera orientation

ultra wide-angle lens

Sensor readings

Corresponding images in rapid serial viewer

dealing with sensecam data: MyLifeBits

status and plans

12 devices built; adding audio trigger and GPS12 devices built; adding audio trigger and GPS But hardware is just the beginningBut hardware is just the beginning Trials and studies are the interesting bitTrials and studies are the interesting bit

domestic probedomestic probe give it to people and see what they do: personal memory give it to people and see what they do: personal memory

enhancement? family fun?enhancement? family fun? clinical trials with memory-loss patientsclinical trials with memory-loss patients

ethics approval for 20 Cambridge patients over 3 yearsethics approval for 20 Cambridge patients over 3 years tourism trialstourism trials

Edinburgh Festival, museums, ...Edinburgh Festival, museums, ... cross-cultural privacy studycross-cultural privacy study

UK, China, India, Egypt, ...UK, China, India, Egypt, ...

fridgecam

MyLifeBits

MyLifeBits is:An experiment in lifetime storageAn experiment in lifetime storage

Digitizing Gordon Bell’s pastDigitizing Gordon Bell’s pastCapturing more of his futureCapturing more of his future

A software systemA software systemCaptureCaptureStorage & retrievalStorage & retrievalOrganization & annotationOrganization & annotation

Minimum requirement: fulfill Vannevar Minimum requirement: fulfill Vannevar Bush’s 1945 “Memex” visionBush’s 1945 “Memex” vision

The guinea pig Has now scanned virtually all:Has now scanned virtually all:

Books written (and read when possible)Books written (and read when possible) Personal documents (correspondence including Personal documents (correspondence including

memos and email, bills, legal documents, papers memos and email, bills, legal documents, papers written, …)written, …)

PhotosPhotos Posters, paintings, photo of things (artifacts, …Posters, paintings, photo of things (artifacts, …

medals, plaques)medals, plaques) Home movies and videosHome movies and videos CD collectionCD collection And, of course, all PC filesAnd, of course, all PC files

Now recording: phone, radio, TV (movies), Now recording: phone, radio, TV (movies), web pages… conversations and meetings to web pages… conversations and meetings to comecome

Paperless throughout 2002. 12” scanned, Paperless throughout 2002. 12” scanned, 12’ discarded12’ discarded..

Only 44 GB, incl. 10 wma, 14 SQL!!! Video: Only 44 GB, incl. 10 wma, 14 SQL!!! Video: o(100) + 500 movo(100) + 500 mov

So you’ve got it – now what do you do with it?

““A record if it is to be useful … must be A record if it is to be useful … must be continuously extended, it must be stored, continuously extended, it must be stored, and above all it must be consulted” and above all it must be consulted”

““The difficulty seems to be, not so much that The difficulty seems to be, not so much that we publish unduly … but rather that we publish unduly … but rather that publication has been extended far beyond publication has been extended far beyond our present ability to make real use of the our present ability to make real use of the record” record”

- Vannevar Bush- Vannevar Bush

Everything goes in a database

You need all the features of a databaseYou need all the features of a database(Consistency, Indexing, Pivoting, Queries, Speed/scalability, Backup, (Consistency, Indexing, Pivoting, Queries, Speed/scalability, Backup, replication)replication)

If you don’t use one, you will find yourself creating one!If you don’t use one, you will find yourself creating one! Files as blobs, also sync with file system for legacy appsFiles as blobs, also sync with file system for legacy apps

SQLSQL

Entities & Links

AnnotatesAnnotates

Caller in Phone CallCaller in Phone Call

Photo of EventPhoto of Event

TranscludesTranscludes

MyLifeBits Software

MyLifeBits store

database

Voice Voice annotation annotation tooltool

Text Text annotation annotation tooltool

Telephone Telephone capture toolcapture tool

TV capture TV capture tooltool

TV EPG TV EPG download download tooltool

Radio Radio capture capture & EPG& EPG

PocketPC PocketPC transfer transfer tooltool

PocketRadio PocketRadio playerplayer

Import filesImport files

MyLifeBits MyLifeBits ShellShell

files

Legacy Legacy applicationsapplications

Browser Browser tooltool

InternetInternet

IM captureIM capture

MAPI MAPI interfaceinterface

Legacy Legacy email clientemail client

GPS import & GPS import & Map displayMap display

SenseCamSenseCam

Screen saverScreen saver

DEMO

A Storocratic Oath

1.1. Do no harm to datesDo no harm to dates(File creation, Photo taken)(File creation, Photo taken)

2.2. Do no harm to device created & Do no harm to device created & other meta-data.other meta-data.

• Camera data & location data are sacred.Camera data & location data are sacred.

3.3. Support & aid the creation of critical meta-Support & aid the creation of critical meta-data. data.

• When/how the user feels like itWhen/how the user feels like it• Auto-magically!Auto-magically!

4.4. Maintain user confidentialityMaintain user confidentiality

www.MyLifeBits.com

research.microsoft.com/hwsystemsresearch.microsoft.com/hwsystems

BONUS SLIDES

Collaborators

Other MSR Collaborators: Kentaro Toyama, Ron Other MSR Collaborators: Kentaro Toyama, Ron Logan, Steve Drucker, Curtis Wong, Mary Logan, Steve Drucker, Curtis Wong, Mary Czerwinski, Brian MeyersCzerwinski, Brian Meyers

Interns: Josh Blumenstock, Evan Salomon, Aleks Interns: Josh Blumenstock, Evan Salomon, Aleks ArisAris

MemexAs We May Think, Vannevar Bush, 1945

““A memex is a device in which an individual stores all A memex is a device in which an individual stores all his books, records, and communications, and which his books, records, and communications, and which is mechanized so that it may be consulted with is mechanized so that it may be consulted with exceeding speed and flexibility”exceeding speed and flexibility”

Full-text search, text & audio annotations, and Full-text search, text & audio annotations, and hyperlinkshyperlinks

The AI Challenge:Content analysis & Data Mining

“Creative thought and essentially repetitive thought are very different things. For the

latter there are, and may be, powerful mechanical aids” – Vannevar Bush

Is MyLifeBits just enough rope to hang yourself with?Is MyLifeBits just enough rope to hang yourself with? MyLifeBits must become MyPersonalAssistantMyLifeBits must become MyPersonalAssistant Content analysis and data miningContent analysis and data mining Doc similarity & “clean living” Doc similarity & “clean living” Document meta-data extractionDocument meta-data extraction

Moving forward:new capture modes/devices

SenseCam

Deja View

Body Media

Quindi

Future work: UI

Focus to date on Focus to date on functionality – now we functionality – now we need to make it need to make it friendlierfriendlier

User studies with User studies with SFSUSFSU

Future work: Visualizations Don't give me a little card Don't give me a little card

image and say, "That's all image and say, "That's all you've got, because that's you've got, because that's what I thought you should what I thought you should want for your virtual want for your virtual shoebox." There have got shoebox." There have got to be multiple modalities to be multiple modalities and the designers have to and the designers have to be able to deal with that. be able to deal with that. … don't metaphor me in, … don't metaphor me in, don't give me only one don't give me only one way of looking at things.way of looking at things.

-Andy van Dam, Hypertext '87 Keynote -Andy van Dam, Hypertext '87 Keynote AddressAddress

Next MediaNext Media

Web ScoutWeb Scout

U. MarylandU. Maryland IN-SPIRE

Dear Appy, How committed are you? Signed, Lost and Forgotten Data

Dear Appy,Dear Appy,I'm having trouble with long-term commitment -- not on my end, I'm having trouble with long-term commitment -- not on my end, heaven knows, but from the apps that created me and with whom I heaven knows, but from the apps that created me and with whom I like to associate. Over time, these pesky apps evolve and they like to associate. Over time, these pesky apps evolve and they simply don't recognize the data that they once helped create! But, simply don't recognize the data that they once helped create! But, we data progeny -- and there are lots of us -- feel that as our we data progeny -- and there are lots of us -- feel that as our creators, these apps should be responsible for eternal support. creators, these apps should be responsible for eternal support. But the little problem with recognition isn't the worst of it – But the little problem with recognition isn't the worst of it – sometimes the apps even disappear altogether. I ask you, is it sometimes the apps even disappear altogether. I ask you, is it expecting too much for 20-something year old data like me to be expecting too much for 20-something year old data like me to be interpretable by my app (e.g. Acrobat, DB2, Draw, Eudora, Office, interpretable by my app (e.g. Acrobat, DB2, Draw, Eudora, Office, Quicken, or RealNetworks), or am I just associating with Quicken, or RealNetworks), or am I just associating with irresponsible apps? irresponsible apps? If things continue on their current path, it seems I will be completely If things continue on their current path, it seems I will be completely un-interpretable within 20 to 50 years! My apps will move to other un-interpretable within 20 to 50 years! My apps will move to other platforms, or evolve to be more Internet- or Next-Big-Thing-centric...platforms, or evolve to be more Internet- or Next-Big-Thing-centric...

By Gordon Bell http://research.microsoft.com/~gbell

Classification wish list Download classifications rather than build themDownload classifications rather than build them Definitions & synonyms should help find what I wantDefinitions & synonyms should help find what I want Today it is too expensive to manually classify my Today it is too expensive to manually classify my

scanned paper. E.g. “right time” meta-data is critical!scanned paper. E.g. “right time” meta-data is critical! Next year I hope “the system” can classify papers Next year I hope “the system” can classify papers

and other documents e.g. billsand other documents e.g. bills In 10 years I expect all documents to appear In 10 years I expect all documents to appear

electronically & classified electronically & classified with a little help from mewith a little help from me

Personal Search is notProfessional or Web search

System sees every entry & accessSystem sees every entry & accessEverything, not just a professional life Everything, not just a professional life Limited to SIS, not an infinite amount, Limited to SIS, not an infinite amount,

covers a profession & personal lifecovers a profession & personal life

Web as seen by search engines

MyLifeBits

Knowledge breadth e.g. Dewey classification

Depth e.g. information item types & coverage

Professional user