Techniques d’etudes et de recherche

Techniques d’etudes et de recherche

Dr DIALLO [email protected]

UFRMI 2016

Overview• 32 hours• ~ 50 percent of lectures / 50 percent of work groups

• Objectives• Training students to research methodology.• Training students to effective written and verbal communications.• Familiarizing students with the research ecosystem• Providing overview of kinds of problems addressed by C.S. research.• Discussing active research areas in C.S.

Workgroups• English session (Elements of style)• Statistics workgroup • Discussions • Around a paper, a topic, your homework.

Homeworks• Homework 1: Write a review.• Homework 2: Write a survey.• Homework 3: Write a paragraph on your research statement.• Homework 4: Tap a dataset.• Homework 5: Oral presentation (Research statement).

Outline• Introduction • Types of Research in computer science• Research methodology• Written and Oral Communication • Tools for Research • Sample research areas in computer science

Warrant from Wayne BoothListen to others as you would have others listen to you. Precise demonstration of truth is important but not as important as the communal pursuit of it. Put in terms of Kant’s categorical imperative, when addressing someone else’s ideas, your obligation is to treat them as you believe all human beings ought to treat one another’s ideas.

IntroductionThis section states the objective of the course, defines what is research, the actors of the research process and the ecosystem of the research. It discusses also the worthiness of PhD for a research career.

What is research ?• “Research is an organized and systematic way of finding answers to

questions”http://linguistics.byu.edu/faculty/henrichsenl/researchmethods/RM_1_01.html

• In everyday life, this includes • Reading a factual book of any sort is a kind of research• Surfing the internet or watching the news is also a type of research

From: http://www.experiment resources.com/definition of research.html‐ ‐ ‐

Who does/needs research ?• Someone in a lab coat peering into a microscope ?• A white-bearded professor taking notes in a silent library ?• Oprah, Jerry Yang (Yahoo), many in business and government• Experts in doing research, and in using the research of others

Why matter ?• As you learn to do research, you will learn to value reliable research

reported clearly and accurately

In science• Goal• Create knowledge• Discovery

• Strict protocols and long established rules• Structure• Formalism

From: http://www.experiment resources.com/definition of research.html‐ ‐ ‐

Research in Computer Science• Scientific practice enables more rapid progress of the field

Explanations — How does that IR system work? Why did theInternet behave in that way?– Guidance — What should we do if we want our data center to have a higher percentage of uptime?– Technologies — How can we build a better integrateddevelopment environment? What networking protocol offers the highest performance for real P2P networks?

From: “Research Methods for Empirical Computer Science”Class by David Jensen at University of Massachussets Amherst

The research ecosystem• Universities / Research labs• Industry R&D• Public research centers (France: CNRS, INRIA, etc.) • Research funding • Industry chairs (Renault, etc.)• National projects funding (ANR, NSF, PASRES)• Regional projects funding (EU, Fonds Macky Sall)

• Publishing • Conferences, Journals

Kinds of things CS researchers do •Design an algorithm•Design an experiment•Run an experiment•Gather data•Make a conjecture•Devise a research question•Construct a theoretical proof•Find flaws in previous experiment

• Identify an important exception to a theory• Construct a general theory• Use a theory to explain an observation• Compare results from theory and experiment• Devise a new measurement or technique• Unify disparate theories• Establish a relationshipbetween variables

From: “Research Methods for Empirical Computer Science” Class by David Jensen atUniversity of Massachusets,Amherst

PhD experience (Education to/by research)What is a doctorate ?• A long, in depth research exploration of one topic• Long = three years (around 6 years in US)• In depth = you will be the world expert or close to it in your particular area• One = typically be working on only one narrow problem

From: M. Harchol - Balter, “Applying to Ph.D. Programs in Computer Science”http://www-2.cs.cmu.edu/~harchol/gradschooltalk.pdf

PhD: Very different than taking classes!In class• Homeworks have known

answers• Techniques for solving problems

introduced in class• Professor pick problems• Close guidance: grades,

professor tells you what to do next

In research• Problems may not be solvable• Invent techniques to solve

problems• You pick problems• Some help from advisor, butneed to be self-motivated and pro-active


Think about life after doctorateAcademia• Teaching in schools• Research in schools• Research institutes

• In France, INRIA, CNRS

• Big vs. small; public vs. private• Country?• Do you love (or at leastlike) to teach? students?

IndustryMany different types of industry settings• Startup• Big industry• Research labs

From: J. Kurose, “10 pieces of advice I wish my PhD advisor had given me”http://www net.cs.umass.edu/kurose/talks/‐

Academia: Research universities/institutes• Doing research on anything you like• Working with graduate students• Teaching classes

• Amount of teaching depends on country

• Applying for grants• Flying around to work with other researchers and to give talks on your

research• Doing service for your department• Doing service for the community

– Reviewing papers, organizing conferencesFrom: M. Harchol - Balter, “Applying to Ph.D. Programs in Computer Science”http://www-2.cs.cmu.edu/~harchol/gradschooltalk.pdf

Industry: Research labs• Doing research• Always need to be useful for company

• Working with other people in the company• Could also have students as interns

• Traveling around a little to give talks and work with others• Doing service for the community• Reviewing papers, organizing conferences

• Importance of each of these tasks depends on the company (more R or more D)


Should you get a doctorate ?• Evaluate• What type of career do you want?• Do you have the elements (personality, drive, passion) to succeed?• Is this the best use of your time?

• If not, it is OK to leave• At any time

• If so, optimize your decisions (life, career, research choices) around making the most of it• If you’re going to “half ass” it, why bother?

From: N. Feamster, A. Gray, “What is a Ph.D.?”http://www.gtnoise.net/classes/cs7001/fall_2008/syllabus.html#Schedule

If you decide to get a doctorate…• Advantages• Ability to have real impact• A lifetime of learning and advancement of knowledge• A job you love• Freedom: much less structure than other jobs• Many people are not so lucky

• But you should conduct High-quality research• You will be evaluated on your publication record and contributions to science,

not on your dissertation• You have an opportunity to fundamentally change the world we live in.

Dissertation is a minimal requirement…think BIG!From: N. Feamster, A. Gray, “What is a Ph.D.?”http://www.gtnoise.net/classes/cs7001/fall_2008/syllabus.html#Schedule

How to do good research ?“A successful person isn’t necessarily better than her less successful peers at solving problems; her pattern recognition facilities have just learned what problems are worth solving.”

– Ray Kurzweil

Factors to consider• Importance How important is the research topic within the larger research and application community?• State of knowledge What do we know already? What is the position of the research with respect to “the frontier”?• Unique competence Are you uniquely qualified to address this research? What is your “secret weapon”?• Interest How much does this research problem interest you personally? Do you have a passion for this problem?

From: “Research Methods for Empirical Computer Science”Class by David Jensen at University of Massachusetts, Amherst

Assessing importance• Audience• Who will care about the answer?

• Impact• Will different answers change...

• what research gets done next?• what is done by practitioners?

• Longevity• How long will the answer be relevant and important?

From: “Research Methods for Empirical Computer Science”Class by David Jensen at University of Massachusetts, Amherst

Choosing a research problem• Pick your problems carefully!• what’s the fundamental issue you’re solving?• will the problem be of interest five, ten years from now?• focus on fundamentals in a world with an increasingly short attention span

• Avoid crowded areas• unless you have a unique talent, viewpoint• low-hanging fruit has been picked• researchers working on “next big thing” are not in the crowd


Complexity of solutions• complexity, sophistication are

themselves not of interest (except to purist)• means, not an end• how is the “story” changed in the

end?


Avoid point solutions• Insights that cut across solution

space vs. Point solution• what broader conclusions can be

drawn from your work?

You are Here (but shouldn’t be)


Be thorough• Always begin with a literature survey• Start with the smallest non-trivial instance• Learn as you go• Prepare to change• Crystallize solutions• Keep an eye open for the unexpected• Carry a notebook

From: S. Kehav, Hints on doing research http://blizzard.cs.uwaterloo.ca/keshav/mediawiki-1.4.7/index.php/Hints_on_doing_research

Be part of a community of scholars• Meet people, listen, collaborate• Good students, colleagues, friends

• Approach, talk with people• Interactions with peer students• research discussions• paper presentations• practice talks….


Homework 1• Select one paper from a conference/journal• This paper should guide your selection of your project’s topic• List of conferences in networking with some statistics http://www.cs.ucsb.edu/~almeroth/conf/stats/• One-page write up with two parts

1. How did you select this paper and why?• Detail your process for selecting paper• Explain why you find it interesting• How this paper inspire your project topic

2. Short description of your research project

Types of research in computer scienceIn this section we discuss the different types of research in computer science research

Computer science research areas• Complexity theory, algorithm design• Automata theory, formal models, logic• Quantum computing• Programming languages• Computer architecture (parallel

programming, many core, GPU)• Operating systems• Distributed systems• Compilers• Networks

• Databases, data mining• Artificial intelligence (machine learning,

natural language processing, computer vision, robotics, social network analysis)

• Computational biology• Security• Computer graphics• Human/computer interaction• Scientific computing, high performance

computing• Visualization

Academic research can be successful• From Stanford to Google • Google search is originally based

on the pagerank algorithm which was developed at Stanford.

Page, L., Brin, S., Motwani, R., & Winograd, T. (1999). The PageRank citation ranking: bringing order to the web.

Designing / Building • Content-centric networking• A new networking paradigm to

better support content distribution at Internet scale.

Jacobson, V., Smetters, D. K., Thornton, J. D., Plass, M. F., Briggs, N. H., & Braynard, R. L. (2009, December). Networking named content. In Proceedings of the 5th international conference on Emerging networking experiments and technologies (pp. 1-12). ACM.

Improving• TCP congestion control• New mechanisms added to TCP to

solve congestion collapse problem and improve performance.

Jacobson, V. (1988, August). Congestion avoidance and control. In ACM SIGCOMM computer communication review (Vol. 18, No. 4, pp. 314-329). ACM.

Modeling• Modeling TCP throughput • A formula to predict a TCP

connection throughput given RTT, loss rate, and other parameters. The formula can be used to design TCP-friendly protocols.

Padhye, J., Firoiu, V., Towsley, D., & Kurose, J. (1998). Modeling TCP throughput: A simple model and its empirical validation. ACM SIGCOMM Computer Communication Review, 28(4), 303-314.

Measuring • Measuring Youtube• Providing various insights about

Youtube measurement data. For instance, provide popularity patterns that can be used by researchers to calibrate simulation models.

Cha, M., Kwak, H., Rodriguez, P., Ahn, Y. Y., & Moon, S. (2007, October). I tube, you tube, everybody tubes: analyzing the world's largest user generated content video system. In Proceedings of the 7th ACM SIGCOMM conference on Internet measurement (pp. 1-14). ACM.

Research ProcessLiterature review (Critical Reading)Problem statementEvaluation methodologyWriting and publication process / Types of communicationsPresentation techniques

How to read research papers ?An introduction to critical reading

Why reading research papers matters ?• To understand state of research on a field• To study seminal work on your field• Learn on another discipline that may offer solution to a problem

Too many papers• Example: Networking conferences• SIGCOMM: ~40 papers• SIGMETRICS: ~20 papers• IMC: ~40 papers• CoNext: ~30 papers• Infocom: ~100 papers• Journals, workshops, …

• Per year: More than 2000 pages to read• Impossible to read it all…• Doesn’t even count cross-disciplinary reading

From: N. Feamster, A. Gray, “Reading and Reviewing Papers”http://www.gtnoise.net/classes/cs7001/fall_2008/syllabus.html#Schedule

Step1: Deciding what to read• Purpose: Learn about hot topics of current research in an area. (searching

for problems, etc.)• Approach: Scan papers in latest conference proceedings

• Purpose: Get up to speed on sub-field• Approach: Transitive closure of related work of papers in a top conference

• Purpose: Learn about an area that is further afield• Approach: Ask expert colleagues


Deciding How to Read• Always top down• First: Abstract, introduction conclusion• Rest of paper if necessary

• If you want to do follow-up research• If you want to better understand the methods/conclusions

• Next steps depend on specific purpose• News reading• Deep diving• Literature survey


Invariant comprehension questions• What is the problem ?• What are the contributions ?• What are the conclusions ?• What is the support for the conclusions ?


Invariant evaluation questions• What is the intellectual nugget ?• Each paper should have a single key intellectual contribution• Remembering this key idea will also give your brain a way to index the paper

• What is the main conclusion / contribution ?• New finding ? Method ? New perspective ?

• Is the conclusion important ?• Does the content support the conclusion ?• Are the methods sound ? In other words, do the main conclusions appear to

be correct ?


Reading the news• Conference proceedings• Goal: Grasp main idea of a collection of a large number of papers. Keep

informed about problems and recent solutions.

• Top-Down method• Skim table of contents: Papers are clustered into sessions which typically

identify the main areas• Consider authors• Prioritize by (1) area of interest (2) reputable authors


Deep Diving• Goal: seek to understand some problem area in greater depth.• Find the seminal paper in the field• Read carefully, including evaluation


Literature Surveys• Create the seed• Recent paper from top conference• Survey paper, if one exists• Seminal paper, if it is different from the above

• Perform transitive closure of cited work• Read related work sections of above papers


Keeping Notes• One-sentence summaries are infinitely better than nothing at all• Primitive approach: Single file of notes• Better: Database with BibTex• There are some existing tools for bibliography management• Will also help you more quickly construct related work sections for your

papers


Some Questions to Ask• Time travel: Will the solution apply n years from now ?• Context switch: Does the solution or technique apply to

other problem domains ?• Unfinished business: Does the paper describe future work or

directions ? Open problem ?• Follow up: Can the claims in the paper be supported using

other methods ? Or, perhaps refuted ?


How to review a paper ?Reviewing in research process

Reading vs. Reviewing Reading• Information gathering, typically

for the benefit of your own research (You are a scientist.)

Reviewing• Goal is to1. Determine a paper’s suitability

for some conference2. Provide feedback to authors to

improve paper

(You are a teacher/evaluator)


A review form• Confidence of the reviewer

• Multiple choice• What is your qualification for reviewing the paper

• Evaluation• Accept, wek accept, can’t decide, weak reject, reject

• Summary of the paper• Your summary of the paper and its main contributions

• Strengths• Weaknesses• Detailed comments

• Comments to back-up your rating and help authors improve paper

• Confidential comments for the committee

Main Question• Does the paper make a significant contribution to the field ?• Are the results surprising ?• Would the paper spark new research ?• Are the ideas clearly expressed ?


First Step: Read and Re-Read • Read the paper once to get the main ideas and contributions • Try to make the one bit decision here

• Read again and take notes (for your review)• Start to organize a review…


Evaluation Method• Motivations and Conclusions• Is the problem important ?• Will a solution advance the state of the art ?• Is there a single important intellectual contribution ?

• Support• Are the results sound, and does the evaluation support the conclusion ?

• Learning• Did you learn anything ? Was it worth learning ?• Will the paper generate discussion ?


Consider the Audience•Will this generate discussion ?• Is this a paper that’s going to send people to the

hallway ?•Will the people who commonly read these

proceedings benefit from the contributions ? •Would people who read other proceedings benefit more

from the paper ?


Consider the Standards•Workshops are typically more permissive as far as accepting

vision without completed, supported work• More emphasis on fostering discussion

• Conference: Depends on quality of papers in the reviewers piles and selectivity• Journals often have the highest standards, especially since

the review process is iterative


Consider the Purpose• Survey• Is the overview complete ?

• Proposal• Does the research that is advocated make sense ? Is it worthwhile ?

• Tutorial• Is the description correct and clearly described


How to Write the Review Itself• Start with a summary• Demonstrates to the authors (and to you!) that you understand the main

point of the paper

• Discuss how authors do or do not deliver on the claims/contributions of paper• Discuss positive aspects (if any)…try to find something• Provide high-level suggestions for improvement• End with details(spelling, punctuation, etc.)


General Tips on Tone and Content• Be polite and respectful• Provide suggestions for how to improve the paper• You may see the paper again !• If the paper is accepted, the flaws should be fixed

• Be positive• The point is not to shoot the paper down


Common Mistake: Being Too Critical•Don’t miss forest for the trees• Papers are never perfect• You job is to determine whether a paper’s flaws invalidate

the contributions (and whether the contributions are significant)

•Being too critical can prevent important research results from being published


Other mistakes and no-nos• Insulting the authors• Criticize the paper, not the authors• The paper did not address

• Revealing your own research agenda• Distributing submitted papers• Spending too much time reviewing a paper• Rule of thumb: Don’t spend more time reviewing a paper than the authors

did writing it !• If a paper is sloppy or flawed, don’t waste your time


Homework 2/3• One PDF file with two parts1. Literature survey for the project (at most two pages, not counting references)

• First, search the literature to identify the most relevant papers to the topic of your project• Second, write two pages summarizing these related papers.

• You should do a critical summary of each paper, highlighting how each paper contributes to the state of the art

• Clearly identify the seminal paper in the area of your project• Prepare using latex/bibtex

2. A paper review (no page limit)• Write a review for the following paper Spathis, P., Belouanas, S. E., & Thai, K. L. (2015, October). Leveraging replication in content-centric networks. In Global Information Infrastructure and Networking Symposium (GIIS), 2015 (pp. 1-6). IEEE.

Recommended readings• S Keshav, How to read a paper, ACM Computer Communication

Review, July 2007• T. Roscoe, Writing reviews for systems conferences

DISCUSSION 1: About your topic•Take into account the instructor’s feedback to move further in your research project.

Writing research papersTechnical writing

Why write it up ?• To remember • To understand• To gain perspective

Why a formal report ?

• It will give you new ways of thinking• Since it is more demanding to write for other, your ideas and

connections among them will be clearer

Credit Booth et al., the craft of research

Conversations among researchers •Research papers and “term” papers are conversations• You must be aware of your audience • What type of relationship do you want with them? • You get to create a “role” for yourself and your reader


Think about your readersWe know things about them: 1. they have their own interests and preconceived ideas 2. Is your question is a “live” issue in your community or readers?3. Where do your readers stand in respect to your answer? 4. What do you expect of your readers ?• accept new knowledge• change beliefs• perform an action


Writers and their common problems• Experts have the same problems as novices, but are better

equipped• Be aware uncertainties will arise •Master your topic by writing about it along the way • Control the complexities of your task• Count on your teacher to understand your struggles • Keep at it


Writing papers is a skill•Many papers are badly written•Good writing is a skill you can learn• It’s a skill that is worth learning• You will get more papers accepted• Your ideas will have more impact• You will have better ideasIn

crea

sing

impo

rtan

ce

From: S. Peyton-Jones, “How to write a good research paper”http://research.microsoft.com/en us/um/people/simonpj/Papers/giving a talk/giving a talk.htm‐ ‐ ‐ ‐ ‐

Writing papers: Model 1


Writing papers: Model 2

• Forces us to be clear, focused• Crystallizes what we don’t understand• Opens the way to dialogue with others: reality check, critique, and

collaborationFrom: S. Peyton-Jones, “How to write a good research paper”http://research.microsoft.com/en us/um/people/simonpj/Papers/giving a talk/giving a talk.htm‐ ‐ ‐ ‐ ‐

Do not be intimidated

Write a paper and give a talk,

about any idea, no matter how weedy and insignificant it may seem to you.


• You need to have a fantastic idea before you can write a paperFallacy

Do not be intimitaded•Writing the paper is how you develop the idea in the

first place.• It usually turns out to be more interesting and

challenging than it seemed at first.


The purpose of your paper

Why bother ?FallacyWe write papers and give talks mainly to impress others, gain recognition and get promoted.


Why bother ?Good papers and

talks are a fundamental part of research excellence.

FallacyWe write papers and give talks mainly to impress others, gain recognition and get promoted.


Papers communicate ideas• Your goal: to infect the mind of your reader with your idea, like a

virus.• Papers are far more durable than programs.

The greatest ideas are (literally) worthless if you keep them to

yourself.


The idea• Figure out what your idea is•Make certain that the reader is in no doubt what the

idea is. Be 100% explicit:• The main idea of this paper is…• In this section we present the main contributions of the

paper•Many papers contain good ideas, but do not distil

what they are.

IdeaA re-usable insight

useful to the reader.


One ping•Your paper should have just one ping: one clear, sharp idea•Read your paper again: can you hear the ping ?•You may not know exactly what the ping is when you start writing; but you must know when you finish• If you have lots of ideas, write lots of papers


The purpose of your paper is not…

•Typically your reader does not have a WizWoz•He is primarily interested in re-usable brain-stuff, not executable artefacts


To describe the WizWoz system

Golden Rule: Storytelling• Every paper tells a story

• Not the chronology of your research

• What’s the big deal ? The main idea ?• What is the problem ?• Why it is hard ?• Why is your solution interesting, significant ?• Why should the reader care ?

• Note: Your story is not a mistery novel• Write top-down

• Note: Nobody is as interested in this topic as you• Make it interesting!

From: N. Feamster, A. Gray, “Communicating Ideas: Writing”http://www.gtnoise.net/classes/cs7001/fall_2008/syllabus.html#Schedule

Papers are not Novels !Tips for simplicity• One idea/topic per logical unit• Simple organization• Short sentences, simple structure

From: N. Feamster, A. Gray, “Communicating Ideas: Writing”http://www.gtnoise.net/classes/cs7001/fall_2008/syllabus.html#Schedule

Economize• I am sorry I have had to write you such a long letter,

but I did not have time to write you a short one. (Blaise Pascal)

• The length of a paper should be correlated with its content.

From: N. Feamster, A. Gray, “Communicating Ideas: Writng”http://www.gtnoise.net/classes/cs7001/fall_2008/syllabus.html#Schedule

The structure of your paper

Your narrative flow• Here is a problem• It’s an interesting problem• It’s an unsolved problem• Here is my idea• My idea works (details, data)• Here’s how my idea compares to other people’s

approaches

I wish I knew how

To solve that

I see how that works ingenious !


Structure (conference paper)• Title (1000 readers)• Abstract (4 sentences, 100 readers)• Introduction (1 page, 100 readers)• The problem (1 page, 10 readers)• My idea (2 pages, 10 readers)• The details (5 pages, 3 readers)• Related work (1-2 pages, 10 readers)• Conclusions and further work (0.5 pages, 100 readers)


The abstract• I usually write the abstract last• Used by program committee members to decide which papers to read• Four sentences [Kent Beck]

1. State the problem2. Say why it’s an interesting problem3. Say what your solution achieves4. Say what follows from your solution


Example1. Many papers are badly written and hard to

understand2. This is a pity, because their good ideas may go

unappreciated3. Following simple guidelines can dramatically

improve the quality of your papers4. Your work will be used more, and the feedback you

get from others will in turn improve your researchFrom: S. Peyton-Jones, “How to write a good research paper”http://research.microsoft.com/en us/um/people/simonpj/Papers/giving a talk/giving a talk.htm‐ ‐ ‐ ‐ ‐

Structure• Title • Abstract (4 sentences)• Introduction (1 page)• The problem (1 page)• My idea (2 pages)• The details (5 pages)• Related work (1-2 pages)• Conclusions and further work (0.5 pages)


Introduction• Summarizes the whole story• The most important part of the paper• If people don’t understand your problem, approach, importance by the end of

the intro, you’re out of luck

• Two schools of thought• Write it first: make certain the story is clear• Write it last: story becomes clear at the end

• My advice: do both• Working introduction• Final introduction


Why to start the Intro Early ?• It’s important to be able to concisely summarize your key

contributions• In as little as a single paragraph

• If you cannot do this, it’s quite possible that your thinking is not clear• Working on the story can improve your thinking

• Muddled writing reflects muddled thinking


The importance of First Impressions• Many readers make up their minds within the first few paragraphs

• The first few paragraphs should state the paper’s purpose with context• Beware This concerns

• The beginning should be intelligible to any reader


Introduction structure (Credit to J. Kurose)•Paragraph 1: Context•Paragraph 2: Problem area•Paragraph 3: This paper•Paragraphs 4-5: Challenges / Solutions•Paragraph 6: Summary of results•Paragraph 7: Outline

The introduction (1 page)1. Describe the problem2. State your contributions

…and that is all

ONE PAGE !


Describe the problem

State your contribution•Write the list of contributions first •The list of contributions drives the entire paper: the paper substantiates the claims you have made•Reader thinks Gosh if they can really deliver this, that’s exciting, I should read this.

State your contributions

Contributions should be irrefutable

Avoid rest of this paper is…

•Not: The rest of this paper is structured as follows. Section 2 introduces the problem. Section 3…Finally, Section 8 concludes.• Instead, use forward references from the narrative in the introduction. The introduction should survey the whole paper, and therefore forward reference every part.

Structure• Title • Abstract (4 sentences)• Introduction (1 page)

•Related work • The problem (1 page)• My idea (2 pages)• The details (5 pages)• Related work (1-2 pages)• Conclusions and further work (0.5 pages)


Avoid early related work section

Credit Simon Peyton Jones – Microsoft Research

Early related work section•Problem 1: the reader knows nothing

about the problem yet; so your description of various technical tradeoffs is absolutely incomprehensible.•Problem 2: describing alternatives

approaches gets between the reader and your idea.


(Un)Related work section• Section 2, or Penultimate section ?• Placing early pushes the meat of the paper later, but can

prevent the reader from discounting your technique•Handwavy rule• Generally better to put towards the end, but…• If the topic of the paper appears similar to others have an

unrelated work section after the intro




Presenting the idea

• Sounds impressive…but• Sends reader to sleep• In a paper, you MUST provide the details but FIRST

convey the idea


Presenting the idea•Explain it as if you were speaking to someone

using the blackboard.•Conveying the intuition is primary, not secondary.•Once your reader has the intuition, it can follow

the details.•Even if it skips the details, it still takes away

something valuable.


Putting the idea first •Do not recapitulate your personal journey of discovery. This route may be soaked with your blood, but that is not interesting to the reader.

• Instead, choose the most direct route to the idea.


The payload of your paperIntroduce the problem,

and your idea, using EXAMPLES

and only then present thegeneral case.


The details: evidence• Your introduction makes claims• The body of the paper provides evidence to support

each claim.•Check each claim in the introduction, identify the

evidence, and forward-reference it from the claim..• Evidence can be: analysis and comparison, theorems,

measurements, case studies


States the result carefully•Clearly state assumptions• Experiment/simulation description• Enough info to nearly recreate experiment

• Simulation / measurements• Statistical properties of your results• Confidence intervals

•Are results presented representative ?

Credit J. Kurose Top 10 tips for writing a paper

Don’t overstate / understate your results• Overstatement mistakes• We show that X is prevalent in the Internet• We show that X is better than Y

when only actually shown for one/small/limited cases

• Understatement mistake: fail to consider broader implications of your work• If your result is small, interest will be small• Rock the world

Credit J. Kurose Top 10 tips for writing a paper

Evaluation Section• Context: clearly state assumptions• In what context do your results hold?• How general are they ?

• Recipe: Clearly describe the setup• Machines, data, scripts, topologies, etc.• You must make this clear

• Rule of thumb: The reader should be able to recreate the experiment and results from the description in the paper


Evaluation section• Many people will skim• Corollary: Make it skimmable !

• Evaluation signposts• Table summarizing key results (and where to find them in the paper)• Declarative subsection heading• Readable graphs

• Captions that summarize the key finding• Each graph should have main point• Big fonts !




Related workTo make my work look good, I have to make other people’s work look bad.


Fallacy

The truth: Credit is not like money

• Warmly acknowledge people who have helped you• Be generous to the competition. In his inspiring paper [Foo98],

Foogle..shows…We develop his foundation in the following ways…• Acknowledge weaknesses in your approach


Credit is not like money

• If you imply that an idea is yours, and the referee knows it is not, then either• You don’t know that it is an old idea (bad)• You do know but are pretending it’s yours (very bad)



Conclusion section• Keep it crisp• Remember how reviewers skim and readers skim papers (intro,

abstract, conclusion)• Two elements• Very concise summary (one paragraph)

• Readers should have now context• Elevation (one paragraph to one page depending on paper)

• What are the takeaways?• General lessons or applications ?• Broader implications ?


Style and compositionElements of Style

Style and composition•Often referred to as flow•How sentences flow together to form paragraph•How paragraphs flow together to form sentences•How sections flow together to form a paper

•The most important aspect of writing


Organizing paper flow•Plan first, write later•Write top-down•Step 1: Outline sections•Step 2: Within a section, outline paragraphs• For each paragraph, write topic sentences


Writing a section, Top-down style• Make a bulletted list of points to include• Cluster the points into related topics/points• For each cluster, write a topic sentence• Organize your topic sentences • Make subsections if necessary• Fill in paragraph details (top down !)• Add paragraph headings.


Style Points•Motivation• Everything that a paper includes should be accompanied

with an explanation for why it is necessary / interesting useful.

•Balance• Topics of equal relevance should be addressed with equal

weight/length.From: N. Feamster, A. Gray, “Communicating Ideas: Writng”http://www.gtnoise.net/classes/cs7001/fall_2008/syllabus.html#Schedule

Writing a Paragraph•A paragraph is group of logically related sentences•Start with a sentence that describes the logical relationship (thread)•Keep continuity•Keep a common verb tense•Don’t string together loosely related sentences


Signposting• The reader must have a clear view of how the

paper/story wil proceed•Allow for top-down reading• Signposts: How is the paper (or section) organized ?• Outline in the introduction• Preamble to each section• Declarative subsection titles• Paragraph headings


Landscaping• Your goal: efficient information transfer• Forcing the reader to block or context switch by taking a break,

falling asleep, or, worse --- skimming over important points --- defeat the purpose

• Avoid consecutive pages of dense text• Tables • Figures• Whitespace• Signposts


Visual structure•Give strong visual structure to your paper using•Sections and sub-sections•Bullets• Italics

•Find out how to draw pictures, and use them


The process of writingSome elements fron the craft of research

The writing process•Start early. Very early.•Hastily-written papers get rejected•Papers are like wine: they need time to mature

•Collaborate•Use SVN/dropbox to support collaboration


The writing process•Prepare first, then write• Take time to crystallize your thoughts• Clear thoughts lead to clear writing•Much more difficult to revise muddled text…

• Shut off all distractions•Writing requires focused, clear thinking• Context switches and interrupts are particularly damaging


The Writing Process: Growth•Practice whenever possible• Email, notes, blogs, publications

•Find a style that you like and try to emulate it

•Experiment


Editing: Reading•Read aloud•Helps identify awkward, and repetitive passages

•Read in reverse•Help bypass your brain’s tendency to fill gaps, mistakes, etc.


Editing: Cutting•Watch out for fancy words and cut them•Toss out redundancy •Each sentence, word, phrase, section, graph, etc. must be justified !•Sleep on it…


Getting help

• Experts are good• Non experts are also very good• Each reader can only read your paper for the time once ! So use them

carefully• Explain carefully what you want (I got lost here is much more

important than Jarva is mis-spelt)


Get your paper read by as many friendly guinea pigs as possible

Getting expert help• A good plan: when you think you are done, send the draft to the

competition saying Could you help me ensure that I describe your work fairly ?• Often they will respond with helpful critique (they are interested in

the area)• They are likely to be your referees anyway, so getting their comments

or criticism up front is Jolly Good


Listening to your reviewers

This is really, really, really hard

But it’sreally, really, really, really, really, really, really, really,

important

Treat every review like gold dustBe graceful for criticism as well as praise


Listening to your reviewers•Read every criticism as a positive suggestion for

something you could explain more clearly.•DO NOT respond YOU STUPID PERSON, I MEANT X. Fix

the paper so that X is apparent even to the stupidest reader.• Thank them warmly. They have given up their time for

you.


Elements of style

Usage: Composing individual sentences•Errors (spelling, grammar, etc.) or deviations in style can cause the reader to context switch• This creates a barrier for information flow• Your goal is to reduce or eliminate these

•Writing in a style the reader expects•Check previous conference proceedings


Simplify your usage• Avoid metaphors and other figure of speech • Never use a long/complex word where a short/simple one will do • If is possible to cut a word out, always cut it out• Never use the passive where you can use the active• Never use a foreign phrase, a scientific word, or a jargon word if you

can think of an everyday english equivalent.

Orwell, Politics and the English Language

Use the active voice• The passive voice is respectable but it DEADENS your paper. Avoid it at

all costs.


Use simple, direct language


Omit Needless Words• In order to To• The question of optimizing Optimizing• The question as to whether Whether• For optimization purpose To optimize• This is a module that This module• In a shorter running time More quickly• This is a subject that This subject• His story is a strange one His story is strange.


Avoid Padding• Adding together Adding• Totally eliminated Eliminated• Separated out Separated• Give a description of Describe• The fact that …• It is important to note …


Writing Bugs• Citations as nouns• In [10], the authors showed…• Problem: forces the reader to context switch• Better: Gray et al. previously showed…[10].

• The naked this• Problem: this is a modifier• Next , we sample every tenth data point, this reduces processing time. • Better: Sampling every tenth data point reduces processing time.


Writing bugs• Passive voice• A request for content is sent to the server.• Who/what performs the action ?• Very important when specifying protocols, experimental setups, etc.


Which vs. That•Which clauses can be removed from the sentence without

changing the meaning• BGP, which is the Internet’s routing protocol,…• They are always offsets by commas• Better: omit which is entirely

• That clauses make the modified noun more specific and cannot be removed without changing meaning• Can you send me the code that performs PCA on BGP routing

updates ?• Not offset by commas


Other Misused words • Less vs. Fewer• Impact vs. Influence• May vs. Can


The articles (a, the, etc.)• A/an• Non specific modifier• I need to work on a paper (implication: any paper)

• The• Specific modifier• I need to write the paper. (implication: specific paper)• I need to read the papers (specific papers)

• Collective nouns often do not take any article• Papers can brovide useful background information• The papers at SIGCOMM are very interesting this year.


PublishingPublish or Perish !Conferences vs Journal

acceptance rate

Basic stuff• Submit by the deadline•Keep to the length restrictions• Do not narrow the margins• Do not use 6 pt font• On occasion, supply supporting evidence (e.g.

experimental data, or a written-out proof) in an appendix•Always use a spell checker


Publishing•Publish where you will get mindshare, impact•There is life beyond sigcomm, infocom•Quality over quantity•Don’t be driven by conference deadlines

Credit J. Kurose – 10 pieces of advice I wish my PhD advisor had given me

Publishers / Sponsors• ACM • IEEE• ELSEVIER

Giving Technical TalksGuidelines for successful presentations

A technical talk is great for conveying• Context• What has been done before ?• Why is the research important ?• What problems are still open ?

• An overview and framework• What does this research contribute ?• What methods were used to solve problems ?

• Enthusiasm and excitement

talk by Scott Drysdale, Dartmouth College

A technical talk is a poor way to convey• Nitty-gritty details• Lots of factual information• Theorems & proofs

Leave those to technical papers…

The parts of a technical talk• Introduction•Body•Technicalities•Conclusion•Question

Introduction•Define the problem•Motivate the problem and hook audience• Introduce needed terminology•Discuss earlier work• Explain the key contributions•Provide a roadmap for the rest of the talk

Body•Describe the main hypothesis, experiments, analysis•List the major results•Explain the significance of the results

Technicalities•Present a key lemma or technical idea…•Descend into detail briefly, slowly and carefully•perhaps convince people that what you have done is not trivial …

Conclusions•Summarize the key points – regain lost audience•Make observations that would have been confusing in the beginning•Give weaknesses, open problems• Indicate that the talk is over

Questions•Genuine request for information –> answer the best

you can•Questioner wants to look smart and knowledgeable

--> be polite and complimentary• Malicious questions

– be polite and measured in response– Move questions “offline” if needed– Say “I don’t know” (with assurance) if needed

Who is your audience ?• General public• CS folks (e.g. colloquium)• Experts in the area of your research (conference/workshop)

What does the audience care about – not what you think is cool!

Addressing your audience ?• General public

Introduction Body Technicalities

• CS folks, e.g, a colloquium

Introduction Body Technicalities• CS folks in your area, e.g., seminar class

Introduction Body Technicalities• Experts, e.g, focused workshop

Introduction Body Technicalities

The Ten commandments•Repeat key concepts•Remind, don’t assume•Give examples, not

proofs•Use logical order• Size talk to the time

• Maintain eye contact•Maintain ear contact• Simple visuals• Employ pictures•Do not get anxious

Use Logical Order• You are telling a story. What order will make the best

sense to an audience? (who is not that familiar with the subject)•Avoid forward references•Motivate each step, tie it back to the storyThe order in which you did the research is irrelevant

Size talk to time•Leave time for audience interaction• Plan to end at least 5 minutes early•Plan what to leave out if you get behind•You can’t include everything. Keep the most important stuff – the rest can be read from the paper

Maintain Eye Contact• It is a way to communicate• It is how you tell if the audience is following, lost, bored, etc.•Talk to your audience - don’t read your talk, talk to your feet, talk to the screen

Maintain Ear Contact•Speak slowly•Speak clearly•Project your voice •Pause after delivering a packet of information or asking a question

Simple Visuals• Make sure that the text is large enough to read

• The purpose of the slide is to give the audience a structure, as something to jogtheir memory as to the point you are tryingto make, or as a concrete expression of aformula, etc. It should not be a verbatimtranscript of what you are saying. If you aresaying exactly what is on the slide then youare doing something wrong.

Pictures Pictures• One picture (graph, diagram) can save 5minutes of explanation• Good picture are worth the (considerable)time to make them

Do not get AnxiousPrepare, Practice, Get Experience• Quietly organize your thoughts before talk• Try out the projection equipmentbeforehand– Think about what you will do if the equipmentdoes not work• Pause and take a deep breath if panic strikes

Seven Deadly Sins• Trying to include too much• Going over your time• Being boring• Speaking unintelligibly• Arrogance• Losing your audience• Including material you don’t really understand

Trying to include too much• Symptom - Time almost up and you are halfway through your talk• Symptom – Too many slides.Tearing through slides faster than the audience can read themDisaster - you left the most important stuff tothe end, and are out of time

Being boringPresentation is a public performance• You have to be energetic, animated, enthusiastic. (You can overdo this.)• If you don’t seem to be interested, why should your audience be interested?• If you find the material boring, so will your audience. Pick more interesting material

Speaking unintelligibly• Don’t mumble• Don’t talk in a monotone• Don’t use jargon or undefined terms• Don’t swallow your words or endin...• Avoid mannerisms that distract your audience from what you are saying• Speak slowly if it helps

Arrogance• The fact that youknow more about yourtalk than the audiencedoes not make yousuperior to them.• Do not put down orbelittle questioners

Losing your audienceOver their heads (slow down, back up)• Beneath their interest (get to the good stuff)• Too big a step (go back and fill in details)• Not enough relevant examples (Explain more)• Loss detector: eye contact

Including material you don’t understand•No excuse for it• It is your talk even if you reference othermaterial

Conclusions• Everybody can learn to give good talks• Plan and organize your talk• Think from the audience’s point of view• Keep the focus on key points and ideas• Practice! Get feedback. Get better.

DISCUSSION 2: BrainstormingRead this paper before next class:Van Jacobson, Diana K. Smetters, James D. Thornton, Michael F. Plass, Nicholas H. Briggs, and Rebecca L. Braynard. 2009. Networking named content. In Proceedings of the 5th international conference on Emerging networking experiments and technologies (CoNEXT '09). ACM, New York, NY, USA, 1-12.

• What is the problem ?• What are the contributions ?• What are the conclusions ?• What is the support for the

conclusions ?• What are the open problems ?

The research process

Significance of a research problem• If you can find a problem that you alone want the solution, you have

achieved something substantial.

• If you can pose a problem that the others recognize not just as your problem, but as their problem as well, a problem whose solution will change their thinking in ways they think significant, then it is excellent.

Credit Booth et al. - The craft of research

First steps to take•Must settle on a topic specific enough to let you master a

reasonable amount of information.• Not “the history of scientific writing,”• but “essays in the proceedings of the royal society (1800-1900) as

a precursor to the scientific article”• Out of the topic, develop questions that will guide your

research and point you toward a problem that you intend to solve.

Credit Booth et al. – The craft of research

Research Interest and TopicInterest• A general area of inquiry that we like to explore• Content-centric networking

Topic• An interest specific enough to support research that one might plausibly

report on a book or article that help others to advance their thinking and understanding.

• Routing in large scale content-centric networks

Narrowing down a broad topic• A topic is probably too broad if you can state it in fewer than four or five words. e.g.,

Routing in large scale content-centric networks

The design of routing protocols in decentralized and large-scale content-centric networks

The contribution of the military to the development of DC-3 in the early years of commercial aviation

The history of commercial aviation

Questions• Asking the right questions is key to successful research.• Question your topic from as many angles as you can think of

– questions give your research purpose and direction.• Listening to other people’s questions might help you

formulate your own.• There are some questions that have no answers.


Picking questions• Emphasise on questions that begin with How and

Why.•Concentrate on questions that need more than one-

or two word answer.•Decide which questions stop you for a moment,

challenge you, spark some special interest.

The Ultimate Question is: So what ?

From a question to its significance – three useful steps:

Name your topic: I am working on/studying …

Suggest a question: I am working on/studying ... because I want to find out how/why ...

Motivate the question/find a rationale: I am working on/studying … because I want to find out how/why … in order to understand how/why …


From a question to its significance1. I am studying repair process for cooling systems,2. Because, I want to find out how experts repairers analyse failures3. In order to understand how to design a computerised system that could diagnose and prevent failures

Informal practice vs Writing papers.

What makes a question researchable ?• Not too big or too small• Question focuses on something that has been discussed• It’s interesting and it matters• It’s in some way answerable• There is a method to answering the question• It raises more questions

From, Ballenger, The Curious Researcher, 4th Edition

Question OR Problem• Answer to questions must solve a problem that others in their

community think needs a solution.

P=NP ?

Nature of problems (1/2)The research cycle

Practical problems lead to research problems, and the answers are intended to help solve the practical problems.

Practical problems: • Originate in the world are based on

some cost to society.• Are solved by taking action in the real

world

Research problems: • Originate in your mind • Are based on incomplete knowledge or flawed understanding• Are solved by gathering useful information

Nature of problems (2/2)Applied Research• The rationale for the research

defines what you wish to DO • The consequences of the

research are tangible • The research is applied because

knowledge gained will be applied to solve an immediate practical problem

Pure Research• The rationale for the research

defines what you wish to KNOW • The consequences of the

research are conceptual • The research is pure because

knowledge is pursued for its own sake

Problem• Your questions should help you solve a research

problem•A problem is something you do not yet know or

understand•A problem might be the origin of your research …•… but you may not be able to formulate your problem

fully at the outset


Evidence•All answers must be based on evidence•What is your evidence?•Always explain: what is self-evident to you might not

be self-evident to others •Always avoid generalizations


Making good argumentComponents• Claims - what the reader is asked to accept (believe) • Evidence - why the evidence is (necessary &) sufficient • Warrants • Qualification - limits on the claims • Can qualify in the hypotheses• can qualify after evidence doesn’t fully support the hypotheses

Researchers tools

Google Scholar

Other sources of documentation• Arxiv• Citeseer• Social networks• Academia.edu• Mendeley

Bibliography Styles• APA• Chicago

Reference manager• BibTEX• Bibdesk• Mendeley / Zotero

LaTex (1/2)• LaTeX makes it very simple to

handle equations, figures, bibliographies, indexes, etc. • With LateX you focus on the

content of the document and let the program handle how the output is formatted.

LaTex (2/2)

Tikz

Gnuplot

From Gnuplot documentation

From my Comnet 2012 paper

Simulation• Network Simulator (ns-2/ns-3)• Omnet• Opnet• Peersim• Build your own…

DEMARCHE1. Caractériser le modèle

• Modèle de trafic / service

2. Définir des criteres de performance3. Définir les objets dont on parle

• Bit / Paquets / Bursts / Messages

4. Validité de la simulation5. Validité du résultat 6. Validité des nombres pseudo-aleatoires

• Reproductibilite

Testbed (1/2)• Planetlab / Onelab

Testbed (2/2)

Theoretical background required• Probability (Modeling)• Queuing theory• Markov chains

• Statistics• Data analysis• Reporting reliably

• Algorithms• Complexity theory

• Operations research• Optimization problems• Algorithms / Heuristics

Various Services• Overleaf• Researchgate

Data and Statistics

Confidence interval

cpntools.org

𝜀=4.5𝜎

Pr [¿∨𝑍𝑁−𝑍∨¿𝜀 ]>1−𝜃 ¿

Intervalle de confianceEx. 95%Methode de LE GALL:

Percentile• A typical SLA required of services

that use Dynamo is that 99.9% of the read and write requests execute within 300ms

Credit Amazon Dynamo Paper

Distribution fitting

Credit Matlab Online Tutorial

From my 2013 ComNet paper

Random numbers generation

Uniformly distributed numbers

Any Distribution

Datasets• Google: Youtube Dataset (Youtube8m)• Caida: Internet traffic data• Simpleweb.org: Traffic traces (Dropbox, Anycast, etc.)• Kaggle.com

https://research.google.com/youtube8m/

http://www.caida.org/home/

https://www.kaggle.com/

Homework 4: Tap the Youtube dataset.• Review the available material• Identify information that can be extracted from the dataset ?• Write a two-page paper (Two columns) presenting and discussing key

knowledge from the dataset.

http://netsg.cs.sfu.ca/youtubedata/

http://netsg.cs.sfu.ca/youtubedata/

Discussion 3: Evaluation methodology• We will discuss and review in this workgroup your project of Réseaux

et Protocoles w.r.t evaluation methodology.

Homework 5: Research proposalQuestions that must be answered: 1. What is the problem? (Literature!) 2. Possibilities:

• new problem -> find a solution • known problem with existing solutions -> find a better solution

3. What has been done (by others) already to solve this problem? (Literature!)4. What is missing? What is not good in other approaches/solutions?

(Literature!)5. What are you planning to do?6. What will be the result(s) in the end?

15 mins PPT presentation2 pages max LaTex proposal

References• Cours Méthodes de travail pour la recherche en réseaux, Master 2ème

année, spécialité “Réseaux”, Université Pierre et Marie Curie 2013• Booth, Colomb, Williams, The Craft of Research, University of Chicago

Press.

Documents

Techniques d’etudes et de recherche