Introduction to Music Informatics

Donald ByrdRev. 4 Nov. 2007

Copyright © 2006-07, Donald Byrd

30 Aug. 2006 2

Welcome

• This is a very exciting time for music!• Music informatics is hard to teach

– Music and technology are both changing quickly

– Few students have much background in both– Solution: compromise

• Spend some time bringing each “up to speed”• Either limit tech. demands in both areas, or…• Work in teams• How much background does everyone have?

29 Aug. 2007 3

Class Background: Results 1

• Music: everyone has good background; several have degrees

• Programming: all over the map– Few know the R language at all

• Experience w/ music recommenders, iTunes, etc.: variable, but most don’t have much

rev. 9 Sep. 07 4

Class Background: Results 2

• Have 10 students, 1 auditor• Music Theory

– Q1: 10; Q2: 10; Q3: 8 + 2 partial• Best of any class I’ve ever had

• Audio & Computer Technology– Q4: 6; Q5: 3 + 7 partial; Q6: 6 + 4 partial– Q7: 5 + 4 partial; Q8: 1 + 4 partial; Q9: 3

+ 4 partial• Q8 is hard, esp. to say briefly: cf. my

Vocabulary!

rev. 9 Sep. 07 5

Class Plans & Procedures 1

• Communication– Web site (has many helpful resources)

• …including “Miscellaneous Class Procedures”

– E-mail– Announcements in class– OnCourse? not for now

• Goals & competencies• Assignments & grading: cf. class syllabus

– Project & presentations most important– Preparation & participation are important– Participation => avoid “death by PowerPoint”– Importance/frequency of quizzes is TBD

rev. 27 Aug. 07 6

Class Plans & Procedures 2

• What about programming?– More “reading knowledge” than writing– …but will write some, in R

• Midterm presentation• Final project: presentation & paper

– List of possibilities to appear “soon”• Some with programming, some without

– Proposal due in a few weeks– Presentations last weeks of class– Paper due last week of classes

• Will have specific grading rubrics

26 Aug. 2007 7

What Is Music Informatics?

• Some definitions1. Like music information technology, but

more research-oriented2. Music information retrieval & related areas

• ISMIR = “International Conference (Symposium) on Music Information Retrieval”, but… Change name?

3. Discipline #1 plus a curriculum (SoI def.)

• Tim Bell’s overview (2006)– Moving music from one “form” or “place”

to another (my words)

30 Aug. 2006 8

Humanmemory

Visualdisplay

Digitalimage

Digitalsemantic

Audio

30 Aug. 2006 9

Humanmemory

Visualdisplay

Digitalimage

Digitalsemantic

Audio

Weak links

rev. 30 Aug. 07 10

We’ll Consider All Kinds of Music• Existing music covers a huge range• I follow Schickele/Ellington philosophy

– Peter Schickele: All musics are created equal– Duke Ellington: If it sounds good, it is good

• Great, but makes many problems much harder for computer– If pop only, can rely on existence of lyrics– If classical only, can rely on dynamic range– If pop or folk, can assume texture is melody

& accompaniment– If many genres, can assume 12 notes/octave

27 Jan. 11

We’ll Consider All Representations of Music (Preview)

Audio (e.g., CD, MP3): like speech

Time-stamped Events (e.g., MIDI file): like unformatted text

Music Notation: like text with complex formatting

Digital Audio

Time-stamped Events

Music Notation

30 Aug. 2006 12

Music & Research 1• Different approaches to research

– Highly controlled environments• More objective, easier to quantify results

– Less controlled• Better for exploration

– Sim. to “naturalists 1st, scientists later” idea

• Music isn’t a science! But…– What music “scholars” do similar to what

“scientists” (esp. social scientists) do– In a way, physical scientists have a much

easier job!

30 Aug. 2006 13

Music & Research 2• Different approaches to research

– What “scholars” do is similar to what “scientists” (esp. social scientists) do

– Good reason: music is subtle & subjective– …but it’s not magic!

• David Huron: Explanatory Goals of Music Analysis– Learned as a music student: “methodology is fetish;

rigour is a form of self-deception”– …as a music scholar: methodology is “simply a way

of internalizing the lessons learned from past scholarly mistakes”

• The “Scholarly Method”

31 Aug. 2007 14

Music & Research 3• Example: Gladwell’s article “The Formula”, on

predicting hit songs or movies by computer– Not a scholarly article, but is it plausible?– Physical aspects of audio are easy for computers– …but what we care about is almost always

perceptual– What are concepts like melody & harmony?

• Core Competency– “Understand the difference between physical

(objective) and perceptual (subjective) parameters of musical sound, and why computers can deal much more easily with the former, while people can deal much more easily with the latter”

rev. 30 Aug. 07 15

Classification: Logician General’s Warning

• Classification is dangerous to your understanding– Almost everything in the real world is messy– Absolute correlations between characteristics are rare– Example: some mammals lay eggs; some are “naked”– Example: is the piano a keyboard, a string, or a percussion

instrument?• People say “an X has characteristics A, B, C…”• Nearly always mean “an X has A, & usually B, C…”• Leads to:

– People who know better claiming absolute correlations– Arguments among experts over which characteristic is

most fundamental– Don changing his mind

• But lack of classification is also dangerous to your understanding!

• Cf. version of this on my Teaching page

rev. 12 Sep. 07 16

Examples of Music Informatics Research/Technology in Action

• Listen Game– Nice example of “games with a purpose”– …but doesn’t seem to be usable anymore!

• sCrAmBlEd?HaCkZ!– Audio mosaicing

• http://www.popmodernism.org/scrambledhackz/

• "Together" Listening Experiments– PhD dissertation research by Matt Wright– on perception of musical rhythm

• “Perceptual Attack Time”• http://ccrma.stanford.edu/~matt/together/

30 Aug. 2007 17

Our Own Music

• “Music We're Interested In, and What's Special About It?”– Everyone contributes something– Will play & briefly discuss (from standpoint

of music informatics) in class– Audio files will be on Web site– Will use for something

• Maybe test lossy compression, or play with audio-recognition programs?

• Definitely, test audio segmentation program!– Thanks to Nina Fales for the idea

31 Aug. 2007 18

Materials for Studying Audio 1

• Waveforms & sounds– Fourier Series Applet (www.falstad.com/fourier/)– Simple “artificial” waveforms

• Sine, square, triangle, etc.• Standard on old analog synths (Moog, etc.)• Sine wave (tuning fork) is the simplest• Sine function from trigonometry

– Fourier’s theorem• Any periodic function (repeating waveform) =

sum of harmonically related sine waves• Add up harmonically related sine ways to make

(approximation to) square wave, etc.

31 Aug. 2007 19

Materials for Studying Audio 2

• What are interesting sounds really like?– Static waveforms are simple, boring

• Not just sine, square, etc.: cf. the “random waveform generator” (R demo)

– Acoustic instrument sounds are never static– …a big reason they’re interesting

• Musical instrument samples• Audacity audio editor

– For Windows, Mac OS 9 and X, Linux– Download from

http://audacity.sourceforge.net/

30 Aug. 2006 20

Programming in R: No Problem!

• R is very interactive: can use as powerful calculator• Assignments will be simple• Much help available: from Don & other students• Why R?

– NOT because it's great for statistics!– easy to do simple things with it, including graphs and

handling audio files• probably not good for complex programs

– free, and available for all popular operating systems– very interactive => easy to experiment– has good documentation– Prof. Raphael is using it, and he thinks it's good for music

informatics– Prof. Raphael is using it, and standardizing is a good thing

9 Sep. 07 21

Rudiments of R

• Originally for statistics; good for far more• How to get R

– Web site: http://cran.us.r-project.org/– Versions for Linux, Mac OS X, Windows– Already on STC computers & in M373

• Tutorial (kind of math-heavy):• http://xavier.informatics.indiana.edu/~craphael/teach/

symbolic_music/

• Can use R interactively as a powerful graphing, musicing, etc. calculator

• …but it’s not perfect: sometimes very cryptic

rev. 8 Sep. 2006 22

Representations of Music & Audio 1

• Three basic forms (representations) of music– Audio: most important for most people (general

public)– MIDI files: often best/essential for some musicians,

especially for pop, rock, film/TV– Notation: often best/ essential for musicians (even

amateurs) & music scholars– Essential difference: how much explicit structure

• Music holdings of Library of Congress: over 10M items– Includes over 6M pieces of sheet music and 100K’s of

scores of operas, symphonies, etc.: all notation!

• Differences are profound

1 Sep. 2006 23

Representations of Music & Audio 2

Audio (e.g., CD, MP3): like speech

Time-stamped Events (e.g., MIDI file): like unformatted text

Music Notation: like text with complex formatting

Digital Audio

Time-stamped Events

Music Notation

rev. 9 Sep. 07 24

Rudiments of Musical Acoustics

• Need some musical acoustics for almost anything in digital audio

• Need a bit now (use with R), more later• Acoustics: part of physics

– Concepts like frequency & amplitude

• Psychoacoustics: part of psychology– Concepts like pitch & loudness (perceptual)

rev. 9 Sep. 07 25

Rudiments of Digital Audio

• Sampling rate => maximum frequency– Human hearing goes up to ca. 15-20K Hz– Need 2 samples per cycle– CDs: 44,100

• Sample width (in bits per sample) => Signal-to-Quantization Noise ratio– About 6 dB per bit– CDs: 16 bits = ca. 96 dB SQNR

rev. 4 Nov. 07 26

Research in Music Informatics 1• Scientific method vs. scholarly “method”

– Wide variation in use from field to field– Music theory doesn’t use scientific method (EJI says)– …but lots of music informatics does

• Debugging as an example of scientific method– Hypothesis: program X has no bugs– Methodology: look for bugs– …but no amount of testing can prove the absence of

bugs, just their presence– Cf. Einstein: “No experiments can prove me right;

one experiment can prove me wrong.”– …& Dijkstra: “Testing can only show the presence of

bugs, never their absence.”– A theorem (in math) can be proven; a theory (like a

program) can’t!

rev. 4 Nov. 07 27

Research in Music Informatics 2

• Evaluating reliability of info sources– Especially difficult on the Web: cf. www.dhmo.org

• It’s easy to jump to wrong conclusion: why?– Backus on why musicians’ explanations in acoustics

are almost always wrong– Cf. Logician General’s Classification Warning– Almost everything in the world is messy

• Def. of “trombone” may be more clearcut than of “piano”

• …but it’s still not well-defined!

rev. 19 Sep. 07 28

Research in Music Informatics 3

• How do you study a question in the most objective possible way?– If it’s a matter of perception, maybe by asking

people what they think or hear!– Example: determining “Perceptual Attack Time” of a

sound

• Cf. D. Huron on what he learned as a music student vs. as a music scholar– Methodology as way to avoid repeating mistakes

• Check out my “Information Sources for Music Informatics Students”