Introduction to MIR

GCT731 Fall 2014Topics in Music Technology- Music Information RetrievalIntroduction to MIRCourse Overview

11InstructorName: Juhan Nam ()

Biography1994-1998: BS in EE in Seoul National University 2001-2006: Engineer in Young Chang (Kurzweil) 2006-2012: PhD in Music (also MS in EE), CCRMA, Stanford University2012-2014: Research Engineer in Qualcomm

22OutlinesIntroduction to MIRBackgroundBrief history of music technologyRecent trends and future direction MIRMusic Data and InformationApplicationsCourse overview

3Human, Music and Technology

4History of Music Technology

5Material Processing TechnologyMold metal and wood in a high-quality form Improved or new musical instruments: e.g. piano, saxophone (in 1841)3D printer: resurrection of material processing technology?

5History of Music Technology

6Electro-Mechanical TechnologyMicrophone and speakers: sound as continuous-time signalAmplifier and effects: loudness and timbre controlRecorder: paradigm shift in music creation and distributionPlayer: listening to music anywhereNew musical instruments: electric guitars

6History of Music TechnologyDigital Signal Processing + Computer TechnologyA/D, D/A converters: sound as discrete-time signals Synthesizers: design musical soundsDigital audio workstation (DAW): music recording, editing and production Audio programming: coding sound and musical events MP3 players7

7Recent TrendsBig dataOnline music services: 20M+ songs Youtube: 100h+ video uploaded per minuteEasy sharing of personal music contentIntelligenceInteractive music notation: transcription, score followingAuto-accompaniment: playing with computers ConnectivityMusic is combined with human data: play history, preference, location, etc. Music is distributed via social networks 8

8Future Directions9ChallengesHow can we find music content in a human-friendly way?How can we enhance our musical activity, specifically performance, by musical interaction with computers? How can we benefit from the connection of human data and network with music?

Need of understanding meaning in music! Intelligent Data Processing + Internet Technology Music Information Retrieval (MIR)Area of research that recently emerged in the background International Society of Music Information Retrieval (ISMIR) since 2000 Aims to infer various types of information from music dataMaking computer understand music as human doesProvide intelligent solutions to enhance human musical activities1010Information in Music11Instrument:Composer:Key:Similar songs (by motif) :

Transcription melody, score Mood: Melancholy, Sad, - ELO After all- Radiohead Exit Music

Chopin PianoE-minor

Factual Informationtrack, artist, yearsMusical Informationtimbre, melody, notes, beat, rhythm, chords, structure Semantic Informationgenre, mood, user preference

11Music DataAudio Datawav or mp3 files, audio streaming from microphonesSymbolic DataMIDI files, Music XMLText DataLyrics, review, tags, blog Image DataCD artwork, media, score (as image files)User Data (human data)Play history, rate/preference1212MIR TasksFingerprinting Cover song detectionMusic Transcription: melody, notes, tempo, chordsSegmentation, structure, alignmentSimilarity retrieval, playlists, recommendationClassification: genre, mood, tags, Query by hummingSource separation: vocal removalSymbolic MIR: score retrieval or harmony analysisOptical Music Recognition (OMR)

MIREX: http://www.music-ir.org/mirex/wiki/MIREX_HOME

1313Applications of MIRMusical listeningMusic search and recommendationPerformanceInteractive music performanceEducationInstrument learningEntertainmentSinging evaluation, GameMedia production and music creation Sound sample search in sound libraries14Music SearchQuery by musicSearch a single unique song identified by the queryAudio fingerprint Applied to movies, TV and ads, too

Query by hummingSing with humming and find closest matchesMelody match

15

15Music RecommendationPersonalized RadioGenerate Playlist Based on user data, similarity and context

16

16Music PerformanceScore-FollowingListen to performance and track the notesExamplesJKU: http://www.youtube.com/watch?v=Yf05nzix3_wTonara: http://www.youtube.com/watch?v=HBXJZKTOcpwAutomatic accompanimentScore following + Interactive PerformanceExamplesIRCMAs Antefesco: http://www.youtube.com/watch?v=YkMGtpcAA04Sonations Cadenza: http://www.youtube.com/watch?v=UuZUNTEvfhM

1717Music Education and EntertainmentFocus on performance evaluationLearning musical instrumentExamples Ovelins GuitarBots: http://www.youtube.com/watch?v=-396LJrqYRkMakeMusics smartmusic: http://www.youtube.com/watch?v=b-D00OO7KjYKaraokeSinging evaluation: pitch, beat, enthusiasm, etc ExamplesBMATs SkoreTJ media V-scanner (Perfect Singer VS)

1818Media Production and Music CreationSound Sample searchImagine Researchs MediaMind: search sound effect sample for media production (e.g. film, drama)Izotopes Breaktweaker: search similar timbre of drum sounds Automatic Song writingAlgorithmic compositionAutomatic arrangementMSRs Songsmith: http://research.microsoft.com/en-us/um/redmond/projects/songsmith/

19MIR Research DisciplinesDigital Signal ProcessingAcousticsMusic theoryMachine LearningNatural language processing / Computer visionPsychologyHuman-Computer Interaction 2020About This CourseFocus on inferring information from audio dataSurvey topics in MIRMusic and audio representationsPitch tracking Timbre analysisOnset detection and beat trackingChord recognitionSource separation and polyphonic pitchMusic classification: genre, mood and tagsMusic search and recommendationApplications in music performance 2121Course InformationPrerequisiteBasic digital signal processing: filters, FFT, spectral analysisProgramming: MATLAB Basic music theory: scale, interval, major/minor chordsStrong interest in musicStrong plusMachine Learning: GMM, HMM, SVM, DNN, No required textbookSuggested readings or reference book lists are provided in the course web site.

22Course InformationFormatLecture + Project (Proposal + Presentation + Report)GradingAssignment: 50 % Class participation: 10 % Discussions, attendance and enthusiasmFinal research project: 40 % Presentation (20%) and Report (20%)2323Course InformationCourse website: http://juhannam.github.io/gct731mir2014/

24