Seminar by Bob Sturm on the evaluation of music content analysis systems
Seminar by Bob Sturm on the evaluation of music content analysis systems
01.02.2016
4 Feb 2016
Bob Sturm, from Queen Mary - University of London, will give a talk on “The scientific evaluation of music content analysis systems: Toward valid empirical foundations for future real-world impact" on Thursday February 4th 2016 at 4:30pm in room 52.123 of the Communication Campus of the UPF.
Abstract: Music content analysis research aims to meet at least three goals: 1) connect users with music and information about music; 2) help users make music and information about music; and 3) help researchers develop content analysis technologies. Standard empirical practices used in this discipline, however, have serious problems (as noted in the MIReS 2013 Roadmap, and [2-5]). I present three case studies that exemplify these problems, and discuss them within a design of experiments framework. I argue that problems with MIR evaluation cannot be satisfactorily addressed until the practice installs the formal design of experiments [1]. I also propose new ways to think about what we do, which is very preliminary work.
[1] R. A. Bailey, Design of comparative experiments. Cambridge University Press, 2008.
[2] G. Peeters, J. Urbano, and G. J. F. Jones, “Notes from the ISMIR 2012 late-breaking session on evaluation in music information retrieval,” in Proc. ISMIR, 2012.
[3] B. L. Sturm, “Classification accuracy is not enough: On the evaluation of music genre recognition systems,” J. Intell. Info. Systems, vol. 41, no. 3, pp. 371–406, 2013.
[4] B. L. Sturm, “The state of the art ten years after a state of the art: Future research in music information retrieval,” J. New Music Research, vol. 43, no. 2, pp. 147–172, 2014.
[5] J. Urbano, M. Schedl, and X. Serra, “Evaluation in music information retrieval,” J. Intell. Info. Systems, vol. 41, pp. 345–369, Dec. 2013.