RepoVizz: Online Multimodal Recordings' Repository & Visualizer

Main features / specifications

RepoVizz is a data repository and visualization tool for structured storage and user-friendly browsing of music performance multi-modal recordings. The primary purpose of RepoVizz is to offer means for the scientific community to gain on-line access to a music performance multi-modal database shared among researchers.

RepoVizz is designed to hold synchronized streams of heterogeneous data (audio, video, motion capture, physiological measures, extracted descriptors, etc.), annotations, and musical scores. Data streams are stored as single-channel PCM files. The motion-sequence format MJPEG is chosen for video. Annotations are stored as text files. MusicXML is used for musical scores. Data is structured by customizable XML skeleton files enabling meaningful retrieval. Skeleton files are first created during data gathering, and provide means to (re-)organize acquired data in any desired hierarchical structure, as they only hold pointers to stored data files. Once a dataset is created and uploaded to the server, each skeleton file defines a view. Multi-track data visualization is done remotely via a powerful HTML5-based environment that enables web-driven editing (add annotations, extract descriptors) and downloading of datasets.

DEMO (& GUI)

Use cases and applications We can foresee some usages and applications:
  • Storage, organization and Sharing of Multi-modal data
  • Collaborative Multimodal annotation
  • Music research
  • Music education
  • Health research
  • Sports training
System requirements HTML5 and WebSockets powered web browser
Acknowledgement

EC FET Open project SIEMPRE

EC ICT Project PHENICX

TECNIO network (ACC1Ó - Generalitat de Catalunya)

Team involved Oscar Mayor, Quim Llimona, Esteban Maestre, Marco Marchini, Panos Papiotis and Jordi Llop
Related publications

Mayor, O., Llop J., & Maestre E. (2011).  RepoVizz: A multimodal on-line database and browsing tool for music performance research. 12th International Society for Music Information Retrieval Conference (ISMIR 2011).

Mayor, O., Llimona, Q., Marchini, M., Papiotis, P. & Maestre, E. (submitted). repoVizz: a framework for remote storage, browsing, annotation, and exchange of multi-modal data. 21st ACM international conference on Multimedia, Barcelona.

Commercialization options The software (both server and client) is available under proprietary license
More details SIEMPRE repository
Contact Oscar Mayor < oscar dot mayor at upf dot edu >

 

intranet