Renato Panda

selected publications

1. SMC
   
   

   How Does the Spotify API Compare to the Music Emotion Recognition State-of-the-Art?

   Panda, Renato, Redinho, Hugo, Gonçalves, Carolina, Malheiro, Ricardo, and Paiva, Rui Pedro

   In Proceedings of the 18th Sound and Music Computing Conference (SMC 2021) 2021

   Conference Rank: B2 (Qualis)

   Abs Bib HANDLE PDF Code

   Features are arguably the key factor to any machine learning problem. Over the decades, myriads of audio features and recently feature-learning approaches have been tested in Music Emotion Recognition (MER) with scarce improvements. Here, we shed some light on the suitability of the audio features provided by the Spotify API, the leading music streaming service, when applied to MER. To this end, 12 Spotify API features were obtained for 704 of our 900-song dataset, annotated in terms of Russell’s quadrants. These are compared to emotionally-relevant features obtained previously, using feature ranking and emotion classification experiments. We verified that energy, valence and acousticness features from Spotify are highly relevant to MER. However, the 12-feature set is unable to meet the performance of the features available in the state-of-the-art (58.5% vs. 74.7% F1-measure). Combining Spotify and state-of-the-art sets leads to small improvements with fewer features (top5: +2.3%, top10: +1.1%), while not improving the maximum results (100 features). From this we conclude that Spotify provides some higher-level emotionally-relevant features. Such extractors are desirable, since they are closer to human concepts and allow for interpretable rules to be extracted (harder with hundreds of abstract features). Still, additional emotionally-relevant features are needed to improve MER.

   @inproceedings{Panda2021,
     author = {Panda, Renato and Redinho, Hugo and Gon{\c{c}}alves, Carolina and Malheiro, Ricardo and Paiva, Rui Pedro},
     booktitle = {Proceedings of the 18th Sound and Music Computing Conference (SMC 2021)},
     doi = {10.5281/zenodo.5045100},
     editor = {Mauro, Davide Andrea and Spagnol, Simone and Valle, Andrea},
     file = {:./Panda et al. - 2021 - How Does the Spotify API Compare to the Music Emotion Recognition State-of-the-Art.pdf:pdf},
     keywords = {Spotify,audio features,emotion,music,music emotion recognition},
     month = jul,
     pages = {238--245},
     publisher = {Axea sas/SMC Network},
     title = {{How Does the Spotify API Compare to the Music Emotion Recognition State-of-the-Art?}},
     code = {https://github.com/renatopanda/SMC2021},
     year = {2021},
     abbr = {SMC},
     selected = {true},
     rank = {B2 (Qualis)},
     handle = {http://hdl.handle.net/10316/95161}
   }

2. TAFFC
   
   

   Audio Features for Music Emotion Recognition: a Survey

   Panda, Renato, Malheiro, Ricardo Manuel, and Paiva, Rui Pedro

   IEEE Transactions on Affective Computing 2020

   Impact Factor (JCR): 10.506*

   Abs Bib HANDLE URL PDF

   The design of meaningful audio features is a key need to advance the state-of-the-art in Music Emotion Recognition (MER). This work presents a survey on the existing emotionally-relevant computational audio features, supported by the music psychology literature on the relations between eight musical dimensions (melody, harmony, rhythm, dynamics, tone color, expressivity, texture and form) and specific emotions. Based on this review, current gaps and needs are identified and strategies for future research on feature engineering for MER are proposed, namely ideas for computational audio features that capture elements of musical form, texture and expressivity that should be further researched. Finally, although the focus of this article is on classical feature engineering methodologies (based on handcrafted features), perspectives on deep learning-based approaches are discussed.

   @article{Panda2020,
     author = {Panda, Renato and Malheiro, Ricardo Manuel and Paiva, Rui Pedro},
     doi = {10.1109/TAFFC.2020.3032373},
     file = {:./Panda, Malheiro, Paiva - 2020 - Audio Features for Music Emotion Recognition a Survey.pdf:pdf},
     issn = {1949-3045},
     journal = {IEEE Transactions on Affective Computing},
     pages = {1--1},
     title = {{Audio Features for Music Emotion Recognition: a Survey}},
     url = {https://ieeexplore.ieee.org/document/9229494/},
     year = {2020},
     abbr = {TAFFC},
     selected = {true},
     jcr = {10.506*},
     handle = {http://hdl.handle.net/10316/95975}
   }

3. TAFFC
   
   

   Novel Audio Features for Music Emotion Recognition

   Panda, Renato, Malheiro, Ricardo, and Paiva, Rui Pedro

   IEEE Transactions on Affective Computing 2020

   Impact Factor (JCR): 10.506

   Abs Bib HANDLE URL PDF Code

   This work advances the music emotion recognition state-of-the-art by proposing novel emotionally-relevant audio features. We reviewed the existing audio features implemented in well-known frameworks and their relationships with the eight commonly defined musical concepts. This knowledge helped uncover musical concepts lacking computational extractors, to which we propose algorithms - namely related with musical texture and expressive techniques. To evaluate our work, we created a public dataset of 900 audio clips, with subjective annotations following Russell’s emotion quadrants. The existent audio features (baseline) and the proposed features (novel) were tested using 20 repetitions of 10-fold cross-validation. Adding the proposed features improved the F1-score to 76.4% (by 9%), when compared to a similar number of baseline-only features. Moreover, analysing the features relevance and results uncovered interesting relations, namely the weight of specific features and musical concepts to each emotion quadrant, and warrant promising new directions for future research in the field of music emotion recognition, interactive media, and novel music interfaces.

   @article{Panda2018a,
     author = {Panda, Renato and Malheiro, Ricardo and Paiva, Rui Pedro},
     doi = {10.1109/TAFFC.2018.2820691},
     file = {:./Panda, Malheiro, Paiva - 2020 - Novel Audio Features for Music Emotion Recognition.pdf:pdf},
     issn = {1949-3045},
     journal = {IEEE Transactions on Affective Computing},
     keywords = {Affective computing,audio databases,emotion recognition,feature extraction,music information retrieval},
     month = oct,
     number = {4},
     pages = {614--626},
     title = {{Novel Audio Features for Music Emotion Recognition}},
     url = {https://ieeexplore.ieee.org/document/8327886/},
     code = {https://github.com/renatopanda/TAFFC2018},
     volume = {11},
     year = {2020},
     abbr = {TAFFC},
     selected = {true},
     jcr = {10.506},
     handle = {http://hdl.handle.net/10316/94071}
   }

4. TAFFC
   
   

   Emotionally-Relevant Features for Classification and Regression of Music Lyrics

   Malheiro, Ricardo, Panda, Renato, Gomes, Paulo, and Paiva, Rui Pedro

   IEEE Transactions on Affective Computing – TAFFC 2018

   Impact Factor (JCR): 6.288

   Abs Bib HANDLE URL PDF

   This research addresses the role of lyrics in the music emotion recognition process. Our approach is based on several state of the art features complemented by novel stylistic, structural and semantic features. To evaluate our approach, we created a ground truth dataset containing 180 song lyrics, according to Russell’s emotion model. We conduct four types of experiments: regression and classification by quadrant, arousal and valence categories. Comparing to the state of the art features (ngrams - baseline), adding other features, including novel features, improved the F-measure from 69.9%, 82.7% and 85.6% to 80.1%, 88.3% and 90%, respectively for the three classification experiments. To study the relation between features and emotions (quadrants) we performed experiments to identify the best features that allow to describe and discriminate each quadrant. To further validate these experiments, we built a validation set comprising 771 lyrics extracted from the AllMusic platform, having achieved 73.6% F-measure in the classification by quadrants. We also conducted experiments to identify interpretable rules that show the relation between features and emotions and the relation among features. Regarding regression, results show that, comparing to similar studies for audio, we achieve a similar performance for arousal and a much better performance for valence.

   @article{Malheiro2018,
     author = {Malheiro, Ricardo and Panda, Renato and Gomes, Paulo and Paiva, Rui Pedro},
     doi = {10.1109/TAFFC.2016.2598569},
     file = {:./Malheiro et al. - 2018 - Emotionally-Relevant Features for Classification and Regression of Music Lyrics.pdf:pdf},
     isbn = {978-989-758-203-5},
     issn = {1949-3045},
     journal = {IEEE Transactions on Affective Computing – TAFFC},
     keywords = {lyrics feature extraction,lyrics music,lyrics music classification,lyrics music emotion recognition,music information retrieval,our approach is based,process,regression,role of lyrics in,the music emotion recognition,this research addresses the},
     month = apr,
     number = {2},
     pages = {240--254},
     title = {{Emotionally-Relevant Features for Classification and Regression of Music Lyrics}},
     url = {http://ieeexplore.ieee.org/document/7536113/},
     volume = {9},
     year = {2018},
     abbr = {TAFFC},
     selected = {true},
     jcr = {6.288},
     handle = {http://hdl.handle.net/10316/94353}
   }

5. CMMR
   
   

   Multi-Modal Music Emotion Recognition: A New Dataset, Methodology and Comparative Analysis

   Panda, Renato, Malheiro, Ricardo, Rocha, Bruno, Oliveira, António Pedro, and Paiva, Rui Pedro

   In 10th International Symposium on Computer Music Multidisciplinary Research – CMMR 2013 2013

   Conference Rank: B5 (Qualis)

   Abs Bib HANDLE PDF

   We propose a multi-modal approach to the music emotion recognition (MER) problem, combining information from distinct sources, namely audio, MIDI and lyrics. We introduce a methodology for the automatic creation of a multi-modal music emotion dataset resorting to the AllMusic database, based on the emotion tags used in the MIREX Mood Classification Task. Then, MIDI files and lyrics corresponding to a sub-set of the obtained audio samples were gathered. The dataset was organized into the same 5 emotion clusters defined in MIREX. From the audio data, 177 standard features and 98 melodic features were extracted. As for MIDI, 320 features were collected. Finally, 26 lyrical features were extracted. We experimented with several supervised learning and feature selection strategies to evaluate the proposed multi-modal approach. Employing only standard audio features, the best attained performance was 44.3% (F-measure). With the multi-modal approach, results improved to 61.1%, using only 19 multi-modal features. Melodic audio features were particularly important to this improvement.

   @inproceedings{Panda2013a,
     address = {Marseille, France},
     author = {Panda, Renato and Malheiro, Ricardo and Rocha, Bruno and Oliveira, Ant{\'{o}}nio Pedro and Paiva, Rui Pedro},
     booktitle = {10th International Symposium on Computer Music Multidisciplinary Research – CMMR 2013},
     file = {:./Panda et al. - 2013 - Multi-Modal Music Emotion Recognition A New Dataset, Methodology and Comparative Analysis.pdf:pdf},
     pages = {570--582},
     title = {{Multi-Modal Music Emotion Recognition: A New Dataset, Methodology and Comparative Analysis}},
     year = {2013},
     abbr = {CMMR},
     selected = {true},
     rank = {B5 (Qualis)},
     handle = {http://hdl.handle.net/10316/94095}
   }