Generating Natural Language Descriptions in Visual Contexts
Overview
The project concerns the interaction between language production and images. Our aim is to collect and annotate corpora of descriptions in visual domains, and then use these resources to build novel computational models of reference production.
Theoretical issues to be addressed include the generation of more complex (e.g., relational) descriptions, with varying degrees
of under and overspecification. and non-determinism.
Current status
The project ran from June 2014 to May 2016 under FAPESP grant nr. 2013/23169-0 (Generating Natural Language Descriptions in Visual Contexts).
Deliverables
The Stars corpus contains 704 descriptions produced by 64 subjects. The corpus is described in Paraboni, Yamasaki, da Silva & Teixeira (2014) and it is available for download here as a .zip file. The corpus has been used in Teixeira, Paraboni, da Silva & Yamasaki (2014) and Ferreira & Paraboni (2014).
The Stars2 corpus (which is a different corpus altogether!) contains 884 descriptions produced by 56 subjects. The corpus is described in Paraboni, Galindo & Iacovelli (2017) and it is available for download here as a .zip file. The corpus has been used in Ferreira & Paraboni (2014) and Paraboni, Galindo & Iacovelli (2015).
Both Stars and Stars2 may be freely used for research purposes.
Publications
Ferreira, Thiago Castro; Ivandré Paraboni (2017) Improving the generation of personalised descriptions. 10th International Conference on Natural Language Generation (INLG-2017). Santiago de Compostela, Spain.
Paraboni, Ivandré; Alex Gwo Jen Lan; Matheus Mendes de Sant'Ana; Flávio Luiz Coutinho (2017) Effects of cognitive effort on the resolution of overspecified descriptions. Computational Linguistics 43 (2), pp.451-459. 10.1162/COLI_a_00288.
Ferreira, Thiago Castro; Ivandré Paraboni (2017) Generating Natural Language Descriptions using Speaker-dependent Information. In: Natural Language Engineering 23(6), pp. 813-834. 10.1017/S1351324917000079.
Paraboni, Ivandré; Michelle Reis Galindo; Douglas Iacovelli (2017) Stars2: a corpus of object descriptions in a visual domain. Language Resources and Evaluation 51(2), pp. 49-62. The original publication is available at http://www.springerlink.com.
da Silva, Diego Santos and Ivandré Paraboni (2015) Generating Spatial Referring Expressions in Interactive 3D Worlds. Spatial Cognition and Computation 15(3), pp. 186-225.
Paraboni, Ivandré; Michelle Reis Galindo; Douglas Iacovelli (2015) O córpus Stars2 de Expressões de Referência. Relatório Técnico PPgSI-001/2015. University of São Paulo.
Paraboni, Ivandré; Michelle Reis Galindo; Douglas Iacovelli (2015) Generating overspecified referring expressions: the role of discrimination. The 53rd Annual Meeting of the Association for Computational Linguistics (ACL-2015), pp. 76-82, Beijing.
Altamirano, Romina; Thiago Castro Ferreira; Ivandré Paraboni; Luciana Benotti (2015). Zoom: a corpus of natural language descriptions of map locations. The 53rd Annual Meeting of the Association for Computational Linguistics (ACL-2015), pp. 69-75, Beijing.
Paraboni, Ivandré; Alan K. Yamasaki; Adriano S. R. da Silva; Caio V. M. Teixeira (2014) Generating Underspecified Descriptions of Landmark Objects. 17th International Conference on Text, Speech and Dialogue (TSD-2014). Brno, Czech Republic. Lecture Notes in Artificial Intelligence vol. 8655, pp. 76-83. Springer International Publishing. The original publication is available at http://www.springerlink.com.
Ferreira, Thiago Castro; Ivandré Paraboni (2014). Referring Expression Generation: taking Speakers' Preferences into account. 17th International Conference on Text, Speech and Dialogue (TSD-2014). Brno, Czech Republic. Lecture Notes in Artificial Intelligence vol. 8655, pp. 539-546. Springer International Publishing. The original publication is available at http://www.springerlink.com.
Silva, Diego Santos; Ivandré Paraboni (2014) Geração de Expressões de Referência em Ambientes Virtuais Interativos. Linguamática 6(1), pp. 15-27.
Ferreira, Thiago Castro; Ivandré Paraboni (2014). Classification-based Referring Expression Generation. 15th International Conference on Intelligent Text Processing and Computational Linguistics (CiCLing-2014). Kathmandu, Nepal. Lecture Notes in Computer Science vol. 8403, pp. 481-491. Springer-Verlag Berlin Heidelberg. The original publication is available at http://www.springerlink.com.
Teixeira, Caio V. M.; Ivandré Paraboni; Adriano S. R. da Silva; Alan K. Yamasaki (2014) Generating Relational Descriptions involving Mutual Disambiguation. 15th International Conference on Intelligent Text Processing and Computational Linguistics (CiCLing-2014). Kathmandu, Nepal. Lecture Notes in Computer Science vol. 8403, pp. 492-502. Springer- Verlag Berlin Heidelberg The original publication is available at http://www.springerlink.com.
Paraboni, Ivandré (2014) O córpus Stars de expressões de Referência. Relatório Técnico PPgSI-002/2014. University of São Paulo.
Iacovelli, Douglas; Michelle Reis Galindo; Ivandré Paraboni (2014) Lausanne: a Framework for Collaborative online NLP Experiments. 11th International Conference on Computational Processing of Portuguese (PROPOR-2014) São Carlos, Brazil. Lecture Notes in Computer Science vol. 8775, pp. 280-285. Springer International Publishing. The original publication is available at http://www.springerlink.com.
