BibTex

Provide the BibTex-formatted reference for the dataset. Please use the correct published version (ACL anthology, etc.) instead of google scholar created Bibtex.

@inproceedings{cs_restaurants,
address = {Tokyo, Japan},
title = {Neural {Generation} for {Czech}: {Data} and {Baselines}},
shorttitle = {Neural {Generation} for {Czech}},
url = {https://www.aclweb.org/anthology/W19-8670/},
urldate = {2019-10-18},
booktitle = {Proceedings of the 12th {International} {Conference} on {Natural} {Language} {Generation} ({INLG} 2019)},
author = {Dušek, Ondřej and Jurčíček, Filip},
month = oct,
year = {2019},
pages = {563--574},
}

Intended Use

What is the intended use of the dataset?

The dataset was created to test neural NLG systems in Czech and their ability to deal with rich morphology.

Dataset Creators

Who created the original dataset? List the people involved in collecting the dataset and their affiliation(s).

Ondrej Dusek and Filip Jurcicek

Funding

Who funded the data creation?

This research was supported by the Charles University project PRIMUS/19/SCI/10 and by the Ministry of Education, Youth and Sports of the Czech Republic under the grant agreement LK11221. This work used using language resources distributed by the LINDAT/CLARIN project of the Ministry of Education, Youth and Sports of the Czech Republic (project LM2015071).

Who added the Dataset to GEM?

Who contributed to the data card and adding the dataset to GEM? List the people+affiliations involved in creating this data card and who helped integrate this dataset into GEM.

Simon Mille wrote the initial data card and Yacine Jernite the data loader. Sebastian Gehrmann migrated the data card and loader to the v2 format.

Reason for Structure

How was the dataset structure determined?

The data originated as a translation and localization of Wen et al.'s SF restaurant NLG dataset.

How were labels chosen?

How were the labels chosen?

The input MRs were collected from Wen et al.'s SF restaurant NLG data and localized by randomly replacing slot values (using a list of Prague restaurant names, neighborhoods etc.).

The generated slot values were then automatically replaced in reference texts in the data.

Splitting Criteria

Describe any criteria for splitting the data, if used. If there are differences between the splits (e.g., if the training annotations are machine-generated and the dev and test ones are created by humans, or if different numbers of annotators contributed to each example), describe them here.

The creators ensured that after delexicalization of the meaning representation there was no overlap between training and test.

The data is split at a 3:1:1 rate between training, validation, and test.

Why is the Dataset in GEM?

What does this dataset contribute toward better generation evaluation and why is it part of GEM?

This is one of a few non-English data-to-text datasets, in a well-known domain, but covering a morphologically rich language that is harder to generate since named entities need to be inflected. This makes it harder to apply common techniques such as delexicalization or copy mechanisms.

Difference from other GEM datasets

What else sets this dataset apart from other similar datasets in GEM?

The dialog acts in this dataset are much more varied than the e2e dataset which is the closest in style.

Technical Terms

Technical terms used in this card and the dataset and their definitions

• utterance: something a system or user may say in a turn
• meaning representation: a representation of meaning that the system should be in accordance with. The specific type of MR in this dataset are dialog acts which describe what a dialog system should do, e.g., inform a user about a value.

Proposed Evaluation

List and describe the purpose of the metrics and evaluation methodology (including human evaluation) that the dataset creators used when introducing this task.

This dataset uses the suite of word-overlap-based automatic metrics from the E2E NLG Challenge (BLEU, NIST, ROUGE-L, METEOR, and CIDEr). In addition, the slot error rate is measured.

Creation Process

If created for the dataset, describe the creation process.

Six professional translators translated the underlying dataset with the following instructions:

• Each utterance should be translated by itself
• fluent spoken-style Czech should be produced
• Facts should be preserved
• If possible, synonyms should be varied to create diverse utterances
• Entity names should be inflected as necessary
• the reader of the generated text should be addressed using formal form and self-references should use the female form.

The translators did not have access to the meaning representation.

Justification for Using the Data

If not, what is the justification for reusing the data?

It was not explicitly stated but we can safely assume that the translators agreed to this use of their data.

Details on how Dataset Addresses the Needs

Describe how this dataset addresses the needs of underserved communities.

The dataset may help improve NLG methods for morphologically rich languages beyond Czech.

Links and Summaries of Analysis Work

Provide links to and summaries of works analyzing these biases.

To ensure consistency of translation, the data always uses formal/polite address for the user, and uses the female form for first-person self-references (as if the dialogue agent producing the sentences was female). This prevents data sparsity and ensures consistent results for systems trained on the dataset, but does not represent all potential situations arising in Czech.

Technical Limitations

Describe any known technical limitations, such as spurrious correlations, train/test overlap, annotation biases, or mis-annotations, and cite the works that first identified these limitations when possible.

The test set may lead users to over-estimate the performance of their NLG systems with respect to their generalisability, because there are no unseen restaurants or addresses in the test set. This is something we will look into for future editions of the GEM shared task.