BibTex

Provide the BibTex-formatted reference for the dataset. Please use the correct published version (ACL anthology, etc.) instead of google scholar created Bibtex.

@inproceedings{hayashi-etal-2019-findings,
title = "Findings of the Third Workshop on Neural Generation and Translation",
author = "Hayashi, Hiroaki  and
Oda, Yusuke  and
Birch, Alexandra  and
Konstas, Ioannis  and
Finch, Andrew  and
Luong, Minh-Thang  and
Neubig, Graham  and
Sudoh, Katsuhito",
booktitle = "Proceedings of the 3rd Workshop on Neural Generation and Translation",
month = nov,
year = "2019",
address = "Hong Kong",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/D19-5601",
doi = "10.18653/v1/D19-5601",
pages = "1--14",
abstract = "This document describes the findings of the Third Workshop on Neural Generation and Translation, held in concert with the annual conference of the Empirical Methods in Natural Language Processing (EMNLP 2019). First, we summarize the research trends of papers presented in the proceedings. Second, we describe the results of the two shared tasks 1) efficient neural machine translation (NMT) where participants were tasked with creating NMT systems that are both accurate and efficient, and 2) document generation and translation (DGT) where participants were tasked with developing systems that generate summaries from structured data, potentially with assistance from text in another language.",
}

Intended Use

What is the intended use of the dataset?

Foster the research on document-level generation technology and contrast the methods for different types of inputs.

Dataset Creators

Who created the original dataset? List the people involved in collecting the dataset and their affiliation(s).

Graham Neubig (Carnegie Mellon University), Hiroaki Hayashi (Carnegie Mellon University)

Funding

Who funded the data creation?

Graham Neubig

Who added the Dataset to GEM?

Who contributed to the data card and adding the dataset to GEM? List the people+affiliations involved in creating this data card and who helped integrate this dataset into GEM.

Hiroaki Hayashi (Carnegie Mellon University)

Reason for Structure

How was the dataset structure determined?

• Structured data are directly imported from the original RotoWire dataset.
• Textual data (English, German) are associated to each sample.

Splitting Criteria

Describe any criteria for splitting the data, if used. If there are differences between the splits (e.g., if the training annotations are machine-generated and the dev and test ones are created by humans, or if different numbers of annotators contributed to each example), describe them here.

• English summaries are provided sentence-by-sentence to professional German translators with basketball knowledge to obtain sentence-level German translations.
• Split criteria follows the original RotoWire dataset.

What does an outlier of the dataset in terms of length/perplexity/embedding look like?

• The (English) summary length in the training set varies from 145 to 650 words, with an average of 323 words.

Why is the Dataset in GEM?

What does this dataset contribute toward better generation evaluation and why is it part of GEM?

The use of two modalities (data, foreign text) to generate a document-level text summary.

Difference from other GEM datasets

What else sets this dataset apart from other similar datasets in GEM?

The potential use of two modalities (data, foreign text) as input.

Modification Details

For each of these changes, described them in more details and provided the intended purpose of the modification

• Added GEM ID in each sample.
• Normalize the number of players in each sample with "N/A" for consistent data loading.

Pointers to Resources

Getting started with in-depth research on the task. Add relevant pointers to resources that researchers can consult when they want to get started digging deeper into the task.

• Challenges in Data-to-Document Generation
• Data-to-Text Generation with Content Selection and Planning
• Findings of the Third Workshop on Neural Generation and Translation

Technical Terms

Technical terms used in this card and the dataset and their definitions

• Data-to-text
• Neural machine translation (NMT)
• Document-level generation and translation (DGT)

Proposed Evaluation

List and describe the purpose of the metrics and evaluation methodology (including human evaluation) that the dataset creators used when introducing this task.

To evaluate the fidelity of the generated content to the input data.

Relevant Previous Results

What are the most relevant previous results for this task/dataset?

See Table 2 to 7 of (https://aclanthology.org/D19-5601) for previous results for this dataset.

Creation Process

If created for the dataset, describe the creation process.

Professional German language translators were hired to translate basketball summaries from a subset of RotoWire dataset.

Language Producers

What further information do we have on the language producers?

Translators are familiar with basketball terminology.

Data Preprocessing

How was the text data pre-processed? (Enter N/A if the text was not pre-processed)

Sentence-level translations were aligned back to the original English summary sentences.

Annotation Values

Purpose and values for each annotation

Sentence-end indices for the tokenized summaries. Sentence boundaries can help users accurately identify aligned sentences in both languages, as well as allowing an accurate evaluation that involves sentence boundaries (ROUGE-L).

Quality Control Details

Describe the quality control measures that were taken.

Token and number overlaps between pairs of aligned sentences are measured.

Justification for Using the Data

If not, what is the justification for reusing the data?

Reusing by citing the original papers:

• Sam Wiseman, Stuart M. Shieber, Alexander M. Rush: Challenges in Data-to-Document Generation. EMNLP 2017.
• Hiroaki Hayashi, Yusuke Oda, Alexandra Birch, Ioannis Konstas, Andrew Finch, Minh-Thang Luong, Graham Neubig, Katsuhito Sudoh. Findings of the Third Workshop on Neural Generation and Translation. WNGT 2019.

Potential PII Risk

Considering your answers to the PII part of the Data Curation Section, describe any potential privacy to the data subjects and creators risks when using the dataset.

• Structured data contain real National Basketball Association player and organization names.

Technical Limitations

Describe any known technical limitations, such as spurrious correlations, train/test overlap, annotation biases, or mis-annotations, and cite the works that first identified these limitations when possible.

Potential overlap of box score tables between splits. This was extensively studied and pointed out by [1].

[1]: Thomson, Craig, Ehud Reiter, and Somayajulu Sripada. "SportSett: Basketball-A robust and maintainable data-set for Natural Language Generation." Proceedings of the Workshop on Intelligent Information Processing and Natural Language Generation. 2020.

Unsuited Applications

When using a model trained on this dataset in a setting where users or the public may interact with its predictions, what are some pitfalls to look out for? In particular, describe some applications of the general task featured in this dataset that its curation or properties make it less suitable for.

Users may interact with a trained model to learn about a NBA game in a textual manner. On generated texts, they may observe factual errors that contradicts the actual data that the model conditions on. Factual errors include wrong statistics of a player (e.g., 3PT), non-existent injury information.

Discouraged Use Cases

What are some discouraged use cases of a model trained to maximize the proposed metrics on this dataset? In particular, think about settings where decisions made by a model that performs reasonably well on the metric my still have strong negative consequences for user or members of the public.

Publishing the generated text as is. Even if the model achieves high scores on the evaluation metrics, there is a risk of factual errors mentioned above.