BibTex

Provide the BibTex-formatted reference for the dataset. Please use the correct published version (ACL anthology, etc.) instead of google scholar created Bibtex.

@inproceedings{parikh-etal-2020-totto,
title = "{ToTTo}: A Controlled Table-To-Text Generation Dataset",
author = "Parikh, Ankur  and
Wang, Xuezhi  and
Gehrmann, Sebastian  and
Faruqui, Manaal  and
Dhingra, Bhuwan  and
Yang, Diyi  and
Das, Dipanjan",
booktitle = "Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)",
month = nov,
year = "2020",
address = "Online",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2020.emnlp-main.89",
doi = "10.18653/v1/2020.emnlp-main.89",
pages = "1173--1186",
abstract = "We present ToTTo, an open-domain English table-to-text dataset with over 120,000 training examples that proposes a controlled generation task: given a Wikipedia table and a set of highlighted table cells, produce a one-sentence description. To obtain generated targets that are natural but also faithful to the source table, we introduce a dataset construction process where annotators directly revise existing candidate sentences from Wikipedia. We present systematic analyses of our dataset and annotation process as well as results achieved by several state-of-the-art baselines. While usually fluent, existing methods often hallucinate phrases that are not supported by the table, suggesting that this dataset can serve as a useful research benchmark for high-precision conditional text generation.",
}

Leaderboard Details

Briefly describe how the leaderboard evaluates models.

This dataset has an associated, active leaderboard maintained by the authors. The test set ground truth targets / references are private, i.e they are not publicly shared or downloadable - hence, leaderboard submission is necessary for test set evaluation. To evaluate your model on the dev or test set AND/OR submit to the leaderboard, you need to submit your model files through this form (The form provides an option to opt-out of going on the leaderboard).

The leaderboard reports three sets of BLEU, PARENT and BLEURT scores for each submission - on the overall test set, the Overlap subset of the test set and the non-Overlap subset of the test set.

Intended Use

What is the intended use of the dataset?

ToTTo is a Table-to-Text NLG task, as the paper title says. The task is as follows: Given a Wikipedia table with row names, column names and table cells, with a subset of cells highlighted, generate a natural language description for the highlighted part of the table . The table need not be exactly rectangular in that - cells can sometimes be multi-row or multi-column.

An earlier example of a Table-to-Text NLG task is Wikibio - here the inputs were Wikipedia infoboxes (from the top right corner of entity-related Wiki pages). In contrast, ToTTo mostly has Wikipedia tables from the main article content itself. In general, Table-To-Text NLG tasks can be seen as a subclass of Data-To-Text NLG tasks - where the task is to generate natural language descriptions of inputs which are in the form of structured or semi-structured data. In general, all Data-To-Text NLG tasks need not have an explicit table or other structure - e.g the input in WebNLG is simply a list of triples.

Importantly, ToTTo differs from earlier examples of Table-To-Text NLG in that:

1. It does not suffer from the problem of divergent references - where ground truth descriptions themselves have additional information not found in the table. ToTTo overcomes this by having a multi-step annotation process to edit the initial, free-form table descriptions (which are from Wikipedia) to make them faithful, unambiguous and independent of article context.

2. Since it provides control in the form of highlighted table cells, it prevents the problem of there being a large number of valid descriptions focussing on different parts of the table.

Dataset Creators

Who created the original dataset? List the people involved in collecting the dataset and their affiliation(s).

Ankur Parikh, Xuezhi Wang, Sebastian Gehrmann, Manaal Faruqui, Bhuwan Dhingra, Diyi Yang, Dipanjan Das

Funding

Who funded the data creation?

Google Research

Who added the Dataset to GEM?

Who contributed to the data card and adding the dataset to GEM? List the people+affiliations involved in creating this data card and who helped integrate this dataset into GEM.

Varun Gangal created the initial data card and Yacine Jernite wrote the data loader. The data card was updated with new splits by Simon Mille. Sebastian Gehrmann ported the data card and loader from the v1 to the v2 version and extended it with the new fields.

Reason for Structure

How was the dataset structure determined?

The structure is aimed to encode highlighted tables in a way that allows rows and columns to span multiple fields in width. The other fields are meta-data about the source and the annotations

How were labels chosen?

How were the labels chosen?

The initial table-description pairs are tables from Wikipedia articles, extracted through heuristics such as Number Matching (tables and sentences that overlap with a non-date number of atleast 3 non-zero digits) (Refer to Section 4 of the paper for more)

1. Table Readability: Tables which are deemed non-readable (due to foreign language, poor formatting etc - a very small fraction of 0.5%) are removed from the dataset here.
2. Cell Highlighting: The annotator highlights the cells of the table which support the description.
3. Deletion: The annotator removes phrases in the description which are not supported by the highlighted cells
4. Decontextualization: Descriptions may contain pronouns or other forms of anaphora, or other phenomena which depend on the overall article topic - these are fixed by replacement (e.g replacing pronouns with the entity, provided it occurs in the table). The replacements allowed are limited to one, and annotators are also instructed to conserve fluency.
5. Secondary Annotation: A second set of annotators is shown the output of Stage 4, and asked to fix it if required to ensure it is grammatical.

Splitting Criteria

Describe any criteria for splitting the data, if used. If there are differences between the splits (e.g., if the training annotations are machine-generated and the dev and test ones are created by humans, or if different numbers of annotators contributed to each example), describe them here.

The dev and test splits are further equally distributed between Overlap and non-Overlap . The examples in the Overlap set are harder on account of the domain shift resulting from them having none of their header (row and column) names in common with those seen during training.

Refer to Table 5 in the paper for a more extensive list of properties about table size, target vocabulary etc and their aggregates.

What does an outlier of the dataset in terms of length/perplexity/embedding look like?

There are some very large tables in the dataset with thousands of rows. Table 7 shows some of the challenges of the dataset, showing that very few examples require access to the table description itself which makes those examples an outlier.

Why is the Dataset in GEM?

What does this dataset contribute toward better generation evaluation and why is it part of GEM?

ToTTo is one of the two datasets representing Table-to-Text NLG in GEM, the other one being DART. Unlike DART, which combines datasets from multiple sources and furnishes them in a unified setting, ToTTo is from a homogeneous source. As explained in the Task Summary above, it also has an annotation process explicitly crafted to reduce divergent descriptions, which is not true of DART.

Furthermore, ToTTo is also an instance of a controlled generation task - where in addition to the input (in this case the table) an additional control (in this case the highlighted cells) is given as an additional goal for the generation. The DART task formulation does not include controls.

Difference from other GEM datasets

What else sets this dataset apart from other similar datasets in GEM?

The input is much more complex and the quality much better than that of comparable datasets. The highlighted table cells provide a unique challenge to models.

Pointers to Resources

Getting started with in-depth research on the task. Add relevant pointers to resources that researchers can consult when they want to get started digging deeper into the task.

• The highest spot on the leaderboard is currently held by an anonymous method, with BLEU=49.2, PARENT=58.7 and BLEURT=0.249 on the Overall test set.

• The highest scoring non-anonymous method is the T5-based method of Kale, 2020. This method uses a simple row-major linearization scheme to convert the table (it chooses only the highlighted cells and ignores the other cells - table titles and section titles are prefixed at the start of the respective section table) to a flat string. The linearized input - output description pairs from training examples are then used to finetune T5, with BLEU being used as the dev metric to pick checkpoints, and beam search with beam size 10 being the decoding method.

  Though the best numbers from this method are naturally from the largest T5-pretrained architecture (T5-3B), the paper shows improvements over the next-highest BERT-to-BERT method even when using T5-Base or T5-Small, which have the same and lesser parameters than BERT-to-BERT respectively.

• The Supplementary Repo provides several useful modules to get started with for new approach implementation:

  1. Code for the particular preprocessing / linearization scheme used to linearize the tables into flat sequences for the baseline approaches described in the paper has been described and shared herein

  2. An evaluation script for locally scoring BLEU and PARENT system outputs on dev (or train) sets. Since BLEURT is a model-based metric, a slightly separate set of instructions is provided to evaluate on the same.

Proposed Evaluation

List and describe the purpose of the metrics and evaluation methodology (including human evaluation) that the dataset creators used when introducing this task.

The metrics are used as in the leaderboard. The original paper additionally conducted a human evaluation focusing on fluency, faithfulness, and coverage. Faithfulness was measured as whether facts in the text are not supported by the input, and coverage as the number of highlighted cells that were considered. They thus represent precision and recall of the content.

Relevant Previous Results

What are the most relevant previous results for this task/dataset?

See leaderboard.

Language Producers

What further information do we have on the language producers?

The basic source language producers are Wikipedia authors and/or editors, since the annotation starts with the natural text description near the Wikipedia table. The auxiliary source language producers are the annotators (two per example) who iteratively revise these descriptions to make them unambiguous and faithful to a subset of highlighted cells in the table.

Data Preprocessing

How was the text data pre-processed? (Enter N/A if the text was not pre-processed)

The initial table-description pairs are tables from Wikipedia articles, extracted through heuristics such as Number Matching (tables and sentences that overlap with a non-date number of atleast 3 non-zero digits) (Refer to Section 4 of the paper for more)

1. Table Readability: Tables which are deemed non-readable (due to foreign language, poor formatting etc - a very small fraction of 0.5%) are removed from the dataset here.
2. Cell Highlighting: The annotator highlights the cells of the table which support the description.
3. Deletion: The annotator removes phrases in the description which are not supported by the highlighted cells
4. Decontextualization: Descriptions may contain pronouns or other forms of anaphora, or other phenomena which depend on the overall article topic - these are fixed by replacement (e.g replacing pronouns with the entity, provided it occurs in the table). The replacements allowed are limited to one, and annotators are also instructed to conserve fluency.
5. Secondary Annotation: A second set of annotators is shown the output of Stage 4, and asked to fix it if required to ensure it is grammatical.

The paper does not specifically describe the annotation platform or location profiles of the annotators.

Filter Criteria

What were the selection criteria?

After construction of the splits, the data curators filtered training examples that had rare table header combinations (<=5 examples) and which had an overlap with the validation or test splits.

Consent Policy Details

What was the consent policy?

Annotators were full time employees that were aware of the goal of the project and consented to having the data released as part of the dataset.

Maintenance Plan Details

Describe the original dataset's maintenance plan.

For submissions, you can delete your data by emailing totto@google.com from the email account used to sign up for the submission. Deletion requests will be responded to within 60 days.

Links and Summaries of Analysis Work

Provide links to and summaries of works analyzing these biases.

The original work as well as our GEM paper analyzes some biases

Potential PII Risk

Considering your answers to the PII part of the Data Curation Section, describe any potential privacy to the data subjects and creators risks when using the dataset.

Since the source data is from wikipedia, only data in the public domain is included in the dataset.

Technical Limitations

Describe any known technical limitations, such as spurrious correlations, train/test overlap, annotation biases, or mis-annotations, and cite the works that first identified these limitations when possible.

The dataset is limited to topics that are present in Wikipedia, more specifically those topics that are present in articles which contain atleast one table Sports and Countries form 53.4% of the dataset. The remaining fraction is made up of broader topics like Europe, North Americaand Politics