ART
Abductive reasoning is inference to the most plausible explanation. For example, if Jenny finds her house in a mess when she returns from work, and remembers that she left a window open, she can hypothesize that a thief broke into her house and caused the mess, as the most plausible explanation. This data loader focuses on abductive NLG: a conditional English generation task for explaining given observations in natural language.
You can load the dataset via:
import datasets
data = datasets.load_dataset('GEM/ART')
The data loader can be found here.
website
paper
authors
Chandra Bhagavatula (AI2), Ronan Le Bras (AI2), Chaitanya Malaviya (AI2), Keisuke Sakaguchi (AI2), Ari Holtzman (AI2, UW), Hannah Rashkin (AI2, UW), Doug Downey (AI2), Wen-tau Yih (AI2), Yejin Choi (AI2, UW)
Quick-Use
Contact Name
If known, provide the name of at least one person the reader can contact for questions about the
dataset.
If known, provide the name of at least one person the reader can contact for questions about the dataset.
Chandra Bhagavatulla
Multilingual?
Is the dataset multilingual?
Is the dataset multilingual?
no
Covered Languages
What languages/dialects are covered in the dataset?
What languages/dialects are covered in the dataset?
English
License
What is the license of the dataset?
What is the license of the dataset?
apache-2.0: Apache License 2.0
Additional Annotations?
Does the dataset have additional annotations for each instance?
Does the dataset have additional annotations for each instance?
automatically created
Contains PII?
Does the source language data likely contain Personal Identifying Information about the data creators
or
subjects?
Does the source language data likely contain Personal Identifying Information about the data creators or subjects?
no PII
Dataset Overview
-
Where to find the Data and its Documentation
-
Languages and Intended Use
-
Credit
-
Dataset Structure
-
Where to find the Data and its Documentation
-
Languages and Intended Use
-
Credit
-
Dataset Structure
Where to find the Data and its Documentation
Webpage
What is the webpage for the dataset (if it exists)?
What is the webpage for the dataset (if it exists)?
Download
What is the link to where the original dataset is hosted?
What is the link to where the original dataset is hosted?
Paper
What is the link to the paper describing the dataset (open access preferred)?
What is the link to the paper describing the dataset (open access preferred)?
BibTex
Provide the BibTex-formatted reference for the dataset. Please use the correct published version
(ACL
anthology, etc.) instead of google scholar created Bibtex.
Provide the BibTex-formatted reference for the dataset. Please use the correct published version (ACL anthology, etc.) instead of google scholar created Bibtex.
@inproceedings{
Bhagavatula2020Abductive,
title={Abductive Commonsense Reasoning},
author={Chandra Bhagavatula and Ronan Le Bras and Chaitanya Malaviya and Keisuke Sakaguchi and Ari Holtzman and Hannah Rashkin and Doug Downey and Wen-tau Yih and Yejin Choi},
booktitle={International Conference on Learning Representations},
year={2020},
url={https://openreview.net/forum?id=Byg1v1HKDB}
}
Contact Name
If known, provide the name of at least one person the reader can contact for questions about the
dataset.
If known, provide the name of at least one person the reader can contact for questions about the dataset.
Chandra Bhagavatulla
Contact Email
If known, provide the email of at least one person the reader can contact for questions about the
dataset.
If known, provide the email of at least one person the reader can contact for questions about the dataset.
Has a Leaderboard?
Does the dataset have an active leaderboard?
Does the dataset have an active leaderboard?
no
Languages and Intended Use
Multilingual?
Is the dataset multilingual?
Is the dataset multilingual?
no
Covered Languages
What languages/dialects are covered in the dataset?
What languages/dialects are covered in the dataset?
English
Whose Language?
Whose language is in the dataset?
Whose language is in the dataset?
Crowdworkers on the Amazon Mechanical Turk platform based in the U.S, Canada, U.K and Australia.
License
What is the license of the dataset?
What is the license of the dataset?
apache-2.0: Apache License 2.0
Intended Use
What is the intended use of the dataset?
What is the intended use of the dataset?
To study the viability of language-based abductive reasoning. Training and evaluating models to generate a plausible hypothesis to explain two given observations.
Primary Task
What primary task does the dataset support?
What primary task does the dataset support?
Reasoning
Credit
Curation Organization Type(s)
In what kind of organization did the dataset curation happen?
In what kind of organization did the dataset curation happen?
industry
Curation Organization(s)
Name the organization(s).
Name the organization(s).
Allen Institute for AI
Dataset Creators
Who created the original dataset? List the people involved in collecting the dataset and their
affiliation(s).
Who created the original dataset? List the people involved in collecting the dataset and their affiliation(s).
Chandra Bhagavatula (AI2), Ronan Le Bras (AI2), Chaitanya Malaviya (AI2), Keisuke Sakaguchi (AI2), Ari Holtzman (AI2, UW), Hannah Rashkin (AI2, UW), Doug Downey (AI2), Wen-tau Yih (AI2), Yejin Choi (AI2, UW)
Funding
Who funded the data creation?
Who funded the data creation?
Allen Institute for AI
Who added the Dataset to GEM?
Who contributed to the data card and adding the dataset to GEM? List the people+affiliations
involved
in creating this data card and who helped integrate this dataset into GEM.
Who contributed to the data card and adding the dataset to GEM? List the people+affiliations involved in creating this data card and who helped integrate this dataset into GEM.
Chandra Bhagavatula (AI2), Ronan LeBras (AI2), Aman Madaan (CMU), Nico Daheim (RWTH Aachen University)
Dataset Structure
Data Fields
List and describe the fields present in the dataset.
List and describe the fields present in the dataset.
observation_1: A string describing an observation / event.observation_2: A string describing an observation / event.label: A string that plausibly explains why observation_1 and observation_2 might have happened.
How were labels chosen?
How were the labels chosen?
How were the labels chosen?
Explanations were authored by crowdworkers on the Amazon Mechanical Turk platform using a custom template designed by the creators of the dataset.
Example Instance
Provide a JSON formatted example of a typical instance in the dataset.
Provide a JSON formatted example of a typical instance in the dataset.
{
'gem_id': 'GEM-ART-validation-0',
'observation_1': 'Stephen was at a party.',
'observation_2': 'He checked it but it was completely broken.',
'label': 'Stephen knocked over a vase while drunk.'
}
Data Splits
Describe and name the splits in the dataset if there are more than one.
Describe and name the splits in the dataset if there are more than one.
train: Consists of training instances.dev: Consists of dev instances.test: Consists of test instances.
Dataset in GEM
-
Rationale for Inclusion in GEM
-
GEM-Specific Curation
-
Getting Started with the Task
-
Rationale for Inclusion in GEM
-
GEM-Specific Curation
-
Getting Started with the Task
Rationale for Inclusion in GEM
Why is the Dataset in GEM?
What does this dataset contribute toward better generation evaluation and why is it part of GEM?
What does this dataset contribute toward better generation evaluation and why is it part of GEM?
Abductive reasoning is a crucial capability of humans and ART is the first dataset curated to study language-based abductive reasoning.
Similar Datasets
Do other datasets for the high level task exist?
Do other datasets for the high level task exist?
no
Ability that the Dataset measures
What aspect of model ability can be measured with this dataset?
What aspect of model ability can be measured with this dataset?
Whether models can reason abductively about a given pair of observations.
GEM-Specific Curation
Modificatied for GEM?
Has the GEM version of the dataset been modified in any way (data, processing, splits) from the
original curated data?
Has the GEM version of the dataset been modified in any way (data, processing, splits) from the original curated data?
no
Additional Splits?
Does GEM provide additional splits to the dataset?
Does GEM provide additional splits to the dataset?
no
Previous Results
-
Previous Results
-
Previous Results
Previous Results
Measured Model Abilities
What aspect of model ability can be measured with this dataset?
What aspect of model ability can be measured with this dataset?
Whether models can reason abductively about a given pair of observations.
Metrics
What metrics are typically used for this task?
What metrics are typically used for this task?
BLEU, BERT-Score, ROUGE
Previous results available?
Are previous results available?
Are previous results available?
no
Dataset Curation
-
Original Curation
-
Language Data
-
Structured Annotations
-
Consent
-
Private Identifying Information (PII)
-
Maintenance
-
Original Curation
-
Language Data
-
Structured Annotations
-
Consent
-
Private Identifying Information (PII)
-
Maintenance
Original Curation
Sourced from Different Sources
Is the dataset aggregated from different data sources?
Is the dataset aggregated from different data sources?
no
Language Data
How was Language Data Obtained?
How was the language data obtained?
How was the language data obtained?
Crowdsourced
Where was it crowdsourced?
If crowdsourced, where from?
If crowdsourced, where from?
Amazon Mechanical Turk
Language Producers
What further information do we have on the language producers?
What further information do we have on the language producers?
Language producers were English speakers in U.S., Canada, U.K and Australia.
Topics Covered
Does the language in the dataset focus on specific topics? How would you describe them?
Does the language in the dataset focus on specific topics? How would you describe them?
No
Data Validation
Was the text validated by a different worker or a data curator?
Was the text validated by a different worker or a data curator?
validated by crowdworker
Was Data Filtered?
Were text instances selected or filtered?
Were text instances selected or filtered?
algorithmically
Filter Criteria
What were the selection criteria?
What were the selection criteria?
Adversarial filtering algorithm as described in the paper
Structured Annotations
Additional Annotations?
Does the dataset have additional annotations for each instance?
Does the dataset have additional annotations for each instance?
automatically created
Annotation Service?
Was an annotation service used?
Was an annotation service used?
no
Annotation Values
Purpose and values for each annotation
Purpose and values for each annotation
Each observation is associated with a list of COMET (https://arxiv.org/abs/1906.05317) inferences.
Any Quality Control?
Quality control measures?
Quality control measures?
none
Consent
Any Consent Policy?
Was there a consent policy involved when gathering the data?
Was there a consent policy involved when gathering the data?
no
Private Identifying Information (PII)
Contains PII?
Does the source language data likely contain Personal Identifying Information about the data
creators
or subjects?
Does the source language data likely contain Personal Identifying Information about the data creators or subjects?
no PII
Justification for no PII
Provide a justification for selecting no PII above.
Provide a justification for selecting no PII above.
The dataset contains day-to-day events. It does not contain names, emails, addresses etc.
Maintenance
Any Maintenance Plan?
Does the original dataset have a maintenance plan?
Does the original dataset have a maintenance plan?
no
Broader Social Context
-
Previous Work on the Social Impact of the Dataset
-
Impact on Under-Served Communities
-
Discussion of Biases
-
Previous Work on the Social Impact of the Dataset
-
Impact on Under-Served Communities
-
Discussion of Biases
Previous Work on the Social Impact of the Dataset
Usage of Models based on the Data
Are you aware of cases where models trained on the task featured in this dataset ore related tasks
have been used in automated systems?
Are you aware of cases where models trained on the task featured in this dataset ore related tasks have been used in automated systems?
no
Impact on Under-Served Communities
Addresses needs of underserved Communities?
Does this dataset address the needs of communities that are traditionally underserved in language
technology, and particularly language generation technology? Communities may be underserved for
exemple because their language, language variety, or social or geographical context is
underepresented
in NLP and NLG resources (datasets and models).
Does this dataset address the needs of communities that are traditionally underserved in language technology, and particularly language generation technology? Communities may be underserved for exemple because their language, language variety, or social or geographical context is underepresented in NLP and NLG resources (datasets and models).
no
Discussion of Biases
Any Documented Social Biases?
Are there documented social biases in the dataset? Biases in this context are variations in the
ways
members of different social categories are represented that can have harmful downstream consequences
for members of the more disadvantaged group.
Are there documented social biases in the dataset? Biases in this context are variations in the ways members of different social categories are represented that can have harmful downstream consequences for members of the more disadvantaged group.
no
Considerations for Using the Data
-
PII Risks and Liability
-
Licenses
-
Known Technical Limitations
-
PII Risks and Liability
-
Licenses
-
Known Technical Limitations
PII Risks and Liability
Potential PII Risk
Considering your answers to the PII part of the Data Curation Section, describe any potential
privacy
to the data subjects and creators risks when using the dataset.
Considering your answers to the PII part of the Data Curation Section, describe any potential privacy to the data subjects and creators risks when using the dataset.
None
Licenses
Copyright Restrictions on the Dataset
Based on your answers in the Intended Use part of the Data Overview Section, which of the following
best describe the copyright and licensing status of the dataset?
Based on your answers in the Intended Use part of the Data Overview Section, which of the following best describe the copyright and licensing status of the dataset?
public domain
Copyright Restrictions on the Language Data
Based on your answers in the Language part of the Data Curation Section, which of the following
best
describe the copyright and licensing status of the underlying language data?
Based on your answers in the Language part of the Data Curation Section, which of the following best describe the copyright and licensing status of the underlying language data?
public domain