BibTex

Provide the BibTex-formatted reference for the dataset. Please use the correct published version (ACL anthology, etc.) instead of google scholar created Bibtex.

@inproceedings{xu2022fairytaleqa, author={Xu, Ying and Wang, Dakuo and Yu, Mo and Ritchie, Daniel and Yao, Bingsheng and Wu, Tongshuang and Zhang, Zheng and Li, Toby Jia-Jun and Bradford, Nora and Sun, Branda and Hoang, Tran Bao and Sang, Yisi and Hou, Yufang and Ma, Xiaojuan and Yang, Diyi and Peng, Nanyun and Yu, Zhou and Warschauer, Mark}, title = {Fantastic Questions and Where to Find Them: Fairytale{QA} -- An Authentic Dataset for Narrative Comprehension}, publisher = {Association for Computational Linguistics}, year = {2022} }

Leaderboard Details

Briefly describe how the leaderboard evaluates models.

The task was to generate questions corresponding to the given answers and the story context. Success on the Question Generation task is typically measured by achieving a high ROUGE-L score to the reference ground-truth question.

Intended Use

What is the intended use of the dataset?

The purpose of this dataset is to help develop systems to facilitate assessment and training of narrative comprehension skills for children in education domain. The dataset distinguishes fine-grained reading skills, such as the understanding of varying narrative elements, and contains high-quality QA-pairs generated by education experts with sufficient training and education domain knowledge to create valid QA-pairs in a consistent way.

This dataset is suitable for developing models to automatically generate questions and QA-Pairs that satisfy the need for a continuous supply of new questions, which can potentially enable large-scale development of AI-supported interactive platforms for the learning and assessment of reading comprehension skills.

Dataset Creators

Who created the original dataset? List the people involved in collecting the dataset and their affiliation(s).

Ying Xu (University of California Irvine); Dakuo Wang (IBM Research); Mo Yu (IBM Research); Daniel Ritchie (University of California Irvine); Bingsheng Yao (Rensselaer Polytechnic Institute); Tongshuang Wu (University of Washington); Zheng Zhang (University of Notre Dame); Toby Jia-Jun Li (University of Notre Dame); Nora Bradford (University of California Irvine); Branda Sun (University of California Irvine); Tran Bao Hoang (University of California Irvine); Yisi Sang (Syracuse University); Yufang Hou (IBM Research Ireland); Xiaojuan Ma (Hong Kong Univ. of Sci and Tech); Diyi Yang (Georgia Institute of Technology); Nanyun Peng (University of California Los Angeles); Zhou Yu (Columbia University); Mark Warschauer (University of California Irvine)

Funding

Who funded the data creation?

Schmidt Futures

Who added the Dataset to GEM?

Who contributed to the data card and adding the dataset to GEM? List the people+affiliations involved in creating this data card and who helped integrate this dataset into GEM.

Dakuo Wang (IBM Research); Bingsheng Yao (Rensselaer Polytechnic Institute); Ying Xu (University of California Irvine)

Reason for Structure

How was the dataset structure determined?

[N/A]

How were labels chosen?

How were the labels chosen?

A typical data point comprises a question, the corresponding story content, and one answer. Education expert annotators labeled whether the answer is locally relevant to one story section or requires summarization capabilities from multiple story sections, and whether the answers are explicit (can be directly found in the stories) or implicit (cannot be directly found in the story text). Additionally, education expert annotators categorize the QA-pairs via 7 narrative elements from an establish framework.

Splitting Criteria

Describe any criteria for splitting the data, if used. If there are differences between the splits (e.g., if the training annotations are machine-generated and the dev and test ones are created by humans, or if different numbers of annotators contributed to each example), describe them here.

The books are randomly split into train/validation/test splits. We control the ratio of QA-pair numbers in train:validation:test splits close to 8:1:1

What does an outlier of the dataset in terms of length/perplexity/embedding look like?

[N/A]

Why is the Dataset in GEM?

What does this dataset contribute toward better generation evaluation and why is it part of GEM?

The dataset distinguishes fine-grained reading skills, such as the understanding of varying narrative elements, and contains high-quality QA-pairs generated by education experts with sufficient training and education domain knowledge to create valid QA-pairs in a consistent way.

Modification Details

For each of these changes, described them in more details and provided the intended purpose of the modification

The original data contains two answers by different annotators in validation/test splits, we removed the 2nd answer for GEM version because it is not being used for the Question Generation task.

Pointers to Resources

Getting started with in-depth research on the task. Add relevant pointers to resources that researchers can consult when they want to get started digging deeper into the task.

[N/A]

Proposed Evaluation

List and describe the purpose of the metrics and evaluation methodology (including human evaluation) that the dataset creators used when introducing this task.

The task was to generate questions corresponding to the given answers and the story context. Success on this task is typically measured by achieving a high ROUGE score to the reference ground-truth questions.

Relevant Previous Results

What are the most relevant previous results for this task/dataset?

A BART-based model currently achieves a ROUGE-L of 0.527/0.527 on valid/test splits, which is reported as the baseline experiment for the dataset paper.

Language Producers

What further information do we have on the language producers?

The fairytale story texts are from the Project Gutenberg website

Data Preprocessing

How was the text data pre-processed? (Enter N/A if the text was not pre-processed)

Due to a large number of fairytales found, we used the most popular stories based on the number of downloads since these stories are presumably of higher quality. To ensure the readability of the text, we made a small number of minor revisions to some obviously outdated vocabulary (e.g., changing “ere” to “before”) and the unconventional use of punctuation (e.g., changing consecutive semi-colons to periods).

These texts were broken down into small sections based on their semantic content by our annotators. The annotators were instructed to split the story into sections of 100-300 words that also contain meaningful content and are separated at natural story breaks. An initial annotator would split the story, and this would be reviewed by a cross-checking annotator. Most of the resulting sections were one natural paragraph of the original text.

Filter Criteria

What were the selection criteria?

For each story, we evaluated the reading difficulty level using the textstat Python package, primarily based on sentence length, word length, and commonness of words. We excluded stories that are at 10th grade level or above.

Annotation Values

Purpose and values for each annotation

The dataset annotation distinguishes fine-grained reading skills, such as the understanding of varying narrative elements, and contains high-quality QA-pairs generated by education experts with sufficient training and education domain knowledge to create valid QA-pairs in a consistent way.

Quality Control Details

Describe the quality control measures that were taken.

The annotators were instructed to imagine that they were creating questions to test elementary or middle school students in the process of reading a complete story. We required the annotators to generate only natural, open-ended questions, avoiding “yes-” or “no-” questions. We also instructed them to provide a diverse set of questions about 7 different narrative elements, and with both implicit and explicit questions.

We asked the annotators to also generate answers for each of their questions. We asked them to provide the shortest possible answers but did not restrict them to complete sentences or short phrases. We also asked the annotators to label which section(s) the question and answer was from.

All annotators received a two-week training in which each of them was familiarized with the coding template and conducted practice coding on the same five stories. The practice QA pairs were then reviewed by the other annotators and the three experts, and discrepancies among annotators were discussed. During the annotation process, the team met once every week to review and discuss each member’s work. All QA pairs were cross-checked by two annotators, and 10% of the QA pairs were additionally checked by the expert supervisor.

For the 46 stories used as the evaluation set, we annotate a second reference answer by asking an annotator to independently read the story and answer the questions generated by others.

Consent Policy Details

What was the consent policy?

During the annotation process, the team met once every week to review and discuss each member’s work. All QA pairs were cross-checked by two annotators, and 10% of the QA pairs were additionally checked by the expert supervisor.

Other Consented Downstream Use

What other downstream uses of the data did the original data creators and the data curators consent to?

Aside from Question Generation task, the data creators and curators used this data for Question Answering, and QA-Pair Generation tasks, and to identify social stereotypes represented in story narratives.

Maintenance Plan Details

Describe the original dataset's maintenance plan.

We plan to host various splits for the FairytaleQA dataset to better serve various types of research interests. We have the original data for 2 different split approaches including train/validation/test splits and split by fairytale origins. We are also plan to host the dataset on multiple platforms for various tasks.

Social Impact Observations

Did any of these previous uses result in observations about the social impact of the systems? In particular, has there been work outlining the risks and limitations of the system? Provide links and descriptions here.

[N/A]

Details on how Dataset Addresses the Needs

Describe how this dataset addresses the needs of underserved communities.

From the educational perspective, given that reading comprehension is a multicomponent skill, it is ideal for comprehension questions to be able to identify students’ performance in specific sub-skills, thus allowing teachers to provide tailored guidance.

Potential PII Risk

Considering your answers to the PII part of the Data Curation Section, describe any potential privacy to the data subjects and creators risks when using the dataset.

[N/A]

Technical Limitations

Describe any known technical limitations, such as spurrious correlations, train/test overlap, annotation biases, or mis-annotations, and cite the works that first identified these limitations when possible.

We noticed that human results are obtained via cross-estimation between the two annotated answers, thus are underestimated. One possibility for future work is to conduct a large-scale human annotation to collect more answers per question and then leverage the massively annotated answers to better establish a human performance evaluation.

Unsuited Applications

When using a model trained on this dataset in a setting where users or the public may interact with its predictions, what are some pitfalls to look out for? In particular, describe some applications of the general task featured in this dataset that its curation or properties make it less suitable for.

The QA-pairs annotated by education experts are targeting the audience of children from kindergarten to eighth grade, so the difficulty of QA-pairs are not suitable to compare with other existing dataset that are sourced from knowledge graphs or knowledge bases like Wikipedia.

Discouraged Use Cases

What are some discouraged use cases of a model trained to maximize the proposed metrics on this dataset? In particular, think about settings where decisions made by a model that performs reasonably well on the metric my still have strong negative consequences for user or members of the public.

[N/A]