GEM Workshop at EMNLP 2023

The Third Version of the Generation, Evaluation & Metrics (GEM) Workshop will be held as part of EMNLP, December 6-10, 2023.

Overview

Many new NLP applications are cast through the lens of natural language generation. With the advent of these new approaches, many opportunities arise: generation in previously less studied languages, new evaluation paradigms, methods for corpus creation, more efficient architectures, strategies for safe deployments, among many others. At the same time, we can learn from the rich history of NLG research to further improve generation methods. These developments require robust and sound NLG evaluation processes. To that end, the GEM workshop aims to encourage the development of model auditing and human evaluation strategies, and to popularize model evaluations in languages beyond English.

We welcome submissions related, but not limited to, the following topics:

  • 💎 Automatic evaluation of generation systems (example, example, example)
  • 💎 Creating NLG corpora and challenge sets (example, example, example)
  • 💎 Critiques of benchmarking efforts and responsibly measuring progress in NLG (example, example)
  • 💎 Effective and/or efficient NLG methods that can be applied to a wide range of languages and/or scenarios (example, example, example)
  • 💎 Application and evaluation of generation models interacting with external data and tools (example, example, example)
  • 💎 Sociotechnical perspectives of employing large language models (example)
  • 💎 Standardizing human evaluation and making it more robust (example, example, example)

We further invite submissions that conduct in-depth analyses of outputs of existing systems, for example through error analyses, by applying new metrics, or by testing the system on new test sets. While we encourage the use of the infrastructure the organizing team has developed as part of the GEM benchmark, its use is not required.

If you are interested, you can check out last year's workshop websites from ACL 2021 and EMNLP 2022.

Industrial Track - Unleashing the Power of NLP: Bridging the Gap between Academia and Industry

GEM 2023 is proud to announce the launch of its Industrial Track, which aims to provide actionable insights to industry professionals and to foster collaborations between academia and industry. This track will address the unique challenges faced by non-academic colleagues, highlighting the differences in evaluation practices between academic and industrial research, and explore the challenges in evaluating generative models with real-world data.

The Industrial Track invites submissions covering the following topics, including (but not limited to):

  • 💎 Breaking Barriers: Bridging the Gap between Academic and Industrial Research (example)
  • 💎 From Data Diversity to Model Robustness: Challenges in Evaluating Generative Models with Real-World Data (example)
  • 💎 Beyond Metrics: Evaluating Generative Models for Real-World Business Impact (example, example, example)

How to submit?

Submissions can take either of the following forms:

  • 💎 Archival Papers Papers describing original and unpublished work can be submitted in a between 4 and 8 page format.
  • 💎 Non-Archival Abstracts To discuss work already presented or under review at a peer-reviewed venue, we allow the submission of 2-page abstracts.

All submissions are allowed unlimited space for references and appendices and should conform to EMNLP 2023 style guidelines. Archival paper submissions must be anonymized while abstract submissions may include author information.

You can submit directly through SoftConf. Please select the track you are submitting to during the submission.

We additionally welcome presentations by authors of papers in the Findings of the EMNLP. The selection process is managed centrally by the workshop chairs for the conference and we thus cannot respond to all individual inquiries. However, we will try our best to accomodate your requests.

Shared Task

We are organizing a shared task focused on multilingual summarization, including human and automatic evaluation. The Shared Task will be run "Backwards": the workshop will serve as a platform to pre-register your hypotheses. More info on how to participate to come!

Important Dates

Note: For any questions, please email gem-benchmark-chairs@googlegroups.com.

Paper Submission Dates

  • 📅 8 September 2023: Workshop paper submission deadline
  • 📅 20 October 2023: Workshop paper notification deadline
  • 📅 3 November 2023: Workshop paper camera ready deadline

Note The website showed wrong dates for notication and CR deadlines. Apologies for any inconvenience.

Workshop Dates

  • 📅 6 December 2022: Workshop

Organization

Contact: gem-benchmark-chairs@googlegroups.com

General Chairs

Khyathi Raghavi Chandu (AI2)

Elizabeth Clark (Google Deepmind)

Kaustubh Dhole (Emory University)

Sebastian Gehrmann (Bloomberg)

João Sedoc (NYU)

Alex Wang (Cohere)

Industry Track Chairs

Enrico Santus (Bloomberg)

Hooman Sedghamiz (Bayer)