Just as pre-trained Large Language Models (LLMs) like GPT-4 have revolutionized the field of AI with their remarkable capabilities in natural language understanding and generation, LLM-powered systems also see great potential for contributing to the wellbeing of our society through public sector applications, which often feature governmental, publicly-funded and non-profit organizations. This workshop brings together researchers and practitioners from AI, HCI, and social sciences to present their research around LLM-powered tools in the context of public sector applications.

Our workshop's target audience consists of

Important Dates

Be mindful of the following dates:

Note: all deadlines are AoE (Anywhere on Earth).

Invited Speakers

Malihe Alikhani
Malihe Alikhani
Northeastern University

Malihe Alikhani is an assistant professor of AI and social justice at the Khoury School of Computer Science, Northeastern University. Her research goal is to design inclusive and equitable language technologies. This involves developing systems that communicate effectively with diverse populations, especially those from underserved communities. By integrating insights from cognitive science and social sciences with machine learning, she designs computational models that capture the diversity of interpretation and reflect the success of language as a communicative tool. She aspires to discern and address biases in learning models, especially in applications such as education, health, and social justice. This aspiration has led her to collaborate with educators, healthcare experts, and community leaders to create inclusive technology-enabled experiences.

Talk title: Towards Inclusive and Equitable Language Technologies

Talk abstract: With the increasing deployment of language technologies to users, understanding the impact of natural language processing models on society and behaviors is critical. We aim to design culturally responsible, equitable, and inclusive technologies that serve a diverse population. This talk provides an overview of our projects in democratizing Large Language Models, focusing on making language technologies accessible and equitable for education, health, and social justice. We explore two case studies: 1) developing discourse-aware models for inclusive social media moderation, and 2) creating equitable machine learning frameworks for multimodal communication.

Kevin Leyton-Brown
Kevin Leyton-Brown
University of British Columbia

Kevin Leyton-Brown is a professor of Computer Science and a Distinguished University Scholar at the University of British Columbia. He also holds a Canada CIFAR AI Chair at the Alberta Machine Intelligence Institute and is an associate member of the Vancouver School of Economics. He received a PhD and an M.Sc. from Stanford University (2003; 2001) and a B.Sc. from McMaster University (1998). He studies artificial intelligence, mostly at the intersection of machine learning and (1) the design and operation of electronic markets and (2) the design of heuristic algorithms.

Talk title: Rationality Report Cards: Assessing the Economic Rationality of Large Language Models

Talk abstract: There is increasing interest in using LLMs as decision-making ``agents''. Doing so includes many degrees of freedom: which model should be used; how should it be prompted; should it be asked to introspect, conduct chain-of-thought reasoning, etc? Settling these questions---and more broadly, determining whether an LLM agent is reliable enough to be trusted---requires a methodology for assessing such an agent's economic rationality. In this paper, we provide one. We begin by surveying the economic literature on rational decision-making, taxonomizing a large set of fine-grained ``skills'' that an agent should exhibit, along with dependencies between them. We then propose a benchmark distribution that quantitatively scores these skills and, combined with a user-provided rubric, produces ``rationality report card''. Finally, we describe the results of a large-scale empirical experiment with 21 different LLMs, characterizing the both current state of the art and the impact of different model sizes on models' ability to exhibit rational behavior.

Niloufar Salehi
Niloufar Salehi
UC Berkeley

Niloufar Salehi is an assistant professor in the School of Information at UC, Berkeley. She studies human-computer interaction, with her research spanning education to healthcare to restorative justice. Her research interests are social computing, participatory and critical design, human-centered AI, and more broadly, human-computer interaction (HCI). Her work has been published and received awards in premier venues including ACM CHI and CSCW and has been covered in Venture Beat, Wired, and the Guardian. She is a W. T. Grant Foundation scholar. She received her PhD in computer science from Stanford University in 2018.

Diyi Yang
Diyi Yang
Stanford University

Diyi Yang is an Assistant Professor in Computer Science at Stanford University. Professor Yang's research interests are Computational Social Science and Natural Language Processing. Her research goal is to understand the social aspects of language and then build socially aware NLP systems to better support human-human and human-computer interaction. Professor Yang received her PhD from the School of Computer Science, Carnegie Mellon University, and her bachelor's degree from Shanghai Jiao Tong University, China. Her work has received multiple best paper nominations or awards at ICWSM, EMNLP, SIGCHI, ACL, and CSCW. She is a recipient of Forbes 30 under 30 in Science, IEEE “AI 10 to Watch”, the Intel Rising Star Faculty Award, Microsoft Research Faculty Fellowship, and NSF CAREER Award.

Program

8:45am - 9:00am Opening Remarks
Contributed Talks Session 1
9:00am - 9:15am Continued pre-training of LLMs for Portuguese and Government domain: A proposal for product identification in textual purchase descriptions
Eduardo Soares de Paiva, Fernando Sola Pereira, David da Guia Carvalho, Nilson Romero Michiles Junior, Rennis Sousa de Oliveira, Stella Mendes Meireles Bonifacio, Andre Luiz Monteiro da Rocha, Hamilton Luiz Rodrigues de Oliveira, Felipe de Abreu Moreira Cezar, Helio Theodoro Junior
9:15am - 9:30am Using a Large Language Model to Choose Effective Climate Change Messages
Thomas Benchetrit, Iris Kremer, Erik Hemberg, Aruna Sankaranarayanan, Una-May O'Reilly
9:30am - 10:30am Invited Talk: Malihe Alikhani
10:30am - 11:00am Coffee Break
11:00am - 11:30am Invited Talk: Diyi Yang
Contributed Talks Session 2
11:30am - 11:45am An Approach for Targeted Analysis of Topic Portrayals in Public Service Media Corpora
Kristen Marie Scott, Felix Mercer Moss
11:45am - 12:00pm SAGE: System for Accessible Guided Exploration of Health Information
Sabriya Maryam Alam, Haodi Zou, Reya Vir, Niloufar Salehi
12:00pm - 12:15pm Where It Really Matters: Few-Shot Environmental Conservation Media Monitoring for Low-Resource Languages
Sameer Jain, Sedrick Scott Keh, Shova Chhetri, Karun Dewan, Pablo Izquierdo, Johanna Prussmann, Pooja Shrestha, César Suárez, Zheyuan Ryan Shi, Lei Li, Fei Fang
12:15pm - 12:30pm LLM-Assisted Modeling and Simulations for Public Sector Decision-Making: Bridging Climate Data and Policy Insights
Charles Cao, Jie Zhuang, Qiang He
12:30pm - 2:00pm Lunch Break & Poster Session
2:00pm - 3:00pm Invited Talk: Kevin Leyton-Brown
3:00pm - 3:30pm Invited Talk: Niloufar Salehi
3:30pm - 4:00pm Coffee Break
Contributed Talks Session 3
4:00pm - 4:15pm The Case for Animal-Friendly LLMs
Sankalpa Ghose, Tse Yip Fai, Kasra Rasaee, Jeff Sebo, Peter Singer
4:15pm - 4:30pm InstructPart: Affordance-based Part Segmentation from Language Instruction
Zifu Wan, Yaqi Xie, Ce Zhang, Zhiqiu Lin, Zihan Wang, Simon Stepputtis, Deva Ramanan, Katia P. Sycara
4:30pm - 4:45pm Try That in a Small Town: Large Language Models for Modest Municipalities
David Huggins-Daines
4:45pm - 5:00pm Designing LLM-Based Support for Homelessness Caseworkers
Whitney Nelson, Min Kyung Lee, Eunsol Choi, Victor Wang
5:00pm - 5:15pm Heterogeneous Value Alignment Evaluation for Large Language Models
Zhaowei Zhang, Ceyao Zhang, Nian Liu, Siyuan Qi, Ziqi Rong, Song-Chun Zhu, Shuguang Cui, Yaodong Yang
5:15pm Closing Remarks

Call for Papers

We invite papers from the broad intersection of LLM-powered systems and public sector applications, described in the following two tracks. Submissions that simultaneously touch on both tracks are also encouraged.

Track 1: Developing LLM-powered tools for positive outcomes.

This track welcomes use-inspired research that develops LLM-powered tools to make positive real-world impact in the public sector. Application domains include, but are not limited to, education, urban planning, public health, agriculture, environmental sustainability, and social welfare and justice. We also welcome submissions that tackle the common challenges in public sector LLM research, such as: costly data collection, significant problem scoping process with domain experts, impact evaluation, high-stake domains yet far less resources (budget, allocated staff effort, AI expertise), and vulnerable stakeholders. We particularly encourage submissions that report on practically realizing these LLM-based systems in the real world, success stories and lessons learned.

Track 2: Designing, deploying and evaluating LLM-powered services with directly impacted communities.

This track invites submissions that investigate how LLM-powered systems fit (or misfit) into the human organizations that implement them, as well as how those systems might directly impact community stakeholders who are on the receiving end of those services. Topics of interest include, but are not limited to, participatory design on LLM-based tools/systems, understanding public sector’s needs and challenges of integrating LLMs, interfacing LLMs with public sector organizations and end users, human oversight of LLMs, the role of LLMs in social justice and equity. We particularly encourage submissions from HCI and social science communities, as well as public sector organizations.

Both tracks welcome two types of submissions:

All papers must be submitted as PDF documents. Author names and affiliations should be listed on the submission. Additional supplemental materials may be submitted along with the paper. However, the main body of the paper should be self-contained. Submissions violating these formatting requirements will be desk-rejected. The AAAI-24 Latex/Word template guideline can be found here.

Submission should be made through OpenReview. This workshop is non-archival. All accepted papers will be uploaded to the workshop website. If any author do not wish to make their camera-ready versions public (because of some conflict of interest), please reach out to the workshop chairs and we would be happy to accommodate such requests. Rejected papers will remain private and not posted in public.

Organization

Ryan Shi
Ryan Shi
University of Pittsburgh
Hong Shen
Hong Shen
Carnegie Mellon University
Sera Linardi
Sera Linardi
University of Pittsburgh
Lei Li
Lei Li
Carnegie Mellon University
Fei Fang
Fei Fang
Carnegie Mellon University

For any enquiry please contact publlm@googlegroups.com