Zalo AI contributes a set of Vietnamese language proficiency assessment standards to AI Vietnam

Why does AI Vietnam need a complete set of Vietnamese language proficiency assessment standards?

The explosive growth of GPT chat has created a new race: Generative AI. According to statistics, there are currently about 16,000 models similar to GPT chat in the world. Vietnam is not out of that trend when many research groups also want to experiment with Generative AI using Vietnamese.

This leads to the need for a Vietnamese proficiency assessment set for these AI models to measure the level of knowledge and thinking in Vietnamese.

However, most LLM (Large language models) research groups in Vietnam have to build their own evaluation toolkits with their own standards for their models. These are internal evaluation tools that have not been made public.

Zalo AI's evaluation set is aimed at general needs, can be a common standard for LLM models and is provided to the AI ​​community. This helps research groups access a comprehensive evaluation dataset and allows parties to compare results with each other, thereby creating motivation to improve the model.

Motivation for Vietnamese AI to join the world's Generative AI wave

In November 2023, Zalo AI officially announced the Vietnamese Multitask Language Understanding (VMLU) Vietnamese proficiency assessment standards. This is a set of standards researched and developed by Zalo AI engineers in collaboration with the Japan Advanced Institute of Science and Technology (JAIST) to assess the ability to understand and apply the Vietnamese language of AI models, especially Generative AI.

The birth of VMLU has motivated individuals, startups or research groups to develop new Vietnamese AI models, laying the foundation for measuring the accuracy and upgrading the results of basic models, helping to complete the development process of Vietnamese language AI applications, created by Vietnamese people to serve Vietnamese people.

This is also one of the important factors promoting the development of Generative AI in Vietnam to catch up with the AI ​​wave in the world.

What are the Vietnamese language proficiency assessment standards?

VMLU is a multi-faceted, multi-level Vietnamese language assessment standard set that meets the most diverse needs in the Vietnamese Generative AI research and development market, consisting of two main parts: data (test dataset) and a set of assessment standards, as a basis for testing AI models applying the Vietnamese language.

Specifically, the dataset includes 10,880 multiple-choice questions with 58 topics. Each topic has about 200 questions and is distributed across 4 areas: STEM, Social Sciences, Humanities and a broad category “Expanded”.

With this data set, VMLU has a difficulty stratification into 4 levels: Primary, Secondary, High School and Vocational - for university and postgraduate. From there, the toolkit helps to effectively evaluate the Vietnamese language proficiency of AI models in both elementary and complex knowledge.

To help research teams easily evaluate the capabilities of their Vietnamese AI models, the Zalo AI engineering team has designed detailed instructions with simple operations.

Continue to contribute to the Vietnamese AI community

The VMLU standard set is a product researched with the aim of contributing to and developing the Vietnamese AI research community in particular and the information technology community in general, without charging any users, research groups or businesses.

Previously, Zalo AI has deployed and organized a series of competitions and programs for the Vietnamese AI community such as: Zalo AI Challenge, Zalo AI Hackathon, Zalo AI Summit... These activities not only create a playground for the Vietnamese AI community but also encourage the application of AI in life, solve urgent social problems, and serve the needs of millions of Vietnamese people.

Dr. Chau Thanh Duc - Head of Zalo AI Research Department - Lecturer at the University of Natural Sciences, Ho Chi Minh City National University affirmed: "Zalo AI always aims to contribute to the Vietnamese AI community, creating motivation for Vietnamese AI to develop. From there, we expect more and more AI products by Vietnamese people, for Vietnamese people".