SELECT * FROM DAFc WHERE DAFc="UC4R1IsRVKs_qlWKTm9pT82Q"
Julia Kempe - Synthetic Data – Friend or Foe in the Age of Scaling?
· 26.05.2024 · 20:32:29 ··· Sonntag ⭐ 32 🎬 1072
📺 Institut des Hautes Etudes Scientifiques (IHES)
As AI and LLM model size grows, neural scaling laws have become a crucial tool to predict the improvements of large models when increasing capacity and the size of original (human or natural) training data. Yet, the widespread use of popular models means that the ecosystem of online data and text will co-evolve to progressively contain increased amounts of synthesized data.
In this talk we ask: How will the scaling laws change in the inevitable regime where synthetic data makes its way into the training corpus? Will future models, still improve, or be doomed to degenerate up to total (model) collapse? We develop a theoretical framework of model collapse through the lens of scaling laws. We discover a wide range of decay phenomena, analyzing loss of scaling, shifted scaling with number of generations, the ''un-learning" of skills, and grokking when mixing human and synthesized data. Our theory is validated by large-scale experiments with a transformer on an arithmetic task and text generation using the LLM Llama2.
Julia Kempe (NYU Center for Data Science and Courant Institute of Mathematical Sciences)
===
Find this and many more scientific videos on https://www.carmin.tv/ - a French video platform for mathematics and their interactions with other sciences offering extra functionalities tailored to meet the needs of the research community.
===
· 26.05.2024 · 20:32:29 ··· Sonntag
U
U
L
L
T
* 1716748349
* 1716748349
X 1072
Y 32
P
C 19762
B 7
V 90
* · 28.07.2022 · 00:00:00 ···
2 · 30.04.2023 · 05:39:01 ···
L · 30.06.2024 · 10:08:39 ···
C · 05.07.2024 · 10:08:42 ···
💘 🖱️ * · 01.01.1970 · 01:00:00 ···
* · 01.01.1970 · 01:00:00 ···
· 01.01.1970 · 01:00:00 ···
**##
*** · 01.01.1970 · 01:00:00 ··· ::
*2* · 01.01.1970 · 01:00:00 ··· ::
*L* · 01.01.1970 · 01:00:00 ··· ::
*C* · 01.01.1970 · 01:00:00 ··· ::
********
