An interactive walkthrough of a KL loss kernel that computes exact distillation losses and gradients chunk by chunk, without storing the vocabulary-sized logits: 11.6× less memory and 3.3× faster at 256K tokens.
Iker García-Ferrero. Based on joint work with Bakbergen Ryskulov, David Montero, David Jansen, Ali Hashemi, Jezabel R. Garcia, Antonio Tiene, Román Orús.
We study how to train LLMs to refuse harmful requests within a topic while still answering legitimate ones. Self-generated training data and harmful-benign prompt pairs help control that boundary, and evaluating both sides reveals the trade-off between safety and over-refusal.
Antonio Tiene, Alejo López-Ávila, Iker García-Ferrero.
How offline teacher logits and a fused KL loss make distillation more efficient in memory and compute, with benchmarks and an open-source implementation.
Antonio Tiene, Iker García-Ferrero, Ali Hashemi, Bakbergen Ryskulov.
The training process behind FLUX.1 Krea, an open image-generation model developed with Black Forest Labs, with a focus on aesthetics and photorealism.
Sangwu Lee, Titus Ebbecke, Erwann Millon, Will Beddow, Le Zhuo, Iker García-Ferrero, Liam Esparraguera, Mihai Petrescu, Gian Saß, Gabriel Menezes, Victor Perez
We present GoLLIE, a Large Language Model trained to follow annotation guidelines. It outperforms previous approaches on zero-shot IE and supports schemas defined on the fly.
Oscar Sainz, Iker García-Ferrero, Rodrigo Agerri, Oier Lopez de Lacalle, German Rigau and Eneko Agirre