Note llama's feed forward is a bit different too: self.w2(F.silu(self.w1(x)) * s... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		jorlow on May 28, 2024 \| parent \| context \| favorite \| on: Reproducing GPT-2 in llm.c Note llama's feed forward is a bit different too: `self.w2(F.silu(self.w1(x)) * self.w3(x))` I.e. the nonlinearity is a gate. https://github.com/meta-llama/llama3/blob/14aab0428d3ec3a959...

soraki_soladead on May 29, 2024 [–]

Fwiw, that's SwiGLU in #3 above. Swi = Swish = silu. GLU is gated linear unit; the gate construction you describe.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact