Discussion about this post

User's avatar
Zac Hill's avatar

Love these kinds of succinct but deep-cut summaries. How much is known either mechanically or tacitly about the drivers behind why the OAI models scale quadratically on long contexts? Like in other words have people derived or inferred many of the relevant distinctions in the architecture?

No posts

Ready for more?