Twishmay Shankar’s Post

Recent research highlights how the density of self-attention graphs within llm models significantly impacts their expressive power. Key Insights: ... Intrinsic Dimension Matters: A higher intrinsic dimension of input boosts the model's expressive capacity. ... Self-Attention is Crucial: Increasing the number of attention heads and context length enhances reasoning capabilities. ... Adaptive Partitioning: LLMs adaptively partition their input space during training, leading to better function approximation and improved reasoning. ... Scalable Improvements: Larger models and more extensive context inputs correlate with higher attention density, translating to smarter AI models. https://lnkd.in/gN7kWtVE

The Abstraction and Reasoning Corpus (ARC) challenges AI to exhibit abstract reasoning and adaptability, testing pattern recognition, transformations, and generalization. The o-metric measures task complexity based on grid size, aiding strategy.

The New Ideas Needed for AGI The strength of LLMs is also their weakness. A truly general intelligence may require a certainty in thinking and reasoning that cannot be found in language.

Hard to fathom its only been a decade since this book was released.

Unlike traditional AI tests, which often rely on vast amounts of data and specific tasks, ARC pushes AI to understand and apply abstract concepts, much like how a child learns through experience and intuition.

Unlearning to Learn Experimentation is about deliberate invalidation, tearing down mental walls to uncover new understanding. Embrace discomfort, reframe failure, and cultivate the addiction to discovery for breakthroughs through strategic questioning and consistent wrongness. https://lnkd.in/gW74in4k