Rameswar Panda

Research

Most of what I work on tries to answer a simple question: how much capability can you get out of a given amount of compute? In practice that means building open models, finding architectures that are cheaper to train and serve, and working out training recipes and data that hold up at scale.

Granite: We started with Granite Code in 2024, a family of open code models from 3B to 34B parameters trained on 116 programming languages, and later extended them to 128K context. Since then the team has shipped general-purpose models through Granite 3.0 to 3.3, adding long context and reasoning that can be switched on or off. Granite 4.0 moved to a hybrid Mamba-2/Transformer architecture and came with Nano models as small as 350M. With Granite 4.1 we went back to dense models, where the 8B matches our previous 32B mixture-of-experts, and Granite 4.2 added built-in reasoning and tool calling. Everything is released under Apache 2.0 on Hugging Face, and Granite received the highest score in the history of Stanford's Foundation Model Transparency Index (95/100 in 2025).

Efficient architectures: Together with collaborators at MIT, we have spent a lot of time on alternatives to standard softmax attention. This includes Gated Linear Attention, PaTH attention for position encoding, stick-breaking attention, cross-layer attention for shrinking the KV cache, and dynamic short convolutions. We have also worked on distilling Transformers into hybrid attention models. On the sparse side, we have looked at training mixture-of-experts models densely and released ScatterMoE, a fast MoE implementation.

Training recipes: Hyperparameters that work at one scale often break at another. The Power Scheduler is a learning-rate schedule that does not need to be retuned when batch size or token count changes. On the data side, CodeAlchemy rewrites real code into synthetic training data that teaches more per token than the original. We have also studied which data matters for long context, and how much model size implicit reasoning needs. PRISM looks at mid-training across Granite, LLaMA, Mistral, and Nemotron-H models: a short mid-training phase on about 27B high-quality tokens is what makes later reinforcement learning work, lifting reasoning scores three to four times over applying RL to the base model directly.

Agents: Models are only as good at tool use as the data they learn from, and good data is scarce. Toucan is an open dataset of 1.5M tool-use trajectories collected from real MCP environments, and API Pack covers API calls across programming languages. Earlier we trained Granite-20B-FunctionCalling.

Vision (earlier work): Before moving to language models I worked on efficient computer vision, including the vision transformers CrossViT and RegionViT, adaptive computation for video such as AR-Net, and multi-task learning with AdaShare.

Open source

Most of our work is released openly. Some of the projects people use the most:

Publications

The full list of my publications is on Google Scholar. Here is a short selection:

Press

Service