NVIDIA’s Fast AI Agentic Model

Long-running AI agents often spend most of their time on routine execution rather than difficult reasoning. After making a plan, they may perform hundreds of tool calls, file reads, validations, commands, and formatting steps, so using a frontier reasoning model for every action can become unnecessarily slow and expensive.

NVIDIA’s Nemotron 3.5 Lightning takes a different approach: a fast, efficient model designed for high-volume agent execution. The idea is simple: use the expensive model to think and the fast model to work. In this article, we examine whether that architecture can reduce cost without sacrificing agentic performance.

What is NVIDIA Nemotron 3.5 Lightning?

Furthermore, NVIDIA Nemotron 3.5 Lightning is an open-weight reasoning and instruction model designed primarily for the execution layer of agentic systems.

Its core specifications are:

NVIDIA’s official NVFP4 model card also lists single-GPU deployment on a DGX Spark GB10 or H100, with support spanning Blackwell, Hopper and Ampere hardware depending on quantization.

The model is primarily intended for English and programming languages, while Spanish, French, German, Italian and Japanese are also officially supported.

This is important because Nemotron 3.5 Lightning should not be evaluated as simply “another 30B model.”

Of course, its intended job is much more specific.

Why NVIDIA Built an Execution-Focused Model

Consider a coding agent.

It may first need to understand a bug and develop a plan. That is a difficult reasoning problem.

But after the plan exists, the agent may need to:

The first step may deserve a frontier model.

Do all the others?

Probably not.

NVIDIA argues that long-running agents spend a substantial portion of their workloads on exactly these high-volume execution operations, such as tool calls, validation and delegation. Using a frontier reasoning model for every execution step increases both cost and latency.

In short, Nemotron 3.5 Lightning is NVIDIA’s answer.

A possible production architecture becomes:

Next, this changes how we should think about model selection.

Instead of asking:

Finally, which single model should power my agent?

the more useful question becomes:

Similarly, which model should handle each type of work inside my agent?

That is the architectural idea behind Lightning.

Architecture Deep Dive

Meanwhile, Nemotron 3.5 Lightning uses one of the more interesting architectures among current smaller agent models.

NVIDIA describes it as a hybrid:

Mamba-2
   +
Mixture-of-Experts
   +
Selective Attention
   +
Multi-Token Prediction

The combination matters because each component solves a different efficiency problem.

1. Mixture-of-Experts: 30B Parameters, Only 3B Active

Nemotron 3.5 Lightning contains approximately 30 billion total parameters but activates only around 3 billion for each token.

In a dense 30B model, essentially the whole network participates in inference.

In an MoE model:

On the other hand, the router chooses only a small subset of experts.

You therefore retain much of the representational capacity of a larger model while doing computation closer to a significantly smaller model.

That is central to Lightning’s throughput advantage.

Published runtime configuration also exposes 128 routed experts plus a shared expert, with six routed experts selected per token. The configuration contains 52 hidden layers. Its hybrid layer pattern resolves to Mamba, MoE and sparse Attention components rather than using full self-attention at every layer. These are implementation-level configuration details, so developers should verify them against the exact checkpoint and runtime they deploy.

2. Mamba-2 Layers

Traditional Transformers rely heavily on attention.

Attention is extremely powerful, but long sequences become computationally expensive.

Although Mamba is based on state-space modeling and can process sequences more efficiently.

Nevertheless, Nemotron 3.5 Lightning does not abandon attention entirely. Instead, NVIDIA uses Mamba-2 for much of the sequence processing while preserving selected Attention layers where global token interaction remains valuable.

Conceptually:

This hybrid design is particularly relevant for long-context agents.

Instead of paying full attention costs throughout the entire network, the model mixes mechanisms optimized for different jobs.

3. Selective Attention

Attention is still important when tokens must directly compare information across distant parts of the sequence.

That matters for:

• long documents
• source-code repositories
• multi-step tool trajectories
• conversation history
• retrieved documents
• agent memory

Nemotron therefore keeps selected attention layers instead of switching to a pure state-space architecture.

The architectural philosophy is not “Mamba instead of Transformer.”

It is use expensive global attention only where it adds sufficient value.

4. Multi-Token Prediction

Normal autoregressive LLMs learn:

Token 1 → predict Token 2
Token 2 → predict Token 3
Token 3 → predict Token 4

Nemotron 3.5 Lightning includes Multi-Token Prediction, or MTP, layers that learn to predict multiple future tokens during training. Moreover, NVIDIA added a dedicated continued-pretraining stage for these MTP layers.

MTP improves training signals, but it also becomes useful during inference.

Instead of proposing only:

next token

the system can speculate about:

token t+1
token t+2
token t+3
...

Those candidates can then be verified efficiently.

As a result, this is one of the mechanisms behind Lightning’s high generation throughput.

Why Is Nemotron 3.5 Lightning So Fast?

On the other hand, its speed does not come from one optimization. It is the combination of several.

The best option therefore depends on concurrency.

There is no universally fastest configuration.

• MoE Sparsity: 30B parameters provide capacity, but only about 3B are active.
• In short, Hybrid Mamba Architecture: Mamba reduces the need to perform full attention across every layer.
• Furthermore, NVFP4 Quantization: Lower-precision inference reduces memory and compute requirements.
• Instead, Multi-Token Prediction: Several future tokens can be proposed together.
• Of course, Speculative Decoding: NVIDIA provides three speculative approaches:
• MTP: Integrated directly into the model. NVIDIA recommends it particularly for medium to high concurrency.
• In particular, DSpark: A dedicated draft model optimized for DGX Spark and lower-concurrency data-center inference.
• As a result, DFlash: An additional draft model that developers can benchmark against MTP and DSpark for their workload.

While Nemotron 3.5 Lightning combines strong intelligence with up to 4x output speed of similar-sized models, placing it on the accuracy-speed Pareto frontier for high-volume agent workloads.

NVIDIA Nemotron 3.5 Lightning Benchmark Results

NVIDIA publishes both BF16 and NVFP4 results across knowledge, reasoning, coding, agents, instruction following and long context.

In fact, the important observation is that quantization does not dramatically collapse model quality.

Here are the official reported results. Benchmark-native units are preserved, so not every value should be interpreted as a percentage.

Moreover, NVIDIA says these evaluations were run through a consistent NeMo Gym and NeMo Evaluator-based harness and has published benchmark recipes for reproducibility.

In contrast, an interesting result is how close NVFP4 remains to BF16.

Nemotron 3.5 Lightning Pricing

However, pricing is slightly more complicated than a single number because the model is open-weight and available through multiple routes.

The following reflects publicly listed pricing on August 12, 2026.

Next, NVIDIA currently offers a free API endpoint for prototyping through build.nvidia.com.

Meanwhile, OpenRouter lists both a free Nemotron 3.5 Lightning route with a 1M context and a standard route currently priced at $0.05 per million input tokens and $0.20 per million output tokens. The standard OpenRouter route currently advertises a 262K context rather than the full 1M model capability.

Finally, Fireworks currently lists exactly $0.05 per million input tokens, $0.01 per million cached input tokens and $0.20 per million output tokens, with a 262K serverless context window.

Pricing and context limits can change quickly, particularly during the first weeks after a model release.

How to Access NVIDIA Nemotron 3.5 Lightning

First, at launch, there are already several practical ways to use the model.

Option 1: NVIDIA API

1. Go to https://build.nvidia.com/ and login or sign up
2. Click on your profile picture and then API keys.
3. Generate a new API key.
4. Now use this API for inference.

Option 2: Ollama

1. Install Ollama in your system from
2. Run the following command in terminal to download and run Nemotron 3.5 lightening locally.

ollama run nemotron-3.5-lightning”

Option 3: OpenRouter

You can also use OpenRouter to run this model. Of course, its listed as a Free model on OpenRouter. Instead, grab an API key and start to use it

Hands-on: Using Nemotron 3.5 Lightning Through NVIDIA API

Nevertheless, NVIDIA exposes the model through an OpenAI-compatible endpoint. The official example uses nvidia/nemotron-3.5-lightning-30b-a3b.

Install the client:

pip install openai

Set your API key:

export NVIDIA_API_KEY="your_api_key"

Now create a simple request:

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://integrate.api.nvidia.com/v1",
    api_key=os.environ["NVIDIA_API_KEY"]
)

response = client.chat.completions.create(
    model="nvidia/nemotron-3.5-lightning-30b-a3b",
    messages=[
        {
            "role": "user",
            "content": """
            A customer has submitted a warranty claim.

            Purchase date: 2025-04-12
            Claim date: 2026-03-02
            Warranty duration: 12 months
            Damage type: manufacturing defect

            Determine whether the claim is within the warranty period.
            Return JSON with:
            decision
            rationale
            """
        }
    ],
    temperature=1.0,
    top_p=0.95,
    max_tokens=2000,
    extra_body={
        "chat_template_kwargs": {
            "enable_thinking": True
        },
        "reasoning_budget": 4000
    }
)

print(response.choices[0].message.content)

Output:

{
"decision": "approved",
"rationale": "The warranty period begins on the purchase date of 2025-04-12 and lasts for 12 months, ending on 2026-04-12. The claim was submitted on 2026-03-02, which falls within the active warranty period. Additionally, the damage is listed as a manufacturing defect, which is typically covered under standard warranty terms."
}

• This is a better first test than asking: Write a poem about AI.
• Nemotron 3.5 Lightning is designed for structured agent workloads, so test it accordingly.

Conclusion

NVIDIA’s main argument is that future production AI systems may depend less on a single giant model and more on a coordinated architecture of planners, routers, specialized workers, fast execution models, and verification layers. This represents a shift from maximizing model size to optimizing how different models work together.

In that architecture, Nemotron 3.5 Lightning does not need to be the smartest model available. Its value comes from being efficient, fast, and capable enough to handle most routine agent tasks while recognizing when harder work should be escalated. NVIDIA is therefore optimizing for practical, scalable agent execution rather than simply competing for the largest or most intelligent model.

Frequently Asked Questions

Harsh Mishra is an AI/ML Engineer who spends more time talking to Large Language Models than actual humans. Passionate about GenAI, NLP, and making machines smarter (so they don’t replace him just yet). When not optimizing models, he’s probably optimizing his coffee intake. 🚀☕