The Jetson model.NVFP4Quantization and speculative decoding were combined, and a configuration was shown to reduce processing load and verification times at the same time.NVIDIAAccording to the measurement,BF16than the largest.6.28I got double decoding.
Run the logic loop locally on Jetson.
The reasoning and agent functionality with a compact open model can now be run locally on the edge AI platform Jetson. According to NVIDIA, the model that appeared in the year 2026 has features that previously required a large-scale data center system that could be executed on Jetson.
The model can be deployed locally using the inference framework vLLM and llama.cpp. The inference loop will not depend entirely on the data center and will be operated by sensors and devices close to the processed system.
Different calculation range per token in Dense and MoE
The volume of processing of local inferences is not determined by the total number of parameters of the model alone.Dense model Qwen3.8-27B enables all 270 billion parameters whenever a token is generated.
The Nemotron 3.5 Lightning of the Mixture of Experts (MoE) model has a total number of parameters of 300 billion, while the 30 billion parameters are activated per token.
NVFP4 quantification reduces load per model run
In normal decoding, the response is generated by each 1 token. There are ways to reduce the load of each execution to speed up the generation, and to increase the amount of production that proceeds with each 1 execution, so that the model can be re-executed every time you move to the next token.
NVFP4Quantization applies to the former, using values of low accuracy, in each execution.GPUReduce the amount of data processed and transferred.NVIDIAThis low accuracy will increase the production speed and reduce memory usage.
Candidate tokens are verified in multiple stages.
Speculative decoding increases the number of tokens that can be verified by 1. A small draft model proposes multiple candidate tokens, and the main model that takes the final decision on output checks the candidate.
If the main model receives multiple candidates, the generator will generate multiple tokens through verification of 1 times.The Jetson measurement of NVIDIA combined this mechanism with NVFP4 and used the fastest configuration in each model to maximize the decoding output of BF16 compared to 6.28.
Select the configuration according to the target model and processing conditions.
The speculative decoding techniques that work with Jetson include MTP, DFlash, and DSpark. The methods of generating and evaluating candidates vary, so the quickest methods and draft checkpoints vary by model.
The decoding output also varies according to the type of workload.NVIDIA requires models and configurations to be verified using a prompt that represents the target application, according to the necessary judgment, tool use, and response patterns.