> What the NVIDIA collaboration means, and a look inside the research: how we turned sign language video into tokens a language model can read and write. Blog Rylo News October 21, 2026 19 mins # Rylo partners with NVIDIA to build the future of AI-powered sign language translation What the NVIDIA collaboration means, and a look inside the research: how we turned sign language video into tokens a language model can read and write. ![](https://cdn.sanity.io/images/cp6a9qoi/production/899331851abd8f6517e9646e8fe12ef176837a06-800x800.jpg?w=48&h=48&fit=crop&auto=format) Dr. Amit Moryossef ![Rylo and NVIDIA logos side by side](https://cdn.sanity.io/images/cp6a9qoi/production/db599cc1440351ed3ef5c23e8aac23fd67a1b3a4-2880x1620.png?w=1280&fit=crop&auto=format) - [Why we're working with NVIDIA](https://rylo.com/post/rylo-partners-with-nvidia#why-we-re-working-with-nvidia) - [One consistent video format](https://rylo.com/post/rylo-partners-with-nvidia#one-consistent-video-format) - [Store what changes](https://rylo.com/post/rylo-partners-with-nvidia#store-what-changes) - [Spend fewer bits on the background](https://rylo.com/post/rylo-partners-with-nvidia#spend-fewer-bits-on-the-background) - [Keep the movement as poses](https://rylo.com/post/rylo-partners-with-nvidia#keep-the-movement-as-poses) - [Learn what to keep](https://rylo.com/post/rylo-partners-with-nvidia#learn-what-to-keep) - [Our tokenizer: Group of Tokens](https://rylo.com/post/rylo-partners-with-nvidia#our-tokenizer-group-of-tokens) - [Real-time low-latency infinite video generation model](https://rylo.com/post/rylo-partners-with-nvidia#real-time-low-latency-infinite-video-generation-model) - [Just the beginning](https://rylo.com/post/rylo-partners-with-nvidia#just-the-beginning) Table of Contents - [Why we're working with NVIDIA](https://rylo.com/post/rylo-partners-with-nvidia#why-we-re-working-with-nvidia) - [One consistent video format](https://rylo.com/post/rylo-partners-with-nvidia#one-consistent-video-format) - [Store what changes](https://rylo.com/post/rylo-partners-with-nvidia#store-what-changes) - [Spend fewer bits on the background](https://rylo.com/post/rylo-partners-with-nvidia#spend-fewer-bits-on-the-background) - [Keep the movement as poses](https://rylo.com/post/rylo-partners-with-nvidia#keep-the-movement-as-poses) - [Learn what to keep](https://rylo.com/post/rylo-partners-with-nvidia#learn-what-to-keep) - [Our tokenizer: Group of Tokens](https://rylo.com/post/rylo-partners-with-nvidia#our-tokenizer-group-of-tokens) - [Real-time low-latency infinite video generation model](https://rylo.com/post/rylo-partners-with-nvidia#real-time-low-latency-infinite-video-generation-model) - [Just the beginning](https://rylo.com/post/rylo-partners-with-nvidia#just-the-beginning) ![](https://rylo.com/_astro/hero-sky-background.qaFu5Zgx_Z2c7tg0.webp?dpl=dpl_4eMb1ZnesKCBVtygrnNJDQfT6KhP) No more feeling left out. Rylo lets your voice be heard. Join the thousands of people who use Rylo! [Dr. Amit Moryossef](https://www.linkedin.com/in/amit-moryossef/) Dr. Amit Moryossef is a researcher and entrepreneur in sign-language technology. He completed his Ph.D. at Bar-Ilan University and a postdoc at the University of Zurich. He founded sign.mt, a real-time sign-language translation platform, which was acquired by Rylo, where he now leads research. His work has received multiple best paper awards at ACL and EMNLP, focusing on making signed languages accessible through machine learning. ## Why we're working with NVIDIA Real-time sign language translation needs AI that can watch a person sign, understand them, and sign back. Training models like that takes serious compute. NVIDIA is giving us access to their supercomputer, so we can train bigger models, faster. Less than a year ago, Rylo acquired [sign.mt](https://rylo.com/sign), founded by Dr. Amit Moryossef. Amit now leads Rylo Research, the team doing this work. The team built a compact way to represent sign language video. It's based on NVIDIA's [Cosmos Tokenizer](https://github.com/nvidia-cosmos/cosmos-predict1/blob/main/examples/inference_tokenizer.md) models and plugs into GPT-style language models. The same representation can rebuild video, generate new video, and serve as the base for sign language understanding and translation. Training runs on [NVIDIA NeMo Megatron Bridge](https://docs.nvidia.com/nemo/megatron-bridge/latest/index.html), part of the open NVIDIA NeMo suite. > “Training models that can reliably interpret the nuances of sign language and translate them into text or speech in real time requires significant computing resources. Access to Nvidia compute gives us the ability to train these models at greater scale and precision, and at a much faster pace than was previously possible. But computing power alone is not enough. This progress is made possible by an exceptional team: Ariel Aharonson, who is responsible for developing our tools and infrastructure; Ziv Lazarov and Sarah Bekrar, who are responsible for bringing in high-quality data; and Shaltiel Shmidman, who brings deep experience in training models at scale.” *Dr. Amit Moryossef, Head of Research, Rylo* The rest of this post is the research team's account of how it works, from classic video compression to a model that generates sign language video in real time. It gets technical, and the videos do most of the explaining. We can preserve a conversation as video, audio, or text. Each keeps different details. A recording often captures more than we need to understand what was said, so what should we keep? That depends on what we want to preserve. We call the relevant information signal, and the rest noise. Consider Hamlet's “to be or not to be” as audio and text. The text keeps the words in far fewer bytes. Notice what it leaves out: the voice, rhythm, and intonation. ![Audio waveform of the Hamlet soliloquy beside its text, with file sizes](https://cdn.sanity.io/images/cp6a9qoi/production/7e2dd54b2231679ffd7b452271eed506c75efb54-1600x400.png?w=740&fit=crop&auto=format) *The same words, different details. This audio file is roughly 1,475× larger than the text version. Audio from berniegourley.com.* For signing, handshapes, facial expressions, body movement, and timing carry meaning. These are signal we need to preserve. The signer's exact appearance and the room behind them aren't needed to understand the signing. They matter only if we want to reconstruct the entire video. Better compression comes from better understanding: knowing what carries meaning, what we can leave out, and what we can predict. That helps us separate signal from noise and encode the signal in fewer bits. ### Why associate compression with understanding? Predictable data needs fewer bits, an idea formalized by [Shannon's entropy (1948)](https://onlinelibrary.wiley.com/doi/10.1002/j.1538-7305.1948.tb00917.x). The [Hutter Prize](http://prize.hutter1.net/) explores this connection, and [Huang et al. (2024)](https://arxiv.org/abs/2404.09937) found that text compression correlated closely with average benchmark scores across the language models they studied. That motivates learning representations; it does not make a small video file proof of understanding. For the prediction intuition, see 3Blue1Brown's [Reinventing Entropy](https://www.youtube.com/watch?v=l6DKRf-fAAM). How compact can we make a video while keeping the signing intact? We'll start with general video compression, then focus on representations built for signing. The goal is a representation compact enough for downstream sign language understanding, while preserving the details that carry meaning. Maybe it could even support video generation? Let's explore! ## One consistent video format Sign language videos come from phones, webcams, and other cameras, in portrait, landscape, and square formats. To simplify our work, we'll use one format: a **1:1** aspect ratio, made with a center crop. Here's what that crop keeps from each recording shape: ![Seven recording aspect ratios from 9:16 to 16:9, with the 1:1 square highlighted](https://cdn.sanity.io/images/cp6a9qoi/production/d7940a6a588e7a518eb452ab56c9f9ba54b14fa1-1600x282.png?w=740&fit=crop&auto=format) In our centered, single-signer clip, the hands, face, and torso fit inside a square crop. Other recordings may need a wider or tracked crop to preserve the signing space. The compression comparisons use one [PopSign](https://www.popsign.org/) clip: the ASL sign *"TV"*, 53 frames @ 29.97 fps, center-cropped to **1:1**. Following the same clip lets us see what each representation loses. It is an example, not a benchmark. Unless noted otherwise, video re-encodings use `libx264` at **CRF 19**. The encoder setting stays fixed; the resulting visual quality can still differ. ![The source PopSign clip beside its 1:1 center crop](https://cdn.sanity.io/images/cp6a9qoi/production/ee607e10054609c69a9612a0fb0970a82bb6623d-1600x1094.png?w=740&fit=crop&auto=format) ## Store what changes Video is predictable. Most pixels look a lot like the pixels beside them; most frames look a lot like the frame before. Traditional video codecs exploit that redundancy, storing what changes and referring back to what doesn't. In H.264, for example, the codec stores an occasional I frame, a picture that can be decoded without other pictures. A P frame predicts from earlier reference pictures and stores motion information and corrections; B frames can use references from both directions. These pictures form a [**group of pictures (GOP)**](https://docs.nvidia.com/video-technologies/video-codec-sdk/13.0/nvenc-video-encoder-api-prog-guide/index.html). Here is the GOP in our H.264 clip, stored in an MP4 file: ![Top: the clip's I frame (frame 0, 256 × 256) and four of its P frames, visualized as differences to the previous frame (amplified 4×, small differences suppressed)](https://cdn.sanity.io/images/cp6a9qoi/production/7ad1733f4330d5264f487178a9f7b4633965025a-1704x824.png?w=740&fit=crop&auto=format) *Top: the clip's I frame (frame 0, 256 × 256) and four of its P frames, visualized as differences to the previous frame (amplified 4×, small differences suppressed). These illustrate what changes, rather than the codec's actual motion-compensated residuals. Bottom: the actual bytes H.264 spends on every frame of the file, colored by frame type.* This clip stores one I frame at **11.1 KB**. Its 16 P frames average **4.4 KB**, and the 36 B frames average **974 B**: 12× smaller than the I frame. A full picture is expensive; predictions and corrections cost much less. To make the file smaller still, we can discard detail. We will track the loss with [PSNR](https://www.mathworks.com/help/vision/ref/psnr.html) (peak signal-to-noise ratio): higher values mean less pixel error. It is a useful yardstick, but it does not tell us whether the signing is still readable. ### How PSNR measures pixel error **Definition.** PSNR is defined from **MSE** (mean squared error) as PSNR = 10 log10(MAXI2 / MSE), where MAXI is the maximum possible pixel value. For 8-bit images, MAXI = 255. ([MathWorks](https://www.mathworks.com/help/vision/ref/psnr.html)) PSNR compares pixel values, not meaning. For a visual explanation of how a traditional codec decides what to keep, see Sophie Wang's [JPEG essay](https://www.sophielwang.com/blog/jpeg). Start with **resolution**. As the image shrinks, watch the fingers and face lose detail: ![Spatial downsampling of the original 1944 × 1944 video](https://cdn.sanity.io/images/cp6a9qoi/production/aecc86e217ead6650ef866f5529c1988bba8084c-1704x626.png?w=740&fit=crop&auto=format) *Spatial downsampling of the original 1944 × 1944 video. Each step is downsampled with ffmpeg's area filter; PSNR is measured after Lanczos-upscaling each video back to the original resolution.* Fewer pixels means fewer bytes, but not in direct proportion. The dashed lines show proportional scaling; the measured curve between them is flatter: ![File size vs pixel count, both axes log-scaled](https://cdn.sanity.io/images/cp6a9qoi/production/3dc6c9d45d6513dbadc6cedfa3242f1bc6ace8fd-1600x652.png?w=740&fit=crop&auto=format) *File size vs pixel count, both axes log-scaled. Solid line: actual encoded size. Dashed lines: slope-1 reference projections from each end of the ladder.* In this clip, the codec's **per-pixel cost is 4.8× lower** at full resolution than at the smallest. File overhead and spatial redundancy both affect the result; pixel count alone does not determine file size. Reducing frame rate loses a different kind of detail: motion between the retained frames. These clips keep the same playback duration, but become visibly choppier: ![Temporal downsampling from 29.97 fps down to 8](https://cdn.sanity.io/images/cp6a9qoi/production/c50be9774428ec5827ae2bc33573e90b149fbd31-1704x764.png?w=740&fit=crop&auto=format) *Temporal downsampling from 29.97 fps down to 8. PSNR is measured after resampling each variant back to the original frame rate.* At **256 × 256** the hands and face remain recognizable. Around **128 × 128** the face blurs and edges fade. Reducing frame rate instead leaves sharper frames but loses the movement between them. Reducing each side by 4× (**512 → 128**) shrinks the clip to 13.3% of its size; reducing frame rate by 3× (**24 → 8 fps**) only to 69.7% of its size. Resolution saves more bytes here, but loses a different kind of detail. PSNR ranks the 24 fps clip below the blurry 128 × 128 clip. That single score cannot explain which lost details matter for signing. Both changes affect the whole video. Could we save bytes by removing the background instead of detail on the signer? ## Spend fewer bits on the background First, locate the signer. A geometric crop can remove much of the surrounding scene. Our [sign/image-to-human-avatar](https://github.com/sign/image-to-human-avatar) pipeline detects the body and crops a 2.5× shoulder-width area centered on the neck: ![Detect-and-crop pipeline from sign/image-to-human-avatar](https://cdn.sanity.io/images/cp6a9qoi/production/4e6b007c33a56c73b58026dd7466a550e4d67ba2-1704x560.png?w=740&fit=crop&auto=format) *Detect-and-crop pipeline from sign/image-to-human-avatar. A body-based crop can still cut off raised hands, as it does here.* Segmentation separates the person from the background pixel by pixel. MediaPipe's *Selfie Multiclass* model classifies every pixel of a 256 × 256 image into one of six classes: - `background` - `hair` - `body-skin` - `face-skin` - `clothes` - `others` We apply that model through [sign/selfie-segmentation](https://github.com/sign/selfie-segmentation). The colored overlay shows its predictions. We then replace the background with flat green or blur it, keeping the selected foreground pixels. All versions use the same H.264 settings: ![Segmentation outputs from sign/selfie-segmentation running MediaPipe's Selfie Multiclass 256 × 256 segmenter](https://cdn.sanity.io/images/cp6a9qoi/production/7b7edbc8466a9a80e653b8c2d92711147531d85b-1704x548.png?w=740&fit=crop&auto=format) *Segmentation outputs from sign/selfie-segmentation running MediaPipe's Selfie Multiclass 256 × 256 segmenter.* Compared to the unmasked 256 × 256 input, the smaller of the two saves 42% of the bytes. Here, the blurred output is slightly smaller than the masked one. A flat background is cheap to encode, but the file must also describe the moving boundary around the signer. Watch the moving fingers: the mask sometimes removes part of them. The savings only help if the signer remains intact. Another approach is to store the movement directly. ### The same idea in self-driving video Focusing on the ROI isn't unique to signing. In self-driving, the **driving corridor** is a fixed geometric foreground: top entries in the [comma.ai video compression challenge](https://github.com/commaai/comma_video_compression_challenge) denoise everything outside it before the codec sees the frame. ## Keep the movement as poses Instead of storing pixels, we can track landmarks on the body, hands, and face. Watch what survives when the signer becomes a skeleton: ![Raw video beside MediaPipe Holistic and reduced skeleton renderings, with file sizes](https://cdn.sanity.io/images/cp6a9qoi/production/e8522e0ddc4bcf2f5b0c2fac1b2ab6cf4f63e34e-1704x764.png?w=740&fit=crop&auto=format) The coordinate count per frame is **fixed**, regardless of image resolution. Size scales with frames, not pixels. Stored as `float16`, the reduced skeleton for this clip uses 55.28 KiB of coordinate values, before file headers. The reduced version keeps selected landmarks on the body, hands, and face. That selection is **handcrafted**. A fixed set of coordinates cannot preserve every hand shape or facial detail, and a missed landmark becomes an error in the representation. Poses can still support strong translation. Google's [SL2T](https://deepmind.google/blog/putting-sign-language-ai-into-users-hands/) uses MediaPipe Holistic landmarks and was trained on over 100,000 hours across more than 50 sign languages. It now powers [ASL-to-English input on Pixel 11](https://blog.google/products-and-platforms/devices/pixel/american-sign-language-sign-to-text-pixel-11/). Compact landmarks can support useful translation at scale. ### What can we learn from pose based translation studies? An evaluation of eight pose estimators found that the choice of extractor affected sign-language translation, with missing hand keypoints associated with worse scores. ([O'Brien et al., 2026](https://arxiv.org/abs/2604.24609)) An earlier sign-language recognition study also found a gap between pose and image-based systems, highlighting limitations of available pose estimates. ([Moryossef et al., 2021](https://arxiv.org/abs/2104.10166)) These are evaluations of particular pipelines, not proof that every pose representation must fail. [MediaPipe Holistic](https://chuoling.github.io/mediapipe/solutions/holistic.html) uses tracking between frames and crops the hands and face from the full-resolution input. Its 256 × 256 body-pose input is not a resolution limit on the entire pipeline. Video compression usually aims to recover a faithful reconstruction. Ideally, our representation would also offer a way back to video. Coordinates alone do not specify the signer's appearance or every detail of a hand shape. We want a compact representation learned from video, paired with a decoder trained to reconstruct it. ## Learn what to keep A **neural video tokenizer** learns a compact representation instead of using a fixed set of landmarks. Its encoder turns video into tokens; its decoder turns them back into pixels. Training determines which details survive that round trip. Continuous tokens are vectors of numbers. Discrete tokens are IDs from a finite vocabulary. A language model can read either, but discrete IDs also fit its usual next-token prediction task: ![Table comparing continuous and discrete video tokens](https://cdn.sanity.io/images/cp6a9qoi/production/fbe7442df6e158bb0ad5f41c8be370683f3b90a9-1600x630.png?w=740&fit=crop&auto=format) Compare four pretrained tokenizers on our clip. LTX and Cosmos CV are continuous; Cosmos DV and AToken are discrete. At 512 × 512, compare the face and the moving fingers: ![Four tokenizer reconstructions of the same clip at 512 by 512: LTX, Cosmos CV, Cosmos DV, and AToken](https://cdn.sanity.io/images/cp6a9qoi/production/6aae842cc32c687fae1d77c771103bb8a3c2124a-1704x1044.png?w=740&fit=crop&auto=format) *Size means the stored token tensor, not the MP4 preview. Model weights are excluded. Stride lists the reduction in time × height × width.* ### Bytes and tokens tell different stories **PSNR** measures pixel agreement; [**LPIPS**](https://arxiv.org/abs/1801.03924) measures learned visual similarity. Neither tells us whether the signing remains intelligible. ![One clip, several resolutions: H.264 files versus stored neural latents](https://cdn.sanity.io/images/cp6a9qoi/production/fe4add89ec591cb269f8dce8e3bc63827f1e31cd-1604x902.png?w=740&fit=crop&auto=format) *One clip, several resolutions: H.264 files versus stored neural latents. Lower LPIPS or higher PSNR is better; farther left costs fewer bytes.* At the small end, Cosmos DV at 256 × 256 uses **5.7 KB** of stored latents, versus **16.9 KB** for H.264 at 64 × 64, with slightly lower LPIPS. At the other end, the largest H.264 version has the best measured fidelity in this sweep. There is no single winner at every size. These video tokens could become inputs to a vision-language model for sign language understanding. But a language model also pays for **sequence length**. Counting tokens instead of bytes changes the picture: ![A continuous token is one vector; a discrete token is one ID](https://cdn.sanity.io/images/cp6a9qoi/production/202379d9736898eb168eac1b472759e0c32b2c34-1604x902.png?w=740&fit=crop&auto=format) *A continuous token is one vector; a discrete token is one ID. Equal token counts do not mean equal byte budgets.* LTX uses few tokens here, but each contains 128 values. Each Cosmos DV token is a single ID. A short sequence can still contain many bytes; a small file can still require many prediction steps. ### How we compare storage size, reconstruction quality, and token count The [saved results](https://github.com/sign/video-compression-blog/blob/main/public/assets/popsign/manifest.json) compare H.264 MP4 bytes with serialized PyTorch latent tensors, including serialization overhead but excluding model weights. We vary resolution, not the encoders' rate settings at matched quality. These points illustrate this clip, not a general codec ranking. PSNR uses reconstructions resized to the original with Lanczos. LPIPS uses the AlexNet variant, antialiased bilinear resizing, and the mean over corresponding leading frames. Measurements include the display video's re-encoding loss. The [LPIPS paper](https://arxiv.org/abs/1801.03924) evaluates agreement with human image-similarity judgments, rather than sign-language comprehension. For continuous latents, we count vectors rather than their individual values. For discrete latents, we count IDs; AToken emits eight IDs per latent location. For these models, the latent grid grows with resolution and duration. The listed stride describes that reduction, not an independent receptive field. ### Tokens a language model can write If we want a language model to generate video by writing tokens, we'll need discrete IDs it can predict one at a time from a fixed vocabulary. Continuous video generators such as [LTX](https://github.com/Lightricks/LTX-2) use a different approach, based on diffusion. The compact Cosmos DV reconstruction above loses fine detail. Could we keep a richer representation of the signer's appearance, without making the model generate it again in every frame? ## Our tokenizer: Group of Tokens Borrow the structure of a group of pictures: keep a rich reference, then encode the continuation compactly. Our [**Group of Tokens (GoT)**](https://github.com/sign/video-tokenizer/pull/63) pairs continuous I tokens from a reference frame with discrete P tokens for the video. The decoder uses both to reconstruct the signer. We store them together in the [`.got` format](https://github.com/sign/video-tokenizer/blob/dda150f5fa1dc1af612d25f23ea78655456fc2e0/group_of_tokens/got.proto). We build on [NVIDIA Cosmos tokenizers](https://github.com/nvidia-cosmos/cosmos-predict1/blob/main/examples/inference_tokenizer.md), fine-tuning the discrete tokenizer on **768 × 768** video and using the continuous tokenizer for the appearance reference. The input includes the room; the reconstruction target is the**segmented signer on a flat background**. The decoder learns to leave out the room, so no separate segmenter is needed at inference. Source GOP boundaries define the training clips. ![Unmasked video in; the signer reconstructed with the background masked](https://cdn.sanity.io/images/cp6a9qoi/production/7974bfa8a86495c441b3fe7c40421c36088f66bb-1600x842.png?w=740&fit=crop&auto=format) *Unmasked video in; the signer reconstructed with the background masked.* ### One rich reference, a small discrete stream The reference is a 48 × 48 grid of vectors, each holding 16 numbers. The P stream uses a 16 × 16 grid of IDs per eight frames: **32 discrete tokens per frame** in steady state. Here is one slice of that stream beside the reference it shares: **Why 768 × 768?** We wanted more hand and face detail than 512 × 512 gave us, at a size suited to smaller screens and signer insets. Token count can be tuned separately through downsampling. ![Actual tokens from the .got file](https://cdn.sanity.io/images/cp6a9qoi/production/401d943c1fdf0674fdc3f7e623ccf88d93ee05d1-1600x1052.png?w=740&fit=crop&auto=format) *Actual tokens from the .got file. PCA projects all 16 reference values into three color channels. The original video frames show beneath the P grid. The reference costs 72 KiB once; each 256-ID slice costs 480 bytes before headers. The first slice covers one frame, then each covers up to eight.* These IDs are learned codes, not motion vectors or independent image patches. The decoder reads the grid together with the reference and recent context. An ID can change even over the background; its position alone does not tell us what it encodes. For all 53 frames, GoT uses **2,048 discrete tokens**, the same discrete count as Cosmos at 256 × 256 and nine times fewer than Cosmos at 768 × 768. GoT also stores **2,304 continuous reference tokens**. Compare the face, glasses, and fingers: ![Same unmasked input, different targets: Cosmos reconstructs the scene; GoT reconstructs the foreground](https://cdn.sanity.io/images/cp6a9qoi/production/98f631d4f73e6f004cd763d3223c10fba7ff2203-1704x912.png?w=740&fit=crop&auto=format) *Same unmasked input, different targets: Cosmos reconstructs the scene; GoT reconstructs the foreground. Counts cover all 53 frames, with one I-frame kept throughout. Previews use H.264 CRF 18. Run details · Download .got* GoT keeps more facial detail than the 256 × 256 Cosmos reconstruction at the same discrete count. The additional continuous reference helps make that possible; these are not equal total storage budgets. The moving fingers still need work. This clip illustrates the trade-off, rather than establishing a ranking across videos. ### Training details and token counts Our [GoT tokenizer](https://huggingface.co/sign/GoT-Tokenizer-8x48x48) combines fine-tuned **Cosmos DV8×16×16** with a new bottleneck that downsamples the discrete path by another 3× along each spatial axis, for a total stride of 8 × 48 × 48 (time × height × width). The [Cosmos continuous](https://github.com/nvidia-cosmos/cosmos-predict1/blob/main/examples/inference_tokenizer.md) reference path keeps its 48 × 48 grid. The tokenizer has roughly **133 million parameters**: 111 million in the shared Cosmos backbone and 22 million in our learned bottleneck. These counts cover the inference model, excluding training-only components. [Parameter-count source and breakdown.](https://github.com/sign/video-compression-blog/blob/main/public/model-sizes.json) Every [bottleneck decoder block cross-attends to the I tokens](https://github.com/sign/video-tokenizer/blob/dda150f5fa1dc1af612d25f23ea78655456fc2e0/video_tokenizer/training/bottleneck.py). This encourages the discrete codes to focus on change, without requiring a perfect separation of motion and appearance. All frames, including the first, use the discrete reconstruction path. The [49-frame training window](https://github.com/sign/video-tokenizer/blob/dda150f5fa1dc1af612d25f23ea78655456fc2e0/configs/got_israel1_r14.yaml) becomes seven temporal slices: 7 × 16 × 16 = 1,792 IDs from a 32,768-entry finite scalar quantization (FSQ) vocabulary. Our 53-frame demo pads to eight slices, so it uses 2,048 IDs. Both share one I-frame: 2,304 vectors of 16 values each. At 16 bits per value, the reference costs 72 KiB; the training window's 1,792 IDs use 3.28 KiB at 15 bits each, before headers and entropy coding. Fewer discrete IDs do not imply the same reduction in total file size. [Cosmos used adversarial training](https://arxiv.org/html/2501.03575v1#S4.SS2), but we do not have its trained discriminator. We fine-tune [Lightricks' pretrained LTX VAE discriminator](https://github.com/sign/video-tokenizer/blob/dda150f5fa1dc1af612d25f23ea78655456fc2e0/video_tokenizer/training/models/ltx.py) instead, to help improve reconstruction quality. ### Causal tokens, longer videos The tokenizer is **causal**: it can use recent context without waiting for future frames. Each slice sees itself and earlier slices within its attention window. The reference stays available separately: ![One layer's attention: filled cells are visible, dots are masked](https://cdn.sanity.io/images/cp6a9qoi/production/025a2fefbb1c33ef6e4c245634f65e0d9283f3e7-1600x566.png?w=740&fit=crop&auto=format) *One layer's attention: filled cells are visible, dots are masked. At slice seven, the oldest leaves the six-slice span. Nothing sees the future.* Bounded recent context lets us [encode](https://github.com/sign/video-tokenizer/pull/62) and [decode](https://github.com/sign/video-tokenizer/pull/61) beyond the training length. Here, each attention layer spans six temporal token slices. The 49 frames are the training length, not a fixed inference buffer. ### When should the reference change? The reconstruction above keeps the first reference throughout. When encoding an existing video, we can keep that reference across source GOPs or refresh it at their boundaries: ![One token group spans the video; every coded frame stays conditioned on I₁](https://cdn.sanity.io/images/cp6a9qoi/production/2072320af3132bc3ddbcc553fe59b8e1f53b43a2-1600x628.png?w=740&fit=crop&auto=format) *One token group spans the video; every coded frame stays conditioned on I₁. Arrows show which I-frame conditions each stream. Within a group, tokens also use recent causal context.* ## Real-time low-latency infinite video generation model So far, we have reconstructed existing video. Now we train a language model to generate the P tokens itself. It reads the continuous I tokens through a two-layer adapter and predicts each discrete P token from the reference and preceding tokens. The causal decoder turns that stream into video. Our [Qwen3.5-2B-derived model](https://huggingface.co/sign/qwen3.5-vl-2b-got-videogen) has **1.45B parameters at inference**. The reference supplies the person; next-token prediction supplies the continuation. This gives us a way to generate video without predicting a whole clip at once. We trained both the NVIDIA Cosmos-based GoT tokenizer and the Qwen-derived video language model on NVIDIA's [supercomputer](https://developer.nvidia.com/blog/accelerating-ai-storage-by-up-to-48-with-nvidia-spectrum-x-networking-platform-and-partners/) using NVIDIA H100 GPUs. For distributed training of the video language model, we use [NVIDIA NeMo Megatron Bridge](https://docs.nvidia.com/nemo/megatron-bridge/latest/index.html) with [NVIDIA Collective Communication Library (NCCL)](https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/index.html) handling communication between GPUs. Cosmos supplies the video representation; the language model learns to generate its tokens. ![First, train the tokenizer](https://cdn.sanity.io/images/cp6a9qoi/production/0451b391b7248c7892c4264dd964e80ab88f9ffd-1600x796.png?w=740&fit=crop&auto=format) *First, train the tokenizer. Then train the video language model on its tokens. At inference, the same I reference conditions generation and decoding.* ### Model sizes and parameter counts The [video generation model](https://huggingface.co/sign/qwen3.5-vl-2b-got-videogen/tree/930c283c0aa7523ae0f14cd0e235fc4b6709e27b) has roughly **1.45 billion parameters at inference**, including a 7-million-parameter adapter for the continuous reference tokens. Adapting the vocabulary and replacing the vision tower makes it smaller than the original Qwen3.5-2B model. Auxiliary prediction heads used during training bring the total to roughly **1.57 billion parameters**. They're not used for generation. The separate GoT tokenizer has roughly **133 million parameters**. See the [parameter breakdown](https://github.com/sign/video-compression-blog/blob/main/public/model-sizes.json) for exact counts and sources. ### Conditioning on appearance A separate image model created these nine reference appearances. Each conditions the same video model. Watch how the continuations retain the reference's face, hair, and clothing as the person moves: ![One continuation per avatar](https://cdn.sanity.io/images/cp6a9qoi/production/5e845b6d631d00bdc5d8bc2ebc7202a59db7ea9e-1602x1610.png?w=740&fit=crop&auto=format) *One continuation per avatar. Each uses the avatar reference and an eight-frame prefix of P tokens from real video; the model generates the tokens that follow. Hand detail and texture artifacts remain.* ### Start immediately. Keep generating. **Low latency** means a short wait between requesting video and seeing the first generated frames. Playback can begin while the rest is still being generated. **Real time** means generating new frames at least as fast as playback consumes them, so the video can keep playing without waiting for more. Tokens are generated on an NVIDIA B200; a separate NVIDIA RTX PRO 6000 decodes them into pixels. **Infinite videos** means there is no fixed clip-length limit. With **sliding-window attention**, the model reads a bounded window of recent tokens instead of the entire video history. As it generates new tokens, the window advances and older tokens leave it. The appearance reference stays available, while the recent context stays bounded as the video grows. This demonstration runs for **15 minutes**. It is one long continuation, not the maximum the model can generate: ![15-minute demonstration](https://cdn.sanity.io/images/cp6a9qoi/production/6d6a230020b868b9ecab66ad8d48201695be13e5-1600x1120.png?w=740&fit=crop&auto=format) *15-minute demonstration. 768 × 768, 24 fps.* **Parallelism** lets us generate several video streams at once. Our team reports that one NVIDIA B200 can generate tokens for up to eight parallel streams, each at up to 24 fps equivalent. That measures token generation only. A separate NVIDIA RTX PRO 6000 decodes eight parallel streams into pixels in real time. These are the reported capacities of the two stages. ## Just the beginning This research is part of Rylo’s broader effort to build real-time sign language translation. We've moved from traditional video compression to a representation that brings sign language video into language modeling pipelines. Our generation demos show how this approach can start with low latency, keep pace with playback, and continue without a fixed clip-length limit. This opens a path toward sign language understanding and generation, and ultimately streaming automatic interpreters. We're just getting started! Hands and other fine details still need work, and we're working with native signers to ensure the signing is intelligible. Better data, stronger tokenizers, and more capable language models give us concrete ways to move forward. We're excited to build toward sign language systems that can follow a conversation and respond as it happens. Keep following our progress at [rylo.com/sign](https://rylo.com/sign). [Dr. Amit Moryossef](https://www.linkedin.com/in/amit-moryossef/) Dr. Amit Moryossef is a researcher and entrepreneur in sign-language technology. He completed his Ph.D. at Bar-Ilan University and a postdoc at the University of Zurich. He founded sign.mt, a real-time sign-language translation platform, which was acquired by Rylo, where he now leads research. His work has received multiple best paper awards at ACL and EMNLP, focusing on making signed languages accessible through machine learning. Share this article ## Read next [Deaf Culture ![](https://cdn.sanity.io/images/cp6a9qoi/production/268e7bfe0e05ce61a42063249c37e7a960761a78-1999x1141.jpg?rect=144,0,1712,1141&w=480&h=320&fit=crop&auto=format) How to Call a Deaf Person on the Phone: The Complete 2026 Guide Need to call a deaf person on the phone? Learn how 711 relay services, VRS, captioned phones, and apps like Nagish make communication simple in 2026. June 12, 2026](https://rylo.com/post/how-to-call-a-deaf-person-on-the-phone)[Hearing Loss ![](https://cdn.sanity.io/images/cp6a9qoi/production/8b2cce37205478470de29d744603fc2d46d6f26b-1999x1311.jpg?rect=17,0,1967,1311&w=480&h=320&fit=crop&auto=format) How to Know If You're Going Deaf in One Ear: Common Signs, Causes, and What to Do Next Wondering if you're going deaf in one ear? Learn the early signs of single-sided hearing loss, common causes, when it's an emergency, and what to do next. June 11, 2026](https://rylo.com/post/how-to-know-if-youre-going-deaf-in-one-ear)[Deaf Culture ![](https://cdn.sanity.io/images/cp6a9qoi/production/0406b884d6c9ee39034f7e7bcf2ae19e0053834d-2000x1200.jpg?rect=100,0,1800,1200&w=480&h=320&fit=crop&auto=format) How to Sign "Deaf" in ASL Want to learn how to sign "deaf" in ASL? Follow step-by-step instructions, see accepted variations, avoid common mistakes, and understand Deaf culture. June 11, 2026](https://rylo.com/post/how-to-sign-deaf-in-asl) --- - HTML version of this page: https://rylo.com/post/rylo-partners-with-nvidia - Every page also answers `Accept: text/markdown`, and has a `.md` twin at its own URL. - Agent index: https://rylo.com/llms.txt - Site map: https://rylo.com/sitemap.xml