ICTC: Instruction-Conditioned Token Compression for 3D CT

The clinical picture

A chest CT is not one image. It is a stack of hundreds of slices, and a radiologist scrolls through all of them, holding the anatomy together in their head, before writing a report that a referring clinician will act on.

That reading time is the bottleneck in a lot of health systems. The scanners are faster than the people who interpret what comes off them, and in places where radiologists are scarce the queue is measured in weeks rather than hours.

So there is real interest in models that can read a volume and draft a report, or answer a specific question about a scan. Not to replace the radiologist, but to triage, to draft, to flag.

Why it's hard

The obstacle is arithmetic. Vision-language models work by turning an image into tokens that a language model can consume alongside text. That is manageable for a single photograph. It is not manageable for a CT volume.

Encode every slice of a scan into patch tokens and the count runs into the tens of thousands before the language model has processed a single word. The context window fills up, the compute cost climbs, and the model spends most of its capacity on redundant slices where nothing clinically interesting is happening.

The obvious response is to compress the visual tokens before handing them over. But compression is lossy by definition, and in radiology the thing you cannot afford to lose is the small finding: the nodule occupying a handful of voxels on two adjacent slices. Compress badly and you throw away exactly the evidence the report depends on.

There is a second, subtler problem. Most compression schemes are fixed. They squeeze the volume the same way regardless of what is being asked, which means a question about the adrenal glands and a question about the lung bases receive identical visual input.

The idea

ICTC compresses in two stages, and the split is the point.

The first stage is deliberately task-agnostic. Each slice is reduced to a small set of latents independently of any instruction, which produces a reusable representation that any downstream question can draw on.

The second stage is where the question enters. The compression across slices is conditioned on the encoded instruction, so the visual tokens that reach the language model are already shaped toward what is being asked. Ask about the adrenal glands and the compression allocates its budget differently than if you had asked for a full report.

Whether that is the right trade is genuinely open. Steering the visual tokens toward the query starts the model closer to the answer, but it also means the language model never sees a neutral representation of the scan. That tension is the most interesting question the project raises.

How it works

A frozen ViT-Base encodes each axial slice into patch tokens, with no gradients flowing through it. Stage one applies a Perceiver-style cross-attention block that compresses each slice to 32 latents, applied identically regardless of the instruction.

Stage two stacks the latents from all slices and aggregates them through cross-attention into a final set of M tokens. The query vectors are modulated by a FiLM layer conditioned on the encoded instruction text. M is a swept hyperparameter, evaluated from 16 up to 512, so the compression-versus-quality trade-off can be measured rather than assumed.

The language model is OPT-1.3B, adapted with LoRA at rank 16 targeting the query and value projections. Only the projector, the vision-to-language bridge, and the LoRA adapters train: 13.4 million parameters, with the LoRA component accounting for 0.24 percent of the language model's weights. The vision encoder and the language backbone stay frozen throughout.

Training is multi-task over CT-RATE v2, mixing report generation, entity-conditioned response, and binary abnormality questions in a single run. Evaluation reports multi-label AUROC and F1 across 18 abnormality labels at each of six token budgets, alongside RadGraph-F1 for clinical entity extraction and RaTEScore for radiology-specific semantic similarity.

The whole pipeline is YAML-configured with a 368-test suite, because a result you cannot rerun is not a result.

What happened

This is early work, and the honest headline is that the system runs end to end, not that it performs well yet.

The first full run covered a single epoch on 1,200 training volumes. Data streaming, multi-task training, the six-budget evaluation sweep and checkpointing all completed without error, which for a first integration is the milestone that matters.

The numbers are what under-training looks like. Classification AUROC sits between 0.49 and 0.53 across all six token budgets, statistically indistinguishable from chance. That means there is no compression-versus-accuracy trade-off to report yet, because there is no learned signal to trade off. Claiming otherwise from these numbers would be dishonest.

Two findings from the run are worth keeping. The training loss is strongly bimodal: yes-or-no classification steps converge to near zero within about 40 steps, while report-generation steps stay elevated throughout. That is the multi-task sampler working correctly, and it also diagnoses a real problem, because the easy task is consuming gradient steps the harder one needs.

The generation metrics tell a coherent story too. RaTEScore holds around 0.70 while RadGraph-F1 sits far lower. The model is producing text that is recognizably in-domain radiology language but is not yet naming the right entities and relations. That gap is precisely what more training should close.

Free-running generation initially collapsed to an immediate end-of-sequence token. Diagnosis identified it as a greedy-decoding degeneracy typical of lightly trained models, and a standard inference-time fix improved output quality across every task without any retraining.