AI.Cruz - Learn AI Through Interactive Visualizations

GEN AI · JEV SERIES 1/4

JEV: A Qwen-Powered Decision Network

How to build a network that answers multiple-choice, yes/no, and score questions using a Qwen backbone and a lightweight ChoiceHead.

1What is a JEV network?

A JEV (Joint Embedding Vision) network is a decision-making model built on top of a frozen or partially fine-tuned language backbone. Instead of generating free-form text, it scores a fixed set of candidate answers for each question and selects the best one. This design is compact, fast to train, and directly optimisable with cross-entropy loss against the correct choice.

choice — Pick the best option from a labeled list. Direct classification over candidates.
noul — Answer yes or no. Binary prediction; AUC is reported in addition to accuracy.
score — Rate something on a numeric scale. Treated as ordinal choice selection.

All three task types flow through the same architecture. A task token injected at the head’s input tells the ChoiceHead which regime it is operating in, so a single model handles all three without any task-specific branches.

2The two-part architecture

The network has two components wired in series.

Backbone — Qwen3-0.6B

Runs the first N decoder layers (default: 20 of 28).
Converts input tokens into contextual hidden states.
Can be frozen (head-only training) or partially fine-tuned with LoRA.

Head — ChoiceHead

Reads one hidden state per choice and one for the answer slot.
Runs a small self-attention stack over those embeddings.
Scores every choice with a bilinear dot product.

The head runs in fp32 even under bf16 autocast because bf16 logits are too coarse for a stable cross-entropy loss — a detail that matters in practice when training on modern GPUs.

3Why not just generate an answer token?

Autoregressive generation is expensive at inference time (one forward pass per new token) and hard to optimise directly (requires RLHF or similar). Scoring a fixed candidate set is a single forward pass — the same complexity regardless of how long the choices are — and cross-entropy provides a dense, well-behaved gradient on every example.

The BEV-Decision dataset (avbiswas/bev-decision-150K) contains 150,000 questions across all three task types derived from bird’s-eye view driving scenarios — a natural benchmark for this architecture because the label space is naturally bounded and the questions are linguistically rich.

4A minimal forward pass

At a high level, a single example flows through the network in three stages.

Tokenise. The question and each choice are tokenised separately. Choices are wrapped in <option>...</option> tags and concatenated after the question. The string "The answer is:" closes the sequence. Position IDs are assigned so that all choices start at the same position (parallel encoding — covered in post 2).
Backbone. The full sequence runs through the Qwen layers. The attention mask is crafted so each choice is isolated from the others but can attend to the question (covered in post 2).
Head. The ChoiceHead reads the hidden state at the last real token of each choice and at the final answer token, then scores every choice with a bilinear product. The highest-scoring choice is the prediction.

5Key design decisions at a glance

Decision	Why
Partial backbone (first N layers)	Reduces memory and FLOPS; the lower layers provide rich token representations without the cost of the full model.
LoRA on last K kept layers	Allows the backbone to adapt to the task domain with a tiny fraction of trainable parameters.
fp32 head under bf16 autocast	Prevents numerical instability in the logits without giving up the speed of bf16 in the backbone.
Bilinear scoring in the head	Order-invariant: works correctly regardless of how many choices appear or in what order.