Prompt Engineering for Small Language Models (≈7 B Parameters)

What is Prompt Engineering for Small Language Models?

Prompt engineering is the practice of shaping the input given to a language model so that the output meets specific quality, factuality, and formatting requirements.

• It treats the prompt as a contract between user and model.
• For 7 B‑parameter models, the contract must compensate for limited world knowledge and weaker multi‑step reasoning.
• Key components include context injection, few‑shot examples, and explicit output schemas.

Why Prompt Engineering Is Critical for 7 B Models

Smaller models exhibit systematic weaknesses that can be mitigated through disciplined prompting.

• Patchy knowledge coverage: they often miss niche facts and may hallucinate.
• Logic breaks on multi‑step tasks: they can skip steps or contradict themselves.
• Low instruction adherence: they tend to obey only part of a complex request.
• Format instability: output can drift from the desired structure (e.g., JSON, tables).

How to Design Effective Prompts for 7 B Models

Follow a repeatable workflow that combines clear contracts, incremental validation, and repair loops.

• One task per prompt: isolate a single objective to avoid overload.
• Inject missing knowledge: prepend a “FACTS” block containing all domain‑specific data the model may need.
• Provide a few‑shot example: show a short, correctly formatted instance that the model can imitate.
• Declare a format contract: state the exact output type (e.g., “Output JSON only”) and include a failure token such as INSUFFICIENT_DATA.
• Enforce step‑by‑step execution: ask the model to list each step, then validate each step before proceeding.
• Use a repair prompt: when a field is missing or malformed, request regeneration of only that fragment.

Knowledge Injection (Context Injection)

Supply the model with a concise knowledge base to eliminate hallucination.

• Wrap facts in a clearly marked block: FACTS (use only these): - … - …
• Keep the block short (≤ 150 tokens) to stay within context limits.
• Reference the block explicitly in the task description.

Format Contracts and Schema Enforcement

Define the exact structure the model must emit.

• Specify the schema in plain language and with a tiny example.
• Include a stop sequence or sentinel (e.g., “END_JSON”) to prevent trailing prose.
• If JSON proves brittle, fall back to a markdown table or “Key: Value” list that can be parsed with regex.

Step‑by‑Step Validation and Repair Loops

Treat each generation as a unit test.

• After each step, check for completeness and correctness.
• When a defect is found, issue a targeted repair prompt: “Regenerate field price using the same format.”
• Iterate until all checklist items pass.

Scoring and Iteration Loop

Measure prompt performance with a lightweight scorecard.

• Adherence: % of mandatory requirements satisfied.
• Factuality: count of statements that contradict injected facts.
• Format pass‑rate: % of outputs that parse without error.
• Stability: variance of key decisions across runs.
• Cost: average token usage and latency.

Use the scorecard to guide incremental changes: modify one constraint, re‑run, and keep improvements that raise adherence/format scores without inflating cost.

Hardware and Inference Considerations

Even with optimal prompting, deployment choices affect reliability.

• Quantization: INT8/INT4 reduces memory and speeds inference; 4‑bit quant + LoRA adapters retain most accuracy.
• Frameworks: llama.cpp for local CPU/GPU, vLLM for high‑throughput servers, Transformers.js for client‑side experiments.