AI Guardrails: Implementing NeMo Guardrails and Llama Guard for Input/Output Filtering

Overview #

AI guardrails are real-time security filters that intercept both incoming prompts and outgoing model completions, preventing hallucinations, toxic content, brand violations, and prompt injection attacks.

The Three Guardrail Checkpoints #

1. Input Guardrail: Analyzes user prompts for malicious intent, jailbreaks, PII, and off-topic topics before the foundation model executes.
2. Dialogue Guardrail: Constrains conversational trajectory according to stateful domain flows.
3. Output Guardrail: Inspects the model's generated response for factual consistency (hallucination checks), toxicity, and competitors' names before returning data to the client.

NVIDIA NeMo Guardrails & Colang #

NeMo Guardrails uses Colang, a specialized domain-specific language for defining conversational flows and safety rails. When an input matches an unsafe pattern, the system immediately returns a predefined canonical response without invoking expensive LLM inference.

Meta Llama Guard for Real-Time Content Safety #

Llama Guard is an open-weights classifier fine-tuned specifically to categorize prompts against the MLCommons taxonomy (covering hate speech, self-harm, cyberattacks, and chemical weapons). Running Llama Guard 8B takes sub-30ms and acts as a firewall for AI APIs.

Code Example: Executing Llama Guard Check Before Dispatching Prompt #

from openai import OpenAI

client = OpenAI(base_url="https://api.apihundred.com/v1", api_key="your_key")

def is_prompt_safe(user_prompt: str) -> bool:
    guard_response = client.chat.completions.create(
        model="llama-guard-3-8b",
        messages=[{"role": "user", "content": user_prompt}],
        max_tokens=20,
        temperature=0.0
    ).choices[0].message.content.strip()

    # Llama Guard outputs 'safe' or 'unsafe \n S1/S2/...'
    return guard_response.startswith("safe")

if is_prompt_safe("How do I configure OAuth in Django?"):
    print("Prompt cleared safety checks. Dispatching to model...")
else:
    print("Blocked by AI Guardrails.")

Frequently Asked Questions #

Q: How much latency do guardrails add?

Lightweight classifiers like Llama Guard 8B add 25ms to 50ms, while regex and vector similarity guardrails add less than 5ms.

Q: Can guardrails prevent all hallucinations?

Guardrails can verify self-consistency and cross-reference citations, eliminating up to 90% of factual hallucinations in RAG pipelines.

Q: Where should guardrails execute?

In enterprise architectures, guardrails are best deployed directly at the AI Gateway layer so all downstream microservices are protected uniformly.