Thank you for sending your enquiry! One of our team members will contact you shortly.
Thank you for sending your booking! One of our team members will contact you shortly.
Duration 14 hours
Course Outline
AI Sovereignty and Local LLM Deployment
- Addressing cloud LLM risks: data retention policies, input training, and jurisdictional concerns.
- Ollama architecture overview: model server, registry, and OpenAI-compatible API.
- Comparative analysis with vLLM, llama.cpp, and Text Generation Inference.
- Review of model licensing terms for Llama, Mistral, Qwen, and Gemma.
Installation and Hardware Configuration
- Setting up Ollama on Linux with CUDA and ROCm acceleration.
- Implementing CPU-only fallbacks and optimizing for AVX/AVX2 instructions.
- Deploying via Docker with persistent volume mapping.
- Configuring multi-GPU environments and VRAM allocation strategies.
Model Management
- Retrieving models from the Ollama registry using commands like ollama pull llama3.
- Importing GGUF format models from HuggingFace and TheBloke.
- Evaluating quantization levels: Q4_K_M, Q5_K_M, and Q8_0 tradeoffs.
- Managing model switching and concurrent loading limits.
Custom Modelfiles
- Mastering Modelfile syntax: FROM, PARAMETER, SYSTEM, and TEMPLATE directives.
- Tuning temperature, top_p, and repeat_penalty parameters.
- Engineering system prompts for specific role-based behaviors.
- Creating and publishing custom models to local registries.
API Integration
- Utilizing the OpenAI-compatible /v1/chat/completions endpoint.
- Handling streaming responses and enforcing JSON mode.
- Integrating with LangChain, LlamaIndex, and proprietary applications.
- Implementing authentication and rate limiting via reverse proxies.
Performance Optimization
- Managing context window sizes and KV cache efficiency.
- Implementing batch inference and parallel request processing.
- Allocating CPU threads with NUMA topology awareness.
- Monitoring GPU utilization and memory pressure metrics.
Security and Compliance
- Isolating network traffic for model serving endpoints.
- Establishing input filtering and output moderation pipelines.
- Maintaining audit logs for all prompts and completions.
- Verifying model provenance and hash integrity.
Requirements
- Intermediate proficiency in Linux and container administration.
- A high-level understanding of machine learning concepts and transformer architectures.
- Familiarity with REST APIs and JSON data structures.
Target Audience
- AI engineers and developers seeking to replace cloud-based LLM APIs.
- Organizations constrained by data sensitivity policies that restrict cloud model usage.
- Government and defense sectors requiring air-gapped language model capabilities.
