Thank you for sending your enquiry! One of our team members will contact you shortly.
Thank you for sending your booking! One of our team members will contact you shortly.
Duration 14 hours
Course Outline
Deployment Options
- Trade-offs between hosted APIs and self-hosting
- Patterns for on-premises, edge, and air-gapped environments
- Single model versus model portfolio approaches
- Comparison of cost per decision
Hardware Sizing and Capacity
- Memory footprint relative to parameter count
- CPU-only versus GPU acceleration strategies
- Planning for concurrency and throughput
- Allowing headroom for task length growth
Serving Stacks
- vLLM and SGLang options
- llama.cpp and GGUF execution
- Ollama for local development
- Selecting a stack based on decision workload shape
Quantization Without Losing Calibration
- Precision formats and memory savings
- Measuring quality and calibration loss
- Per-tensor and importance-aware variants
- Acceptance testing after quantization
Latency Engineering
- Median versus tail latency considerations
- Latency growth with context length
- Warm start and model caching techniques
- Batching multiple decision questions
Edge and Constrained Environments
- Running small decision models on CPUs
- Deployment on Apple Silicon and embedded systems
- Offline operation and update strategies
- Distributing weights in air-gapped settings
Containerization and Scaling
- Designing images for model serving
- Managing volumes and caches
- Horizontal scaling and routing strategies
- Resource limits and GPU sharing
Observability
- Monitoring request, latency, and error metrics
- Tracking confidence distributions
- Logging decision outcomes and escalations
- Alerting on calibration drift
Versioning and Rollout
- Versioning models and adapters
- Shadow and canary deployment methods
- Pinning model files for audit purposes
- Rollback procedures
Security and Multi-Tenancy
- Authentication and rate limiting mechanisms
- Tenant isolation for prompts and logs
- Network isolation and egress control
- Audit logging of decisions
Requirements
- Experience with Linux administration and containerization.
- Familiarity with model inference servers and GPU management.
- Basic understanding of REST APIs, load balancing, and monitoring.
Audience
- MLOps and platform engineers managing inference infrastructure
- SRE teams supporting AI services
- AI engineers deploying decision models on-premises
- Architects designing air-gapped or regulated AI deployments
