Posts

Introduction Tool calling (also called function calling or plugins) is the capability that turns large language models from text predictors into general-purpose controllers for software. Instead of only generating natural language, an LLM can: Decide when to call a tool (e.g., “get_weather”, “run_sql_query”) Decide which tool to call Construct arguments for that tool Use the result of the tool to continue its reasoning or response This post is a deep dive into the anatomy of tool calling: the moving parts, how they interact, what can go wrong, and how to design reliable systems on top of them. ...

Introduction In the world of large language models (LLMs), memory is a critical bottleneck. RAM (system memory) and VRAM (video RAM on GPUs) serve distinct yet interconnected roles in training and running models like GPT or Llama. While RAM handles general computing tasks, VRAM is optimized for the massive parallel computations required by LLMs.[1][3][4] This detailed guide breaks down their differences, impacts on LLM workflows, and optimization strategies, drawing from hardware fundamentals and real-world AI applications. ...

CUDA (Compute Unified Device Architecture) is NVIDIA’s powerful parallel computing platform that unlocks the immense computational power of GPUs for general-purpose computing. Mastering CUDA enables developers to accelerate applications in AI, scientific simulations, and high-performance computing by leveraging thousands of GPU cores.[1][2] This detailed guide takes you from beginner fundamentals to advanced optimization techniques, complete with code examples, architecture insights, and curated resources. Why Learn CUDA? GPUs excel at parallel workloads due to their architecture: thousands of lightweight cores designed for SIMD (Single Instruction, Multiple Data) operations, contrasting CPUs’ focus on sequential tasks with complex branching.[3] CUDA programs can achieve 100-1000x speedups over CPU equivalents for matrix operations, deep learning, and simulations.[1][4] ...

FastAPI’s speed, async capabilities, and automatic API documentation make it ideal for building production-grade APIs serving Large Language Models (LLMs). This guide details best practices for deploying scalable, secure FastAPI applications handling LLM inference, streaming responses, and high-throughput requests.[1][3][5] LLM APIs often face unique challenges: high memory usage, long inference times, streaming outputs, and massive payloads. We’ll cover project structure, async optimization, security, deployment, and LLM-specific patterns like token streaming and caching. ...

Ollama is an open-source framework that enables running large language models (LLMs) locally on personal hardware, prioritizing privacy, low latency, and ease of use.[1][2] At its core, Ollama leverages llama.cpp as its inference engine within a client-server architecture, packaging models like Llama for seamless local execution without cloud dependencies.[2][3] This comprehensive guide dissects Ollama’s internal mechanics, from model management to inference pipelines, quantization techniques, and hardware optimization. Whether you’re a developer integrating Ollama into apps or a curious engineer, you’ll gain actionable insights into its layered design. ...