Karn Wong's Docs

Appearance

LLM ​

Specs ​

Demos ​

Tools ​

Frameworks ​

  • ADK
  • genkit - Open-source framework for building agentic apps in JavaScript, Go, Dart, and Python, built and used in production by Google
  • sherpa-onnx - Speech-to-text, text-to-speech, speaker diarization, speech enhancement, source separation, and VAD using next-gen Kaldi with onnxruntime without Internet connection. Support embedded systems, Android, iOS, HarmonyOS, Raspberry Pi, RISC-V, RK NPU, Axera NPU, Ascend NPU, x86_64 servers, websocket server/client, support 12 programming languages

Libraries ​

  • langextract - A Python library for extracting structured information from unstructured text using LLMs with precise source grounding and interactive visualization.

Diagrams ​

  • paperbanana - Open source implementation and extension of Google Research’s PaperBanana for automated academic figures, diagrams, and research visuals, expanded to new domains like slide generation.
  • tufte-claude-skill - Claude Code skill that turns 'make me a chart' into a Tufte-compliant chart. Distilled from Edward Tufte's three foundational books.

Serving ​

  • Lemonade
  • LMCache - Supercharge Your LLM with the Fastest KV Cache Layer
  • ollama - Get up and running with Llama 3, Mistral, Gemma, and other large language models
  • sglang - SGLang is a high-performance serving framework for large language models and multimodal models.
  • vllm - A high-throughput and memory-efficient inference and serving engine for LLMs

llama.cpp ​

Disable thinking:

bash
--chat-template-kwargs '{"enable_thinking":false}'

Guardrails ​

  • Guardrails - NeMo Guardrails is an open-source toolkit for easily adding programmable guardrails to LLM-based conversational systems.

API Gateways ​

  • bifrost - Fastest LLM gateway (50x faster than LiteLLM) with adaptive load
  • litellm - Python SDK, Proxy Server (AI Gateway) to call 100+ LLM APIs in OpenAI (or native) format, with cost tracking, guardrails, loadbalancing and logging. [Bedrock, Azure, OpenAI, VertexAI, Cohere, Anthropic, Sagemaker, HuggingFace, VLLM, NVIDIA NIM]

Serialization ​

  • toon - 🎒 Token-Oriented Object Notation (TOON) – Compact, human-readable, schema-aware JSON for LLM prompts. Spec, benchmarks, TypeScript SDK.

API ​

Chat ​

bash
curl -X POST http://example.com/v1/responses \
  -u "username:password" \ # basic auth
  -H "Authorization: Bearer $OPENAI_API_KEY" \ # api key auth
  -H "Content-Type: application/json" \
  -d '{
        "model": "Llama-3.2-1B-Instruct-Hybrid",
        "input": "What is the population of Paris?",
        "stream": false
      }'

Basicauth can also be provided as request header:

bash
echo -n "username:password" | base64 # generate base64 string
curl -H "Authorization: Basic xxxx" # use above base64 string as auth token

Image Generation ​

bash
curl https://xxxx/v1/images/generations \
  -H "Authorization: Bearer xxx" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "auto",
    "prompt": "a hearty stew"
  }' | jq -r .data[0].b64_json | base64 --decode > image.png

Misc ​

Hardware ​

Setting up NVIDIA DGX Spark with ggml ​

bash
bash <(curl -s https://ggml.ai/dgx-spark.sh)

Vendors ​

Google ​

Apps ​

  • gallery - A gallery that showcases on-device ML/GenAI use cases and allows people to try and use models locally.

Resources ​

Must Reads ​

Security ​