A small playground to explore tokenization and decoding with GPT-style models. Based on the LLM Playground idea from AI engineering projects: tokenize text, inspect token IDs, and try different decoding strategies.

Tech: Hugging Face Transformers, Jupyter/local or Colab. The live app will be hosted at tik-tokenizer.aag1091.com when deployed.

What it does

  • Tokenize raw text and see token IDs (e.g. with GPT-2 tokenizer).
  • Experiment with temperature, top-k, and other decoding settings.
  • Compare completion vs instruction-tuned behavior.
  • Try it: tik-tokenizer.aag1091.com (subdomain; deploy when ready)
  • Source: Local project under ai-eng-projects-2/project_1 (notebook + env).