TokenLens

LLM token optimization toolkit by Sharad Khare






> TokenLens helps developers and AI teams cut OpenAI, Claude, and other LLM API bills by converting bulky JSON prompts into compact TOON (Token-Oriented Object Notation) payloads — often 30–60% fewer tokens on tabular data.

Built and maintained by Sharad Khare — AI strategist, full-stack developer, and builder of practical AI tooling.

---

Why Sharad Khare built TokenLens

Large language model costs grow with every input token. Teams repeatedly send product catalogs, CRM exports, analytics tables, and survey batches inside prompts — and JSON syntax adds expensive overhead.

TokenLens makes the savings visible before you ship to production:

• Compare JSON vs TOON side by side
• Estimate token counts and dollar cost per request
• Test your own payloads in a Streamlit dashboard
• Optionally validate with OpenAI GPT-4 or Anthropic Claude

---

Features

| Feature | Description |

|---------|-------------|

| Payload comparison | Byte, token, and cost diff for structured datasets |

| Cost calculator | GPT-4, GPT-4 Turbo, GPT-3.5, Claude Opus & Sonnet pricing |

| CLI analysis | run_analysis.py for terminal-based walkthroughs |

| Interactive UI | dashboard.py Streamlit app with live conversion |

| LLM integration | Optional live prompts when API keys are configured |

| Encoding options | Key folding, comma/tab/pipe delimiters |

---

Quick start

git clone https://github.com/sharadkhare/tokenlens.git
cd tokenlens
python -m venv .venv

# Windows
.\.venv\Scripts\Activate.ps1

# macOS / Linux
source .venv/bin/activate

pip install -r requirements.txt
python verify_setup.py

Run the toolkit

python run_analysis.py          # full CLI walkthrough
streamlit run dashboard.py        # interactive dashboard

Optional API keys

export OPENAI_API_KEY="your-key"
export ANTHROPIC_API_KEY="your-key"

---

JSON vs TOON example

JSON (verbose):

{
  "inventory": [
    {"sku": "NB-2401", "name": "NovaBook Air", "price": 1189.0},
    {"sku": "MS-1108", "name": "SwiftGlide Mouse", "price": 64.5}
  ]
}

TOON (compact — fewer LLM tokens):

inventory[2]{sku,name,price}:
  NB-2401,NovaBook Air,1189
  MS-1108,SwiftGlide Mouse,64.5

Same data. Smaller prompt. Lower API bill.

---

Project structure

tokenlens/
├── config.py              # Brand & project metadata (Sharad Khare)
├── sample_data.py         # Demo datasets
├── verify_setup.py        # Installation check
├── run_analysis.py        # CLI analysis runner
├── dashboard.py           # Streamlit UI
├── utils/
│   └── token_metrics.py   # Token & cost helpers
├── requirements.txt
├── pyproject.toml
└── README.md

---

When to use TOON

| Best for | Avoid when |

|----------|------------|

| Tabular LLM prompts | Deeply nested irregular JSON |

| High-volume API calls | Systems that require strict JSON only |

| Product / order / analytics data | One-off tiny payloads |

| Cost-sensitive AI workflows | Maximum cross-tool compatibility needed |

---

Python usage

from toon import encode
from utils.token_metrics import count_tokens, calculate_cost

data = {
    "inventory": [
        {"sku": "NB-2401", "name": "NovaBook Air", "price": 1189.0},
        {"sku": "MS-1108", "name": "SwiftGlide Mouse", "price": 64.5},
    ]
}

compact = encode(data)
tokens = count_tokens(compact)
cost = calculate_cost(tokens, model="gpt-4")

print(compact)
print(f"Tokens: {tokens} | Est. cost: ${cost:.6f}")

---

Tech stack

• Python 3.8+
• toonify — TOON encode/decode
• tiktoken — OpenAI-compatible token counting
• Streamlit — dashboard UI
• OpenAI / Anthropic SDKs — optional live LLM demos

---

About the author

Sharad Khare is an AI strategist and full-stack developer who builds practical tools at the intersection of software engineering and intelligent automation.

• Website: sharadkhare.in

---

Keywords

Sharad Khare · LLM token optimization · OpenAI cost reduction · Claude API savings · TOON format · Python AI tools · prompt engineering · Streamlit dashboard · structured data serialization · AI cost optimization

---

Who This Is For

• AI engineers optimizing prompt payloads and API cost
• Product teams running high-volume LLM workflows
• Founders shipping token-efficient SaaS copilots
• Developers benchmarking JSON vs compact serialization

Use Cases

• Reduce OpenAI and Claude spend for tabular prompts
• Compress CRM, catalog, and analytics payloads
• Estimate cost impact before production rollout
• Build internal LLM cost-optimization dashboards

---

License

MIT © Sharad Khare — see [LICENSE](LICENSE).