Hebrew LLM Eval Suite
Trusted76/100Benchmark and compare LLMs on Hebrew reasoning, comprehension, sentiment, translation, and Israeli cultural knowledge. Wraps the HuggingFace Open Hebrew LLM Leaderboard tasks (HeQ, HebrewSentiment, Hebrew Winograd, translation) plus DictaLM 3.0 benchmark tasks (Summarization, Nikud, Israeli Trivia) into a reproducible evaluation harness. Runs evals against Claude, GPT, Gemini, AI21 Jamba, DictaLM, Llama, and local HuggingFace models. Produces comparison scorecards in JSON and markdown. Use when choosing an LLM for a Hebrew product, answering procurement questions about Hebrew performance, validating a fine-tuned Hebrew model, or tracking Hebrew regressions after a model upgrade. Do NOT use for Arabic NLP, ASR benchmarking, or general English benchmarks.
Trust score 76/100 (Trusted) · 53+ installs · 4 GitHub contributors · MIT license
Israeli product teams pick LLMs blind. There is no standardized Hebrew benchmark that a PM can run in an afternoon to compare Claude against GPT against DictaLM against AI21 Jamba on their actual use case. The HuggingFace Open Hebrew LLM Leaderboard is built for base models and few-shot prompts, not for API-hosted chat models. DictaLM publishes benchmark results but only for its own suite. Teams end up guessing, testing informally, or trusting marketing claims.
skills-ilHow to use this skill
Not sure how? Read the guide- 1. Click "Download ZIP" to download the skill files.
- 2. Open Claude Desktop and go to Customize > Skills.
- 3. Click "+" and select "Upload a skill", then upload the ZIP file.
- 4. Start a new conversation. The skill will activate automatically when relevant.
Developers? Install via command line (CLI)
npx skills-il add skills-il/developer-tools@v1.2.0-hebrew-llm-eval-suite --skill hebrew-llm-eval-suite -a claude-codeWhen to Apply
- When choosing an LLM for a new Hebrew product and needing to justify the choice to leadership
- When answering enterprise procurement questions about Hebrew performance
- When validating whether a provider upgrade improved or regressed Hebrew quality
- When validating a fine-tuned Hebrew model against a baseline
- When comparing providers on a specific task: comprehension, translation, summarization, or diacritization
Try These Prompts
We are building a Hebrew news summarization feature and need to pick between Claude Sonnet, GPT-5, and DictaLM-3.0-24B. Run the relevant benchmarks (HeQ, DictaLM Summarization, Winograd) with 1000 samples and 3 runs, and recommend a model with reasoning.
Anthropic released a new version of claude-sonnet. Run the hebrew-core suite on the new and previous versions and tell me if there was any regression over 2 points on any benchmark.
I am building a Hebrew chatbot and deciding between Claude Haiku and AI21 Jamba 1.5 Mini. Compare them on HeQ, HebrewSentiment, and HebNLI with 500 samples and 3 runs, and provide a scorecard with a recommendation.
We have a data residency constraint requiring a local model. Run Hebrew benchmarks on DictaLM-3.0-Nemotron-12B-Instruct and compare to Claude Sonnet quality. How much quality am I giving up?
Frequently Asked Questions
Changelog
Added Gemini 3, Jamba 1.6, and Jamba-Reasoning-3B to the model roster; reconciled SKILL.md and run_eval.py model lists; relabeled scorecard table as illustrative placeholders, not measured results; added evidence.json.
May 20, 2026
HEBREW-MMLU, lm-evaluation-harness + inspect_ai cross-refs, verified DictaLM 2.0/3.0, Aya/Hebrew-Mistral/Hebrew-Gemma comparators, claude-opus-4-7, fixed HE table row, tokenizer fairness section.
Apr 25, 2026
Related Skills
Author: skills-ilCompare cloud hosting costs for Israeli startups and developers across AWS (il-central-1 Tel Aviv), Azure (Israel Central), GCP (me-west1 Tel Aviv), Oracle Cloud (il-jerusalem-1 Jerusalem), and Israeli providers like Kamatera. Use when the user needs to evaluate cloud pricing with Israel-specific considerations including data residency under Privacy Protection Law Amendment 13, latency from Tel Aviv, NIS billing options, startup credit programs (AWS Activate, Google for Startups, Microsoft Founders Hub, Israel Innovation Authority Telem program with subsidized Nvidia B200 GPUs), and FinOps cost optimization strategies. Do NOT use for comparing on-premise hosting, colocation services, or non-cloud SaaS pricing.
Author: skills-ilGuide developers in integrating Israeli agritech tools and precision agriculture platforms including CropX (soil monitoring), Netafim GrowSphere (IoT irrigation), Taranis (crop intelligence), and the broader Israeli agritech ecosystem (approximately 600-750 companies per Start-Up Nation Central agrifoodtech). Use when user asks about agritech APIs, precision agriculture, smart irrigation, "hashkaya cham", crop monitoring, pest detection, Israeli agriculture tech, or needs to build farm management software. Covers irrigation optimization, pest detection, climate data integration, and Israeli agricultural context. Do NOT use for general gardening advice or non-agricultural IoT projects.
Author: skills-ilConvert between Hebrew (Jewish) calendar and Gregorian dates, look up Israeli holidays, format dual dates for Israeli documents, and calculate Israeli business days. Use when user asks about Hebrew dates, "luach ivri", Jewish calendar, Israeli holidays, "chagim", Shabbat times, or needs dual-date formatting for Israeli forms. Do NOT use for Islamic Hijri calendar or non-Israeli holiday calendars.
Use at your own risk. Terms of Use · Security
Want to build your own skill? Try the Skill Creator · Submit a Skill