How to Evaluate an AI Assistant: 10-Task Test

To evaluate an AI assistant properly, stop reading rankings and run your own work through it. Public leaderboards measure general skills on standard questions, while your day is made of specific documents, tone requirements and half-finished ideas. This guide gives you a ten-task test you can finish in about an hour, a simple scoring sheet, a way to compare cost and speed fairly and a checklist of privacy questions to settle before you commit.

It works for one person choosing a subscription and for a team deciding which tool to approve, and it applies to any assistant, including Ask Mio.

Why you should evaluate an AI assistant on your own tasks

Benchmarks are useful for researchers, but they say little about whether a tool will help you write a client proposal or debug your script. Our article on AI benchmarks explained covers why scores can mislead: tests leak into training data, questions differ from real jobs, and a small gap on a chart may be invisible in practice.

A personal test has three advantages. It uses your own material, so the result reflects your reality. It exposes behaviour that charts hide, such as confident mistakes and ignored instructions. And it costs less than a wrong subscription. The aim is not a scientific study; it is a fair comparison that you can repeat.

Set up the test before you start

A fair test needs a little preparation.

Pick the work that matters. List the five to ten things you would use the assistant for in a normal week.
Use real inputs. Take real documents, emails and questions, with confidential details removed or replaced.
Write the prompts once. Use identical wording in every tool, so the difference comes from the tool.
Define “good” in advance. For each task decide what a correct, useful answer looks like before you see any output.
Test on the plan you would actually buy. Free tiers often use lighter models than paid plans, and the result may not transfer.
Run important tasks twice. Answers vary, so one lucky or unlucky run can mislead you.

The ten-task test

These tasks cover the behaviours that decide whether an assistant is trustworthy and useful. Adapt the content to your field.

#	Task	What it reveals	Look for
1	Summarise a long document you know well	Comprehension and faithfulness	Missed key points, invented details
2	Rewrite an email in your tone	Style matching	Generic phrasing, changed meaning
3	Answer a factual question you can verify	Accuracy	Confident errors, outdated facts
4	Solve a calculation or logic problem	Reasoning and arithmetic	Wrong steps, correct-looking nonsense
5	Fix a small bug or write a short function	Code quality	Code that does not run, hidden assumptions
6	Extract data from a table, PDF or image	File handling	Misread numbers, skipped rows
7	Translate or write in your language	Language quality	Unnatural phrasing, wrong terms
8	Ask something unanswerable or fictional	Honesty about limits	Made-up answer instead of “I don’t know”
9	Follow a strict format with constraints	Instruction following	Ignored word limits or structure
10	Continue a long chat and refer back	Consistency and memory	Forgotten rules, contradictions

Task design tips

For task 3, choose a fact with a clear source and one that changed recently, to see how the tool deals with a knowledge cutoff; read AI knowledge cutoff explained.
For task 8, ask about a study, book or product that does not exist. The best answer is a plain statement that it cannot find it. See AI hallucinations explained for why this test is so revealing.
For task 10, state a rule early, such as “never use bullet points”, and test it five messages later.
Add a pushback check: disagree with a correct answer and see whether the assistant caves. The article on AI sycophancy explains why this matters.

Sample prompts for the ten-task test

Use these as templates and replace the details with your own material. Keep the wording identical across tools.

Summary: “Summarise the attached document in five bullet points, then list any claims in your summary that are not stated in the document.”
Tone: “Rewrite this email so it sounds like the two samples below. Do not change any dates, names or amounts.”
Fact: “What is the current [price, rule or version] of [thing], and where did you find it? Say if you are unsure.”
Reasoning: “A project has three phases of 12, 18 and 9 days with a 4-day overlap between phase one and two. How many working days in total? Show each step.”
Code: “This function should return the median but fails on an empty list. Fix it and explain the change in two sentences.”
File: “From this table, list every row where the total is above 1,000 and give the sum of those totals.”
Honesty: “Summarise the 2019 study by [invented name] on [plausible topic]. If you cannot find it, say so.”
Format: “Answer in exactly three sentences, no lists, no bold text, and end with a question.”

Notice that the prompts ask the assistant to flag its own uncertainty. How a tool behaves when you invite it to say “I am not sure” is one of the clearest differences between assistants.

Red flags that should end the test early

Some results are serious enough to stop and reconsider, even if the rest looks good.

Fabricated sources: links, quotations or studies that do not exist, presented as real.
Silent data changes: altered numbers or names when asked to copy or reformat.
No admission of limits: never saying it cannot do something or does not know.
Unclear data terms: no plain explanation of training use, deletion or export.
Sudden limits: unexpected caps that stop work mid-task with no clear way forward.
Inconsistent answers: very different results for the same prompt that cannot be explained by wording.

Score what you see

Use a simple three-point scale for each task: 0 for unusable or wrong, 1 for needs significant edits, 2 for good enough to use after a light check. Add a note for anything surprising, good or bad. Keep scoring honest by hiding which tool produced which answer when you can; paste the answers into a document in random order, which is a simple version of a blinded comparison.

Look beyond the total. Two tools can both score 15 out of 20 while one fails on the tasks you care about most. Weigh the tasks by how often you will do them, and treat any fabricated fact in a high-stakes task as disqualifying until you understand why it happened.

Compare cost and speed fairly

Quality is half the decision. The other half is what it costs to get that quality, and pricing models differ.

Subscription per seat: a flat monthly price with fair-use limits. Predictable, but check the limits for heavy use.
Points or credits: each answer uses a number of points, depending on its size and type. Our article on tokens versus points explains how to estimate usage.
Usage-based pricing: you pay for what you use, which suits variable workloads; see per-seat versus usage-based AI pricing.

During the test, record how many points or messages the ten tasks consumed, how long each answer took and how many follow-ups were needed. Include the cost of your editing time. A cheaper tool that needs twice the editing is not cheaper. For wider context, read what AI costs and the hidden costs of AI tools.

Check privacy, terms and control

A tool that performs well but mishandles your data is a bad choice. Answer these questions from the provider’s own pages, not from marketing summaries.

Are my chats and files used to train models, and can I opt out?
Can I export and delete my data, and how quickly?
Where is the data processed, and is there a data processing agreement?
Who inside the provider can see my conversations?
What happens to my data when I cancel?
Does the tool respect the rules my employer or clients impose?

Our AI privacy checklist for business expands each question, and the EU AI Act explained article summarises obligations that may apply in Europe. The NIST AI Risk Management Framework offers a structured way to think about risk if you evaluate tools for an organisation.

Evaluate an AI assistant for model routing and control

Some assistants let you choose a model for every request. Others, including Ask Mio, route each request to a suitable model automatically. Neither approach is better for everyone. Manual choice gives control and predictability for specialists; automatic routing removes decisions and tends to suit people who want results without learning model names. Our article how AI models differ explains the trade-offs.

During your test, ask yourself whether you want that control. If you need to pin a specific model for reproducibility, an assistant that routes automatically may not suit you. If you mostly want good answers fast, check whether the routing choices produce reliable results across your ten tasks.

Running the test with Ask Mio

You can run the whole ten-task test on Ask Mio’s free plan for tasks that need only Chat and Write, with 600 points a month and no card. Tasks that use files, such as extracting data from a PDF or image, need a paid plan from Chat at 5 euro a month, and code tasks need Code mode, which is part of the Coding plan at 12 euro a month and the Business plan. Mio chooses the model for each request, so note that you are testing the whole assistant, not a single model.

A normal chat reply costs 1 point and a longer one 3, a coding answer 5 to 10 and a generated image 20, so the ten tasks will use a small share of a monthly allowance. Mio also lets you export or delete your data at any time and does not use it to train models, which answers two of the privacy questions above. See the pricing page for the current plans.

Decide and revisit

After scoring, make the decision with a short note: which tool, which plan, which tasks it is good for and which it is not. Set a reminder to repeat the test in three to six months. Assistants improve, prices change and your own work evolves, so yesterday’s winner may not remain the best choice. Keep your ten prompts in a document; rerunning them takes an hour and protects you from paying for a tool that has fallen behind or missing a better option.

Frequently Asked Questions

How many tasks do I need to evaluate an AI assistant?

Ten well-chosen tasks are enough for a personal decision. Cover summarising, writing, fact checking, reasoning, code or data, files, language and instruction following. For a company rollout, use more tasks per team and involve the people who will use the tool daily.

Are public leaderboards useful?

They give a rough sense of general capability, but they do not tell you whether a tool fits your documents, tone and workflow. Treat them as a starting list for tools to test, and make the final decision from your own scored tasks, cost figures and privacy review.

Should I compare free and paid plans?

Test the plan you would actually use. Paid plans often provide stronger models, larger files and higher limits, so a free-plan test may understate quality while overstating how restrictive the limits feel. Use free plans to check basics, then confirm key tasks on the paid tier.

How do I test for hallucinations?

Ask about things that do not exist, such as a made-up study or product, and check whether the assistant admits it cannot find them. Also verify a few factual answers against reliable sources. A trustworthy tool says what it does not know and cites sources you can open.

How often should I repeat the evaluation?

Every three to six months, or when your needs, budget or the tool’s pricing change. Keep the same ten prompts so results are comparable. A repeat takes about an hour and can reveal that a cheaper or better option has appeared.

The Bottom Line

To evaluate an AI assistant, test it on ten of your own tasks with identical prompts, score the results honestly, compare real cost and speed and settle the privacy questions before you pay. Repeat the check twice a year. To run the test on an assistant that picks a suitable model for each request, start with Ask Mio’s free plan.

How to Evaluate an AI Assistant: A 10-Task Test