LearnAI

LearnAI turns a topic or an uploaded document into a structured course: modules with explanations and examples, a ten-question quiz, and a timed assessment graded on submission.

It is a complete application rather than a pipeline demo — an agent system, a retrieval layer, a document processor, authentication, a relational schema and a server-rendered interface, built end to end.

What it does

Two inputs. Name a topic and the planner researches it and designs the curriculum itself. Upload a PDF, DOCX, TXT or Markdown file up to 20 MB and the course is built from that document, with [Page N] and [Section: X] citations carried into the generated text.

Eight output formats. The same course scaffolding renders as a standard course, a mentor-and-learner conversation, a podcast script with [INTRO]/[SEGMENT]/[OUTRO] markers, an interview of 8–12 Q&A pairs per module, condensed study notes, a flashcard deck, a case study, or a step-by-step tutorial with Try This checkpoints.

Seven of those have their own system prompt and their own Pydantic output class; standard routes to the base chain. The formats are differentiated almost entirely through the JSON schema’s field descriptions, which is what actually reaches the model through the function definition.

For the three dialogue formats, speaker names are generated once per course and injected into every module prompt, so the cast stays consistent instead of drifting between modules.

Assessment. Ten multiple-choice questions with explanations, plus a longer timed paper mixing multiple-choice and short answer, worth roughly fifty points. Short answers are graded at submission by comparing the learner’s response against the expected one, returning a verdict and written feedback. Every attempt is persisted with score and time taken.

The generation pipeline

Five agents in a LangGraph StateGraph, sharing a Pydantic state object:

1. Planner — a ReAct loop. With a document it calls search_document; without one it calls research_subtopics. It decides how many modules the course needs rather than taking a fixed count, and is capped at eight tool iterations for document courses, five for topic-only.
2. Writers — every module written concurrently through asyncio.gather, each with its format-specific prompt and its own retrieved context.
3. Reviewer — scores each module out of ten and rewrites anything below seven. For document-based courses it then runs a separate faithfulness check against the source passages, and rewrites a second time if the module is judged unfaithful.
4. Quiz validator — generates ten questions, then independently re-solves every one.
5. Assessment validator — the same cross-check across the longer paper, skipping short-answer questions, which cannot be verified this way.

The two validators fan out from the reviewer and run in the same superstep.

Two decisions here I would defend. The graph has no conditional edges — seven static edges and one diamond. Every branch, including the rewrite threshold and the faithfulness verdict, is plain Python inside a node. Routing logic in the graph would have made the topology describe the control flow twice, once in edges and once in the code that decides them.

The ReAct loop is hand-written rather than create_react_agent: bind_tools, a bounded for loop, and explicit handling of each tool result. A tool that raises returns its error to the model as a ToolMessage instead of propagating, so the agent can recover inside its own loop.

Every LLM call in the system runs at temperature=0, and all twenty-two chains use with_structured_output against a Pydantic schema — there is no free-text parsing in the generation path.

Document processing

Extraction is structure-aware and differs per format.

PDF is read with PyMuPDF. Headings are detected from the span dictionary using font size and weight — a line qualifies at 16pt and above, or at 12pt when bold and under 100 characters — and the heading level is derived from the size. Tables are located with find_tables() and converted to GitHub-style Markdown pipe tables. Page numbers are embedded inline as [Page N] markers, which the chunker later parses back out to attach page metadata.

DOCX is read with python-docx, where headings come from paragraph style names and are rewritten into Markdown # syntax in the text stream — which is what lets one chunker handle both formats.

Chunking is paragraph-based rather than a sliding character window: 1,000 characters with 200 of overlap, split on blank lines. A heading forces a chunk break whenever the section name changes, and no overlap is carried across that break — a section boundary is a hard cut, so unrelated sections never merge into one chunk. Any chunk that still exceeds 1,500 characters is split with an 800-character stride.

Indexes are cached per document with a one-hour TTL, so analysing a file and then generating from it reuses the embeddings rather than paying for them twice.

Retrieval

Retrieval is hybrid. FAISS handles semantic similarity, BM25 handles exact keyword matching, and the two ranked lists are fused with Reciprocal Rank Fusion at the standard constant of 60, each retriever over-fetching three times the requested depth before fusion. Semantic search alone misses the specific term a learner typed; keyword search alone misses the paraphrase. Both arms are individually guarded, so if one fails the search degrades to the other rather than erroring.

Before embedding, each chunk receives a one-line LLM-written description of where it sits in the document — Anthropic’s Contextual Retrieval, which they report cuts retrieval failures by around 49%. Contextualization runs across the corpus concurrently under a semaphore of five.

That technique has a trap worth naming. Those synthetic context lines now live inside the chunks, and handing them to the writer makes it describe the document instead of teaching from it. So both indexes are built from the contextualized text while the store retains the originals alongside them, and retrieval returns the originals:

# Return original chunks (better for LLM synthesis)
for idx, _score in scored[:k]:
    if idx < len(self.original_chunks):
        "text": self.original_chunks[idx],

Same index, two jobs, deliberately separated. Nothing in the generation path ever touches the contextualized text.

Verifying the answer key

Everything above is engineering. This part took the thinking.

A model asked to write ten multiple-choice questions produces ten plausible questions, four plausible options each, and one marked correct. Some of those answer keys are wrong, and nothing in the output indicates which ones. A wrong key looks exactly like a right one until a learner answers correctly and is told otherwise.

The obvious defence — asking the model to check its own answer — does not work. Shown its own prior answer, a model agrees with it, because the answer is in the context and agreeing is the path of least resistance. The check returns output shaped like verification while carrying none of the information.

The requirement was therefore not “check the answer” but get a second opinion that cannot see the first one. Each question goes to a separate call that solves it from scratch, with the options re-numbered from zero and the claimed answer withheld:

async def verify_answer(question: str, options: str) -> str:
    """Independently solve a quiz/assessment question step by step
    WITHOUT knowing the claimed answer. Evaluates each option to determine
    the correct one. Use this to cross-check generated answers."""

It returns a structured verdict, the index is parsed and bounds-checked against the four available options, and where the two disagree the independent answer wins and the correction is logged:

if verified_idx is not None and verified_idx != q["correct_index"]:
    logger.info("[Agent:QuizValidator] Corrected: '%s' %d→%d",
                q["question"][:50], q["correct_index"], verified_idx)
    q["correct_index"] = verified_idx

Two calls to the same model, differing only in what one of them is permitted to see, disagree often enough to justify the call.

Security

The application is session-based and multi-tenant, so the controls are part of the design rather than an afterthought:

• Passwords — bcrypt with a per-password salt, behind a policy requiring 8–128 characters with upper case, lower case, a digit and a symbol.
• Sessions — signed cookies via itsdangerous, set httponly and samesite=lax, marked secure over HTTPS, with a thirty-day lifetime. Login clears any existing cookie before issuing a new one.
• CSRF — every non-GET request must present a signed, time-limited token, checked in middleware before the route runs. The token is accepted from an x-csrf-token header or a form field, with the request body parsed directly so the route still receives its stream intact.
• Rate limiting — a sliding window over failed logins, keyed by IP and email, blocking after five attempts and cleared on success.
• Authorization — every route that loads a record checks ownership against the session user and returns 403, so a valid ID belonging to another account is not accessible.
• Prompt injection — user-supplied topics and module titles are stripped of control characters, collapsed, and truncated before entering any prompt.
• XSS — Jinja autoescaping throughout, with no template bypassing it, and all AI-generated Markdown parsed from textContent and passed through DOMPurify before it reaches the DOM. The seven format-specific post-processors each re-sanitize their output before assignment.
• Uploads — a MIME allow-list with an extension fallback, a 20 MB cap, and an empty-file check.

Interactive API docs are disabled, and a catch-all middleware logs tracebacks server-side while returning a generic message to the browser. Login and registration return deliberately indistinguishable messages so neither confirms whether an account exists.

Data and delivery

Seven SQLAlchemy models with application-generated UUID keys:

Relationships cascade all, delete-orphan the full depth, so deleting a course removes its modules, quizzes, assessments and every attempt against them. Questions live in JSON columns rather than their own tables, which keeps the model’s structured output intact end to end. Every route that walks a relationship eager-loads it with selectinload, so no page issues N+1 queries.

The interface is server-rendered Jinja2 with HTMX swapping fragments returned by the same routes — new course cards prepend into the grid, module bodies swap into a viewer pane, quiz submissions replace themselves with results. There is no build step and no SPA framework. Markdown is rendered client-side, and the CSRF token is injected into every form and every HTMX request by a global hook that re-runs after each swap, so swapped-in fragments stay wired up.

The principle behind it

The same idea appears twice in this system, and it is the one I would carry to the next project.

A model reviewing its own output is not a second opinion — it is the same opinion with more confidence. Getting a real check means controlling what the second call is allowed to see: hide the claimed answer and re-solve the question; search one representation of a document and generate from another.

Both improvements came from restricting context, not from writing a better prompt.