Project Blog: Florence

A project blog by

Yohann Ian

AI/ML Engineer

I initially ran Florence on AWS. If you want to try it out, visit the link below. But bear in mind, I turned it off on AWS because it takes money to keep it running and to make API calls. Instead, I deployed the UI on Netlify for free… call it a mannequin. It ain’t the real thing, but I thought it would be nice to put it out there.

Netlify LINK: https://florence-ui-preview.netlify.app/
GitHub LINK: https://github.com/Yohann-Ian/florence-literature-rag

Florence: Retrieval Augmented Generation Agent

I built a RAG agent for literary analysis and deployed it on AWS. Yes, I like literature.

  • RAG Pipelines
  • LangGraph
  • FastAPI
  • Docker
  • AWS ECS
  • ChromaDB
  • Orchestration
  • Ragas Evaluation
  • HITL Systems

How do I combine my love for creating things with my love for literature?

I develop a RAG.

Delightful for writers.

Demonstrative for AI Engineers.

I built this Literary Analysis RAG agent from scratch and ran it on AWS. It’s a highly technical end-to-end project, showcasing the highly-sought skills and tools for deploying AI-enhanced systems in the workplace.

Let me take you through.

Project Florence

An AI Agent that Runs Comparative Analysis on Literature Books

“Whatever talents a person may possess to amuse and instruct others, be they ever so inconsiderable, he is yet bound to exert them: if his attempt be ineffectual, let the punishment of an unaccomplished purpose have been sufficient; let none trouble themselves to heap the dust of oblivion upon his efforts.”

― Percy Bysshe Shelley, Prometheus Unbound

This project touches many technical concepts and offers a first-hand experience of building an AI Agent using Langgraph.

Demo videos. But seriously, read the rest of the page.

A full Florence run on localhost. I did not cut short or speed up the video. You’'ll see it exactly as it is. You can ask Florence just about anything. Out of convenience, I included buttons to prefill the input with a literature question. Florence is NOT hardcoded to answer these questions from memory.

Running it on Amazon AWS

I successfully served Florence on Amazon ECS, with CloudWatch Logs in the background.

The Machine

A quick look at the inner machinery. I’ll go into detail further below. It’s quite intricate. If you come from electrical engineering, you’d recognise this as a state machine. (My impression going into AI agents and Langgraph)

The RAG Agent

Just to be clear with you, I’m not building a town.


The RAG agent’s name, Florence, was inspired by the Renaissance period, that originated in Florence, Italy.

Before I get technical, I want you to know what it does and see how it looks like from a UI.

Answers are drawn from the Corpus, the collection of books that were vectorized, usable by the agent.

So How Does It Work?

You Ask the Question

Florence Searches the Corpus

You Get the Answer

You Ask

“How does the Party in Nineteen Eighty-Four control what people think?”

Florence answers:

In Nineteen Eighty-Four, Orwell shows the Party controlling thought through three overlapping mechanisms. First, through the mutability of the past: Winston's own job at the Ministry of Truth is to rewrite historical records so that whatever the Party currently says has always been true, leaving no external record against which memory can be checked. Second, through the narrowing of language: Newspeak is designed so that dissenting thoughts become literally unsayable, and eventually unthinkable. Third, through doublethink, the trained mental discipline of holding two contradictory beliefs at once and accepting both, which lets a Party member accept a lie as truth while still functioning in the world.

Confidence 0.82
Faithfulness 0.78
Sources 4 passages from Nineteen Eighty-Four

Florence UI

For simplicity, I’m just gonna show you the image of the UI. Running the UI on Squarespace is not feasible.

Again, if you wanna view a dummy version in your browser, click here. It’s a dummy, though. It doesn’t have a backend.

Under the Hood

This is technical. A few subsystems: the preprocessing step, the main agent, AWS navigation.

Offline ingestion  ·  runs once, ships with the image Books EPUB · PDF · TXT Chunking HIERARCHICAL/SEMANTIC/FIXED Embed BGE-LARGE · LOCAL GPU ChromaDB VECTOR INDEX · SHIPPED IN IMAGE Serving layer  ·  AWS  ·  public URL You ASK A QUESTION AWS ALB HTTPS · 300S IDLE AWS ECS · Docker FARGATE · 1 vCPU · 2 GB FastAPI UVICORN · PYDANTIC graph.invoke() LANGGRAPH ENTRY Enters agent Florence RAG  ·  agent state machine (LangGraph) Domain router HAIKU CLASSIFIER · YES/NO Valid Invalid END Retrieve TOP-K = 5 · CHROMA QUERY READS Grade documents 5× HAIKU · PARALLEL · YES/NO Enough Rewrite query SONNET · REPHRASE Insufficient Retry Generate SONNET · MAX 4096 · TEMP 0.3 Check hallucination LLM-AS-JUDGE · JSON SCORES Grounded Ungrounded HITL checkpoint interrupt() · CONFIDENCE ≥ 0.7 Pause You (review) APPROVE OR REJECT Reject + feedback (≤ 3) Approve You (read answer) GROUNDED · SOURCES · SCORES
Process Data store You / endpoint
Offline ingestion This is the library work. Books get broken into chunks, turned into vectors with my GPU, and stored in ChromaDB. Done once. The whole index rides inside the Docker image so the deployed agent carries her memory with her.
Serving layer · AWS This is the front of the house. Public URL, HTTPS, AWS load balancer sitting in front of an ECS Fargate task running a single Docker container. FastAPI takes your request, Pydantic validates it, then hands it into the agent graph. One container running the whole show.
Florence RAG · agent This is the librarian. Seven nodes, three loops. Bad retrievals get rewritten. Ungrounded answers get regenerated. Low-confidence answers pause and ask you to weigh in. Nothing leaves the machine that hasn't been checked.

Check out my GitHub

Here’s a peek of the agent states in the image on the right. You’ll see the full thing on my GitHub. Continue downwards for my insights on each step I took.

Let’s Begin

The Process Map

We’ll get lost without it.

Florence  ·  The apparatus

Technical Narrative: Use-Case Flowchart

From a closed literary corpus to a grounded, self-checking answer. Seven stages, wired as a state machine with two retry loops and a human-in-the-loop pause. The same question travels the same path each time.

  1. 01

    Ingest & chunk the corpus

    Two books are broken into hierarchical passages: small child chunks for precise retrieval, larger parent chunks for surrounding context. Two levels, one index. Intially, I had 12 books in mind, but I had to reduce the scope for feasibility.

    Libraries LangChain text splitters, EbookLib, PyMuPDF, BeautifulSoup
  2. 02

    Embed & store

    Each chunk is embedded locally with BAAI/bge-large-en-v1.5 and stored in a ChromaDB vector index. The index is built once and shipped with the container; the deployed agent carries its own memory.

    Libraries sentence-transformers, langchain-huggingface, ChromaDB, PyTorch
  3. 03

    Route by domain

    An incoming query first meets a fast LLM classifier (Haiku) that decides whether the question is plausibly about the corpus. Prompt-injection attempts are filtered here too.

    Libraries langchain-anthropic, LangGraph
  4. 04

    Retrieve & grade in parallel

    Top-K passages are pulled from the index. Each one is graded for relevance by five concurrent Haiku calls. If too few survive, the query is rewritten and retrieval is retried up to twice.

    Libraries ChromaDB, langchain-anthropic, concurrent.futures
  5. 05

    Generate & audit

    Sonnet drafts the answer from the surviving passages. The result is then audited against those same passages by an LLM-as-judge step that returns faithfulness and confidence scores. Ungrounded answers are regenerated once.

    Libraries langchain-anthropic, Ragas (evaluation)
  6. 06

    Pause for human review HITL loop

    If confidence is low, the graph pauses and surfaces the draft to a human. Approve accepts it. Reject with feedback loops back to generation with the feedback in the prompt. Capped at three rejections.

    Libraries LangGraph (interrupt, checkpointer)
  7. 07

    Serve & deploy

    The whole graph lives behind a FastAPI service and a single-page frontend. The image is containerised, pushed to ECR, and run on ECS Fargate — no GPU, model baked in, API key injected at runtime.

    Libraries FastAPI, Pydantic, Uvicorn, Docker, AWS CLI
    DockerImage
    ECRRegistry
    ECSRuntime
    ALBIngress

    ▲ Every stage above runs inside the same container the URL is pointing at.

City map of Florence, Italy, by cartographer W. Hollar, originally printed in 1660, Arte of the Booke

London city map, by cartographers G. Braun and F. Hogenberg, originally printed in 1572, Arte of the Booke

Step 1: Ingest and Chunk the Corpus

Break it down.

This is where we break the source books into chunks. You vectorize the chunks with your embedding model and store them in ChromaDB. Broadly speaking, the performance of the RAG is influenced by your chunking strategy.

I got this GIF from Weavegate. They've got an amazing , in-depth blog post on chunking.

The particular challenge of a literature RAG is that “meaning” is captured differently by different authors. This is the challenge. In more modern prose, meaning is quite straight forward. But in say, Shakespeare, you would have to read an entire stanza and abstract rather broadly to get the meaning conveyed.

Almost all languages have recursion. Meaning can be several clauses long or captured in a single clause within a very long sentence. Consider this line from 1984: "The past, he reflected, had not merely been altered, it had been actually destroyed, for how could you establish even the most obvious fact when there existed no record outside your own memory?" This sentence, for example, conveys several points of information. Chunking it too short would miss out on some details that could be important. Chunk 1: The past, he reflected, had not merely been altered, it had been actually destroyed, for‍ ‍Chunk 2: for how could you establish even the most obvious fact when there existed no record outside your own memory?


If you took Chunk 2 by itself, you'd see a rhetorical question about establishing facts, but you'd lose the claim that grounds it. The reader would not know that Orwell is arguing the past has been destroyed, only that memory is unreliable. As a writer, he combines the fact in the 1st sentence, and the meaning of the fact in the 2nd.

The Hierarchical Chunking Strategy is Selected for Comparing 1984 and Brave New World

Both are science fiction dystopias, but in opposite directions. In 1984, pain is used to control people. In Brave New World, pleasure is used to control people. Thematically, they’re ripe for comparison.

However, their proses are markedly different. A passage in 1984 contains large ideas, which contain sub-ideas. A passage in Brave New World typically contains two contradicting ideas as the means of illustrating cognitive dissonance, for example:

“I have all the pleasures in the world. I am terribly unhappy today.”

Semantic Chunking would not capture this. Nor would it capture the nested meaning in 1984. Hierarchical Chunking proves more versatile and neutral, giving the best result.

I list 3 types of chunking strategies in my code. Honestly speaking, the best approach is to choose the chunking strategy that fits the book you are trying to analyze:

  • Fixed

    • Great for Peter and Jane. But we ain’t reading Peter and Jane. Still, it’s a simplistic way to work it. Great for speed and predictable resource usage. Not good for complex literary/linguistic analysis.

  • Semantic

    • Split into sentences using grammar and punctuation. Then, vectorize using the embedding model you have. Compare the cosine distances between vectors. Distances below a preset threshold signal that chunks are semantically similar. This method hinges on adjacency, though. It doesn’t work so well for ideas that carry contradictions or tensions.

  • Hierarchical

    • Cut the text at two sizes: small child chunks (roughly 256 characters) for precise search, and larger parent chunks (roughly 1024 characters) for context. Every child sits inside a parent and knows which one. Embed only the children and store them in the vector index. When a query comes in, match against the children, but return the parent to the LLM. Search happens at high resolution, generation happens at wider scope.

“A really efficient totalitarian state would be one in which the all-powerful executive of political bosses and their army of managers control a population of slaves who do not have to be coerced, because they love their servitude.”
― Aldous Huxley, Brave New World

The BATCH_SIZE figure controls how many chunks get processed at once during embedding. Higher takes up more RAM. Lower means slower.

If you ever wanted to see vectors, this is a really interesting one for MNIST. Check it out and more here https://projector.tensorflow.org/

Step 2: Embed and Store

Vectorize, baby.

I used ChromaDB. https://www.trychroma.com/

Interesting Ramble: One time, this product owner I spoke to told me Content Writers would be ousted by LLMs because LLMs used vectors. She made a gesture with her hands to indicate that the vector moved somehow upon retrieval… like it had a magnitude and a direction and it had been sent to the surface.

And I thought to myself….what a strange thing to say.

Firstly, Philosophically: Spades, however technically sound, do not replace gardeners.

Secondly, Meta-physically and Mathematically: Classically, vectors are entities used to describe object movement in the physical world, hence they have magnitudes and directions. However, in computer science, in this scenario, vectors are just tuples, mate. They don’t move.

Okay, back to the build

Every chunk is embedded locally using BAAI/bge-large-en-v1.5, running on GPU during ingestion and on CPU inside the deployed container. Vectors go into ChromaDB as two persistent collections (children and parents), cosine similarity, HNSW indexing. The whole ChromaDB folder is baked into the Docker image, so the deployed container has no external database to phone home to. Libraries: sentence-transformers, langchain-huggingface, ChromaDB, PyTorch.

My first build used all-MiniLM-L6-v2 and retrieval was so weak that the LLM was answering from its own memory of these books instead of from the passages I gave it. Swapping to BGE-large was the largest single quality improvement in the project. For the store: local, open-source, and carried inside the container means Florence has no external dependencies at query time.

BGE-large is 1024-dimensional, meaning each chunk is represented by 1024 numbers that capture what it means. MiniLM was 384-dimensional. More dimensions means finer distinctions but larger index and slightly slower search. On this corpus size (~5,500 chunks), the tradeoff is trivial.

Step 3: Route by Domain

That’s a fancy way of saying “Check to see if the user’s question falls within the domain”

A user could ask Florence anything. Florence needs to decide at the door whether the question is one she should try to answer. Florence only accepts literature questions. It’s a design constraint. Without constraints you have no product.

What I did: It’s basically gatekeeping

  1. The query is first checked for prompt-injection patterns. If it passes, it goes to a Haiku classifier (temperature 0, ten-token ceiling) that decides in one call whether the query is about the literary corpus. YES routes to retrieval, NO returns a polite rejection. Libraries: langchain-anthropic, LangGraph.

  2. Also: The prompt-injection check runs before anything else, because a query that tries to override the system prompt does not deserve a Haiku call. It is caught at the door with pattern matching.

  3. Haiku's system prompt for domain routing tells it explicitly what the corpus is and instructs it to be generous. Shorthand titles ("1984", "BNW"), character names ("Winston", "Bernard Marx"), and thematic questions ("how does totalitarianism work in fiction") all pass.

Step 4: Retrieve and Grade

“Have you got the goods?”

Retrieval finds passages that might inform the answer. Grading decides which of those passages are good enough.

What I did Retrieval pulls the top five child chunks from ChromaDB. Grading then runs five concurrent Haiku calls (one per chunk) via ThreadPoolExecutor, each answering yes or no on relevance. Chunks that pass proceed. If fewer than two pass, the query is rewritten (Sonnet, one call) and retrieval retries. The rewrite loop is capped at two.

Why I did it this way

An earlier version graded chunks sequentially with Opus and took about twenty seconds. Parallel Haiku takes about two. That is a 10x speedup.

Retrieval uses the child chunks for search precision but fetches their parents for the LLM to work with. This is the whole point of hierarchical chunking realised at query time.

Two rewrites is the cap because a third one usually means the corpus genuinely does not contain the answer and further grinding just wastes tokens.

For example....

Query How does the Party control what people think in Nineteen Eighty-Four?
Grader 1 · Chunk 1
"Who controls the past controls the future: who controls the present controls the past. And yet the past, though of its nature alterable, never had been altered."
Pass · Relevant
Grader 2 · Chunk 2
"The whole aim of Newspeak is to narrow the range of thought. In the end we shall make thoughtcrime literally impossible, because there will be no words in which to express it."
Pass · Relevant
Grader 3 · Chunk 3
"His mother gave him a small piece of chocolate. He remembered the way she had looked at him as he took it, and the taste of it on his tongue, sweet in a way nothing had been sweet since."
Fail · Not relevant
Grader 4 · Chunk 4
"To know and not to know, to be conscious of complete truthfulness while telling carefully constructed lies, to hold simultaneously two opinions which cancelled out, knowing them to be contradictory and believing in both of them."
Pass · Relevant
Grader 5 · Chunk 5
"There was of course no way of knowing whether you were being watched at any given moment. You had to live, did live, from habit that became instinct, in the assumption that every sound you made was overheard."
Pass · Relevant
Result 4 pass · 1 fail → Four chunks proceed to generate

Step 5: Generate and Audit

RAGs need LLMs…

Evening Meeting, Vladimir Makovsky, 1875 - 1897, Russian Federation

Not exactly an Evening Meeting, but it’s things get moving on a computer.

Generate and Audit

This is where we get answers from the agent. However, as you and I know, LLM-generated answers, however supported, can be incorrect. So we have to audit them.…to deliberate on the answer, so to speak.

The awkward truth about running a RAG system on famous books is that the model already knows them. Sonnet has read Nineteen Eighty-Four. If we let it blend the retrieved passages with what it already remembers, the whole system becomes theatre: the answer looks grounded but is actually being produced from parametric memory, and the passages we worked so hard to retrieve are decoration. Generation and audit are the two moves that stop this.

Generation
Uses Sonnet at temperature 0.3 with a 4096-token ceiling. The system prompt is written aggressively against Sonnet drawing from Anthropic’s training. My generation rules force Florence to:

  1. Only make claims supported by the provided context

  2. Cite specific passages or details

  3. Acknowledge genuine ambiguity rather than collapse tensions the texts preserve

  4. Not be afraid to express an idea even if it is politically inconvenient. Literature is not always cozy.

The generate node also injects any human feedback from an earlier rejection directly into the next system prompt. This is what makes the HITL loop actually change the answer rather than produce a rerun of the same one.

Auditing
Runs immediately after generation. How it goes is: You have the just-written answer that you would like to audit. So, you run another Sonnet call to score it against the same passages. It’s kinda circular, I know. When you score, you get:

  • Faithfulness: How supported the claims are (Score 0 to 1)

  • Confidence: How well the answer addresses the question (Score 0 to 1)

If faithfulness falls below 0.7, the answer is regenerated once.

The audit uses Sonnet as an LLM-as-judge because there is no cheaper adversarial evaluator available at query time. Haiku was tried and produced unreliable JSON. Sonnet is worth the extra tokens for the reliability.

If the answer fails the audit twice in a row, it proceeds anyway with the low score attached, honestly labelled. Low scores are a good starting point for higher scores :)

Step 6: HITL… the Human Reviews

Human judgement.

That’s what this step is for.

Real production systems escalate low-confidence outputs to humans. Think support tickets to senior agents, medical diagnoses to specialists, legal drafts to partners. Florence would not be complete without it.

This is where the human is essential. My principle is: if the human cares, they should be at the center of the decision-making process, no matter the machine they’re steering.

If the confidence score falls below 0.7, the graph calls LangGraph's interrupt() primitive. Execution pauses.

  1. The state (query, draft answer, confidence score, retrieved passages) is checkpointed by MemorySaver under thread_ID, so we can survive a stateless HTTP boundary.

  2. The API returns "paused" to the frontend.

  3. The user picks Approve or Reject-with-feedback.

  4. Reject loops back to Generate with the feedback injected into the next prompt. Capped at three rejections. State is restored if run on the same thread_ID.

Captain Albert Crowe, by Yves Berube

A ship is a machine. The Captain is… well, the Captain does what he has always done.

Step 7: Serve and Deploy

Serve it on AWS or Azure or Google Cloud. I had to show Florence doesn’t just work on my computer, running on localhost. Deployment is a world of its own. Doing this surfaces issues with cold starts, health checks, timeouts, secret injection, image size, cost. I forced myself to touch all of them.

Okay, you got me…
I chose AWS specifically because ECR, ECS, and ALB fluency is more useful for the job market (on average in the U.S.) than boutique platforms. These things are NOT taught in engineering school and yet businesses are paying top dollar for people who can operate them. Funny huh? Teach your kids how to navigate hyperscalers in Grade school.

Here’s the breakdown

FastAPI
The graph is wrapped in a FastAPI service exposing GET /, GET /health, POST /query, and POST /resume. Pydantic validates requests and responses.

Frontend
The frontend is plain HTML, CSS, and vanilla JavaScript, served from a static folder. Simplicity wins here.

Docker
You gotta pack it. The whole application ships as a Docker image built from python:3.12-slim, with CPU-only PyTorch, BGE pre-downloaded at build time, and the ChromaDB index baked in. Around 4 GB total.

Amazon Web Services (AWS)
Deployed on AWS: pushed to Elastic Container Register (ECR), run on ECS Fargate (1 vCPU, 2 GB RAM), behind an Application Load Balancer with a 300-second idle timeout, logs to CloudWatch, API key injected at runtime.


BGE-large is pre-downloaded into the image at build time as its own Docker layer, so container startup does not stall on a first-run model download. This is what keeps cold starts to seconds rather than a minute.

The ANTHROPIC_API_KEY is injected as an environment variable at task runtime, never baked into the image. One does not include his API key in the Docker image.

I set up an ECS instance.

In the Japanese art of Kintsugi, you assemble broken pottery pieces using gold to make it more beautiful. A bit of an abstraction here, but real learning and value comes from doing something, especially with imperfect pieces.

My Findings and What Would Be Cool

What I learned

Three findings worth stating plainly.

One
The embedding model matters more than any other single choice in the pipeline. Everything downstream is downstream of what the retriever surfaces.

Two
There is a literary asymmetry in what RAG can do. Orwell answers well because he narrates his themes plainly. Huxley answers less well because his themes live in irony and imagery. Dense retrieval matches on semantic surface, not interpreted meaning.

Three
Using a big model for every step is expensive and often wrong. Domain routing and grading are yes/no decisions and Haiku handles them at a fraction of the cost. Generation and auditing warrant Sonnet. Choosing the right size for each step is the difference between a demo that costs cents and one that costs dollars per query.

Four
This development scaffolding generalises. Swap the corpus of novels for legal case law and you have a legal research agent; for internal documents and you have an internal knowledge tool. The mechanism (hierarchical chunking, LLM-graded retrieval, grounded generation, self-audit, human-in-the-loop) is not about literature. I just happen to like Literature over Law.

Honest limits

Florence has a small corpus, cut from ten books to make the scope of this project feasible. Expanding is straightforward but takes time.

Three things:

  • The HITL checkpointer is currently in-memory (MemorySaver). If the container dies for some reason, the user loses his session memory. A more durable backend (DynamoDB, Redis) would be good if you want to survive container restarts and perform horizontal scaling via multiple containers (to keep the service perpetually free of downtime, planned or otherwise).

  • The current confidence threshold and retry caps are hand-set. It would be good to tune them against a larger evaluation set if you wanna get more involved in the quality of output.

  • And honestly, I would love to run this on my own local LLM. You save on cost.


I like building automated things like this, especially if business logic and operations are involved.

I’ve made some I can’t share for privacy reasons.

My name is Yohann.

If you’re interested, do drop me a mail. :)

Fear not, first-world person, I speak the Queen's English and eat hamburgers. Ford Coppola is one of my favorite film directors.

I’ve worked in multiple fields, technical and non-technical, across different nationalities and styles. That’s valuable human experience. Something you don’t get, even for $100/month on Claude Max:

  • Navigating difficult stakeholders

  • Taking a joke

  • Scaling up my intelligence for the same salary

  • Succeeding in domains I wasn’t trained by a GPU for

  • Handling problems that’ll attack you on Monday

  • Joining the meeting when the product is on fire

Also, if you're gonna call me, best drop me a message on WhatsApp first introducing yourself. Scam calls have been rampant since 2020.

Email: yohannian774@gmail.com
H/P: +60122807264