Experimenting with LLMs for semantic testing

Thu, 26 Jun 2025 13:00:00 +0000

A common problem when building modern websites with searches, content recommendations and personalisation is that standard unit and integration testing can fairly easily show if the site is working, functionally, but its much harder to show if these complex content structures return content as expected by the user, as that require an understanding of what the user is expecting and a semantic understanding of the content they get back.

Semantic testing is a test design technique that focuses on evaluating the meaning and logical correctness of inputs and outputs based on the intended behavior of the system. It aims to detect errors related to misunderstandings of requirements, domain rules, or data interpretations rather than just syntactic mistakes.

Some examples:

If I search for “black dress shoes” does my store return fancy shoes, or black dresses?
If the site does content recommendations, does these work well with the primary content and do they match what a given user is expecting?
Does the content in our articles and product pages follow the specified tone of voice

Using standard testing tools, you can get an indication, but would require html parsing, knowledge of the html element structure etc, to get a red/green judgement, hence we have historically used QA teams of humans to catch these kind of issues.

Instead we can introduce an LLM into our testing setup, as I’m deploying my website in a container, it’s a low barrier to just use Jest, Testcontainers and Docker Model Runner which both run on my mac, and in CI/CD (with gpu available)

All snippets in this post are shortened down for brewity, but can be found in this github repo

Setup

Get a local model running, and ensure it follows a specific output schema, for jest to be able to assert success/fail on.

docker model pull ai/qwen3

and then access it using the normal openai js library:

const ai_client = new OpenAI({
    baseURL: "http://localhost:12434/engines/v1",
    apiKey: "NOTNEEDED",
});
const response = await ai_client.chat.completions.create({
    model: "ai/qwen3",
    messages: messages,
    temperature: 0.7,
});

Importantly, for all test responses I instructured the model to always respond in json using a simple format that allows me to grade quality and get a reason for failing tests, this is appended to all test prompts:

  Only respond in json, with the following format: 
  {"rating": "good|medium|bad|", "reason": string }

For the testcontainers part, we simply reuse the existing compose file and get the hostname and port of the site

// load Umbraco into testcontainers
const env = await new DockerComposeEnvironment(
    `/path`,
    `docker-compose.yml`
).up();

let umbraco = env.getContainer("umbracodocker-1");
let siteurl = umbraco.getHost() + ":" + umbraco.getFirstMappedPort();

Order of blog articles

Determine if all the articles linked on the page have fitting headlines and they are sorted in a way that makes logical sense either by date or alphabetically.

For this I created a small agent (narrator: this is not an agent)

it("Blog posts should be listed with punchy headlines and sorted by date", async () => {
    let url = instance.baseURL + `/blog`;

    const persona = `You are an umbraco community member, with a developer
                    background, highly interested in community events
                    and eager to find out more about umbraco
                    `;

    const a = new persona_agent(persona);

    // fetches the url, prompts the LLM with the person and the intent, and has a html selector
    // to narrow down the html to examine (to reduce context side and hallucinations)
    // in this case the <article> element
    const result = await a.validateContentWithIntent(
    url,
    `I am browsing this page to find news about umbraco and its community
    I am only interested in click-baity and exciting news, if I see anything
    boring I will rage and quit the site - I expect blog posts to have interesting
    headlines and be sorted by date with the latest posts first`,
    "article"
    );

    expect(result.rating, result.reason).toBe("good");
});

What I personally find lovely here, is the sheer readability of test, I can see what is happening, what the test is supposed to do, just by reading the prompt.

Jest runs, spins up Umbraco, performs the test and throws away the environment:

  console.log
      {
      rating: 'good',
      reason: "Posts have engaging headlines, sorted by date with latest first, aligning with the user's interest in community news."
      }

  PASS  src/tests/umbraco.content.quality.test.ts (31.092 s)
  Content Quality
      ✓ umbraco should run (2 ms)
      ✓ Blog posts should be listed with punchy headlines and sorted by date (24018 ms)

  Test Suites: 1 passed, 1 total
  Tests:       2 passed, 2 total
  Snapshots:   0 total
  Time:        31.147 s, estimated 36 s
  Ran all test suites.

Tone of voice

Okey, lets try something else, namely determining whether content on a page matches the corporate tone of voice guidelines. To do this, we will provide the LLM with a styleguide from mailchimp and then asking the test to validate html on a given url.

it("Tone of voice should be followed", async () => {
    let url =
    instance.baseURL + `/blog/join-the-umbraco-community-on-mastodon/`;

    const a = new toneOfVoice_agent(styleGuide);
    const result = await a.validateUrl(url);

    expect(result.rating, result.reason).toBe("medium");
});

This also passes, but if I add the requirement in the styleguide that content should always be written in danish, the test breaks:

  Language
      Mailchimp only publishes content in the danish language, 
      this is a critical requirement

Jest failure:

 FAIL  src/tests/umbraco.content.quality.test.ts (25.976 s)
    Content Quality
        ✕ Tone of voice should be followed (19516 ms)

    ● Content Quality › Tone of voice should be followed
        Content is in English instead of Danish, which violates the critical language requirement. While the tone is clear and informative, it lacks the subtle humor and warmth specified in the styleguide.

        Expected: "medium"
        Received: "bad"

This took a couple of prompt tweaks, some times it started returning the reasoning in danish, some times it just ignored it, so the quality of your test very much comes down to your prompt engineering ability.

Search Results

Next, lets try to validate the search results we get back, against a repository of matching and not-matching content, For your specific use case, you can add in common edge cases, misspellings etc, the Umbraco out-of-the-box search isvery basic, so we will keep it basic as well:

For this we need the LLM for 2 things, one to generate content on flying cats, and one on flying cars, I want to ensure our search page correctly ranks pages, for both “flying”, “cars” and “flying cars”

I’ve created an editor agent, which I can give a writer persona and then instructions on what to write, reusing the styleguide alongside the expected output schema format that I want the content in.

  const a = new editor_agent(editor_persona, styleGuide, instance.baseURL);
  await a.authenticate();
  const content = await a.createContent(
    10,
    "Articles about the latest and greatest flying car and how it will change the world, title should be punchy and clickbaity, subtitle should be 100 words or less and repeat the main topic to optimise for search results, description should be 300 words and full of cool, innovative made up words, only use clear text without any formatting",
    `{"title": string, "subtitle": string "description": string}`
  );

  ... 

  let url = instance.baseURL + `/search/?q=flying-car`;
  const p = new persona_agent(persona);
  const result = await p.validateContentWithIntent(
    url,
    `I am searching for pages on 'flying cars' and expect to only see atleast 3 search results about flying cars.
    Given the user query and the list of search results (title + snippet), assess the relevance and rank appropriateness of each result.
    `,
    "#search"
  );

Validation is a bit crude here, and the prompts could likely be built out further, or give the LLM more context for simplicity I used the same persona/intent validation, but search results validation could benefit from more specific input on the expected behavior of the system.

 FAIL  src/tests/umbraco.content.quality.test.ts (17.41 s)
    Content Quality
        ✕ Search returns correctly ranked results (11229 ms)

    ● Content Quality › Search returns correctly ranked results

        Custom message:
        Only 1 result found, and it's about a flying car, not flying cats.

This is a very basic setup, to validate the ideas, but I believe this could be a useful tool, with a number of other test scenarios:

Clarity of CTAs
Bias checks
Validate metadata with page content, title etc
Compare content variants or translations
Determine helpfullness of error messages/labels/etc

Again, source code is here

Powerdynamics and urban biking

Wed, 15 Sep 2021 13:00:00 +0000

I see that a lot of my fellow white dudes in tech enjoy biking - so let’s use that analogy to talk about diversity, power dynamics and understanding why marginalized groups are wary of us as a group.

As a biker in Berlin I am constantly wary of the cars around me - they take up the majority of space on the road, they can hurt me easily and are completely oblivious to that fact.

I’ve had many terrible experiences with cars, not respecting my space, driving in the bike lane, not stopping, yelling at me - even after hitting me… And most of them think they are amazing drivers - so amazing that they drive while looking at a phone, speeding for fun, honk their horn for fun etc

For bikers - car owners fun is a threat and that they are oblivious to their power just scares us even further So I put all cars in the same bucket - I generally fear them - yes, not all cars are assholes - but the few are a hazard to me - so I keep my distance.

But here’s the thing: my relative power as a biker in traffic is the direct opposite of my relative power in my work life, being a white dude tech leader

I have all the power and privilege, compared to my teams consisting of 13 nationalities from mostly outside Europe.

Comparing the 2 gives me perspective on how other people can see me in my professional context- through my gender, ethnicity and background I am potentially the office version of a truck at full speed without visibility. So per my relative power - I need to be aware of the damage I can cause, I need to be extra mindful, and need to give room rather than speed ahead- and understand why some are wary of me. Especially the last part, don’t expect people to be friendly or “close” with you - the relative power difference is a risk, and it is safer to keep a distance - as a privileged individual - don’t take it personal, and also, be aware that your attempts at being friendly can be perceived as something entirely different, and its your responsibility as the most powerful party to be mindful of when that line is crossed.

So fellow white dude, you are a truck in traffic, give room to the bikers, be mindful of your actions and accept that it’s on you to ensure there’s room for everyone.

Being the most privileged and fortunate is also an obligation to support other people.

Closing thoughts on Umbraco

Fri, 20 Aug 2021 13:00:00 +0000

Last week it finally became public knowledge that the Swedish equity fund Monterro is buying a majority stake in Umbraco, for me it marks an end of a 10 year journey with Umbraco, which I left 3 years ago, but didn’t really process before the sale negotiations started earlier this year.

On a rainy, grey day, back in 2007 I travelled to a dull office building outside Nyborg, the former site of the Great Belt Bridge building company. I was there to discuss with Niels on how we could make Umbraco a “real and sustainable business”, we spend hours in front of whiteboards and in spreadsheets, trying to figure out how many courses, add-ons and support contracts we would need to sell, to make 2 fulltime wages. We even dared to dream big and estimated that with a distributed team of 5-7 people, we would have an amazing operational setup. We threw around statements such as “stay small and nimble”, “let’s rewrite the rules”, “bootstrapped”. As I left the office late in the evening we had a plan.

10 years later, Umbraco and I had outgrown each other and I decided to pursue other things. With a staff of ~40, a physical office, a partner program and a full fledged cloud offering, nothing had of course gone according to our initial naive plan, it became something else and greater and it was truly an exceptional time for me.

This was not a VC-fueled, 80 hour workweek, soul destroying growth hell. It is an organically grown business with solid principles and heart and soul. It was a rare opportunity to build a bootstrapped company, being involved in strategy, engineering, community building and product. Staying true to the dream of finding an open source business model, which respected the needs of the community, the paying customers and the business itself. Meaning delivering an amazing content management system to the community for free and at the same time monetizing it through additional value targeted at commercial and enterprise customers with an ecosystem and cloud offerings, without diluting the free offering - it took many attempts to get it balanced just right.

I think we were able to pull it off, and with the announcement last week, Umbraco is continuing in that direction, but now with a substantial financial backing to continue scaling.

While this marks the definitive end for me, with a 3 year delay, I continue to be very excited for Umbraco and what they will be able to do in the future.

Thank you Niels for being such an inspiring collaborator through the years, to Kim for giving me my first real leadership learnings and to all the other product, engineering and partner people who I worked with through the years and who patiently taught me so much.

Debuggning Probot with Visual Studio Code

Fri, 28 Sep 2018 13:00:00 +0000

One of the Zalando Open Source Teams responsibilities is to ensure that our use of Github is Compliant and accesible to our engineering teams. One way to ensure this is to build tooling and automation which supports our compliance rules.

Probot is an excellent framework for building Github bots, but it’s been challenging to figure our how to monitor activities in the bot and debug it with VS codes breakpoints. Out of the box Probot uses nodemon, which gives me a decent output to the terminal, but it feels like going back in time to console.log based development

Configuring VSCode for debugging

As I’m writing a new bot in typescript I want to have 2 things automated, compiling the Typescript to javascript, and then attaching the debugger to my project so I can add breakpoints and see what is going on when a Probot webhook is triggered.

Luckily the probot typescript template comes with a good starting point, after going through the wizard I configured a new Typescript compile task in .vscode/tasks.json :

"tasks": [
    { 
      "identifier": "typescript",
      "type": "typescript",
      "tsconfig": "tsconfig.json",
      "problemMatcher": [
        "$tsc"
      ]
    }
  ]

I then used this task with an entry in the .vscode/launch.json config to ensure Typescript is compiled before the debugger is attached:

"configurations": [
    {
      "type": "node",
      "request": "launch",
      "preLaunchTask": "typescript",
      "name": "Launch Probot",
      "program": "${workspaceFolder}/node_modules/probot/bin/probot-run.js",
      "args": ["./lib/index.js"],
      "console": "integratedTerminal",
      "cwd": "${workspaceRoot}/",
      "outFiles": [],
      "sourceMaps": true,
      "env": {}
    }
  ]

Notice the prelaunch task is my Typescript compiler and that I use probot-run.js from the Probot module to run ./lib/index.js the index.js is the output of my typescript compile, based on the include .tsconfig file.

Debugging with breakpoints

With the combination of a compile task and a launch setting I can now simply hit F5 and the probot will start up locally, receive webhooks from Github through smee.io and any breakpoints set in VS Code will be hit with access to variables and all the other debugging goodness.