""

Just a game? How the AI Act found its way into how we build software

At PyCon Italia 2026, we introduced a challenge that tests how good you are at spotting the bugs AI leaves behind. It's aimed at developers, tech leads and anyone working in tech who increasingly has to deal with AI in their daily work. Before releasing it, we asked ourselves a few questions: the same ones, as it turned out, that the AI Act asks everyone to consider.

A simple game, some serious questions

At Nephila we built a game: the AI Debug Challenge. Players have to find the bug hidden in a snippet of Python code across three levels of increasing difficulty, "from a junior-level slip to a senior-level trap". You get three shots per level, and at the end the game gives you a verdict on your skills as a debugger of AI-generated code. Are you an ace bug hunter, or do you still have some ground to cover?

The idea is simple and meant as a bit of fun: a game for people who write code every day and want to enjoy themselves, maybe have a laugh, or wince, when they read their final verdict.

There's one detail that changes things, though: both the code to fix and the verdicts are written by AI. So the challenge doesn't just generate exercises: it passes "judgement" on a person. Light-hearted as the intent is, we asked ourselves a few questions right from the design stage:

  • What effect can that feedback have on the person receiving it, and what happens if someone takes it more seriously than expected?

  • Is it clear that it was written by a machine and not by a person?

  • Could the game be taken out of context and misused?

Answering these questions felt like a priority. And so, perhaps without being fully aware of it, we put into practice the idea the AI Act is built on: risk assessment. Before building and releasing an AI-based tool, we tried to understand who would use it, whether it lent itself to misuse, and whether the way it worked or what it produced could be misunderstood or, worse, cause someone harm.

Each of these questions turned into a concrete design choice. Before we get to those, though, let's take a step back: what do we talk about when we talk about the AI Act?

The AI Act in brief

The AI Act is the EU regulation on artificial intelligence, the world's first legal framework devoted entirely to this technology. It applies to anyone who develops, distributes or uses AI systems on the EU market, and its goal is to promote trustworthy, human-centric AI.

It has been in force since August 2024 and has been widely discussed, but it's easy to dismiss as "yet another EU regulation", something for lawyers to worry about. Yet its logic is simpler than the sheer length of the text suggests: it classifies AI systems not by the technology they use, but by the impact they can have on people, especially on their health, safety and fundamental rights.

It sets out four levels of risk:

  • unacceptable: banned practices, such as social scoring and exploiting people's vulnerabilities, for example those linked to age or disability;

  • high: systems used in critical areas such as safety or critical infrastructure, or in sensitive ones such as recruitment, education or access to credit;

  • limited: systems such as chatbots or text generators, which are mainly subject to transparency obligations;

  • minimal: everything else.

The underlying principle is proportionality: the more a tool can affect a person's rights, opportunities or sensibilities, the stricter the obligations.

Three key principles run through the entire regulation:

  • assessing the impact a system can have;

  • making it clear when AI is involved;

  • maintaining human oversight of how the system works.

Read this way, what looked like a job for lawyers becomes a practical guide for anyone working with AI, at any scale.

From questions to design choices

A game like ours is clearly not a high-risk system. Still, the questions we had asked ourselves called for precise answers, and each answer became a design choice.


Boundaries: keeping the game a game

Could the game be misused?

We gave the system very clear boundaries: it won't comment on anything outside its task, and it's designed to spot attempts to trick it or push it beyond its intended scope. When players try to "cheat", it replies with a quip and goes back to the game without getting sidetracked.

""

*tongue-in-cheek replies to cheating attempts

To find out more about how the AI Debug Challenge was designed and how these safeguards were built, read these two posts on our blog:


Transparency: disclosing the AI from the start

Is it clear that the verdict is written by a machine?

On this point the AI Act is explicit. Article 50, whose transparency obligations have applied since 2 August 2026, requires that people interacting with an AI system be informed of it at the latest at the time of their first interaction, and that AI-generated content be identifiable as such.

That's why we chose to say so up front, before showing the first code snippet: an opening notice states that the exercises and verdicts are AI-generated. Players know this before they receive their final verdict.

""

*the opening notice stating that the exercises and verdicts are AI-generated


Tone and human oversight: verdicts that don't sting

What effect can that feedback have on the person receiving it?

We worked on two fronts. The first is tone: the final feedback was designed to always come across as friendly and constructive, never critical or judgemental. It's generated by AI but managed through a prompt registry (Phoenix), where our Communications team stepped in to fine-tune the tone of voice. For more on how we set this up, see We built a game about AI bugs. The hard part wasn't the AI.

""

*an example of the final verdict players receive after completing the challenge

The second is oversight: both during development and after launch, we kept an eye on what the AI was generating. In practice, this meant investing heavily in the design of the architecture behind the challenge, which you can read about in "Can you jailbreak this LLM?" The dev community tried: here's what held up, and why.

This is the principle of human oversight, which the AI Act makes mandatory for high-risk systems (Article 14) and which we applied to our challenge anyway: AI generates, a human reviews.

Learning comes first: knowing what to ask

There's one more thing that comes before any technical choice: to ask the right questions, you first need to know which questions to ask.

Article 4 of the AI Act deals with exactly this: AI literacy, meaning people's ability to understand how these systems work, what opportunities they offer and what risks they carry. The obligation has applied since February 2025 and was amended this year by the Digital Omnibus (Regulation (EU) 2026/1744). It now requires those who develop and use AI systems to take measures that support AI literacy among their staff, without having to guarantee a specific level for each individual.

At Nephila, we've put this into practice in three ways:

  • training: internal courses and workshops on artificial intelligence;

  • guidelines: shared rules on which data should never be passed to external services, how to check an output before using it, and when a person needs to sign off;

  • a shared knowledge base, where the lessons and best practices from each project become available to everyone.

Our take

This experience taught us something that we feel is very much in the spirit of the AI Act: documenting the limits and assessing the risks of an AI system matters, but bringing those questions into the design process from the very start matters just as much.

Anyone who makes an AI-based tool available, however simple or marginal it may seem, takes on a responsibility: for what happens, or could happen, when people use it. Even if "it's just a game".

Nephila's first public AI project started out as a technical experiment, yet the questions we asked ourselves are the same ones the AI Act asks everyone to consider. We like to think that's where it all began: with those questions, not with a legal requirement.

Resources

​​​​​

***

We wrote this article with the support of AI. If you're interested in how we use these tools as part of our writing process, you can read more here.

Related posts