Credit decision support: what the audit found, and how we fixed it
We ran our audit tool on a credit-decision assistant. This case study sets out what the audit found, what it means for a lender, and what our forward-deployed engineers did after the report.
- Tier High-risk Annex III(5), essential services
- AI components detected 2 One AI SDK, one vector store
- Findings 2 Both medium
- Annex III point 5 Essential services, including creditworthiness
At a glance
- Sector
- Financial services, consumer lending to natural persons, including creditworthiness assessment
- System
- A credit-decision assistant. It takes in applications, calls Google's AI SDK (google.genai) and connects to a Qdrant vector store.
- EU AI Act scope*
- Annex III, point 5: essential services, including creditworthiness. The audit tool places the use in the high-risk tier.
- What was audited
- The AI components, the risk tier, and the records, human oversight and documentation the use would need
- What the team now has
- A decision record with fixed fields, a technical file for each release, a tested stop path and a named monitoring owner
- Stack detected
- Google AI SDK (google.genai)
Qdrant (qdrant_client)
Context
A system that helps decide who gets a loan can change a person's life.
The EU AI Act places systems that assess creditworthiness under Annex III, point 5. That brings duties on data, records, human oversight and explanations (Articles 9 to 15, 26 and 86). Data protection law adds its own safeguards for automated decisions (GDPR Article 22).
Lenders and credit teams ask the same four questions:
- Which AI models, SDKs and data stores does the system use?
- Who checks a decline before the applicant hears it?
- What do we keep, for how long, and who can see it?
- What do we tell an applicant who asks why?
What we audited
Our audit tool reads the code.
It builds an inventory of AI components (SDKs, models, vector stores and prompt templates), places the use in an EU AI Act tier with the Annex III point that applies, and lists what the team must show for that tier. Each finding is tied to an article or control and given a severity. Each one goes to the team with the evidence behind it.
AI inventory
Which AI SDKs, vector stores and prompt templates the system uses, and where they sit in the code.
Risk tier
Whether the use is high-risk under Annex III, and which point applies.
Obligations to evidence
Records of each decision (Article 12), human oversight (Article 14), technical documentation (Article 11 and Annex IV) and explanations (Article 86).
Next steps
Items that sit outside the code, such as what runs in production, what data is stored and who can stop the system.
What the audit found
The two medium findings concern what data leaves the system and what is kept.
-
MEDIUMModel calls in the application code
AI SDK detected: Google (google.genai)
Why it mattersEach call sends application data to an outside provider. The decision record needs the model version, and the data sent needs a lawful basis.
-
MEDIUMThe memory module
Vector store detected: Qdrant (qdrant_client)
Why it mattersStored vectors can carry personal data. Article 10 data governance and GDPR apply, and retention has to be set.
-
TIERAudit summary
Risk tier: high-risk, Annex III(5)
Why it mattersBrings duties on data, records, human oversight and explanations (Articles 9 to 15, 26 and 86)
Open items
-
OPEN ITEM
Next step: decision records
Not visible in the code Article 12 needs an automatic log of each decision, with fixed fields
-
OPEN ITEM
Next step: override and stop
Reviewer workflow Article 14(4)(d) and (e): a person can override the output and stop the system
-
OPEN ITEM
Next step: technical documentation
Release process Annex IV documentation should match the version that runs
-
OPEN ITEM
Next step: explanations for applicants
Decision output Article 86 and GDPR Article 22 safeguards for automated decisions
The open items are the questions a lender is asked next: who decides, what is recorded, and what the applicant is told.
What we did next
After the report, our forward-deployed engineers either fixed the issues hands-on or advised and worked alongside the team to close them.
That is how we work. Each finding is triaged by severity with the team. An engineer then either fixes it hands-on, or advises and works with the team to close it. The team keeps the decisions that belong to a lender, such as the legal basis and the wording of an explanation. After the change, we run the audit tool again on the same system, and the re-audit report goes into the technical file.
-
Fixed hands-on
Our engineers put the model version into every decision record, so each decision can be traced to the model that made it. They also moved the Qdrant connection behind a configuration file, with a retention setting and a route for deleting records about a person.
-
Advised and worked alongside the team
Our engineers worked with the team on the fields of the decision record, the override reason and the stop control. The team's risk and legal staff decided the wording of explanations and the legal basis for sending data to the provider.
-
What we would advise a lender
- Record the model version with every decision.
- Set a retention period for any vector store that holds personal data.
- Require a reason for every override. A blank field is a gap.
- Test the stop control before each release, and keep the test record.
- Keep the technical file in step with the model version.
What the team now has
A decision record.
Each decision writes fixed fields, including the model version and any override reason.
A retention rule for the vector store.
Retention is set, and a route exists for deleting a person's records.
A technical file for each release.
It matches the version that was tested, and a person signs it off.
A tested stop path.
Reviewers can override a recommendation, record why, and stop the system.
Monitoring with a named owner.
Drift, bias and error alerts go to a person, and each change follows the same loop.
How it is built
The sections below describe how the audited system is built and how we would govern it, using the components the audit detected.
Architecture
The system has two layers. The top layer is the decision flow. Data comes in from sources, passes through ingestion and quality gates, and reaches a feature store. The scoring service and the decision service then use those features. Loan officers work in the reviewer console.
The bottom layer is governance. It holds the model registry, the monitoring service, the event log store and the audit export. It watches the decision flow and keeps the record. It does not replace the checks in the flow itself.
How a decision flows
An application is checked against basic rules. The model then scores it and gives the main reasons. An approval goes to the applicant, and the decision is recorded.
A recommended decline goes to a loan officer first. The officer can confirm the decline or overturn it. Either way, the applicant receives the decision and the reasons.
An applicant who disagrees can ask for a review. A reviewer who was not involved in the first decision looks at the case again. Every step writes to the event log.
Human oversight and explanations
Article 14 asks that people can understand the system, notice when they rely on it too much, interpret its output, and override or stop it. Article 26(2) asks deployers to give oversight to people with the right competence, training and authority.
The reviewer console is built for this job. It shows the score, the main reasons, data quality flags and the model version. It records every override with a reason. It has a stop control, which is tested before handover.
Article 86 gives a person the right to clear and meaningful explanations when a decision based on a high-risk system's output significantly affects them. GDPR Article 22 adds safeguards for decisions based solely on automated processing. The build stores the reasons for each decision, so a plain explanation can be given on request. Wording is checked with your legal advisers.
Data and model governance
Article 10 asks for careful data governance. The build covers four things:
Lineage
Every feature can be traced back to its source, and to each change made to it.
Quality gates
Missing, out-of-range or stale data is flagged or stopped at ingestion.
Bias checks
Outcomes are compared across groups where your data and legal basis allow it. Where they do not, the file records the gap and the reason.
Drift checks
Inputs and scores are compared with a baseline before each release and during use.
Accuracy, robustness and security tests (Article 15) run before each release. The model registry records each version, its data, its owner and its approval.
Keeping the model under watch
When monitoring shows drift, a bias alert or an incident, the case goes to review. If a change is needed, a change request follows. The change is tested again, approved by a named person and released. Every step is logged.
Logging and evidence trail
Article 12 requires high-risk systems to keep automatic logs. Article 26(6) asks deployers to keep the logs under their control for at least six months. Article 19 sets the same minimum for providers. Your policy can set a longer period, but the build never keeps logs for less than six months.
Each decision writes a record with these fields:
- Time stamp and request ID
- Model name and version
- A reference to the input record, not the raw personal data
- Score, reason codes and data quality flags
- Decision and route (approved, declined or referred)
- Pseudonymised reviewer ID, with any override and its reason
- The explanation sent, and when it was sent
- The release ID of the technical file the record belongs to
The technical file
The technical file is the document set that Annex IV lists for high-risk systems. The build generates most of it from the records above, so each release has a file that matches what was tested. A person reviews the draft before it is signed off.
Fundamental rights impact assessment and DPIA
Article 27 asks deployers of Annex III point 5 systems to assess the impact on fundamental rights before first use. The assessment describes the process the system supports, how often it runs and who is affected. It also covers the specific risks of harm, the human oversight measures and what happens if the risks appear.
A GDPR data protection impact assessment (DPIA, Article 35) covers the risks to personal data. Article 27(4) allows parts of a DPIA to count towards the impact assessment, where the DPIA already covers the same duties.
We design the two assessments together:
- The data inventory and data flows feed the DPIA. The affected groups and possible harms feed the Article 27 assessment.
- The oversight measures in the assessment match what the reviewer console does.
- Validation test reports feed the risk sections of both documents.
- At handover, both documents are final, signed off and linked to the release. Article 27(2) requires them to be updated when a change affects them.
EU AI Act obligations mapped to controls
| Obligation | Control we build | Evidence produced |
|---|---|---|
| Art 9 risk management | Risk register and release gate | Risk file per release |
| Art 10 data governance | Lineage, quality gates, bias and drift checks | Data sheet and bias report |
| Art 11 and Annex IV technical documentation | Documentation generated from the build | Technical file per release |
| Art 12 record-keeping | Event log with fixed fields | Log export and field schema |
| Art 13 transparency to deployers | Model card and instructions for use | Instructions for use |
| Art 14 human oversight | Reviewer console, override and stop control | Override log and oversight design |
| Art 15 accuracy, robustness, cybersecurity | Test suites, input checks, access controls | Test reports and access review |
| Art 17 quality management (providers) | Change and document control, aligned to ISO/IEC 42001 | Change and document records |
| Art 26 deployer duties | Monitoring, reviewer briefing, log retention | Monitoring reports and logs kept six months or more |
| Art 27 impact assessment | Assessment workflow | Approved assessment and notification record |
| GDPR Articles 22 and 35 | Decision safeguards and DPIA workflow | DPIA and explanation records |
| Art 86 explanations | Explanation built from stored reasons | Explanation record for each request |
Which duties apply depends on whether you develop the system yourself (provider) or use it under your authority (deployer). We map the build to your role first.
Delivery plan
We work in five phases. Each phase ends with a review. Sprints run to two weeks.
Discover
Confirm the use case, the data sources and your role. Check the Annex III position.
Design
Agree the architecture, the oversight design and the scope of both assessments.
Build
Build the data pipelines, scoring service, reviewer console and event log. Draft the technical file as the build goes.
Validate
Run the tests and the bias and drift checks. Complete the impact assessment and DPIA drafts.
Handover
Transfer the pack, run the handover sessions and agree the monitoring owner, as set out in your contract.
Handover pack
- Code and infrastructure definitions for the scoring and decision service
- The reviewer console and its user guide
- The event log schema and export tool
- The technical file for the release
- The Article 27 assessment and DPIA, with the sign-off record
- The monitoring runbook and alert rules
- The model card and instructions for use
- Briefing notes for reviewers
Questions teams ask us
Talk through a system like this
Bring your scoring pipeline and your review process. We will run the same audit tool on it and walk through the findings, the fixes and the handover pack with you.
* EU AI Act scope: Annex III, point 5 (essential services, including creditworthiness assessment). The tier is the one the audit tool returned for this system.
