Owning AI Software

Exploring the distinction between creating AI-generated software and owning the responsibility for it.

AI Can Build Software. That Doesn’t Mean Your Business Should Own the Risk.

There is a growing narrative that AI is about to make SaaS companies, software consultancies and perhaps even software development itself obsolete.

The argument seems compelling:

Why pay for software when I can ask AI to build exactly what I need?

Today, an AI coding agent can already generate an impressive amount of software. Give it sufficient instructions and it can create databases, APIs, authentication, user interfaces and integrations in a fraction of the time they would have taken a development team only a few years ago.

And this capability is improving incredibly quickly.

But I think this argument confuses two very different things:

The cost of creating software and the cost of owning software.

AI is dramatically reducing the first.

It hasn’t solved the second.

When you buy software, you aren’t just buying code

Consider something as ordinary as a CRM.

When a company pays for a CRM, it isn’t simply paying for some database tables, forms and business logic.

It is paying somebody else to worry about the system.

The provider worries about availability. Infrastructure. Backups. Security patches. Upgrades. Monitoring. Performance. Disaster recovery.

If something stops working at 2:00 AM, somebody else owns the problem.

There are SLAs, support processes, escalation procedures and contractual obligations.

The customer gets to concentrate on running their business.

This is one of the fundamental propositions of SaaS that is sometimes lost in the discussion around generative AI.

The alternative to a €20,000 annual SaaS subscription isn’t necessarily a €2,000 AI-generated clone.

It is:

€2,000 to generate it + the cost and risk of owning that software indefinitely.

And those are very different propositions.

“But AI will eventually maintain the software too”

Probably.

Imagine telling an AI:

“Build our CRM, deploy it, monitor it, back it up, fix bugs and continuously improve it.”

Technically, I think we’ll get there.

But that creates another question that is much more difficult:

Who is responsible when it gets something wrong?

Let’s say the AI autonomously decides that part of the database needs restructuring.

It generates a migration, tests it, deploys it and continues operating normally.

Six months later, somebody discovers that the migration corrupted five years of customer activity.

Who owns the problem?

The employee who originally prompted the AI?

The company?

The cloud provider?

The AI provider?

The model?

Being technically capable of operating a system isn’t the same as accepting responsibility for operating it.

Unless somebody else contractually accepts that responsibility, your business is effectively self-insuring the software.

For an internal tool used by four people, that could be perfectly reasonable.

For the system containing your entire customer database, financial information or critical business processes, the calculation looks very different.

There is also a difference between AI-generated software and software you understand

Businesses already build their own applications today.

Our customers build solutions with platforms such as Microsoft Power Platform, Dataverse, SharePoint and Azure all the time.

And yes, when you build an application yourself, you assume responsibility for it.

But there is an important distinction.

You know what you built.

You can know its data model.

You can inspect its business rules.

You can understand which Power Automate flow executes when an invoice is approved.

You can define exactly which security role gives a user access to customer records.

And professional software delivery adds layers specifically designed to reduce the possibility of something going catastrophically wrong.

Development, test and production environments.

Source control.

Automated testing.

Deployment pipelines.

Role-based access control.

Data-loss-prevention policies.

Monitoring and telemetry.

Audit logs.

Backups.

Change management.

Rollback procedures.

Peer review and approval gates.

These things sometimes feel like unnecessary bureaucracy when everything works.

They suddenly become extraordinarily valuable when something doesn’t.

The black-box problem

Now compare that with a hypothetical system completely created and continuously modified by autonomous AI.

What changed last Tuesday?

Why?

What other functionality did the change affect?

Which tests were executed?

Who approved the change?

Can we reproduce the previous version?

Can we guarantee that the AI won’t make a similar change again?

Can an engineer reconstruct why the production system looks the way it does today?

If the answers to those questions are unclear, then making AI more capable doesn’t necessarily reduce business risk.

In some circumstances, it increases it.

For autonomous AI development to become appropriate for business-critical software, I believe we need much more than better code generation.

We need AI systems that can tell us exactly what they intend to change, assess the potential impact, prove that changes have been tested, maintain immutable audit trails, provide deterministic rollback and respect policies defining which changes they are allowed to make autonomously.

And high-risk changes should still require explicit approval.

There’s an interesting consequence to this.

Once we introduce source control, testing, deployment gates, separate environments, audit trails, monitoring, security policies, backups and human approval…

We’ve essentially recreated professional software engineering governance.

AI just becomes an extraordinarily capable participant inside it.

And I think that’s much more likely than governance disappearing altogether.

So what are you paying a technology partner for?

This is where I think the discussion around AI and software development often misses the point.

When you hire a professional technology partner, you aren’t purchasing typing.

And increasingly, you aren’t primarily purchasing code either.

You’re paying somebody to determine what should be built in the first place.

You’re paying for architecture.

For understanding business processes.

For choosing where custom development makes sense and where an existing product is the better decision.

For security design.

For governance.

For testing.

For deployment discipline.

For monitoring.

For maintainability.

For documentation and knowledge transfer.

For someone who can look at an AI-generated solution and recognize that, although it works perfectly in the demonstration, its architecture is going to become a serious problem in eighteen months.

And importantly:

You’re paying someone to put their name behind the solution.

This doesn’t mean we should avoid AI

Quite the opposite.

We use AI precisely because it makes software development faster.

If AI can reduce something that previously required five days of development to one, that’s fantastic.

That efficiency should benefit our customers.

But faster code generation doesn’t eliminate the need for architecture, security, governance, testing and accountability.

It allows us to spend proportionally more effort on those things and less effort manually producing code.

That’s progress.

Build the differentiation. Buy the boring stuff.

AI may also change what businesses should build themselves.

It increasingly makes sense to build the things that make your business different:

It makes considerably less sense to reinvent everything underneath them.

Identity, cloud infrastructure, databases, security, monitoring, backup systems and commodity business capabilities already exist as mature managed services and platforms.

A sensible strategy isn’t necessarily buy everything or AI-build everything.

It is:

Buy the boring infrastructure. Build the differentiation. Use AI aggressively. Govern everything.

AI can make software cheap. It cannot make accountability cheap.

This is ultimately why I don’t think AI means businesses suddenly need their technology partners less.

It changes what they need them for.

The value of a software company or technology partner can no longer be “we know how to write the code.”

AI is commoditizing that capability extremely quickly.

The value has to move higher.

Understanding the business.

Designing the right solution.

Choosing the right technologies.

Knowing what not to build.

Making intelligent trade-offs.

Creating secure and maintainable architectures.

Establishing governance.

Reducing operational risk.

And taking professional responsibility for the result.

AI can already build remarkable software.

The more important question for a business isn’t:

“Can AI build this?”

Increasingly, the answer will be yes.

The better questions are:

“Should we build it?”

“Do we understand what we’ve built?”

“Can we govern and safely operate it?”

And ultimately:

“Who owns the consequences if it goes wrong?”

Until AI can provide convincing answers to those questions, generating the software is only one part of the job.