Safety and risk: what can go wrong?

Most tools have natural limits. A person typing emails can only send so many per hour. A human reviewer can only process so many applications per day. Speed is capped by human capacity.

AI removes these limits. A system that generates emails can produce thousands per minute. An analyzer that reviews applications can process the entire queue in hours.

This is the promise. The risk is the same mechanism in reverse: if something goes wrong, it goes wrong at the same scale. One mistake, replicated across every output.

The categories of concern:

Privacy — data that should stay private getting exposed.

Accuracy — confident statements that are factually wrong.

Fairness — patterns that treat people unfairly.

Control — systems acting in ways not anticipated.

None of these are hypothetical. All have happened to real organizations using real AI systems.

AI systems are hungry for data. That appetite creates vulnerability.

How information leaks

When employees paste customer data into AI chat interfaces, that data leaves the organization. Depending on the tool and its terms, the data might be stored externally, used for training, or accessible to the tool provider.

Internal documents, customer details, strategic plans — all at potential risk when used casually with AI tools.

This is not paranoia. In 2023, Samsung employees pasted proprietary source code into ChatGPT, exposing internal systems. The boundaries of what AI "knows" are not always clear or controllable.

Shadow usage

Employees use AI tools without official approval. Personal accounts, free tools, whatever gets the job done. The organization has no visibility into what information is being shared.

In most organizations, this is already happening.

Protective approaches

Some patterns help:

Enterprise versions of AI services with data processing agreements.

Clear policies about what categories of information can and cannot be used with AI.

Local AI deployments that process data without sending it externally.

Training so employees understand the privacy implications.

AI generates text that sounds authoritative but may be completely wrong.

Why AI invents things

AI predicts what text should come next based on patterns. Whether the prediction is factually accurate is not part of the process. AI has no concept of "true" versus "false" — only "likely-sounding" versus "unlikely-sounding."

This means AI will produce plausible-sounding falsehoods. Invented citations that look real. Statistics that sound reasonable but are fabricated. Descriptions of events that never happened. All stated with complete confidence.

The polished prose problem

When text is obviously rough or uncertain, it invites scrutiny. When text is smooth and confident, it passes unexamined more often.

AI produces polished prose regardless of accuracy. The text looks right even when it is wrong.

The scale factor

When a human makes an error in one document, it affects that document. When AI generates reports, emails, or analyses at scale, the same error can appear hundreds or thousands of times.

A wrong customer name in one email is embarrassing. A wrong customer name in ten thousand emails is a crisis.

What verification looks like

For outputs that matter:

Specific claims checked against primary sources.

Names and dates verified independently.

Statistics traced to original data.

Quotes confirmed as actually said.

This takes time. The time cost must be weighed against the risk of errors. But for outputs that will be public or have real consequences, verification is not optional.

AI learns from data that reflects historical unfairness.

How bias enters

Training data contains the patterns of its sources. If historical hiring decisions favored certain groups, an AI trained on hiring data learns those patterns. If text sources contain stereotypes, the AI absorbs them.

This is not intentional malice. The AI is doing exactly what it was trained to do — match patterns.

Where it matters most

Any decision that significantly affects people:

Hiring and recruiting.

Loan and credit decisions.

Housing applications.

Customer service priority.

Pricing and offers.

In these areas, biased AI causes real harm to real people. Beyond the ethical problem, there are often legal exposures as well.

The explanation problem

AI decisions are often difficult to explain. Why did the system score this applicant lower? Why was this customer flagged as higher risk?

For human decisions, there is usually a stated rationale. For AI decisions, the reasoning may be buried in patterns that humans cannot articulate.

Regulations like the EU's GDPR give people the right to an explanation when automated decisions affect them. "The algorithm determined this" is not a sufficient explanation.

Detection and mitigation

Bias is easier to detect than to eliminate. Testing outputs across demographic groups can reveal disparities. But fixing the underlying patterns without degrading performance is technically challenging.

The most reliable safeguard remains human review for consequential decisions. AI can flag and filter; humans can make final calls that require contextual judgment.

Human review checkpoints

The most reliable safety mechanism is human involvement at key points before outputs reach their destination.

AI drafts, human reviews. AI recommends, human approves. The AI handles volume; the human handles verification.

For low-stakes, high-volume work, sampling rather than reviewing everything may be sufficient. For high-stakes outputs, every item requires human eyes.

Monitoring and anomaly detection

Systems should detect when something unusual is happening.

Unusual patterns in output. Customer complaints clustering around AI-generated content. Outputs that match patterns of previous errors.

Catching problems early limits damage.

Clear boundaries

Not everything should be automated. Some decisions are too important, too sensitive, or too context-dependent.

Drawing explicit boundaries — what AI handles versus what requires human judgment — prevents creep into risky territory.

Feedback channels

People who encounter AI outputs should have easy ways to report problems.

"This was wrong." "This seemed unfair." "This did not make sense."

Without feedback mechanisms, problems remain invisible until they escalate to crises.