In the previous page, we explored how file-based data management systems worked. Now we must confront a critical question: If file systems worked, why did we need something else?

The answer lies in understanding that file systems didn't merely have inconveniences—they had fundamental structural limitations that became increasingly problematic as:

• Organizations grew in size and complexity
• Data volumes increased by orders of magnitude
• Business operations became more integrated
• Real-time access became expected rather than exceptional
• Competition demanded data-driven decision-making

These limitations weren't bugs to be fixed; they were inherent to the file-based approach itself.

To systematically understand file system limitations, we can organize them into categories based on the type of problem they create. This taxonomy helps us see not just individual issues but the patterns that connect them:

Let's examine each of these limitations in depth, understanding not just what the problem is but why the file-based architecture makes it inevitable.

Data redundancy occurs when the same piece of information is stored in multiple locations within an organization's data files. In file-based systems, redundancy isn't an accident—it's a structural inevitability.

Why Redundancy Is Unavoidable in File Systems:

A Quantitative Example:

Consider a mid-sized insurance company with 500,000 policyholders. The same customer information might appear in:

The True Cost of Redundancy:

Storage was merely the visible symptom. The real costs were operational:

Data inconsistency is the inevitable consequence of data redundancy. When the same information exists in multiple places, those copies will eventually contain different values. This isn't a matter of 'if'—it's 'when' and 'how badly'.

Inconsistency Patterns:

Data isolation refers to the problem of data being trapped within the boundaries of individual applications or departments, inaccessible to other parts of the organization that need it. In file-based systems, each application was a silo, and breaking down those silos was technically and organizationally difficult.

Why Data Becomes Isolated:

The Business Consequence: Unanswerable Questions

Data isolation meant that many basic business questions—questions that seem trivial today—were nearly impossible to answer:

The Ad-Hoc Integration Problem:

When cross-system queries were truly needed, organizations resorted to ad-hoc integration projects:

Data integrity refers to the accuracy, consistency, and validity of data according to business rules. In file-based systems, integrity enforcement was fragmented, incomplete, and unreliable.

Types of Integrity Constraints:

The Scattered Validation Problem:

In file-based systems, validation logic was duplicated across every application that accessed the data:

The Referential Integrity Problem:

Perhaps the most severe integrity issue in file-based systems was the inability to maintain referential integrity—ensuring that references between related data remain valid.

Example: Order and Customer Files

Consequences of Integrity Failures:

File-based systems offered only primitive security mechanisms, typically limited to what the operating system provided at the file level. This created significant vulnerabilities and compliance challenges.

Operating System File Permissions:

Typical file-level security provided:

These permissions applied to the entire file. You could not:

No Audit Trail:

File systems typically provided no auditing of data access. Organizations couldn't answer:

• Who viewed the customer list last Tuesday?
• Which programs modified the payroll file?
• When was this record last updated, and by whom?
• How many times has this file been accessed this month?

Without audit trails, security breaches couldn't be detected, investigated, or prevented. Insider threats went unnoticed until external consequences surfaced.

As organizations moved toward interactive processing and multiple users needed to access the same data simultaneously, file-based systems faced a fundamental challenge: they had no built-in mechanisms for coordinating concurrent access.

The Lost Update Problem:

Consider two users updating the same customer record:

What Happened:

Both users read the same initial balance ($1000). User 1 calculated the new balance after a deposit ($1200) and wrote it. User 2, still working with the original $1000, calculated the withdrawal ($700) and wrote it. User 1's deposit was completely lost. The correct final balance should be $900 ($1000 + $200 - $300).

Other Concurrency Problems:

Workaround Attempts:

Organizations tried various approaches to manage concurrency:

Atomicity is the property that a group of operations either all succeed together or all fail together—there's no partial completion. File systems provided no atomicity guarantees, making recovery from failures extremely difficult.

The Partial Update Problem:

Consider a funds transfer that must update two files:

Recovery Challenges:

File systems provided no systematic recovery mechanisms for interrupted operations:

The Backup Problem:

Recovery typically relied on restoring from backup and reprocessing transactions. But:

We've now explored the comprehensive set of limitations that made file-based data management increasingly untenable as organizations grew and their data needs became more sophisticated. Let's consolidate these insights:

What's Next:

Now that we understand the comprehensive limitations of file-based systems, we'll examine two of the most critical problems in greater depth: data redundancy and inconsistency (the subject of our next page) and data isolation (the following page). These deep dives will complete our understanding of why Database Management Systems became essential.