Document Analysis: Mining Requirements from Existing Artifacts

Every organization accumulates documents—forms, reports, manuals, emails, spreadsheets, and records—that silently encode decades of operational knowledge. These artifacts are invaluable for database design because they represent formalized, tested, and validated views of organizational data.

Document analysis is the systematic examination of existing written materials to extract data requirements. Where interviews capture what people think they do and believe they need, documents reveal what they actually do and formally require.

Consider a simple invoice form: it defines precisely what data is captured during a transaction, in what format, with what validations. That single document tells you more about the transactional data model than an hour of abstract discussion. When combined with interviews, document analysis creates a requirements base that is both comprehensive and verifiable.

Document analysis complements user interviews by providing concrete, verifiable evidence of organizational data practices. It serves multiple strategic purposes in requirements gathering:

Verification and Validation: Documents confirm or contradict interview findings. When a manager says "we track customer preferences," the customer intake form either contains preference fields or it doesn't. Documents ground abstract descriptions in reality.

Discovery of Implicit Requirements: Documents often contain requirements that users take for granted. The date format on every form, the mandatory fields marked with asterisks, the approval signatures required—these are requirements users no longer consciously remember because they became automatic.

Historical Context: Old versions of documents reveal how requirements evolved. Understanding what changed and why provides insight into organizational priorities and pain points.

Scope Definition: The volume and variety of documents help define project boundaries. If an organization produces 500 distinct report types, that's a scaling factor that affects the design effort.

Organizations produce diverse document types, each valuable for different aspects of requirements gathering. A comprehensive analysis examines multiple document categories to build a complete picture.

Forms: The Richest Requirements Source

Forms deserve special attention because they explicitly define data collection points. Every form represents a deliberate decision about:

• What data is worth capturing
• How that data should be structured and validated
• What relationships exist between data elements
• What is mandatory versus optional

A well-designed database should be able to store every piece of information that appears on organizational forms—plus any contextual data (who filled it, when, why) needed for audit and analysis.

Reports: Defining Information Needs

Reports reveal what the organization considers valuable output. From reports, you can work backward to determine:

• What raw data must be stored to generate these reports
• What calculations, aggregations, and transformations are needed
• What time horizons and historical data are required
• What groupings and categorizations are meaningful

Gathering documents is itself a structured process. Documents may be scattered across departments, archived in different formats, or guarded by protective stakeholders. A systematic approach ensures comprehensive coverage.

Dealing with Document Access Challenges:

Not all documents are readily available:

Confidential Documents: Some documents (financial, HR, legal) may be restricted. Work with project sponsors to obtain appropriate access. Sign NDAs if required. In some cases, redacted samples suffice for requirements purposes.

Informal Documents: Many organizations operate with informal documents—spreadsheets passed between colleagues, personal checklists, undocumented macros. These often contain critical operational knowledge. Ask: "Beyond official forms, what tools do you use daily?"

Digital Documents: Modern organizations have much documentation in electronic formats—PDFs, online forms, database screens. Screenshots and exports capture these artifacts.

Legacy Documents: Old paper forms and outdated system printouts may exist only in physical archives. These are valuable for understanding historical decisions and data that must be migrated.

Forms are the most direct source of data requirements. Each form field maps to potential database attributes. Systematic form analysis extracts maximum requirements value.

Form Analysis Procedure:

1. Identify the Form's Purpose — What transaction, event, or process does this form support?
2. Map the Data Flow — Where does information on this form come from? Where does it go?
3. Extract Data Elements — List every field, its name, label, and apparent purpose
4. Analyze Field Characteristics — Determine data types, lengths, formats, and constraints
5. Identify Relationships — Note references to other entities (customer number, product code)
6. Determine Cardinality — Are there repeating sections (line items, dependents)?
7. Note Derived Values — Identify calculated fields (totals, dates computed from others)
8. Record Validation Rules — Look for format specifications, valid ranges, required fields

Recognizing Data Patterns:

Experienced analysts recognize common data patterns in forms:

Identifiers: Fields like "Order Number," "Employee ID," "Account #" are entity identifiers—candidates for primary keys.

References: Fields prefixed with another entity name ("Customer Number" on an order form) indicate foreign key relationships.

Repeating Groups: Sections with lines for multiple items (order line items, family members, skills) indicate one-to-many relationships requiring separate tables.

Code Fields: Dropdowns, radio buttons, and checkboxes often represent coded values that might become lookup tables.

Calculated Fields: Totals, rates, and derived dates suggest what the database must compute or store.

Date Patterns: Multiple date fields (created, modified, approved, expired) reveal lifecycle requirements.

Reports represent the organization's information output—what they consider valuable to know. Report analysis works backward: from presentation to the underlying data that must be stored to produce the report.

Report Analysis Approach:

Report Requirements Hierarchy:

Reports exist at different organizational levels with different implications:

Dashboards and Visualizations:

Modern organizations use dashboards with charts, gauges, and KPIs. These are reports in visual form. For each visualization element, apply the same analysis: what data produces this chart? What time range? What filters?

Procedure manuals and policy documents encode business rules—the constraints and logic that govern how data is created, modified, and used. These rules must be enforced by the database or the applications that use it.

Extracting Business Rules from Documents:

Business rules appear in various forms within documentation:

Procedure Analysis Technique:

Procedures describe sequences of activities. From these, you can extract:

States and Transitions: Procedures often describe lifecycle stages ("pending → approved → closed"). These become status fields with valid transition rules.

Roles and Permissions: Who performs each step? This maps to user roles and access control requirements.

Triggering Events: What initiates the procedure? Events often correspond to record creation or status changes.

Decision Points: Branching logic in procedures becomes business rules. "If order > $10,000, require manager approval" is a constraint.

Time Constraints: Deadlines and SLAs revealed in procedures become temporal requirements.

Exception Handling: What happens when standard procedures fail? Exceptions often require special data structures (reversal records, exception codes, escalation logs).

When replacing or enhancing existing systems, legacy documentation is invaluable. It reveals what the organization invested engineering effort to build—a strong signal of what matters.

Types of Legacy System Documentation:

Leveraging Legacy Schemas:

Existing database schemas are requirements in themselves—even if poorly designed, they encode decisions made for reasons. When analyzing legacy schemas:

1. Document each table — Record purpose, typical row count, key relationships
2. Analyze columns — Data types, naming conventions, null policies
3. Identify relationships — Explicit foreign keys and implicit references
4. Note anomalies — Unused columns, strange data types, naming inconsistencies
5. Look for patterns — Audit fields, soft deletes, versioning schemes
6. Profile data — Actual values often reveal constraints not declared in the schema

Data Profiling:

Beyond documentation, profiling actual data in legacy systems reveals:

• Real value distributions and ranges
• Null frequencies and patterns
• Referential integrity violations (despite constraints)
• Data quality issues to address in migration