Loading content...
We've explored the external level (what users see) and the conceptual level (the logical structure). But data doesn't live in abstractions—it lives on physical storage devices: spinning hard drives, solid-state drives, network-attached storage systems, and cloud storage volumes.
The internal level is where abstraction meets reality. It defines how the logical structures of the conceptual schema are actually stored, organized, and accessed on physical media. This is the domain of storage engines, file systems, buffer managers, and indexing algorithms.
Why does this matter? Because the difference between a query taking 10 milliseconds and 10 seconds often comes down to physical storage decisions. The conceptual schema might define a table with 10 million rows—but how those rows are stored, indexed, and accessed determines whether the database is usable or unbearably slow.
By the end of this page, you will understand the internal level's responsibility for physical data storage, including file organization methods, indexing structures, storage allocation, and buffer management. You'll see how physical design decisions impact performance and why the three-level architecture keeps these details separate from logical design.
The internal level (also called the physical level or storage level) describes how data is actually stored on storage devices. It is the lowest level of the three-level architecture, hidden from users and even from application programmers.
The internal level contains the internal schema (or storage schema)—a complete specification of how the conceptual schema is mapped to physical storage. This includes:
Formal Definition:
The internal schema is a specification of data storage structures, access paths, and file organizations used to store the database on physical storage devices.
| Characteristic | Description | Impact |
|---|---|---|
| Physical Storage Focus | Deals with bytes on disk, not logical concepts | Performance is primary concern, not semantics |
| Invisible to Users | Completely hidden from external and most conceptual operations | Users never see file structures or index details |
| Performance Critical | Physical decisions determine query speed | Wrong choices can cause 1000x performance difference |
| Hardware Dependent | Must consider device characteristics (HDD vs SSD) | Optimal strategies differ by storage type |
| Tunable | Can be modified without changing conceptual schema | Enables performance optimization without application changes |
Think of the internal level as warehouse management. The conceptual level defines what products exist (tables). The internal level decides: Which warehouse building stores which products? How are shelves organized? Where are the frequently accessed items placed? How do workers (queries) find what they need quickly? Good warehouse organization makes operations efficient; bad organization means workers spend hours searching for items.
To understand internal-level design, we must understand the storage devices the database uses and their characteristics. Modern systems use a memory hierarchy with dramatically different performance at each level.
| Level | Technology | Typical Size | Access Time | $ per GB | Volatility |
|---|---|---|---|---|---|
| CPU Registers | SRAM in CPU | ~1 KB | < 1 ns | N/A | Volatile |
| L1 Cache | SRAM on chip | 32-64 KB | ~1 ns | ~$10,000 | Volatile |
| L2 Cache | SRAM on chip | 256 KB-1 MB | ~4 ns | ~$1,000 | Volatile |
| L3 Cache | SRAM on/near chip | 4-50 MB | ~15 ns | ~$100 | Volatile |
| Main Memory (RAM) | DRAM | 16 GB - 6 TB | ~100 ns | ~$5 | Volatile |
| SSD (NVMe) | NAND Flash | 256 GB - 30 TB | ~100 μs | ~$0.10 | Non-volatile |
| SSD (SATA) | NAND Flash | 256 GB - 8 TB | ~500 μs | ~$0.08 | Non-volatile |
| HDD (Spinning) | Magnetic disk | 1 TB - 20 TB | ~10 ms | ~$0.02 | Non-volatile |
| Tape/Archive | Magnetic tape | Petabytes | seconds-minutes | ~$0.004 | Non-volatile |
The access time difference between RAM (~100 nanoseconds) and HDD (~10 milliseconds) is a factor of 100,000. This means:
Disk I/O is the bottleneck. The entire architecture of database storage is designed to minimize the number of disk reads and writes required for any operation.
On HDDs, sequential reads (reading consecutive bytes) are 100-200x faster than random reads (jumping to different locations). On SSDs, the gap is smaller but still significant (3-10x). This is why the internal level carefully organizes data to maximize sequential access and minimize random seeks.
The internal level must decide how records (rows) are physically arranged in files. This file organization profoundly affects query performance.
Heap organization stores records in no particular order—new records are placed wherever there's space.
123456789101112131415161718192021
Heap File Structure:┌─────────────────────────────────────────────────────────┐│ Page 1 ││ ┌───────────┬───────────┬───────────┬───────────┐ ││ │ Record 1 │ Record 2 │ Record 3 │ Record 4 │ ││ │ ID=7 │ ID=3 │ ID=12 │ ID=1 │ ││ └───────────┴───────────┴───────────┴───────────┘ │├─────────────────────────────────────────────────────────┤│ Page 2 ││ ┌───────────┬───────────┬───────────┬───────────┐ ││ │ Record 5 │ Record 6 │ Record 7 │ (Empty) │ ││ │ ID=9 │ ID=2 │ ID=5 │ │ ││ └───────────┴───────────┴───────────┴───────────┘ │└─────────────────────────────────────────────────────────┘ Notice: Records are NOT sorted by ID.To find ID=5, we must scan from Record 1 through Record 7. Query: SELECT * FROM table WHERE id = 5Plan: Sequential Scan, filter (id = 5)Pages Read: Potentially ALL pages in the tableA table can only be physically sorted by ONE attribute (the clustering attribute). All other attributes cannot benefit from the physical order. This is why indexes are essential—they provide alternative "sorted views" of data without physically reorganizing it.
Indexes are the most important performance tool in the internal level. An index is a separate data structure that enables fast lookup without scanning entire tables.
Think of a phone book. The actual records (people) aren't easily searchable because there are millions of them. But the index (alphabetical listing by last name) lets you quickly find any person. Indexes do the same for databases.
| Index Type | Structure | Best For | Not Good For |
|---|---|---|---|
| B-tree/B+ tree | Balanced tree with sorted keys | Range queries, equality, ORDER BY | Pattern matching, full-text |
| Hash Index | Hash table | Equality lookups only | Range queries, sorting |
| Bitmap Index | Bit vectors per value | Low-cardinality columns, AND/OR | High-cardinality, frequent updates |
| GiST | Generalized search tree | Geometric, full-text, complex types | Simple equality lookups |
| GIN | Generalized inverted index | Full-text search, arrays, JSONB | Simple scalar values |
| BRIN | Block range index | Very large tables with natural order | Random distribution, point queries |
The B+ tree is the dominant index structure in relational databases. It provides:
1234567891011121314151617181920212223242526272829303132
B+ Tree Index Structure (order=3): ┌─────────────┐ │ [30,60] │ (Root Node) └──────┬──────┘ ┌───────────────┼───────────────┐ ▼ ▼ ▼ ┌───────────┐ ┌───────────┐ ┌───────────┐ │ [10,20] │ │ [40,50] │ │ [70,80] │ (Internal Nodes) └─────┬─────┘ └─────┬─────┘ └─────┬─────┘ ┌────┼────┐ ┌────┼────┐ ┌────┼────┐ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ┌───┐┌───┐┌───┐ ┌───┐┌───┐┌───┐ ┌───┐┌───┐┌───┐ │5,8││15 ││25,│ │35,││45 ││55,│ │65,││75 ││85,│ (Leaf Nodes) │ ││18 ││28 │ │38 ││48 ││58 │ │68 ││78 ││90 │ └─┬─┘└─┬─┘└─┬─┘ └─┬─┘└─┬─┘└─┬─┘ └─┬─┘└─┬─┘└─┬─┘ │ │ │ │ │ │ │ │ │ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ [Row][Row][Row] [Row][Row][Row] [Row][Row][Row] (Data Pages/Row Pointers) Key Properties:1. All data pointers are in leaf nodes only2. Leaf nodes are linked for sequential access (→→→)3. Internal nodes only guide the search4. Tree is always balanced (all leaves at same level) Search for key=45: Root: 45 ≥ 30 and 45 < 60, go middle Internal: 45 ≥ 40 and 45 < 50, go middle Leaf: Found! Return row pointer Disk reads: 3 (one per level) - regardless of table size!1234567891011121314151617181920212223242526272829
-- B-tree index (default)CREATE INDEX idx_customer_email ON Customer(email); -- Composite index (multiple columns)CREATE INDEX idx_order_customer_date ON Order(customer_id, order_date);-- Efficient for: WHERE customer_id = ? AND order_date = ?-- Also efficient for: WHERE customer_id = ? (leftmost prefix)-- NOT efficient for: WHERE order_date = ? (not leftmost) -- Unique index (also enforces constraint)CREATE UNIQUE INDEX idx_product_sku ON Product(sku); -- Partial index (only index rows matching condition)CREATE INDEX idx_active_orders ON Order(order_date)WHERE status = 'active';-- Smaller index, faster for common queries -- Covering index (includes all columns needed by query)CREATE INDEX idx_order_covering ON Order(customer_id)INCLUDE (order_date, total_amount, status);-- Query can be answered from index alone, no table access needed -- Hash index (PostgreSQL)CREATE INDEX idx_customer_id_hash ON Customer USING HASH(customer_id);-- Slightly faster equality lookups, no range support -- GIN index for full-text searchCREATE INDEX idx_product_search ON Product USING GIN(to_tsvector('english', product_name || ' ' || description));Indexes are not free. Each index consumes disk space and must be updated on every INSERT, UPDATE, or DELETE. A table with 10 indexes means every write operation triggers 11 writes (table + 10 indexes). Choose indexes wisely based on actual query patterns.
The internal level manages physical space through pages (also called blocks)—fixed-size units of storage that are read and written atomically.
12345678910111213141516171819202122232425262728293031323334353637383940
Typical Database Page Structure (8KB example): ┌────────────────────────────────────────────────────────────────┐│ Page Header (24-100 bytes) ││ ┌────────────┬────────────┬────────────┬────────────────────┐ ││ │ Page ID │ LSN │ Checksum │ Free Space Pointer │ ││ │ [4 bytes] │ [8 bytes] │ [4 bytes] │ [2 bytes] │ ││ └────────────┴────────────┴────────────┴────────────────────┘ │├────────────────────────────────────────────────────────────────┤│ Item Pointers (Line Pointer Array) - grows downward ││ ┌────────┬────────┬────────┬────────┬────────┬───────────┐ ││ │ Ptr 1 │ Ptr 2 │ Ptr 3 │ Ptr 4 │ Ptr 5 │ ... │ ││ │ →Row1 │ →Row2 │ →Row3 │ →Row4 │ →Row5 │ │ ││ └────────┴────────┴────────┴────────┴────────┴───────────┘ ││ ↓ ││ ╔════════════════════════════════════════════════════════════╗ ││ ║ F R E E S P A C E ║ ││ ║ ║ ││ ╚════════════════════════════════════════════════════════════╝ ││ ↑ │├────────────────────────────────────────────────────────────────┤│ Tuple Data (Row Data) - grows upward from bottom ││ ┌───────────────────────────────────────────────────────────┐ ││ │ Row 5: customer_id=105, name='Eve Garcia', email=... │ ││ ├───────────────────────────────────────────────────────────┤ ││ │ Row 4: customer_id=104, name='David Lee', email=... │ ││ ├───────────────────────────────────────────────────────────┤ ││ │ Row 3: customer_id=103, name='Carol White', email=... │ ││ ├───────────────────────────────────────────────────────────┤ ││ │ Row 2: customer_id=102, name='Bob Jones', email=... │ ││ ├───────────────────────────────────────────────────────────┤ ││ │ Row 1: customer_id=101, name='Alice Smith', email=... │ ││ └───────────────────────────────────────────────────────────┘ │└────────────────────────────────────────────────────────────────┘ Key Design Elements:1. Item pointers allow rows to move within page (for compaction)2. Free space between pointers and data allows efficient insertion3. Header contains page metadata for recovery and integrity4. Fixed page size enables efficient I/O and buffer managementAbove individual pages, databases organize storage into:
12345678910111213141516171819202122232425262728293031
-- Create tablespace on specific storage-- PostgreSQL exampleCREATE TABLESPACE fast_ssd LOCATION '/mnt/nvme/pg_data';CREATE TABLESPACE archive_hdd LOCATION '/mnt/hdd/pg_archive'; -- Create table on specific tablespaceCREATE TABLE hot_data ( id SERIAL PRIMARY KEY, data JSONB, created_at TIMESTAMP DEFAULT NOW()) TABLESPACE fast_ssd; CREATE TABLE cold_archive ( id BIGINT PRIMARY KEY, data JSONB, archived_at TIMESTAMP) TABLESPACE archive_hdd; -- Move existing table to different tablespaceALTER TABLE order_history SET TABLESPACE archive_hdd; -- Create index on different tablespace than tableCREATE INDEX idx_hot_data_created ON hot_data(created_at) TABLESPACE fast_ssd; -- Oracle: Create tablespace with specific parameters-- CREATE TABLESPACE sales_data-- DATAFILE '/u01/oradata/sales01.dbf' SIZE 10G-- EXTENT MANAGEMENT LOCAL-- SEGMENT SPACE MANAGEMENT AUTO;Use fast storage (NVMe SSD) for frequently accessed tables and indexes, transaction logs, and temp space. Use slower storage (HDD, object storage) for historical data, backups, and rarely-accessed archives. This tiered storage approach optimizes cost and performance simultaneously.
Since disk I/O is slow, databases maintain a buffer pool (also called buffer cache or shared buffers)—an area of RAM that caches frequently accessed pages.
| Database | Parameter | Recommended Setting | Notes |
|---|---|---|---|
| PostgreSQL | shared_buffers | 25% of RAM | Start here, tune based on workload |
| PostgreSQL | effective_cache_size | 50-75% of RAM | Hint for query planner |
| MySQL InnoDB | innodb_buffer_pool_size | 70-80% of RAM | Most critical InnoDB setting |
| Oracle | SGA_TARGET | 40-80% of RAM | Includes buffer cache + other caches |
| SQL Server | max server memory | Leave 2-4GB for OS | SQL Server manages the rest |
When the buffer pool is full and a new page is needed, which existing page should be evicted?
Most databases use sophisticated variants that prevent sequential scans from flushing the entire useful cache (the "table scan problem").
123456789101112131415161718192021222324252627282930313233343536
-- PostgreSQL: Check buffer cache hit ratioSELECT sum(blks_hit) / nullif(sum(blks_hit + blks_read), 0) * 100 AS cache_hit_ratioFROM pg_stat_database;-- Goal: > 99% cache hit ratio for OLTP workloads -- PostgreSQL: See what's in the buffer cache (requires pg_buffercache extension)CREATE EXTENSION pg_buffercache; SELECT c.relname AS table_name, count(*) AS buffers, round(100.0 * count(*) / (SELECT count(*) FROM pg_buffercache WHERE relfilenode IS NOT NULL), 2) AS percent_of_cacheFROM pg_class cJOIN pg_buffercache b ON b.relfilenode = pg_relation_filenode(c.oid)GROUP BY c.relnameORDER BY buffers DESCLIMIT 20; -- MySQL: InnoDB buffer pool statisticsSHOW STATUS LIKE 'Innodb_buffer_pool%';-- Key metrics: -- Innodb_buffer_pool_read_requests (logical reads)-- Innodb_buffer_pool_reads (physical reads from disk)-- Hit ratio = 1 - (reads / read_requests) -- MySQL: Buffer pool contents summarySELECT TABLE_NAME, ENGINE, TABLE_ROWS, ROUND(DATA_LENGTH / 1024 / 1024, 2) AS data_mb, ROUND(INDEX_LENGTH / 1024 / 1024, 2) AS index_mbFROM information_schema.TABLESWHERE TABLE_SCHEMA = 'your_database'ORDER BY DATA_LENGTH DESC;Both the database and the operating system have caches. PostgreSQL with shared_buffers + OS page cache works well. But some databases (Oracle, SQL Server) prefer direct I/O to avoid double-caching overhead. Understand your database's preference to avoid configuration conflicts.
DBAs make numerous physical design decisions that don't affect logical structure but dramatically impact performance. These are pure internal-level concerns.
12345678910111213141516171819202122232425262728293031323334353637
-- Partitioning by date range (PostgreSQL)CREATE TABLE order_history ( order_id BIGINT, order_date DATE NOT NULL, customer_id INTEGER, total_amount DECIMAL(12,2)) PARTITION BY RANGE (order_date); CREATE TABLE order_history_2023 PARTITION OF order_history FOR VALUES FROM ('2023-01-01') TO ('2024-01-01'); CREATE TABLE order_history_2024 PARTITION OF order_history FOR VALUES FROM ('2024-01-01') TO ('2025-01-01'); -- Query only scans relevant partitionSELECT * FROM order_history WHERE order_date = '2024-06-15';-- Only order_history_2024 is accessed! -- Compression (PostgreSQL TOAST + external compression)-- Large values automatically compressed and stored out-of-line -- MySQL InnoDB compressionALTER TABLE archive_data ROW_FORMAT=COMPRESSED KEY_BLOCK_SIZE=8; -- Fill factor (PostgreSQL)CREATE INDEX idx_product_name ON product(name) WITH (fillfactor = 70);-- Leaves 30% free space per page for updates -- Clustered table (PostgreSQL)CLUSTER order_history USING idx_order_date;-- Physically reorders rows by order_date - one-time operation -- Oracle IOT (Index-Organized Table) - data stored in index structure-- CREATE TABLE hot_data (-- id NUMBER PRIMARY KEY,-- data VARCHAR2(100)-- ) ORGANIZATION INDEX;Physical design decisions should be based on actual workload patterns, not intuition. Use EXPLAIN ANALYZE to see query plans and I/O costs. Monitor buffer hit ratios, wait events, and I/O latencies. The best physical design for your system depends on YOUR data and YOUR queries.
We've explored the internal level—where database abstractions meet physical reality. Here are the key takeaways:
What's Next:
We've now covered all three levels: external (user views), conceptual (logical structure), and internal (physical storage). Next, we'll examine the ANSI-SPARC architecture—the formal framework that defined this three-level model and explains how the levels interact through mappings.
You now understand the internal level—how databases physically store and access data. You can explain file organization methods, indexing structures, buffer management, and physical design decisions. This knowledge enables you to make informed choices about database performance and storage configuration.