– 8 Real Application Clusters, Database server nodes (aka compute nodes) (Sunfire X4170’s)

– 2 quad core, Intel Xeon, E5540 processors (2.53ghz) in each server (Total 64 cores across 8 nodes)

– 72gb of RAM on each node (Total 576gb of Ram on the database servers)

– 3, 36 port QDR Infiniband switches (40gbit infiniband)

– 14 Exadata cells, (Sunfire X4275’s).

– Each cell has

– 2 quad core Intel Xeon, E5540 processors on each cell (Total 112 cpu cores on all 14 cells together)

– 24Gb RAM on each cell (Total 336gb of Ram on all 14 cells together)

– 384Gb of Flash Cache (PCI-E flash card) on each cell (5Tb on all 14 cells together)

– 12x600gb (SAS) disks (7.2Tb) or 12x1Tb (SATA) disks (12Tb) (Total 100Tb with SAS disks, or 168Tb with SATA disks on all 14 cells together)

The above hardware gives the database machine the ability to read data at the rate of 21GigaBytes a second or 1,000,000 Io’s per second.

Each disk drive (SAS) in the Sunfire X4275 server are, 3.5 inch, 15k Rpm, SAS 2.0, 600Gb drives. Each drive has an average read capacity of atleast 125MegaBytes per second for sequential scans. 168 disk drives, can together scan at the rate off (And return) 21,000 MegaBytes per second.

The Infiniband connections between the storage cells and the compute nodes have enough network bandwidth to transport data at the rate of 21GigaBytes per second.

The 64 cpu cores can issue i/o requests at the approx rate of 300megabytes per core, hence requesting about 21GigaBytes of data per second.

So the system is architect ed to optimally read (enough cpu’s to request the i/o, network bandwidth to ship the i/o and enough disk capacity to service that i/o requests). This is why the oracle database machine is a well balanced system.

At the read rate of 21GigaBytes a second, 1Terabyte of data can be read by the database servers in less than 1 minute.

Each database Node and Exadata Cell, has dual port, Quad data rate (QDR) Infiniband connectivity.

Infiniband is used for the database node to exadata cell connectivity and also for the Rac high speed interconnect (Cache fusion network)

Infiniband has the flexibility of a LAN network with the Speed of a SAN.

Oracle’s interconnect protocol uses DMA, to move data from the wire directly to the Memory without any additional copies made.

The exadata cells provide a highly redundant, high performance hardware with very intelligent software to efficiently process database i/o requests.

The hardware capability was discussed in the section “Lots of hardware power for the storage servers”.

The Intelligence in the Exadata cells include “Smart flash cache”, “Smart Scan’s”, “Storage Indexes” and “Exadata column compression”.

Oracle Automatic Storage Management (ASM) ensures that all oracle database files are evenly spread across all the 168 disks available in the database machine.

Oracle Database uses  the protocol iDB (Built by oracle, aptly called the Intelligent Database protocol) to communicate with the exadata cells. iDB is built on the Zero Data Loss, Zero Copy implementation (ZDP) of the industry standard protocol RDSv3 (Reliable datagram socket)

Random read operations are cached on the 5Tb of flash cache available in the database machine, significantly improving oltp performance.

The exadata smart flash cache, working with the database server, keeps track of data access patterns and intelligently manages the caching of blocks from the oracle datafiles.

The oracle database server uses the iDB protocol to push query predicates (That limit data retrieved by the query using filters and join conditions) to the exadata cell.

This enables the cell to do three things

– Identify rows that are needed by the query and ship only those rows back to the database server (Not entire blocks)

– Identify the columns needed by the query and ship only the required columns in the rows back to the database server

– Use Bloom filters and process join conditions and ship only matching rows in queries with joins back to the database server

This drastically reduces the amount of data send back to the database server (Reducing the network usage)

Transferring file blocks from disks inside a cell to the physical memory of the cell can happen relatively fast. If a lot of the data that is not needed by the database server, can be eliminated at the cell, then the amount of data that needs to go over a network to the database server is significantly reduced. So the network i/o usage between the database servers and exadata cells is reduced by smart scans.

Smart scans however do not reduce the disk i/o within the cell (ie transfer from disk to cell physical memory).

Oracle creates an in memory array of structures, that keep track off min and max values of columns (columns used in the where clause that benefit by storage indexes), that let oracle identify if specific 1mb regions are needed based on the filtering conditions applied to the tables.

So storage indexes is a filter oracle applies to prune away 1mb chunks that do not have to be read.

This reduces the i/o within the exadata cell.

Traditional relational databases store data in database blocks in a “Row” format. This type of storage limits the amount of compression that can be achieved

Column store databases, organizes and stores data by column. Storing column data together, with the same datatype and similar charachteristics , results in significant compression that can be achieved. However if the query refers to more than a couple of columns in the table, or does more than modest updates and inserts , those queries and dml tend to have slower performance.

Exadata hybrid columnar compression takes a blended approach. Oracle takes rows that fit into multiple blocks (Called a compression unit), converts the rows into columns and stores the data in a columnar format within a compression unit.

Oracle uses 3 different compression format’s and different transformations, depending on the compression level you have chosen

There are 4 levels of compression (Each level is a tradeoff between compression ratio and compression speeds)

– Query Low

– Query High (Default when you say compress for query)

– Archive Low

– Archive High

With Exadata Hybrid columnar compression

1) The amount of storage required to store massive amounts of data could be potentially decreased by a factor of 10.

2) The amount of i/o to be issued (For queries that scan very very large tables) is significantly reduced.

The above features together make’s the Oracle Database Machine, rock database performance.