InfiniBand Fabric Topology

Oracle Exadata Database Machine includes the verify-topology utility. 

This utility can be used to identify the following network connection problems:

Missing InfiniBand cable

Missing InfiniBand connection

Incorrectly-seated cable

Cable connected to the wrong endpoint

Verifying infiniband topology if it is configured to be current or not. 

The InfiniBand network would operate at degraded efficiency intermittently or fail to operate, if the following desired output is not seen: 

# /opt/oracle.SupportTools/ibdiagtools/verify-topology -t fattree

[ DB Machine InfiniBand Cabling Topology Verification Tool ]

Is every external switch connected to every internal switch..........[SUCCESS]

Are any external switches connected to each other....................[SUCCESS]

Are any hosts connected to spine switch..............................[SUCCESS]

Check if all hosts have 2 CAs to different switches..................[SUCCESS]

Leaf switch check: cardinality and even distribution.................[SUCCESS]

Check if each rack has an valid internal ring........................[SUCCESS]

If anything other than "SUCCESS" is reported, investigate and correct the condition. 

How to find serial number of Exadata Server/Storage

Execute below command on operating system to find the serial number of DB node and Storage cell.

# ipmitool sunoem cli "show /SP system_identifier"

Example:

[root@exadbadb01 ~]#  ipmitool sunoem cli "show /SP system_identifier"
Connected. Use ^D to exit.
-> show /SP system_identifier

  /SP
    Properties:
        system_identifier = Exadata Database Machine X5-2 NS00065378


-> Session closed
Disconnected

Exadata Storage Server Shutdown/Startup procedure

Shutdown Procedure

1. Execute below command to check if there is any offline disk  

cellcli > list griddisk attributes name where asmdeactivationoutcome != ‘ yes’

if the above command provides some output ie some disks are in use still then it's not safe to shutdown or offline the storage server as disk redundancy will not be maintained properly.


2. Inactivate all the griddisk with below command if step one positively executed with no output

Cellcli > alter griddisk all inactive


3. Verify whether all disks are inactive or not with below command

Cellcli > list griddisk where status =! ‘inactive’

If all disk grid disks are inactive then we can proceed to shutdown the storage server.


Startup Procedure

1. Start the cell. All cell services will come up automatically.

2. Bring all disk online

Cellcli > alter griddisk all active

3. Verify whether disks successfully came online or not

Cellcli > list griddisk attributes name, asmmodestatus


All disks should come online. If any of the disk is not online then we need to wait for sometime so that the disk status change gets reflected.

Exadata Operating System

Database Node:

Exadata DB node comes with 4 HDD (Hard Disk) with two OS. Two HDD Linux Configured and two HDD Solaris configured.

After Exadata comes to the floor, customer has to decide that which OS they will be required , accordingly Oracle ACS will configure single OS either Linux or Solaris on all four HDD with RAID 5 configuration.

Note: From X5-2, Solaris is no more available for Exadata machine.

Cell Storage:

Cell storage will be Linux configured. There is no second option for the storage cell.

Cell Storage Processes

There are three processes will be running on the cell stoarge servers.

cellsrv - Cell Server

It is mutlithreaded process. 

The primary storage software component. Responsible for disk I/O with the use of iDB protocol*.

It communicates with Oracle Database to serve simple block requests like database buffer cache reads and Smart Scans with projections and filters. 

CELLSRV also implements IORM and collect statistics on its operation.

MS - Management Server

The Management Server process acts like an coordinator between command line interface and cell storage. 

It provide the interface of storage for configuration and administration purpose. 

It communicates with cellsrv process in providing the required information to the cellcli commands that are being executed.

It is also responsible to send alerts and collect statistics.

RS - Restart Server

Restart service will monitor the cellsrv and MS processes. 

It restarts them if necessary.


* iDB protocol is an intelligent DataBase protocol which is the heart of the Exadata. It is used to do communication between Database node and cell storage. It transfers data between cell storage  and Database Node. Also it's provides interconnection bandwidth aggregation and failover.

cellcli - Create Alter and Drop

Cellcli - Create

CellCLI> CREATE CELL [cellname] [realmname=realmvalue,] [interconnect1=ethvalue,] [interconnect2=ethvalue,][interconnect3=ethvalue,] [interconnect4=ethvalue,] 
( ([ipaddress1=ipvalue,] [ipaddress2=ipvalue,] [ipaddress3=ipvalue,] [ipaddress4=ipvalue,]) | ([ipblock=ipblkvalue, cellnumber=numvalue]) ) --- To configure the Oracle Exadata cell network and starts services. 

CellCLI> create celldisk all harddisk
CellCLI> create celldisk all 
CellCLI> create celldisk all harddisk interleaving='normal_redundancy' 
interleaving -- none(default), normal_redundancy or high_redundancy
CellCLI> create celldisk all flashdisk 
CellCLI> create griddisk RECO_CD_11_cell01 celldisk=CD_11_cell01 
CellCLI> create griddisk RECO_CD_11_cell01 celldisk=CD_11_cell01 size=100M 
CellCLI> create griddisk all prefix RECO 
CellCLI> create griddisk all flashdisk prefix FLASH 
CellCLI> create griddisk all harddisk prefix HARD 
CellCLI> create griddisk all harddisk prefix='data', size='270g'
CellCLI> create griddisk all prefix='data', size='300g'
CellCLI> create griddisk all prefix='redo', size='150g' 
CellCLI> create griddisk all harddisk prefix=systemdg 
CellCLI> create flashcache celldisk='FD_00_cell01' 
CellCLI> create flashcache celldisk='FD_13_cell01,FD_00_cell01,FD_10_cell01,FD_02_cell01,FD_06_cell01, FD_12_cell01,FD_05_cell01,FD_08_cell01,FD_15_cell01,FD_14_cell01,FD_07_cell01,FD_04_cell01,FD_03_cell01,FD_11_cell01,FD_09_cell01,FD_01_cell01' 

CellCLI> create flashcache all 
CellCLI> create flashcache all size=365.25G 
CellCLI> create key 
CellCLI> create quarantine 
CellCLI> create threshold cd_io_errs_min.prodb comparison=">", critical=10 
CellCLI> create threshold CD_IO_ERRS_MIN warning=1, comparison='>=', occurrences=1, observation=1 

Cellcli - Alter

CellCLI> alter cell shutdown services rs --> To shutdown the Restart Server service 
CellCLI> alter cell shutdown services MS --> To shutdown the Management Server service 
CellCLI> alter cell shutdown services CELLSRV --> To shutdown the Cell Services 
CellCLI> alter cell shutdown services all -->To shutdown the RS, CELLSRV and MS services 
CellCLI> alter cell restart services rs 
CellCLI> alter cell restart services all 
CellCLI> alter cell led on
CellCLI> alter cell led off 
CellCLI> alter cell validate mail
CellCLI> alter cell validate configuration 
CellCLI> alter cell smtpfromaddr='cell07@orac.com' 
CellCLI> alter cell smtpfrom='Exadata Cell 07' 
CellCLI> alter cell smtptoaddr='satya@orac.com' 
CellCLI> alter cell emailFormat='text' 
CellCLI> alter cell emailFormat='html' 
CellCLI> alter cell validate snmp type=ASR - Automatic Service Requests (ASRs) 
CellCLI> alter cell snmpsubscriber=((host='snmp01.orac.com,type=ASR')) 
CellCLI> alter cell restart bmc - BMC, Baseboard Management Controller, controls the compoments of the cell. 
CellCLI> alter cell configure bmc 
CellCLI> alter physicaldisk 34:2,34:3 serviceled on
CellCLI> alter physicaldisk 34:6,34:9 serviceled off 
CellCLI> alter physicaldisk harddisk serviceled on 
CellCLI> alter physicaldisk all serviceled on 
CellCLI> alter lun 0_10 reenable 
CellCLI> alter lun 0_04 reenable force 
CellCLI> alter celldisk FD_01_cell07 comment='Flash Disk' 
CellCLI> alter celldisk all harddisk comment='Hard Disk' 
CellCLI> alter celldisk all flashdisk comment='Flash Disk' 
CellCLI> alter griddisk RECO_CD_10_cell06 comment='Used for Reco'
CellCLI> alter griddisk all inactive 
CellCLI> alter griddisk RECO_CD_11_cell12 inactive 
CellCLI> alter griddisk RECO_CD_08_cell01 inactive force 
CellCLI> alter griddisk RECO_CD_11_cell01 inactive nowait 
CellCLI> alter griddisk DATA_CD_00_CELL01,DATA_CD_02_CELL01,...DATA_CD_11_CELL01 inactive 
CellCLI> alter griddisk all active 
CellCLI> alter griddisk RECO_CD_11_cell01 active 
CellCLI> alter griddisk all harddisk comment='Hard Disk' 
CellCLI> alter ibport ibp2 reset counters
CellCLI> alter iormplan active 
CellCLI> alter quarantine 
CellCLI> alter threshold DB_IO_RQ_SM_SEC.PRODB comparison=">", critical=100 
CellCLI> alter alerthistory

Cellcli - Drop

CellCLI> drop cell --->> To reset the cell to its factory settings, removes the cell related properties of the server; it does not actually remove the physical server. 

CellCLI> drop cell force 
CellCLI> drop celldisk CD_01_cell05 
CellCLI> drop celldisk CD_00_cell09 force 
CellCLI> drop celldisk harddisk 
CellCLI> drop celldisk flashdisk 
CellCLI> drop celldisk all
CellCLI> drop celldisk all flashdisk force 
CellCLI> drop griddisk DBFS_DG_CD_02_cel14 
CellCLI> drop griddisk RECO_CD_11_cell01 force 
CellCLI> drop griddisk prefix=DBFS 
CellCLI> drop griddisk flashdisk 
CellCLI> drop griddisk harddisk 
CellCLI> drop griddisk all 
CellCLI> drop griddisk all prefix=temp_dg 
CellCLI> drop flashcache 
CellCLI> drop quarantine 
CellCLI> drop threshold DB_IO_RQ_SM_SEC.PRODB 
CellCLI> drop alerthistory

cellcli Continued...

Cellcli - Describe

It Will display all attributes

CellCLI> describe cell

CellCLI> describe physicaldisk

CellCLI> describe lun

CellCLI> describe celldisk

CellCLI> describe griddisk

CellCLI> describe flashcache

CellCLI> describe flashcachecontent

CellCLI> describe metriccurrent

CellCLI> describe metricdefinition

CellCLI> describe metrichistory 

Cellcli - List

CellCLI> help list 

Enter HELP LIST <object_type> for specific help syntax. 

<object_type>: {ACTIVEREQUEST | ALERTDEFINITION | ALERTHISTORY | CELL | CELLDISK | FLASHCACHE | FLASHCACHECONTENT | GRIDDISK | IBPORT | IORMPLAN | KEY | LUN | METRICCURRENT | METRICDEFINITION | METRICHISTORY | PHYSICALDISK | QUARANTINE | THRESHOLD } 

CellCLI> list cell - Will display Oracle Exadata Storage Servers/Cells information 

CellCLI> list cell detail 

CellCLI> list cell attributes all 

CellCLI> list cell attributes rsStatus 

CellCLI> list physicaldisk - Will display physical disks information 

CellCLI> list physicaldisk detail 

CellCLI> list physicaldisk 34:5 

CellCLI> list physicaldisk 34:11 detail 

CellCLI> list physicaldisk attributes all 

CellCLI> list physicaldisk attributes name, id, slotnumber 

CellCLI> list physicaldisk attributes name, disktype, makemodel, physicalrpm, physicalport, status 

CellCLI> list physicaldisk attributes name, disktype, errCmdTimeoutCount, errHardReadCount, errHardWriteCount 

CellCLI> list physicaldisk where diskType='Flashdisk' 

CellCLI> list physicaldisk attributes name, id, slotnumber where disktype="flashdisk" and status != "not present" 

CellCLI> list physicaldisk attributes name, physicalInterface, physicalInsertTime where disktype = 'Harddisk'

CellCLI> list physicaldisk where diskType=flashdisk and status='poor performance' detail 

CellCLI> list lun  

CellCLI> list lun detail 

CellCLI> list lun 0_8 detail 

CellCLI> list lun attributes all 

CellCLI> list lun attributes name, cellDisk, raidLevel, status 

CellCLI> list lun where disktype=flashdisk 

CellCLI> list celldisk - Will display cell disks information 

CellCLI> list celldisk detail 

CellCLI> list celldisk FD_01_cell07 

CellCLI> list celldisk FD_01_cell13 detail 

CellCLI> list celldisk attributes all 

CellCLI> list celldisk attributes name, devicePartition 

CellCLI> list celldisk attributes name, devicePartition where size>20G

CellCLI> list celldisk attributes name,interleaving where disktype=harddisk 

CellCLI> list griddisk - Will display grid disks information 

CellCLI> list griddisk detail

CellCLI> list griddisk DG_01_cell03 detail 

CellCLI> list griddisk attributes all 

CellCLI> list griddisk attributes name, size

CellCLI> list griddisk attributes name, cellDisk, diskType 

CellCLI> list griddisk attributes name, ASMDeactivationOutcome, ASMModeStatus 

CellCLI> list griddisk attributes name,cellDisk,status where size=476.546875G

CellCLI> list griddisk attributes name where asmdeactivationoutcome != 'Yes' 

CellCLI> list flashcache - Will display flash cache information 

CellCLI> list flashcache detail 

CellCLI> list flashcache attributes all 

CellCLI> list flashcache attributes degradedCelldisks 

CellCLI> help list FLASHCACHECONTENT 

Usage: LIST FLASHCACHECONTENT [<filters>] [<attribute_list>] [DETAIL] 

Purpose: Displays specified attributes for flash cache entries. 

Arguments: 

<filters>: An expression which determines the entries to be displayed. 

<attribute_list>: The attributes that are to be displayed. ATTRIBUTES {ALL | attr1 [, attr2]... } 

[DETAIL]: Formats the display as an attribute on each line, with an attribute descriptor preceding each value. 

CellCLI> list flashcachecontent - Will display flash cache content information

CellCLI> list flashcachecontent detail 

CellCLI> list flashcachecontent where objectnumber=161441 detail

CellCLI> list flashcachecontent where dbUniqueName like 'EX.?.?' and hitcount > 100 attributes dbUniqueName, objectNumber, cachedKeepSize, cachedSize

CellCLI> list flashcachecontent where dbUniqueName like 'EX.?.?' and objectNumber like '.*007'

CellCLI> list flashcachecontent where dbUniqueName like '.*X.?.?' and objectNumber like '.*456' detail 

CellCLI> list metriccurrent - Will display metrics information 

CellCLI> list metriccurrent gd_io_rq_w_sm 

CellCLI> list metriccurrent n_nic_rcv_sec detail 

CellCLI> list metriccurrent attributes name,metricObjectName,metricType, metricValue,objectType where alertState != 'normal' 

CellCLI> list metriccurrent attributes name,metricObjectName,metricType, metricValue,alertState where objectType = 'HOST_INTERCONNECT' 

CellCLI> list metriccurrent attributes all where objectType = 'CELL'

CellCLI> list metriccurrent attributes all where objectType = 'GRIDDISK' - 

> and metricObjectName = 'DATA_CD_09_cell01' and metricValue > 0 

CellCLI> list metricdefinition 

CellCLI> list metricdefinition cl_cput detail 

CellCLI> list metricdefinition attributes all where objecttype='CELL' 

CellCLI> list metrichistory - Will display metric's history

CellCLI> list metrichistory cl_cput 

CellCLI> list metrichistory where objectType = 'CELL' 

CellCLI> list metrichistory where objectType = 'CELL' and name = 'CL_TEMP'

CellCLI> list metrichistory cl_cput where collectiontime > '*2011-10-15T22:56:04-04:00*' 

CellCLI> list alertdefinition detail - Will display alert's definitions 

CellCLI> list alertdefinition attributes all where alertSource!='Metric'

CellCLI> list alerthistory - Will display alert's history

CellCLI> list alerthistory detail

CellCLI> list alerthistory where notificationState like '[023]' and severity like '[warning|critical]' and examinedBy = NULL;

CellCLI> list activerequest 

CellCLI> list ibport - Will display InfiniBand configuration details

CellCLI> list ibport detail 

CellCLI> list iormplan - Will display IORM plan details 

CellCLI> list key

CellCLI> list quarantine 

CellCLI> list threshold - Will display threshold details

Cellcli Utility

What is cellcli utility?

Full form – Cell Command Line Interface

Cellcli used to manage the cell through the cell command line utility.

To invoke the cellcli utility, login to the cell storage through root, celladmin or cellmonitor and type cellcli.

Export                 CellCLI> export celldisk 
Import                 CellCLI> import celldisk 
Assign                  CellCLI> assign key 
Calibrate              CellCLI> calibrate
                           CellCLI> calibrate force 

Set

CellCLI> help set 

Usage: SET <variable> <value> 
Purpose: Sets a variable to alter the CELLCLI environment settings for your current session. 
Arguments: variable and value represent one of the following clauses: 
DATEFORMAT { STANDARD | LOCAL } 
ECHO { ON | OFF } 

CellCLI> set dateformat local
CellCLI> set dateformat standard 
CellCLI> set echo on 
CellCLI> set echo off 

Spool 
CellCLI> spool myCellCLI.txt 
CellCLI> spool myCellCLI.txt append 
CellCLI> spool myCellCLI.txt replace 
CellCLI> spool off 
CellCLI> spool --- Will give spool file name 
Scripts execution 
CellCLI> @listdisks.cli 
CellCLI> start listdisks.cli 
Comments 
REM This is a comment 
REMARK This is another comment 
-- This is yet another comment 
Continuation Character 
CellCLI> list metriccurrent attributes name,metricObjectName,metricValue, - 
objectType where alertState != 'normal' --- continuation character for queries spanned in multiple lines 

Exit/Quit 
CellCLI> exit 
CellCLI> quit 

Available cellcli commands:

        ALTER

        ALTER ALERTHISTORY

        ALTER CELL

        ALTER CELLDISK

        ALTER GRIDDISK
        ALTER IBPORT

        ALTER IORMPLAN

        ALTER LUN
        ALTER PHYSICALDISK
        ALTER QUARANTINE

        ALTER THRESHOLD

        ASSIGN KEY

        CALIBRATE

        CREATE

        CREATE CELL

        CREATE CELLDISK

        CREATE FLASHCACHE

        CREATE GRIDDISK

        CREATE KEY
        CREATE QUARANTINE

        CREATE THRESHOLD

        DESCRIBE

        DROP

        DROP ALERTHISTORY

        DROP CELL

        DROP CELLDISK

        DROP FLASHCACHE

        DROP GRIDDISK
        DROP QUARANTINE

        DROP THRESHOLD

        EXPORT CELLDISK

        IMPORT CELLDISK

        LIST

        LIST ACTIVEREQUEST

        LIST ALERTDEFINITION

        LIST ALERTHISTORY

        LIST CELL

        LIST CELLDISK

        LIST FLASHCACHE

        LIST FLASHCACHECONTENT
        LIST GRIDDISK

        LIST IBPORT
        LIST IORMPLAN

        LIST KEY

        LIST LUN

        LIST METRICCURRENT

        LIST METRICDEFINITION

        LIST METRICHISTORY

        LIST PHYSICALDISK
        LIST QUARANTINE

        LIST THRESHOLD

        SET

        SPOOL

        START

Oracle Exadata X5-2 Key Features and Architecture

Key Features 

Up to 684 CPU Cores and 14.6 TB memory per rack for db processing.

Up to 288 CPU Cores per rack dedicated to SQL processing in cell storage.

From 2 to 19 database servers per rack

From 3 to 18 Oracle Exadata Storage servers per rack

Up to 230 TB of flash storage per rack

Uncompressed and mirrored usable capacity of up to 604 TB per rack.

Hybrid columnar compression often delivers 10x to 15x compression ratios.

Complete redundancy for high availability.

Few useful reference links which can help you to explore further on X5-2 architecture.

Oracle Exadata X5-2 press release

Oracle Exadata X5-2 Server Details

Oracle Exadata X5-2 system architecture

Oracle Exadata Datasheet and Database Machine

Exadata Terms

HP - High Performance

HC - High Capacity

IOPS - IOs Per Second

LUN - Logical Unit Number

MS - Management Server

RS - Restart Server

Cellsrv – Cell Server

SATA - Serial Advanced Technology Attachment

SAS - Serial Attached SCSI

PDU - Power Distribution Unit

ILOM - Integrated Lights Out Manager

IB - InfiniBand

EHCC - Exadata Hybrid Columnar Compression

CU - Compression Unit

UDMs - User Defined Metrics

RDS - Reliable Data Socket protocol

SNMP - Simple Network Management Protocol

iDB - Intelligent Database protocol

EM - Enterprise Manager

EMGC - Enterprise Manager Grid Control

DBRM - Database Resource Manager