Tuesday, July 19, 2016

Infiniband Switch Patching

Exadata Patching - Infiniband Switch


Here we have listed down bullet points to patch IB switches on Exadata Database Machine.



Syntax: #patchmgr -ibswitches [ibswitch_list_file] <-upgrade | -downgrade [-ibswitch_precheck] [-force]]

Here 

ibswitch_list_file contains IP of all the IB switches
Upgrade - to upgrade the switch
Downgrade - to downgrade the switch
ibswitch_precheck - to check the prerequisites

Patchmgr utility would be available in storage server patch directory.

Patching

#./patchmgr -ibswitches ibswitches -upgrade -ibswitch_precheck


How it works?
  • Disable Subnet Manager
  • Copy firmware to switch
  • Check minimal firmware version to upgrade it
  • Verify enough space in /tmp and /
  • Verify for free memory to start upgrade
  • Verify host details in /etc/hosts and /etc/sysconfig/network-scripts/ifcfg-eth0 /etc/sysconfig/network-scripts/ifcfg-eth1
  • Verify for NTP server
  • Pre-upgrade validation
  • Start upgrade
  • Load firmware
  • Disable Subnet Manager
  • Verify that /conf/configvalid is set to 1
  • Set SMPriority to 5
  • Reboot switch
  • Restart Subnet Manager 
  • Start post-update validation
  • Confirmation: Inifiniband switch is at target patching level
  • Verifying host details in /etc/hosts and /etc/sysconfig/network-scripts/ifcfg-eth0 /etc/sysconfig/network-scripts/ifcfg-eth1
  • Verifying NTP Server
  • Firmware verification on InfiniBand switch 
  • Post-check validation on IBSwitch 
  • Final Confirmation: Update switch to 2.1.3_4 (Firmware version)

Once it completes on one switch, it will start to upgrade next available switch and in last it will give overall status of upgrade.

You would also like 

The Order of Exadata Machine Patching

What Should Be The Order of Exadata Machine Patching?

After checking on how to download and prepare for patching, now lets have an overview of patching order for Oracle Exadata Machine with list of utility which need to be used to apply it, as it is not compulsory to maintain this order but usually we should follow it to mitigate risk.

We have given the patch location of particular component based on patch number 19625719 (QFSDP).

---------------------------------------------------------------------------------------------
1. Database Server
  • OS updates
  • Can be applied through YUM or downloaded ISO image

Utility:yum/dbnodeupdate.sh
Patch Directory:/19625719/19625719/Infrastructure/11.2.3.3.1/ExadataDatabaseServer
---------------------------------------------------------------------------------------------
2. Grid and RDBMS patches
  • Grid patches
  • RDBMS patches
  • Rolling or non-rolling

Utility: opatch/oplan
Patch Directory: /19625719/Database/11.2.0.3.25_QDPE_Oct2014
---------------------------------------------------------------------------------------------
3. Exadata Storage Server
  • Storage Server OS update
  • Rolling or non-rolling manner
  • Storage sotware patches

Utility: patchmgr
Patch Directory:/19625719/Infrastructure/11.2.3.3.1/ExadataStorageServer_InfiniBandSwitch
---------------------------------------------------------------------------------------------
4. InfiniBand Switch 
  • It will be available with storage server patches
  • First apply it on spine switch than leaf switches

Utilitypatchmgr
Patch Directory:/19625719/Infrastructure/11.2.3.3.1/ExadataStorageServer_InfiniBandSwitch
---------------------------------------------------------------------------------------------
5. PDU Patches
  • One by one on both PDUs

Utility: ILOM
Patch Directory:/19625719/Infrastructure/SunRackIIPDUMeteringUnitFirmware/1.06
---------------------------------------------------------------------------------------------

Bundle Patch On Exadata

Things To Do Before Applying Bundle Patch On Exadata

One of our reader asked us, what we should take care before applying the bundle patch on Exadata, so thought to write this article where we have mentioned some points based on our patching experience, we can say it's part of bundle patch pre-preparation.
Missing of single point can lead you to difficult stage which we can eliminate by planing better.

               "Always plan for worst to do the best"


  • Make sure that you have backup plan for Exadata box connectivity in case you are doing things remotely
  • Install VNC client on your workstation
  • Install VNC server on Exadata in case if you required GUI. You can also use Xmanager and Xming to take GUI from your workstation
  • Download latest Exadata bundle patch
  • Read all the readme which comes with the bundle patch and prepare yourself how patching will be done
  • Create excel which contain all the IP addresses of Exadata components
  • Prepare list of password which would be required to apply the bundle patches
  • Download latest Exacheck and execute on the database machine
  • Study full Exacheck report and check if any action need to be required against any warning or error
  • If require execute Exacheck utility second time to get the Exacheck report
  • Make sure Oracle support is available in case of any urgency. Proactively you can create SR with Oracle before starting to apply bundle patch
  • Check SSH user equivalence between DB server and Storage node. Configure if not configured
  • Decide which patching method you are going to use e.g rolling, non-rolling
  • Make sure you have all the contact details for all the team in case of any help required e.g. Application, Unix, Network, Power, backup
  • You should have latest backup available of all the Database running on Exadata box
  • Check if there is any NFS storage is mounted on Exadata if yes than do appropriate action prior
  • If you have not rebooted your DB server since long than it would be always better to do reboot first before applying patch as it gives us relax feeling if everything comes up normal.--:)
  • Based on the patch readme, copy required patch to each component wherever required
  • Don't forget to make step by step plan based on readme, believe us you will get a lot confidence by preparing this
Hope we have covered almost everything, you can comment here if we forgot any points.   


You would also love to know How to install VNC server on Exadata.


Happy Patching :)

VNC Server on Exadata

How to install VNC Server on Oracle Exadata?

VNC is an excellent tool to connect GUI interface.


It also has an advantage of running task remotely and keep them alive even once your network connection to a remote VNC disconnected. 

This is a quite useful if you run some important GUI updates, which may take several hours and you do not want to kill this update simply by loosing a network connection which can be accessed by VNC viewer from our work station.

Exadata doesn't come with GUI configured so we can use third party tool to get GUI from Exadata or can configure VNC server on top of Database server.

To install and configure VNC-Server we have to install it separately as it is not a part of Oracle ACS team who do the Exadata initial setup.

Here we have given step by step guide to install and configure VNC-Server on Exadata.

There are multiple method to install VNC server like yum or manual RPM installation.

We have explained here VNC server installation with manual RPM installation.

Click on image to enlarge....

1. List of RPM with dependency which required to install VNC server.

















2. Download all above RPMs from below Linux rpm repository


3. Copy it to Exadata compute node where you want to install VNC-Server







4. Install RPMs by executing below command

#rpm -Uvh --nodeps *.rpm









5. Start VNC Server (Here we have started with Oracle user to take console from Oracle as no need of GUI   from root user)



6. Check VNC Server status and port number on which you can connect through vnc viewer




7. Open VNC viewer from your workstation and connect to DB server
















We have started VNC through Oracle user hence we have to provide Oracle user credentials.










Once you will connect via VNC viewer you will get below screen from where you can invoke GUI












Note: Whatever RPM packages that are used here is for reference purpose only. Best practice is to get the latest version of rpm from given Linux repository.

Exadata Patching

Exadata Patching - Download, Extract, Prepare

Let's go through the Exadata machine patching procedure in bullet points based on our patching experience.

Initially planned to write a article for entire patching activity for all database machine components in single post but it will look bit nasty so thought to write a post for each database machine component separately which would be easy to understand.

Exadata patching activity is very easy task if you plan it well.

Well, let's start from downloading the bundle patch and extract it to compute node. 

1. First go to the Oracle Document ID 888828.1, this is the first read requirement for any exadata database machine patching activity.
2. Download required patch as per your setup requirement, in our case it's QFSDP 19625719

QFSDP releases contain the latest software for the following components:

Infrastructure
Exadata Storage Server
InfiniBand Switch
Power Distribution Unit
Database
Oracle Database and Grid Infrastructure PSU
Oracle JavaVM PSU (as of Oct 2014)
OPatch
OPlan
Systems Management
EM Agent
EM OMS
EM Plug-ins

3. Move bundle patch to one of the compute node

List of files:

4. Extract it with unzip utility

Command : #unzip file_name

It will give five tar files as given below


5. Now extract all the tar files using tar utility

Command: #cat *.tar.* | tar -xvf -

It will make single directory with patch number called 19625719

Inside 19625719 directory you will get following directories 


In tree view it will be look like below

Directory view:


File View:



It contained patches for following stacks
  • Database/Clusterware
  • Database Server
  • Storage Server 
  • Infiniband 
  • PDUs

Don't forget to read thoroughly the all readme multiple times to get proper understanding of database machine patching activity. It will help to complete patching activity smoothly.

Once above steps complete we are ready to go for patching.

Cloud Computing - A way ahead


Cloud is the most popular topic nowadays but it is not simply fashionable term in the technology, it is actually broad term. 

The cloud is something beyond the internet. It can both software and infrastructure. Where you can use technology when you need it and as long as you need it. You just pay when you use it. You do not require installing any software in your desktop at all.

Cloud models
Public cloud

A public cloud service is based on a cloud provider typically offering pre-configured and published offerings. 

They normally have an online storefront that lists all available products, configurations, options, and pricing. Because the public cloud providers are offering services to the general public and a wide variety of customers, they have implemented their own cloud management platform. The cloud platform and services offered are targeted at the widest group of potential consumers; therefore, customization of the service is normally limited.

Private cloud

Private cloud services might begin with the same basic cloud computing services
as offered by a public cloud provider, but the service can be hosted at a customer owned or contracted data center. 

Private clouds offer choices of the cloud services to be deployed, how much integration there is between services, how the management and operations are handled, and the level of security controls and accreditation.

Hybrid cloud

Hybrid cloud is a cloud computing environment which uses both private and public cloud services.

It allows to move workload between private and public clouds as computing needs and costs change. It gives businesses greater flexibility and more data deployment options.

Cloud services
SaaS

It simply provides software on demand. There is no need to install; you can directly can start using required software.

A service offering that provides one or more applications to the consumer. Applications are hosted and managed by the provider in the cloud with the consumer accessing the application services from various end-computing devices such as PCs, laptops, tablets, smartphones, or web browsers.

The consumer does not manage the underlying server farm, applications, or storage. The provider might provide the consumer a web-based self-service control panel to give the consumer some control over certain aspects of the application or user configuration. Pricing to the consumer is often based on a per seat user license with potential up-charges for additional storage or application features.

PaaS

PaaS environments provide you with an infrastructure as well as complete operational and development environments for the deployment of your applications.

IaaS

In simple terms it provides all the data center hardware needs based on the demand. 

A service that includes processing, memory, storage, and networking in the form of a VM. Consumers can deploy operating system and applications as they prefer. Cloud providers often supply OS templates as a quick-start to get the system operational quickly.

The cloud provider also handles all systems management of backend server farms, networking, storage systems, and virtualization hypervisors with the consumer managing the OS, applications, and data.

Top 15 advantages of cloud

Top cloud service providers

Exadata Version Information

How To Get Current Release Version of Exadata Components?

Whenever we plan for patching, upgrade or migration, knowing the current release version of exadata component is first and mandatory thing to proceed ahead. These information  also helps you while doing any configuration level changes on exadata machine.

Following commands will help you to get the current realease version of the exadata machine component.


Database Server

OS Image Version: #imageinfo



Database: $opatch lsinventory



Grid:  $opatch lsinventory


Cell Server

Software/OS Image Version: #imageinfo


Infiniband Switch

#version


Ethernet Switch

#show version


PDU

Open web console using PDU management IP. Click on module info to get the PDU firmware version