OBIEE Tuning - Under the hood

 Monitoring what to tune:

• Monitor performance using Enterprise Manager,
• OBIEE Admin/Session tool
• Usage Tracking
• Fiddler browser based client developer add-on's - see where is most of the time being taken for a page load and plan accordingly.
• Use Oracle Application Testing Suite (OATS) for Load Testing using a test strategy reflecting the load and type of usage.

Understanding Monitoring is very important  because Oracle's documentation is not very clear on how the settings effect the overall performance. They at best provide two values - default, and suggested, without much explanation. So if we want to customize these for the situation at hand, we should be able to monitor the effects of changes and tweak it to the what best applies to our situation.

Apply latest patches

• Upgrade to the latest patch set - follow Support documentation (Upgrade advisory) to plan major upgrades.
• Stay on approved Java version

Delegate work away from OBIEE server

Reverse Proxy/OHS Cache: OBIEE prevents the browsers from caching its content. This is a desired tract to ensure fresh reports data, but overloads the BI Presentation server/OHS as all the static content now needs to be served for each and every request.

This can be mitigated by implementing a Reverse Proxy server or enabling caching on Web server.

Optimized connections

The Connection Pools to the underlying databases should be enough to be able to reasonably service the expected number of concurrent users. Pools also exists for network connection and here, the timeout should be reduced so that they can be freed more often and made available for subsequent calls. There are various parameter based on the underlying OS to achieve this:

• Database Connection Pools for these should be reviewed:
• Database connection in the physical layer:
• Init-Block Connection Pools
• Physical DB Connection Pools
• BIP/MDS Connection Pools (others if Essbase/ODI are installed)
• Inter-Process connection pool:increase the bridge connection pool for Presentation server to sufficient level
• Enable compression at web server to ensure high throughput
• Ensure sufficient search depth within the AD of choice. This can be result in additional complexity if there is a cyclic membership.

Thread Management

•     Manage Stuck thread trigger time: 
• Account for long running queries from the usage tracking parameters 
• Ensure that the system doesn't flag threads as stuck in that time frame.  
•     Increase thread pool for charting engine, job manager

Turn Off Debugging features on production:

•     Log levels
•     Wrap Data Type (JDBC)
•     Lock down production RPD to read only mode
•     Database client libraries - disable signal handling

JVM - Memory

• Increase Heap
• Compressed Reference
• Thread Local sizes based on available physical memory for OBIEE

Cache management:

• Move cache and temp directories(javaHost/BIP/BIPS) to RAM disk (~256 GB)
• Use cache seeding. Automate using SASeedQuery after ETL process.
• Adjust maximum cache size based on frequently used/first page reports
• Increase number of open files limit in OS to allow for multiple concurrent processing of cache/sort/logs etc.

Aggregate data

Reduce Database's processing cycles, and the network bandwidth needs.

Hardware Optimization

• Maximize Memory and Networking capability of each server. 
• OBIEE licensing is core based. Addition servers will result in added licensing cost. So maximize the memory and network capacity on the serve before scaling out.
• Throw in more hardware, vertically scale the system for performance, get high availability as an added benefit.

Sizing of Server

Factors used to determine the capacity of a server:

•     Total Named user vs Concurrent users
•     SSL turned on or not.
•     complexity of the Report formats
•     Underlying database and the connection between the two servers

BI Publisher

•     Ensure appropriate JDK/JVM
•     Optimal memory and disk space allocation
•     Use BI Publisher's XSLT processor (Enable scalable feature of XSLT processor)
•     Divide optimization efforts by
• Online/Concurrent Usage
• Report size - Small(~10 Pages)/Medium(~50 Pages)/Large(> 100 pages)/ Extremely large (> 100,000 rows)
• Bursting - Total number of report vs. time window
•     Run-time optimization
• Enable XSLT Processor, enable scalable features
• Enable Muti-Threading
• Increase Thread Count
• Increase FO Buffer size
• Increase Pages in Caches setting
• Enable Run-Time Cache
• Disable Autorun
•     Use JNDI over JDBC
•     Push database functions like Joins to Database (Use OBIEE models)
•     XML
• Use short names (smaller overall XML)
• Use Unique names across structures
• Avoid XML attributes
• Use Absolute paths for Searches rather than Relative paths
•      Scale out
•     Configure debug settings to minimal for production

Finally some functional optimization

• Rearrange dashboards so that the smaller/faster reports are rendered first
• Show summary reports (from aggregates) first
• Force users to put appropriate filters in prompts
• Partition database
• Index appropriately
• Run Statistics and keep them current

Moniter, Adjust, Repeat!

Sachin
Architect - Oracle Engineered Systems
Exalytics/Exalogic/Exadata
BuzzClan LLC

BuzzClan is a business consulting company collaborating to provide Oracle software advisory services & implementation services. BuzzClan LLC is committed to providing substantive business value on each and every client engagement. We do this through a combination of industry-specific business expertise, technical skills, proven project management methods and our “onsite - off site - offshore” delivery model. We strive to work in partnership with our customers to build high-performance teams and create business solutions that will last.

Engineered Systems VS. NIST Definition of Cloud Computing

According to the NIST,

Cloud computing is a model for enabling ubiquitous, convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services) that can be rapidly provisioned and released with minimal management effort or service provider interaction. This cloud model is composed of five essential characteristics, three service models, and four deployment models.(http://csrc.nist.gov/publications/nistpubs/800-145/SP800-145.pdf)

In this blog, I will examine how each of the five essential characteristics defined by NIST (in bold/italics) applies to the engineered systems - Exalogic, Exadata, and Exalytics.

On-Demand Self-Service:  

A consumer can unilaterally provision computing capabilities, such as server time and network storage, as needed automatically without requiring human interaction with each service provider.

This refers to the ability of cloud services consumers to configure and maintain the landscape presented to them. This allows for distributed administration capabilities. While the centralized team concentrates on managing the infrastructure tasks like initial setup, capacity planning and resource quota allocation, backup, disaster recovery, patching, libraries of Virtual Machine (VM) templates, middleware code, etc., the application-level administration is delegated to the consumers. They can create VMs from the template library and extend them according to their needs.

To enable the consumers to do this, several administration consoles are provided by the Cloud Infrastructure to allow for GUI, Command Line Interface (CLI) or scripted interaction with the system.

Oracle’s engineered systems provide the following self-service capabilities:
 •  Cloud admin account access
• Enterprise Manager Operations Center (EMOC) access 
• Virtual Data Center (vDC) Management tab with access to pre-built Virtual Machine templates for frequently used functions
• Base VM templates to build upon
• Command Line Interfaces (CLI and dCLI) access to most of these tasks for automation and one-touch unilateral provisioning

Broad Network Access:  

Capabilities are available over the network and accessed through standard mechanisms that promote use by heterogeneous thin or thick client platforms (e.g., mobile phones, tablets, laptops, and workstations).

Oracle’s engineered systems are generally used for private cloud offerings within secured corporate or public sector settings. Therefore, access to these is not required on a mobile device. The systems provide the access over SSL/SSH using the following: 

• Browser-based interfaces (EMOC/VM Manager/ZFSManager)
• CLI interface over SSH

Resource Pooling:  

The provider’s computing resources are pooled to serve multiple consumers using a multi-tenant model, with different physical and virtual resources dynamically assigned and reassigned according to consumer demand. There is a sense of location independence in that the customer generally has no control or knowledge over the exact location of the provided resources but may be able to specify location at a higher level of abstraction (e.g., country, state, or datacenter). Examples of resources include storage,  processing, memory, and network bandwidth.

This, I believe, is the most important characteristic of a cloud offering. With commodity servers running independently, we have seen that there is often a heterogeneous usage of resources. Most of the servers are underutilized while some can never have enough resources to service the load, creating a bottleneck referred to as 'underutilized server sprawl'. With an IaaS scenario, this can be fixed by dynamically rearranging the load on a particular resource. The engineered systems allow for resource quota for the users/accounts (Account Resource Limit).

• Oracle Virtual Machine Server - Virtualizer
• Virtual CPU/CPU oversubscription
• Common memory pool
• Common storage
• Shared network access  

Rapid Elasticity: 

Capabilities can be elastically provisioned and released, in some cases automatically, to scale rapidly outward and inward commensurate with demand. To the consumer, the capabilities available for provisioning often appear to be unlimited and can be appropriated in any quantity at any time.

We currently follow these steps in order to change the memory, vCPU, and/or the Network resource allocation for an existing guest vServer (Reference - Exalogic: Changing the Memory, vCPU and Network Resources for an existing guest vServer (Doc ID 1586296.1)):

1. Stop the vServer
2. Create a template
3. Create a vServer type with the required memory and vCPU resources
4. Delete the old server
5. Create a new vServer by using the template in step #2 and the vServer type created in step #3

I am hopeful that upcoming releases will make this process more streamlined and easier to manage.
Ideally, to ensure elasticity, the virtual servers should have memory, vCPU, network bandwidth, and storage quotas within a range. The servers should also have a priority definition which will help mitigate resource contention issue in case multiple vServers are trying. The vServer should be able to negotiate the resources based on its priority among other servers, the current load on the vServer, and the Virtual Server as a whole.
The disk space is shared among all the compute nodes and virtual machines.  Shares and projects can be created by the cloud admins to be mounted on the VMs. These shares can also be shared among multiple VMs.
Resource quotas are established at a cloud-account level. Any VM created within the account has access to the resources allocated to the account.
What’s missing?

• Networking quota
• Prioritization (if a resource is oversubscribed, which account/VM/process gets access to the resource if the physical limit is reached)
• Time slot-based allocation   

Measured service:

Cloud systems automatically control and optimize resource use by leveraging a metering capability at some level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency for both the provider and consumer of the utilized service.

Enterprise Manager 12c provides the capability to meter the usage and calculate chargeback of different resources (targets).  A universal charge plan contains rates for CPU, Memory and storage. Extended charge plans can be used across various target types. Details can be found in the OEM - Chargeback Administration.

What’s missing? 

• This functionality is not available for Exalytics servers yet. However, in an OBIEE implementation, certain Usage Tracking reports can be configured to provide this functionality.

Tech Term of the Day 

NIST - National Institute of Standards and Technology


Sachin
Architect - Oracle Engineered Systems
Exalytics/Exalogic/Exadata
BuzzClan LLC

BuzzClan is a business consulting company collaborating to provide Oracle software advisory services & implementation services. BuzzClan LLC is committed to providing substantive business value on each and every client engagement. We do this through a combination of industry-specific business expertise, technical skills, proven project management methods and our “onsite - off site - offshore” delivery model. We strive to work in partnership with our customers to build high-performance teams and create business solutions that will last.

Certified - Exalytics In-Memory Machine X3-4 Implementation Essentials

Hurrey!!

Today I got the result of the Exam I gave back in Dec. 

I have cleared the "Exalytics In-Memory Machine X3-4 Implementation Essentials" exam!, earning a OPN Certified Specialist designation in this field.

Together with the  "Exalogic Elastic Cloud X2-2 Certified Implementation Specialist" certification, it helps me differentiate myself as a proven expert in the field of Oracle Engineered Systems implementations.

Getting ready to crack the Exadata certification next!

Sachin
Architect - Oracle Engineered Systems
Exalytics/Exalogic/Exadata
BuzzClan LLC

BuzzClan is a business consulting company collaborating to provide Oracle software advisory services & implementation services. BuzzClan LLC is committed to providing substantive business value on each and every client engagement. We do this through a combination of industry-specific business expertise, technical skills, proven project management methods and our “onsite - off site - offshore” delivery model. We strive to work in partnership with our customers to build high-performance teams and create business solutions that will last.

Elastic Cloud vs. Not-So-Elastic Cloud: CPU Over-subscription

This is Part III in my series of articles about implementing Exalytics as a private cloud. The first two that laid the groundwork for this concept are:

Part I – Exalytics: The Promise of BI on Private Cloud

Part II - Classify the Exalytics Application Requirements and Cloud Offerings

The idea is to create a system which can host various OBIEE/EPM implementations and can adjust according to the current load and the VM’s priority/quota. Each VM is assigned a range of resources from the available pool. Instead of saying that a VM can have 2 cores/CPU, we can virtualize the resources by creating a virtual unit?the vCPU?and allocate available threads.  This vCPU is the key ingredient in an elastic cloud.

vCPU Configuration - EMOC vDC Management

 We allocate to each VM the resources it needs to run optimally. However, instead of assigning an absolute number of resources, it is done virtually in terms of CPU threads rather than cores. This allows two important things: 
• dynamic reallocation of the resources at runtime without the need to recycle the server
• VMs compete for the CPU cycles/threads on the basis of priority allocated to them at design time.

 If we have a VM hosting the EPM Suite, another VM hosting BI for the North America, and another VM hosting BI for the Asian operations, we can schedule the priority of the North America VM and the Asian server to be high priority based on time of operations. The EPM server can have overall priority so that users on the EPM server will get a better throughput whenever they are on.

Account Resource Limit

 One caveat is that the vCPU memory and storage quotas (Account Resource Limit) are set at the account level rather than at the VM level. This enables a cloud user to create multiple VMs within the quota allocated to them.

 The vCPU priority and cap are defined when creating VM profiles.

VM Proile - CPU Priority/Cap

 This profile will be used to define a VM hosting a service which can run in the background and yield to a higher priority task when needed. A deployment plan is created with this profile and applied while creating a virtual machine.

When creating a new VM Server, the following can be applied to benefit from a real elastic cloud implementation:

1. Change the CPU/vCPU ratio
2. Configure an Account Resource Limit for individual accounts
3. Create VM profiles with vCPU priority

Further explanation of this concept can be found in Donald Forbes’s article on  Exalogic and CPU Oversubscription.

Sachin
Architect - Oracle Engineered Systems
Exalytics/Exalogic/Exadata
BuzzClan LLC

BuzzClan is a business consulting company collaborating to provide Oracle software advisory services & implementation services. BuzzClan LLC is committed to providing substantive business value on each and every client engagement. We do this through a combination of industry-specific business expertise, technical skills, proven project management methods and our “onsite - off site - offshore” delivery model. We strive to work in partnership with our customers to build high-performance teams and create business solutions that will last.