Sunday, October 27, 2019

OCI-Oracle ATP- Import Data from on-prem

The primary method to move data to the Oracle ATP instance is by using Data Pump.

You export data from your On-Prem database. Use the following parameters for faster and easier migration to ATP
exclude=cluster,db_link
parallel=n
schemas=schema_name
dumpfile=export%u.dmp
For larger data size, use parallelism. Use the number of CPU's you have in your ATP. For a 16 CPU ATP instance, here is an example of export. Other parameters like compression can also be used.
expdp sh/sh@orcl \
exclude=cluster,db_link \
parallel=16 \
schemas=sh \
dumpfile=export%u.dmp
Once the export file(s) is ready, move the files to Oracle Cloud Infrastructure Object Storage.

You can use 10 GiB of Object Storage and 10 GiB of Archive Storage for free in your home region. If you use more than 20 GiB and have not upgraded when your Free Trial ends, your data is deleted.
If You are participating in a Free Trial and can store unlimited data. When your trial ends, you are converted to an Always Free account. An Always Free account is limited to 20 GiB of combined Object Storage and Archive Storage in your home region. If you are using more than 20 GiB when your account is converted, your data is deleted. Reduce your usage to 20 GiB or less before converting to Always Free.
Storage usage is the approximate total size of all objects in the region. Usage is updated periodically. You will see a lag between what is displayed and the actual usage.

Oracle Data Pump v18.3 or above supports importing data into ATP. See (https://erpthings.blogspot.com/2019/10/oci-oracle-atp-connect-via-sqlplus-on.html) for details on how to get Oracle Instance Client.
Steps:
  • Store Credential Argument so that Data Pump can authenticate to the Object Storage services
BEGIN
DBMS_CLOUD.CREATE_CREDENTIAL
  DBMS_CLOUD.CREATE_CREDENTIAL(
    credential_name => 'objectstore',
    username => <'userid'>,
    password => <'password'>
  );

  dbms_cloud.enable_credential(credential_name => 'objectstore');
  ALTER DATABASE PROPERTY SET default_credential = 'ADMIN.objectstore';
End;
  exit;
  • Determine the Native URI for your dump file(s) (See this post)
  • Run Data Pump import
impdp admin/password@ATPC1_high \       
     directory=data_pump_dir \       
     credential=def_cred_name \       
     dumpfile= https://objectstorage.us-ashburn-1.oraclecloud.com/n/atpc/b/atpc_user/o/export%u.dmp \
     parallel=16 \
     transform=segment_attributes:n \
     transform=dwcs_cvt_iots:y transform=constraint_use_default_index:y \
     exclude=cluster,db_link
  I got a few errors that were eluding to the fact that the user is not authenticated. To work around the issue, I used a few support documents:

How to validate Object Storage URI and Credentials Are Correctly Set for ATP (Doc ID 2468298.1)

NOTE:2113967.1 - How To Validate A DataPump Export (EXPDP) Dump File ?
NOTE:2416814.1 - ADW: Getting "ORA-17500: ODM err:ODM HTTP Unauthorized" error during import to ADW Instance
NOTE:2446550.1 - ATP : Import to ATP From Object Store Fails with 'ORA-17500: ODM Err:ODM HTTP Unauthorized'

Finally, I used a pre-authenticated url for the dmp file as discussed in the last note.
Once this was done, here is the new impdb command which imported the data:


./impdp admin@buzztaqdev_high \ dumpfile=default_credential:<pre-authenticated url > \
transform=segment_attributes:n transform=dwcs_cvt_iots:y \
transform=constraint_use_default_index:y exclude=cluster,db_link




Sachin
Chief Architect
Oracle Apex Enthusiast
BuzzClan

Saturday, October 26, 2019

Oracle OCI - Object Storage - Access via Native URI

To be able to access objects in the Oracle Cloud Object Storage, we need a Uniform Resource Identifier (URI) ('a string of characters that unambiguously identifies a particular resource')

  • Native URI

https://objectstorage.region.oraclecloud.com/n/object-storage-namespace/b/bucket/o/filename
Tip:
You can view the Native URI of an object (such as an uploaded file) in its Object Details dialog:

  1. Go to the Bucket Details page of the bucket containing the object.
  2. In the bucket's list of objects, click the action menu action menu icon and choose View Object Details.
    The URL Path (URI) field displays the object's Native URI value.



Sachin
Chief Architect
Oracle Apex Enthusiast
BuzzClan

Tuesday, October 15, 2019

OCI-Oracle ATP- Connect via SQLPlus on Linux


Continuing from my OCI series: Moving On-Premises Oracle Apex application to OCI Oracle Autonomous Transaction Processing

To connect to the ATP instance from a Linux machine, we have to do a little more leg work.

Some things to review before we get started.
The client software should be 11.2.0.4 (full install) or higher to be able to connect to the ATP instance. Instant Client can be downloaded and installed from these links:

Oracle Instant Client Downloads
Instant Client Downloads for Linux x86-64 (64-bit)

Downloads

Download the appropriate version per your requirements. I have installed the 18.5 version.
  • Basic Package - All files required to run OCI, OCCI, and JDBC-OCI applications
  • Tools-optional Package SQL*Plus Package - The SQL*Plus command-line tool for SQL and PL/SQL queries
  • Tools Package - Includes Data Pump, SQL*Loader and Workload Replay Client
I downloaded it to my local machine and then uploaded it to the Linux machine via FTP. There are ways to do "wget", but require more work as Oracle Authentication is required.

Once installed, I unzipped in the oracle user home

/home/oracle/instantClient

If you have the other client also installed on the machine, you may have to set-up things slightly different, but basic steps are the same.

Setup Wallet and configuration files

Download the client credentials (Wallet Zip) to the machine, and uninstall it to the home directory for oracle

drwxrwxr-x 2 oracle oracle  4096 Oct 13 17:28 Wallet_BuzzTAQDev
-rw-rw-r-- 1 oracle oracle 19814 Oct 13 17:25 Wallet_BuzzTAQDev.zip

Change the underlying file permissions to 600

chmod  600 *
[oracle@ip-172-30-0-39 Wallet_BuzzTAQDev]$ ll
total 36
-rw------- 1 oracle oracle 6669 Oct 12 23:21 cwallet.sso
-rw------- 1 oracle oracle 6624 Oct 12 23:21 ewallet.p12
-rw------- 1 oracle oracle 3244 Oct 12 23:21 keystore.jks
-rw------- 1 oracle oracle   87 Oct 12 23:21 ojdbc.properties
-rw------- 1 oracle oracle  114 Oct 12 23:21 sqlnet.ora
-rw------- 1 oracle oracle 1751 Oct 12 23:21 tnsnames.ora
-rw------- 1 oracle oracle 3336 Oct 12 23:21 truststore.jks

If Oracle database or client is already installed, we can put the tnsnames and sqlnet files in the current admin/network directory in ORACLE_HOME
If not, we will move the files to the instant client network/admin/ location

/home/oracle/instantClient/network/admin

Change the sqlnet.ora file :

WALLET_LOCATION = (SOURCE = (METHOD = file) (METHOD_DATA = (DIRECTORY="/home/oracle/Wallet_BuzzTAQDev")))

#SSL_SERVER_DN_MATCH=yes

Make sure that all the tnsnames in the downloaded file are in the final tnsnames.ora file

<>_high
<>_low
<>_mediom
<>_tp
<>_tpurgent

Client configuration:

Add the location of the client install in the current path variable:

export PATH=$PATH:/home/oracle/instantClient >> /home/oracle/.bash_profile

add the library path

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/oracle/instantClient >> /home/oracle/.bash_profile

If you already have sql client installed, you may have to rename the key files to ensure you are running the proper version

mv sqlplus sqlplus18

Going forward, we will use the sqlplus18 instead of sqlplus to run our scripts.

Test your connection

[oracle@ip-172-30-0-39 instantClient]$  sqlplus18 admin@buzztaqdev_medium
SQL*Plus: Release 18.0.0.0.0 - Production on Tue Oct 15 18:13:44 2019
Version 18.5.0.0.0
Copyright (c) 1982, 2018, Oracle.  All rights reserved.
Enter password:
Message 1549 not found;  product=SQLPlus; facility=SP2
Connected to:
Oracle Database 18c Enterprise Edition Release 18.0.0.0.0 - Production
Version 18.4.0.0.0
SQL>


Happy testing!!


Sachin
Chief Architect
Oracle Apex Enthusiast
BuzzClan

Monday, October 14, 2019

OCI - ATP Database - Basic connectivity - SQL Developer

Continuing from my OCI series: Moving On-Premises Oracle Apex application to OCI Oracle Autonomous Transaction Processing

Once the database is provisioned, you can connect from various clients.

  1. Credential Zip File
DB ConnectionsDownload Credentials


The Credential zip contains the following files:
  • Wallet and keystore:
    • cwallet.sso
    • ewallet.p12
    • keystore.jks
    • truststore.jks
  • Configuration
    • sqlnet.ora
    • tnsnames.ora
TNS entries are as follows:
  • High
  • Low
  • Medium
  • TP
  • TPurgent

Connecting to SQLDeveloper

Various versions will have different options when it comes to the connection type. I am sharing a couple of things I have seen. Either way, here are a few pointers. 
Download the latest SQL Developer. Cloud connections and secure connections are not available in older versions.


Open SQL Developer on your local computer. In the Connections panel, right-click Connections and select New Connection.
Note:
Depending on your version of SQL Developer, do not right-click Cloud Connection or Database Schema Service Connections. That menu selection is for connecting to a different Oracle cloud service, the Oracle Database Schema Service.

To start a new connection, start with New Connection, not the Cloud Connection.


The New/Select Database Connection dialog appears. Enter the following information:
  • Connection Name - Enter the name for this cloud connection.
  • Username - Enter the database username. Use the default administrator database account (admin) that is provided as part of the service.
  • Password - Enter the admin user's password that you or your Autonomous Transaction Processing administrator specified when creating the service instance.
  • Connection Type - Select Cloud Wallet. (or Cloud PDB)
  • Configuration File - Click Browse, and select the Client Credentials zip file, downloaded from the Autonomous Transaction Processing service console by you.
  • Service - In the drop-down menu, service selections are prepended with database names. Select the tpurgent, tp, high, medium, or low menu item for your database. These service levels map to the TPURGENT, TP, HIGH, MEDIUM and LOW consumer groups, which provide different levels of priority for your session. 



OR

Click Test, Save and Connect.

Link to the ObE - Connecting SQL Developer to Autonomous Transaction Processing

Next: OCI-Oracle ATP- Connect via SQLPlus on Linux

Sachin
Chief Architect
Oracle Apex Enthusiast
BuzzClan

OCI - Creating a ATP Database instance

Create Database Instance


We will provision a 'Serverless deployment' instance of the database. 

  • On the navigation menu on the left, select Autonomous Transaction Processing under Database
  • Click "Create Autonomous Database"

  • Compartment: Use default (for new accounts, you will need to provide a name)
  • Display Name: Descriptive name for the database
  • Database Name
  • Workload Type: Select 'Autonomous Transaction Processing'
  • Deployment Type: Serverless
  • Configuration: Always Free
  • (OCPU:1, Storage: .02TB-2GB)
  • User Name/Password
  • License Type: Included
  • Tags


Create Database

The Create Autonomous Transaction Processing Database dialog closes. On the console, the State field indicates that the database is Provisioning. When creation is completed, the State field changes from Provisioning to Available.



For details see the official documentation here:

Provisioning Autonomous Transaction Processing

Now that the database is ready, let's connect to it:

OCI - ATP Database - Basic connectivity - SQL Developer
OCI-Oracle ATP- Connect via SQLPlus on Linux

Next: OCI-Oracle ATP- Import Data from on-prem


Sachin
Chief Architect
Oracle Apex Enthusiast
BuzzClan

Overview of OCI services - Focus on Always Free Tier

Continuing from my OCI series:
Moving On-Premises Oracle Apex application to OCI Oracle Autonomous Transaction Processing

Once you have signed up on OCI, you can see the various links to the services being offered:

There are a lot of services available on this list. We will be introduced to the following during the course of this series:

  • Database - Autonomous Transaction Processing (ATP)

This will allow us to get a serverless database instance. Oracle autonomously operates all aspects of the database life cycle from database placement to backup and updates. Here is the link to the Oracle Help Center page for ATP: Getting Started with Autonomous Transaction Processing

  • Object Storage

The Oracle Cloud Infrastructure Object Storage service is an internet-scale, high-performance storage platform that offers reliable and cost-efficient data durability. The Object Storage service can store an unlimited amount of unstructured data of any content type, including analytic data and rich content, like images and videos.

As we will be migrating from an existing application, we will use the OS to import data from our existing database (expdb/impdb)

Here is the link to the official documentation: Overview of Object Storage

The Always Free limitations are:

Database:

  1. 2 Database instances
  2. 1 OCPU per database
  3. 20 Gb Exadata storage per database
  4. Doesn't support full backup and restore
  5. Deactivated after 7 days of inactivity, and maybe permanently deleted after 9- days. Banner and email notifications are sent before this happens.
  6. It can be upgraded to paying status, which will remove these restrictions.

Storage:

You can use 10 GiB of Object Storage and 10 GiB of Archive Storage for free in your home region. 

Next: Creating ATP Database Instance

Sachin
BuzzClan LLC

Sunday, October 13, 2019

Moving On-Premises Oracle Apex application to OCI Oracle Autonomous Transaction Processing

Oracle Apex is now available on Oracle Cloud Infrastructure and is very easy to set up by itself. I will explain the various steps in the following blogs.

As we progress, I will also document the process to migrate an application I have which is currently on a compute node in AWS.

The current architecture is as follows:
AWS - EC2 compute node with AWS Linux
Oracle 11g XE
APEX 18.1

My goal is to migrate the application on this server to OCI cloud Autonomous database.

Oracle is promising the following 'Always Free Tier'
New Always Free Services you can use for an unlimited time.

  • Two Oracle Autonomous Databases with powerful tools like Oracle Application Express (APEX) and Oracle SQL Developer
  • Two Oracle Cloud Infrastructure Compute VMs; Block, Object, and Archive Storage; Load Balancer and data egress; Monitoring and Notifications
  • Object Storage: You can use 10 GiB of Object Storage and 10 GiB of Archive Storage for free in your home region.


We plan to take benefit of the Oracle Autonomous Database instance and provision APEX on it to migrate our application.

So here is your link to signup:
Sign Up for Oracle Cloud

It will need your credit card so be prepared to provide that. Oracle does not charge anything 'just yet'.

Let's come back when you have signed up so we can follow through creating our database instance.

Next Blog: Overview of OCI Services

Sachin
Chief Architect
BuzzClan LLC

What is Zero Trust Architecture?