针对oracle 11g 设置dnfs

郑全

如题 。

郑全

Step by Step - Configure Direct NFS Client (DNFS) on Linux (文档 ID 762374.1)
转到底部



In this Document

Goal

Solution

References


Applies to:
Oracle Database - Enterprise Edition - Version 11.1.0.6 to 11.2.0.3 [Release 11.1 to 11.2]
Oracle Database - Enterprise Edition - Version 12.1.0.2 to 12.1.0.2 [Release 12.1]
Oracle Database - Enterprise Edition - Version 12.2.0.1 to 12.2.0.1 [Release 12.2]
Generic Linux
Goal
The purpose of this document is to provide information about the Direct NFS client and Step by Step to configure Direct NFS client.
Note :-The following steps are purely provided for evaluative and educational purpose only.Any issues with configuring the NFS server/client on Linux ie PART A ,you need to contact your Vendor.Oracle will not provide support/troubleshoot PART A.

WARNING:
The DNFS Guide says to enable the init.ora param filesystemio_options to enable direct I/O support.
Doing this, all DB access to all files will be via DIO.
There is no way to only enable direct IO for certain files and exclude others.
So ensure ALL the places where are located your database files (including pfile/spfile, logs,....) are configured to accept Direct IO.
Solution
Step by Step
To use Direct NFS Client, the NFS file systems must first be mounted and available over regular NFS mounts.
PART A --  SETTING UP THE NFS SERVER ON LINUX

Preparing NFS server for Oracle Direct NFS mount filesystem.

If you are already setup the NFS ,skip this step and go to  PART B :-'Setup DNFS'
i) For setting up the NFS server we need to know the UID of the software owner and GID of the DBA group. The UID and GID should also be the same on all the cluster nodes.
To find out the UID and GID issue the id command as the Oracle software owner (e.g. oracle) on one of the cluster nodes,  
# id uid=500(oracle) gid=500(dba) groups=500(dba)

In this case the UID is 500 and the GID is also 500.

ii) As root, create the directory for the oracle files on the NFS server and set the ownership of this directory to this UID and the GID,
# mkdir /oraclenfs
# chown 500:500 /oraclenfs

iii) Add this directory to the NFS exports file /etc/exports.

This file should now contain a line like this
/oraclenfs *(rw,sync,all_squash,insecure,anonuid=500,anongid=500)

The anonuid and anongid should contain the UID and GID we found for the oracle user and dba group on the cluster nodes; in this case 500 and 500.
The 'insecure' option is needed for dNFS clone ( dbms_dnfs.clonedb_renamefile ) procedure.

iv) Make sure the NFS server will get started during boot of this server. I.E. for RedHat Linux this could be done like this;
chkconfig --level 345 nfs on

v) Now start the NFS server process on the NFS server. On RedHat Linux this could be done like this;
service nfs start




vi) If the new export directory is added to the /etc/exports file while the NFS server process was already running, restart the NFS server or re-export with the command exportfs.
Check if the oraclefs directory is exported correctly by issuing the exportfs command.

This command should return a line like this;
# [root@stgasm ~]# exportfs -v
/oraclenfs *(rw,wdelay,root_squash,all_squash,anonuid=500,anongid=500)

MOUNTING NFS ON THE CLIENT NODE / CLUSTER NODES ON LINUX

i) To be able to mount the NFS export, as the root user create an empty directory on each cluster node named /oradata1

ii) Make sure the NFS export is mounted on the cluster nodes during boot time by adding the following line to the /etc/fstab file on each cluster node;
stgasm:/oraclenfs /oradata1 nfs rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,actimeo=0,vers=3,timeo=600 0 0

IMPORTANT :  'actimeo=0' setting is needed for RAC set up, and not to be used ( left to default cache timeout value ) for nonRAC setup

iii) Mount the NFS export by executing the mount /oradata1 command on each server.

iv) Check if the NFS export is correctly mounted with the mount command.

This should return a line like this;
# mount
stgasm:/oraclenfs on /oradata1 type nfs (rw,bg,hard,intr,rsize=32768,wsize=32768,tcp,noac,nfsvers=3, timeo=600,addr=10.177.52.158)


Note: The mount options when usingDirect NFS are no different from the mount options when Direct NFS is not being used. Here's the excerpt from the Oracle documentation:

You must mount NFS volumes used for storing database files with special mount options on the host where the database server is running. When mounting an NFS file system, Oracle recommends that you use the same mount point options that the NAS vendor used when certifying the device. Refer to the device documentation or contact the vendor for information about recommended mount-point options.

Please contact your NAS vendor for NFS mount option recommendations. Most of them tend to have white papers for best Oracle and Direct NFS practices that may also have details on NFS mount options.



PART B -- Configure Direct NFS Client (DNFS)
i) Configure oranfstab file
Direct NFS Client can use a new configuration file or the mount tab file (/etc/mtab on Linux) to determine the mount point settings for NFS storage devices.

This file is required only for configuring the Direct NFS for load balancing and specfic to single database.You can still enable the Direct NFS without configuring oranfstab file.DNFS will take mount point settings for NFS from /etc/mtab on Linux


In RAC,the oranfstab must be configured on all nodes and keep /etc/oranfstab file synchronized on all nodes.

(When the oranfstab file is placed in $ORACLE_HOME/dbs, the entries in the file are specific to a single database. In this case, all nodes running an Oracle RAC database use the same ORACLE_HOME/dbs/oranfstab file.

When the oranfstab file is placed in /etc, then it is globally available to all Oracle databases, and can contain mount points used by all Oracle databases running on nodes in the cluster, including single-instance databases. However, on Oracle RAC systems, if the oranfstab file is placed in /etc, then you must replicate the file /etc/oranfstab file on all nodes, and keep each /etc/oranfstab file synchronized on all nodes, just as you must with the /etc/fstab file.

For reference, Direct NFS searches for mount entries in the following order:
1. $ORACLE_HOME/dbs/oranfstab
2. /etc/oranfstab
3. /etc/mtab

configure the oranfstab as like below
$ORACLE_HOME/dbs/oranfstab
server: stgasm <=== NFS server Host name
path: 10.177.52.158 <--- First path to NFS server ie NFS server NIC
local: 10.177.52.151 <--- First client-side NIC
path: 10.177.52.159 <--- Second path to NFS server ie NFS server NIC (For load balance purpose)
local: 10.177.52.151 <--- Second client-side NIC (For load balance purpose)
export: /oraclenfs mount: /oradata1

ii) Enable the Direct NFS Client ODM Library
cd $ORACLE_HOME/lib

mv libodm11.so libodm11.so_bak

ln -s libnfsodm11.so libodm11.so

On 11.2 you can enable it with the following make command

cd $ORACLE_HOME/rdbms/lib
make -f ins_rdbms.mk dnfs_on

iii) Re-start the Database instance.

alert log shows Direct NFS client is enabled by checking the below message  
Oracle instance running with ODM: Oracle Direct NFS ODM Library Version 2.0
iv) Connect to DB instance and move existing datafiles to NFS drive or create tablespace on NFS drive to check DNFS is working or not
SQL> select * from v$dnfs_servers;

ID     SVRNAME            DIRNAME         MNTPORT   NFSPORT    WTMAX RTMAX
--     --------------------          ----------             ----------          ----------           ---------- ----------
1       10.177.52.158        /oraclenfs                 700                 2049                0               0

NOTE: From 12c, DNFS is enabled by default for the database

PART C -- DNFS Workshop.
DNFS Workshop is located at: dnfs_workshop.pdf
References
NOTE:1452614.1 - How To Setup DNFS (Direct NFS) On Oracle Release 11.2

郑全

Step by Step - Configure Direct NFS Client (DNFS) on Windows (文档 ID 1468114.1)
转到底部



In this Document

Goal

Solution

References


Applies to:
Oracle Database - Enterprise Edition - Version 11.1.0.6 to 11.2.0.4 [Release 11.1 to 11.2]
Generic Windows
***Checked for relevance on 21-Nov-2013***
Goal
The purpose of this document is to provide information about the Direct NFS client and Step by Step to configure Direct NFS client on Windows Platform.
Note: The following steps are purely provided for evaluative and educational purpose only. Any issues with configuring the NFS Server on Linux ie PART A, you need to contact your Vendor. Oracle will not provide support/troubleshoot PART A.
Solution

Step by Step
To use Direct NFS Client, the NFS file systems must first be mounted and available over regular NFS mounts.
PART A --  SETTING UP THE NFS SERVER ON LINUX

Preparing NFS server for Oracle Direct NFS mount filesystem.

If you are already setup the NFS (wit proper options), please skip this step and go to  "PART B -- Configure Direct NFS Client (DNFS) on Windows Platform"
1) To find out the UID and GID issue the id command as the Oracle software owner (e.g. oracle) on one of the cluster nodes,  
# uid=1000(oracle) gid=1001(oinstall) groups=1001(oinstall),1000(dba),1003(oper)

In this case the UID of "oracle" user is 1000 and "dba" grouyp has also GID 1000

2) As root, create the directory for the oracle files on the NFS server and set the ownership of this directory to this UID and the GID,
# mkdir /oraclenfs
# chown 1000:1000 /oraclenfs

3) Add this directory to the NFS exports file /etc/exports.

This file should now contain a line like this
/oraclenfs *(rw,sync,all_squash,insecure,anonuid=1000,anongid=1000)

The anonuid and anongid should contain the UID and GID we found for the oracle user and dba group on the cluster nodes; in this case 1000 and 1000.

4) Make sure the NFS server will get started during boot of this server. I.E. for RedHat Linux this could be done like this;
chkconfig --level 345 nfs on

5) Now start the NFS server process on the NFS server. On RedHat Linux this could be done like this;
service nfs start

6) If the new export directory is added to the /etc/exports file while the NFS server process was already running, restart the NFS server or re-export with the command exportfs.
Check if the oraclefs directory is exported correctly by issuing the exportfs command.

This command should return a line like this:

# [root@stgasm ~]# exportfs -v
/oraclenfs *(rw,wdelay,root_squash,all_squash,insecure,anonuid=1000,anongid=1000)

@ Please note that in my test case I was not able to acces the resource using dNFS from Windows client without "insecure" option.
@ The answer that I had from DEV related that option was:
@ "insecure" option is required for any NFS export for dnfs as we use userland tcp ports. Otherwise, connect() must originate from <1024 port number.



PART B -- Configure Direct NFS Client (DNFS) on Windows Platform
Oracle 11g software was installed using the Oracle installer on the Windows server.
1) Create and configure oranfstab file
You must add the"oranfstab" file in %ORACLE_HOME%\dbs directory. Please be sure when you create that text file any extention (eg: txt) was added in the file name.
Then configure the oranfstab as like below:
C:\>type %ORACLE_HOME%\dbs\oranfstab
server: lnxnfs         <=== NFS server Host name
path: 10.171.52.54 <--- First path to NFS server ie NFS server NIC
local: 10.171.52.33 <--- First client-side NIC
export: /oraclenfs  mount: y:\
uid:1000
gid:1000
C:\>
The Direct NFS Client uses the uid or gid value to access all NFS servers listed in oranfstab. Direct NFS ignores a uid or gid value of 0.
The UID and GID used in that example is from "oracle" user from NFS Server.The exported path from the NFS server must be accessible for read/write/execute by the user with the UID, GID specified in oranfstab or, if neither UID nor GID is listed, by the user with the uid:65534,gid:65534 - (nfsnobody).

2) Enable the Direct NFS Client ODM Library:
a. Change directory to %ORACLE_HOME%\bin. (in my case ORACLE_HOME was C:\oracle\product\11.2.0\dbhome_1)
C:\TMP>pushd C:\oracle\product\11.2.0\dbhome_1\BIN

b. Enter the following commands:
----------------------------------------------------------------------------
==> copy oraodm11.dll oraodm11.dll.bkp
----------------------------------------------------------------------------
C:\oracle\product\11.2.0\dbhome_1\BIN>copy oraodm11.dll oraodm11.dll.bkp
        1 file(s) copied.

C:\oracle\product\11.2.0\dbhome_1\BIN>
----------------------------------------------------------------------------
==> copy /Y oranfsodm11.dll oraodm11.dll
----------------------------------------------------------------------------
C:\oracle\product\11.2.0\dbhome_1\BIN>copy /Y oranfsodm11.dll oraodm11.dll
        1 file(s) copied.

C:\oracle\product\11.2.0\dbhome_1\BIN>
----------------------------------------------------------------------------

3) Re-start the Database instance.

alert log shows Direct NFS client is enabled by checking the below message  
Oracle instance running with ODM: Oracle Direct NFS ODM Library Version 3.0

4) Connect to DB instance and try to create a tablespace on NFS drive to check DNFS is working or not
SQL> select tablespace_name from dba_tablespaces;

TABLESPACE_NAME
------------------------------
SYSTEM
SYSAUX
UNDOTBS1
TEMP
USERS
TEST_DNFS

SQL>
SQL> column ID format 999
SQL> column SVRNAME format A15
SQL> column DIRNAME format A15
SQL> column PATH format A15
SQL> column LOCAL format A15
SQL> select * from v$dnfs_servers;

  ID SVRNAME         DIRNAME            MNTPORT    NFSPORT      WTMAX      RTMAX
---- --------------- --------------- ---------- ---------- ---------- ----------
   1 lnxnfs          /oraclenfs             909       2049     262144     262144

SQL> select * from v$dnfs_channels;

      PNUM SVRNAME         PATH            LOCAL                CH_ID     SVR_ID      SENDS      RECVS      PINGS
---------- --------------- --------------- --------------- ---------- ---------- ---------- ---------- ----------
        10 lnxnfs          10.171.52.54    10.171.52.33             0          1        132        265      0
        10 lnxnfs          10.171.52.54    10.171.52.33             1          1          1          2      0
        17 lnxnfs          10.171.52.54    10.171.52.33             0          1        415        220      0
        27 lnxnfs          10.171.52.54    10.171.52.33             0          1          2          4      0

SQL>
SQL> alter database datafile 'y:\test_dnfs.dbf' resize 200M;

Database altered.

SQL>

5) Excerpt from Database alertlog.
  diagnostic_dest          = "C:\ORACLE"
Oracle instance running with ODM: Oracle Direct NFS ODM Library Version 3.0    <<<<<<<<<<<< Use dNFS library
Thu Jun 14 16:49:25 2012
PMON started with pid=2, OS id=2132
.
.
Thu Jun 14 16:56:45 2012
create tablespace TEST_DNFS datafile 'y:\test_dnfs.dbf' size 100m
Thu Jun 14 16:56:55 2012
Direct NFS: channel id [0] path [10.171.52.54] to filer [lnxnfs] via local [10.171.52.33] is UP    <<<<<
Direct NFS: channel id [1] path [10.171.52.54] to filer [lnxnfs] via local [10.171.52.33] is UP    <<<<<
Thu Jun 14 16:56:56 2012
Completed: create tablespace TEST_DNFS datafile 'y:\test_dnfs.dbf' size 100m
Thu Jun 14 17:25:52 2012
alter database datafile 'y:\test_dnfs.dbf' resize 200M
Completed: alter database datafile 'y:\test_dnfs.dbf' resize 200M
.
.


@ for troubleshooting dNFS issues, next events can be used:
@
@ - to check process traces as per following events via alter system or init.ora:

@

References
NOTE:762374.1 - Step by Step - Configure Direct NFS Client (DNFS) on Linux (11g)