搭建EXADATA虚拟机CELL节点

郑全

首先需要在虚拟机中安装Oracle Linux 5.7, （Red Hat Enterprise Linux理论也可以，但我没有测试过），内存分配1GB通常就足够了。安装过程很简单，需要注意的是需要选上软件开发包，例如gcc/aio之类的，图形界面（GUI）可不装。推荐使用静态IP地址，我的网络配置如下:

要构建一套Exadata虚拟环境，至少需要两台虚拟机，一台用于Cell节点，一台用于DB节点

• 内存(Memory)至少4G以上，推荐配置8G；
• 磁盘（Harddisk）空余至少在40G以上，当然如果有SSD更好
• 虚拟机：vbox,vmware都可。我这里选择VMWARE
• Oracle Linux 5.7安装介质
• Exadata 11.2.3.2 Cell的安装介质，Exadata 11.2.3.2的Cell介质名为V33693-01.zip解压后文件名为cellImageMaker_11.2.3.2.0_LINUX.X64_120713-1.x86_64.tar
• Oracle Clusterware 11.2.0.3以及Oracle database 11.2.0.3的Linux x86_64的安装介质
• 
  

就可以正式开始我们的Exadata之旅了
首先需要在虚拟机中安装Oracle Linux 5.7，选择 软件开发 组
其他都是默认即可。
推荐使用静态IP地址，我的网络配置如下：
DEVICE=eth0
BOOTPROTO=none
HWADDR=00:0C:29:7B:3F:71
ONBOOT=yes
NETMASK=255.255.255.0
IPADDR=192.168.133.110
GATEWAY=192.168.133.2
TYPE=Ethernet
USERCTL=no
IPV6INIT=no
PEERDNS=yes

[root@cell ~]# cat /etc/hosts
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1                cell localhost.localdomain localhost
::1                localhost6.localdomain6 localhost6

注意： 安装完成以后Oracle Linux默认使用UEK，如果这里使用UEK， 则在后面的步骤中无法正常启动cellsrv服务。可以修改grub的配置将其默认启动内核修改为redhat兼容内核：

[root@cell ~]# vi /etc/grub.conf

将default=0修改为default=1 ，然后重启。

因为默认Oracle Linux启动了很多我们不需要的服务，为了节省资源，建议将以下服务停止并且禁用。

chkconfig --level 2345 auditd off && service auditd stop
chkconfig --level 2345 autofs off && service autofs stop
chkconfig --level 2345 avahi-daemon off && service avahi-daemon stop
chkconfig --level 2345 bluetooth off && service bluetooth stop
chkconfig --level 2345 cups off && service cups stop
chkconfig --level 2345 ip6tables off && service ip6tables stop
chkconfig --level 2345 iptables off && service iptables stop
chkconfig --level 2345 isdn off && service isdn stop
chkconfig --level 2345 kudzu off && service kudzu stop
chkconfig --level 2345 mcstrans off && service auditd stop
chkconfig --level 2345 netfs off && service netfs stop
chkconfig --level 2345 pcscd off && service pcscd stop
chkconfig --level 2345 restorecond off && service restorecond stop
chkconfig --level 2345 rhnsd off && service rhnsd stop
chkconfig --level 2345 sendmail off && service sendmail stop
chkconfig --level 2345 setroubleshoot off && service settroubleshoot stop
chkconfig --level 2345 smartd off && service smartd stop
chkconfig --level 2345 xinetd off && service xinetd stop
chkconfig --level 2345 yum-updatesd off && service yum-updatesd stop

然后我们将Exadata Cell Image V33693-01.zip上传到虚拟机内，解压，得到cellImageMaker_11.2.3.2.0_LINUX.X64_120713-1.x86_64.tar，继续解压得到一个名为dl180的文件夹。

[root@cell ~]# unzip V33693-01.zip
Archive:  V33693-01.zip  
inflating: README.txt               
inflating: cellImageMaker_11.2.3.2.0_LINUX.X64_120713-1.x86_64.tar  

[root@cell ~]# tar -pxvf cellImageMaker_11.2.3.2.0_LINUX.X64_120713-1.x86_64.tar
dl180......

在dl180/boot/cellbits下找到cell.bin文件。这个bin文件实际上是一个zip压缩包， 我们使用unzip来对它进行解压：

[root@cell ~]# unzip cell.bin
Archive:  cell.binwarning [cell.bin]:  6408 extra bytes at beginning or within zipfile  (attempting to process anyway)  
inflating: cell-11.2.3.2.1_LINUX.X64_130109-1.x86_64.rpm   
inflating: jdk-1_5_0_15-linux-amd64.rpm

解压后得到cell-11.2.3.2.1_LINUX.X64_130109-1.x86_64.rpm和jdk-1_5_0_15-linux-amd64.rpm两个rpm包

我们先来安装jdk：

[root@cell ~]# rpm -ivh jdk-1_5_0_15-linux-amd64.rpm

然后再安装cell：

[root@cell ~]# rpm -ivh cell-11.2.3.2.1_LINUX.X64_130109-1.x86_64.rpm

会报错：

加一个 -v 可以看到是需要增加一个目录 /var/log/oracle

[root@cell ~]# mkdir -p /var/log/oracle
[root@cell ~]# chmod -R 775 /var/log/oracle

再次安装cell这次没有报错。

接下来的步骤应该是在cell虚拟机中建立对应的虚拟的磁盘和闪盘:

[root@cell ~]# mkdir -p /opt/oracle/cell/disks/raw
[root@cell ~]cd /opt/oracle/cell/disks/raw
[root@cell ~]vi dd.sh
[root@cell ~]cat dd.sh
dd if=/dev/zero of=disk01 bs=1M count=1000
dd if=/dev/zero of=disk02 bs=1M count=1000
dd if=/dev/zero of=disk03 bs=1M count=1000
dd if=/dev/zero of=disk04 bs=1M count=1000
dd if=/dev/zero of=disk05 bs=1M count=1000
dd if=/dev/zero of=disk06 bs=1M count=1000
dd if=/dev/zero of=disk07 bs=1M count=1000
dd if=/dev/zero of=disk08 bs=1M count=1000
dd if=/dev/zero of=disk09 bs=1M count=1000
dd if=/dev/zero of=disk10 bs=1M count=1000
dd if=/dev/zero of=disk11 bs=1M count=1000
dd if=/dev/zero of=disk12 bs=1M count=1000
dd if=/dev/zero of=FLASH01 bs=1M count=1000
dd if=/dev/zero of=FLASH02 bs=1M count=1000
dd if=/dev/zero of=FLASH03 bs=1M count=1000
dd if=/dev/zero of=FLASH04 bs=1M count=1000

执行dd.sh创建对应的磁盘和闪盘：其中磁盘12块，每块大小为1GB，闪盘4块，每块大小也是1GB。

[root@cell raw]# chmod 660 *
[root@cell raw]# ls -ltr
total 16400068
-rw-rw---- 1 root root        692 May 16 16:24 dd.sh
-rw-rw---- 1 root root 1048576000 May 16 16:24 disk01
-rw-rw---- 1 root root 1048576000 May 16 16:24 disk02
-rw-rw---- 1 root root 1048576000 May 16 16:24 disk03
-rw-rw---- 1 root root 1048576000 May 16 16:24 disk04
-rw-rw---- 1 root root 1048576000 May 16 16:25 disk05
-rw-rw---- 1 root root 1048576000 May 16 16:25 disk06
-rw-rw---- 1 root root 1048576000 May 16 16:25 disk07
-rw-rw---- 1 root root 1048576000 May 16 16:26 disk08
-rw-rw---- 1 root root 1048576000 May 16 16:26 disk09
-rw-rw---- 1 root root 1048576000 May 16 16:27 disk10
-rw-rw---- 1 root root 1048576000 May 16 16:27 disk11
-rw-rw---- 1 root root 1048576000 May 16 16:27 disk12
-rw-rw---- 1 root root 1048576000 May 16 16:27 FLASH01
-rw-rw---- 1 root root 1048576000 May 16 16:27 FLASH02
-rw-rw---- 1 root root 1048576000 May 16 16:27 FLASH03
-rw-rw---- 1 root root 1048576000 May 16 16:28 FLASH04

然后删除dd脚本，切换到celladmin用户，重新启动celld服务。

[root@cell ~]# su - celladmin
[celladmin@cell ~]$ cellcli -e alter cell restart services all

发现cellsrv服务无法启动，查看/opt/oracle/cell11.2.3.2.1_LINUX.X64_130109/log/diag/asm/cell/cell/trace/alert.log发现有类似如下的报错信息：

CELLSRV version=11.2.3.2.1,label=OSS_11.2.3.2.1_LINUX.X64_130109,Wed_Jan__9_06:09:48_PST_2013Non critical error DIA-48913 caught while writing to trace file "/opt/oracle/cell11.2.3.2.1_LINUX.X64_130109/log/diag/asm/cell/cell/trace/svtrc_2244_0.trc"Error message: DIA-48913: Writing into trace file failed, file size limit [0] reached

从错误号就可以判断应该是最大文件数不足， 于是需要再修改操作系统的最大文件数限制：

在/etc/sysctl.conf 最后添加一行： fs.file-max = 65536，然后刷新生效：

[root@cell network-scripts]# sysctl -p
net.ipv4.ip_forward = 0
net.ipv4.conf.default.rp_filter = 2
net.ipv4.conf.default.accept_source_route = 0
kernel.sysrq = 0
kernel.core_uses_pid = 1
net.ipv4.tcp_syncookies = 1
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.shmmax = 68719476736
kernel.shmall = 4294967296
fs.file-max = 65536

在/etc/security/limit.conf文件最后添加两行：

* soft nofile 65536
* hard nofile 65536

然后退出重新登录， 切换到 celladmin，使用ulimit -a进行查看是否生效：

[root@cell ~]# ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 11999
max locked memory       (kbytes, -l) 32
max memory size         (kbytes, -m) unlimited
open files                      (-n) 65536
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 11999
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

再次启动cell所有的服务：

[celladmin@cell ~]$ cellcli -e alter cell restart services all

这次发现cell下的cellsrv, ms, rs服务都可以正常启动了。

接下来需要在cellinit.ora中添加网卡的信息：

[celladmin@cell ~]$ cellcli -e create cell cell1 interconnect1=eth0

执行成功以后，可以看到cellinit.ora文件中添加了一行ipaddress1=192.168.133.110/24类似的信息。

[root@cell config]# cat /opt/oracle/cell/cellsrv/deploy/config/cellinit.ora#CELL Initialization Parameters
version=0.0
HTTP_PORT=8888
bbuChargeThreshold=800
SSL_PORT=23943
RMI_PORT=23791
ipaddress1=192.168.133.110/24
bbuTempThreshold=60
DEPLOYED=TRUE
JMS_PORT=9127
BMC_SNMP_PORT=162

接下来创建celldisk, griddisk, flashcache, flashlog:

[celladmin@cell ~]$ cellcli

CellCLI: Release 11.2.3.2.0 - Production on Tue Oct 20 18:43:34 EDT 2020
Copyright (c) 2007, 2012, Oracle.  All rights reserved.
Cell Efficiency Ratio: 1
CellCLI> alter cell restart services all
Stopping the RS, CELLSRV, and MS services...
The SHUTDOWN of services was successful.
Starting the RS, CELLSRV, and MS services...
Getting the state of RS services...  
runningStarting CELLSRV services...
The STARTUP of CELLSRV services was successful.
Starting MS services...The STARTUP of MS services was successful.

创建celldisk

CellCLI> create celldisk all            
CellDisk FD_00_cell1 successfully created
CellDisk FD_01_cell1 successfully created
CellDisk FD_02_cell1 successfully created
CellDisk FD_03_cell1 successfully created
CellDisk CD_disk01_cell1 successfully created
CellDisk CD_disk02_cell1 successfully created
CellDisk CD_disk03_cell1 successfully created
CellDisk CD_disk04_cell1 successfully created
CellDisk CD_disk05_cell1 successfully created
CellDisk CD_disk06_cell1 successfully created
CellDisk CD_disk07_cell1 successfully created
CellDisk CD_disk08_cell1 successfully created
CellDisk CD_disk09_cell1 successfully created
CellDisk CD_disk10_cell1 successfully created
CellDisk CD_disk11_cell1 successfully created
CellDisk CD_disk12_cell1 successfully created

创建flashlog和flashcache

CellCLI> create flashcache all  size=2G
Flash cache cell1_FLASHCACHE successfully created

CellCLI> create flashlog all
Flash log cell1_FLASHLOG successfully created


CellCLI> list flashcache detail

         name:                   cell1_FLASHCACHE
         cellDisk:               FD_00_cell1,FD_01_cell1,FD_03_cell1,FD_02_cell1
         creationTime:           2020-10-20T18:15:18-04:00
         degradedCelldisks:      
         effectiveCacheSize:     2G
         id:                     17be2224-c157-4062-bb88-4c1b5d70d242
         size:                   2G
         status:                 normalCellCLI>  list flashlog detail
         name:                   cell1_FLASHLOG
         cellDisk:               FD_01_cell1,FD_03_cell1,FD_00_cell1,FD_02_cell1
         creationTime:           2020-10-20T18:15:31-04:00
         degradedCelldisks:      
         effectiveSize:          512M
         efficiency:             100.0
         id:                     7893c40d-41c8-4341-9965-98a4c99becf0
         size:                   512M
         status:                 normal
CellCLI> list cell         cell1         online

CellCLI> list celldisk

         CD_disk01_cell1         normal
         CD_disk02_cell1         normal
         CD_disk03_cell1         normal
         CD_disk04_cell1         normal
         CD_disk05_cell1         normal
         CD_disk06_cell1         normal
         CD_disk07_cell1         normal
         CD_disk08_cell1         normal
         CD_disk09_cell1         normal
         CD_disk10_cell1         normal
         CD_disk11_cell1         normal
         CD_disk12_cell1         normal
         FD_00_cell1             normal
         FD_01_cell1             normal
         FD_02_cell1             normal
         FD_03_cell1             normal

创建griddisk

CellCLI> create griddisk all harddisk prefix=DATA, size=700M
GridDisk DATA_CD_disk01_cell1 successfully created
GridDisk DATA_CD_disk02_cell1 successfully created
GridDisk DATA_CD_disk03_cell1 successfully created
GridDisk DATA_CD_disk04_cell1 successfully created
GridDisk DATA_CD_disk05_cell1 successfully created
GridDisk DATA_CD_disk06_cell1 successfully created
GridDisk DATA_CD_disk07_cell1 successfully created
GridDisk DATA_CD_disk08_cell1 successfully created
GridDisk DATA_CD_disk09_cell1 successfully created
GridDisk DATA_CD_disk10_cell1 successfully created
GridDisk DATA_CD_disk11_cell1 successfully created
GridDisk DATA_CD_disk12_cell1 successfully created

CellCLI>  create griddisk all harddisk prefix=RECO
GridDisk RECO_CD_disk01_cell1 successfully created
GridDisk RECO_CD_disk02_cell1 successfully created
GridDisk RECO_CD_disk03_cell1 successfully created
GridDisk RECO_CD_disk04_cell1 successfully created
GridDisk RECO_CD_disk05_cell1 successfully created
GridDisk RECO_CD_disk06_cell1 successfully created
GridDisk RECO_CD_disk07_cell1 successfully created
GridDisk RECO_CD_disk08_cell1 successfully created
GridDisk RECO_CD_disk09_cell1 successfully created
GridDisk RECO_CD_disk10_cell1 successfully created
GridDisk RECO_CD_disk11_cell1 successfully created
GridDisk RECO_CD_disk12_cell1 successfully created

CellCLI> list griddisk
         DATA_CD_disk01_cell1    active
         DATA_CD_disk02_cell1    active
         DATA_CD_disk03_cell1    active
         DATA_CD_disk04_cell1    active
         DATA_CD_disk05_cell1    active
         DATA_CD_disk06_cell1    active
         DATA_CD_disk07_cell1    active
         DATA_CD_disk08_cell1    active
         DATA_CD_disk09_cell1    active
         DATA_CD_disk10_cell1    active
         DATA_CD_disk11_cell1    active
         DATA_CD_disk12_cell1    active
         RECO_CD_disk01_cell1    active
         RECO_CD_disk02_cell1    active
         RECO_CD_disk03_cell1    active
         RECO_CD_disk04_cell1    active
         RECO_CD_disk05_cell1    active
         RECO_CD_disk06_cell1    active
         RECO_CD_disk07_cell1    active
         RECO_CD_disk08_cell1    active
         RECO_CD_disk09_cell1    active
         RECO_CD_disk10_cell1    active
         RECO_CD_disk11_cell1    active
         RECO_CD_disk12_cell1    active



CellCLI> list griddisk RECO_CD_disk12_cell1 detail
         name:                   RECO_CD_disk12_cell1
         availableTo:            
         cachingPolicy:          default
         cellDisk:               CD_disk12_cell1
         comment:               
         creationTime:           2020-10-20T18:22:30-04:00
         diskType:               HardDisk
         errorCount:             0
         id:                     4009d963-0d5f-4a14-86e0-25a7c82dba45
         offset:                 736M
         size:                   256M
         status:                 active
CellCLI> list griddisk DATA_CD_disk01_cell1
         DATA_CD_disk01_cell1    active
CellCLI> list griddisk DATA_CD_disk01_cell1 detail
         name:                   DATA_CD_disk01_cell1
         availableTo:            
         cachingPolicy:          default
         cellDisk:               CD_disk01_cell1
         comment:               
         creationTime:           2020-10-20T18:21:53-04:00
         diskType:               HardDisk
         errorCount:             0
         id:                     f5159bd8-a267-475a-acc7-21758d6fda47
         offset:                 48M
         size:                   688M
         status:                 active


至此cell节点虚拟机基本创建完毕。