lvs keepalived實(shí)現(xiàn)高可用群集配置詳解Mon 16 April 2012lvs是一個開源的軟件，由畢業(yè)于國防科技大學(xué)的章文嵩博士于1998年5月創(chuàng)立(中國人的項(xiàng)目)，可以實(shí)現(xiàn)LINUX平臺下的簡單負(fù)載均衡。LVS是Linux Virtual Server的縮寫，意思是Linux虛擬服務(wù)器。本文將介紹lvs結(jié)合keepalived實(shí)現(xiàn)一個高科用的Linux群集系統(tǒng). lvs有三種工作模式NAT(地址轉(zhuǎn)換),IP Tunneling(IP隧道)、Direct Routing(直接路由)。 工作效率最低的是NAT模式,但NAT模式可以用于各種系統(tǒng),各種環(huán)境的負(fù)載均衡,只需要一個公網(wǎng)ip即可實(shí)現(xiàn) IP Tunneling模式調(diào)度器將連接分發(fā)到不同的后端real server,然后由real server處理請求直接相應(yīng)給用戶,大大提高了調(diào)度器的調(diào)度效率,后端real server沒有物理位置和邏輯關(guān)系的限制,后端real server可以在Lan/Wlan,但是后端real server必須支持IP隧道協(xié)議. DR(Direct Routing)是效率最高的,與IP Tunneling類似,都是處理一般連接,將請求給后端real server,然后由real server處理請求直接相應(yīng)給用戶,Direct Routing與IP Tunneling相比，沒有IP封裝的開銷，但由于采用物理層,所以DR模式的調(diào)度器和后端real server必須在一個物理網(wǎng)段里,中間不能過路由器(也就是一個交換機(jī)相連). lvs支持8種不同的調(diào)度算法輪叫(rr)、加權(quán)輪叫(wrr)、最小連接(lc)、加權(quán)最小連接(wlc)、基于局部性最小連接(lblc)、帶復(fù)制的基于局部性最少鏈接(lblcr)、目標(biāo)地址散列(dh)和源地址散列(sh). 下面就介紹如何來安裝和配置lvs keepalived 本文使用環(huán)境: 操作系統(tǒng):CentOS 5.5 32bit 主調(diào)度器:192.168.3.101/24 備調(diào)度器:192.168.3.102/24 后端real server: 192.168.3.3/24 |192.168.3.102/24(我們這里使用備用lvs作為一個測試 vip(virtual ip):192.168.3.100/24 lvs在2.6的內(nèi)核中是默認(rèn)支持的,所以我們就不需要在來安裝,但是我們需要安裝用戶配置工具ipvsadm yum -y install ipvsadm # 分別在主從lvs上執(zhí)行安裝ipvsadm
我們查看lvs是否支持: lsmod | grep ip_vs #
ip_vs 78081 1
modprobe -l| grep ip_vs
/lib/modules/2.6.18-194.el5/kernel/net/ipv4/ipvs/ip_vs.ko
/lib/modules/2.6.18-194.el5/kernel/net/ipv4/ipvs/ip_vs_dh.ko
/lib/modules/2.6.18-194.el5/kernel/net/ipv4/ipvs/ip_vs_ftp.ko
/lib/modules/2.6.18-194.el5/kernel/net/ipv4/ipvs/ip_vs_lblc.ko
/lib/modules/2.6.18-194.el5/kernel/net/ipv4/ipvs/ip_vs_lblcr.ko
/lib/modules/2.6.18-194.el5/kernel/net/ipv4/ipvs/ip_vs_lc.ko
/lib/modules/2.6.18-194.el5/kernel/net/ipv4/ipvs/ip_vs_nq.ko
/lib/modules/2.6.18-194.el5/kernel/net/ipv4/ipvs/ip_vs_rr.ko
/lib/modules/2.6.18-194.el5/kernel/net/ipv4/ipvs/ip_vs_sed.ko
/lib/modules/2.6.18-194.el5/kernel/net/ipv4/ipvs/ip_vs_sh.ko
/lib/modules/2.6.18-194.el5/kernel/net/ipv4/ipvs/ip_vs_wlc.ko
/lib/modules/2.6.18-194.el5/kernel/net/ipv4/ipvs/ip_vs_wrr.ko
本文介紹lvs的 在主備服務(wù)器上部署keepalived(因?yàn)榍懊嬉呀?jīng)rpm包安裝了ipvsadm,所以就不需要重復(fù)安裝): vi /etc/sysctl.conf
net.ipv4.ip_forward = 1 # 此參數(shù)改為1
sysctl -p # 使修改生效
安裝依賴: yum -y install openssl-devel
# 下載并安裝keepalived
wget http://www./software/keepalived-1.1.19.tar.gz
tar -zxvf keepalived-1.1.19.tar.gz
cd keepalived-1.1.19
./configure --prefix=/ \ # 安裝在默認(rèn)位置(配置文件,二進(jìn)制文件,啟動腳本放到默認(rèn)位置)
--mandir=/usr/local/share/man/ \
--with-kernel-dir=/usr/src/kernels/2.6.18-194.el5-i686/ # 需要內(nèi)核的頭文件
make && make install
在主備lvs上安裝keepalived完畢后我們先來配置主lvs上的keepalived:
編輯配置文件 ! Configuration File for keepalived
global_defs {
notification_email {
coldnight@ # 發(fā)生故障時發(fā)送的郵箱
}
notification_email_from linuxzen@ # 使用哪個郵箱發(fā)送
smtp_server # 發(fā)件服務(wù)器
smtp_connect_timeout 30
router_id LVS_DEVEL
}
vrrp_instance VI_1 {
state MASTER # 標(biāo)示為主lvs
interface eth0 # HA檢測端口
virtual_router_id 51 # 主備的virtual_router_id 必須相同
priority 100 # 優(yōu)先級,備lvs要比主lvs稍小
advert_int 1 # VRRP Multicast 廣播周期秒數(shù)
authentication { # 定義認(rèn)證
auth_type PASS # 認(rèn)證方式為口令認(rèn)證
auth_pass 1111 # 定義口令
}
virtual_ipaddress { # 定義vip
192.168.3.100 # 多個vip可換行添加
}
}
virtual_server 192.168.3.100 80 {
delay_loop 6 # 每隔6秒查看realserver狀態(tài)
lb_algo wlc # 調(diào)度算法為加權(quán)最小連接數(shù)
lb_kind DR # lvs工作模式為DR(直接路由)模式
nat_mask 255.255.255.0
persistence_timeout 50 # 同一IP 的連接50秒內(nèi)被分配到同一臺realserver(測試時建議改為0)
protocol TCP # 用TCP監(jiān)測realserver的狀態(tài)
real_server 192.168.3.3 80 { # 定義realserver
weight 3 # 定義權(quán)重
TCP_CHECK { # 注意TCP_CHECK和{之間的空格,如果沒有的話只會添加第一個realserver
connect_timeout 3 # 三秒無響應(yīng)超時
nb_get_retry 3
delay_before_retry 3
connect_port 80
}
}
real_server 192.168.3.102 80 {
weight 3
TCP_CHECK {
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
connect_port 80
}
}
}
配置備用lvs的keepalived,只需要將state MASTER 改為state BACKUP,降低priority 100 的值: ! Configuration File for keepalived
global_defs {
notification_email {
coldnight@ # 發(fā)生故障時發(fā)送的郵箱
}
notification_email_from linuxzen@ # 使用哪個郵箱發(fā)送
smtp_server # 發(fā)件服務(wù)器
smtp_connect_timeout 30
router_id LVS_DEVEL
}
vrrp_instance VI_1 {
state BACKUP # 標(biāo)示為備lvs
interface eth0 # HA檢測端口
virtual_router_id 51 # 主備的virtual_router_id 必須相同
priority 99 # 優(yōu)先級,備lvs要比主lvs稍小
advert_int 1 # VRRP Multicast 廣播周期秒數(shù)
authentication { # 定義認(rèn)證
auth_type PASS # 認(rèn)證方式為口令認(rèn)證
auth_pass 1111 # 定義口令
}
virtual_ipaddress { # 定義vip
192.168.3.100 # 多個vip可換行添加
}
}
virtual_server 192.168.3.100 80 {
delay_loop 6 # 每隔6秒查看realserver狀態(tài)
lb_algo wlc # 調(diào)度算法為加權(quán)最小連接數(shù)
lb_kind DR # lvs工作模式為DR(直接路由)模式
nat_mask 255.255.255.0
persistence_timeout 50 # 同一IP 的連接50秒內(nèi)被分配到同一臺realserver
protocol TCP # 用TCP監(jiān)測realserver的狀態(tài)
real_server 192.168.3.3 80 { # 定義realserver
weight 3 # 定義權(quán)重
TCP_CHECK { # 注意TCP_CHECK和{之間的空格,如果沒有的話只會添加第一個realserver
connect_timeout 3 # 三秒無響應(yīng)超時
nb_get_retry 3
delay_before_retry 3
connect_port 80
}
}
real_server 192.168.3.102 80 {
weight 3
TCP_CHECK {
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
connect_port 80
}
}
}
由于使用keepalived就不需要使用腳本來配置lvs調(diào)度器,但是這里我們還是會給出一個腳本內(nèi)容,但我們不會用到這個腳本:lvs已經(jīng)內(nèi)置于內(nèi)核,配置命令是ipvsadm,所以lvs的一些操作是通過ipvsadm來控制.下面我們就編寫腳本來實(shí)現(xiàn)lvs的DR模式: 編寫腳本lvsdr: 我們把lvs #!/bin/sh
# 定義虛擬ip
VIP=192.168.3.100
# 定義realserver,并已逗號分開
RIPS=192.168.3.3,192.168.3.102 #,192.168.3.5,192.168.3.6
# 定義提供服務(wù)的端口
SERVICE=80
# 調(diào)用init.d腳本的標(biāo)準(zhǔn)庫
. /etc/rc.d/init.d/functions
case $1 in
start)
echo 'Start LVS of DR Mode'
# lvs dr模式不需要路由轉(zhuǎn)發(fā),但是keepalived需要
#echo '0' > /proc/sys/net/ipv4/ip_forward
# 開啟icmp包重定向
echo '1' > /proc/sys/net/ipv4/conf/all/send_redirects
echo '1' > /proc/sys/net/ipv4/conf/default/send_redirects
echo '1' > /proc/sys/net/ipv4/conf/eth0/send_redirects
# 綁定虛擬ip
ifconfig eth0:0 $VIP broadcast $VIP netmask 255.255.255.255 up
route add -host $VIP dev eth0:0
# 清除lvs規(guī)則
ipvsadm -C
# 添加一條虛擬服務(wù)器記錄
# -p指定一定的時間內(nèi)將相同的客戶端分配到同一臺后端服務(wù)器
# 用于解決session的問題,測試時或有別的解決方案時建議去掉
ipvsadm -A -t $VIP:$SERVICE -s wlc -p
# 添加真實(shí)服務(wù)器記錄
for RIP in `echo $RIPS |sed -e 's/,/\n/g'`
do
ipvsadm -a -t $VIP:$SERVICE -r $RIP:$SERVICE -g -w 1
done
# 設(shè)置tcp tcpfin udp的超時連接值
ipvsadm --set 30 120 300
ipvsadm
;;
stop)
echo 'Stop LVS DR'
ifconfig eth0:0 down
ipvsadm -C
;;
*)
echo 'Usage:$0 {start | stop}'
exit 1
esac
編輯完畢保存退出,然后給這個腳本執(zhí)行權(quán)限: chmod x /etc/init.d/lvsdr
然后就可以通過service命令來啟動lvs dr模式 service lvsdr start
將這個腳本分別放到主備lvs的/etc/init.d/下,賦予執(zhí)行權(quán)限. 我們真正需要的是realserver的腳本,下面我們來編寫realserver腳本,同樣放在/etc/init.d/下,編輯rs腳本: vi /etc/init.d/lvsrs
#!/bin/sh
VIP=192.168.3.100
. /etc/rc.d/init.d/functions
case $1 in
start)
echo 'lo:0 port starting'
# 為了相應(yīng)lvs調(diào)度器轉(zhuǎn)發(fā)過來的包,需在本地lo接口上綁定vip
ifconfig lo:0 $VIP broadcast $VIP netmask 255.255.255.255 up
# 限制arp請求
echo '1' > /proc/sys/net/ipv4/conf/lo/arp_ignore
echo '2' > /proc/sys/net/ipv4/conf/lo/arp_announce
echo '1' > /proc/sys/net/ipv4/conf/all/arp_ignore
echo '2' > /proc/sys/net/ipv4/conf/all/arp_announce
;;
stop)
echo 'lo:0 port closing'
ifconfig lo:0 down
echo '0' > /proc/sys/net/ipv4/conf/lo/arp_ignore
echo '0' > /proc/sys/net/ipv4/conf/lo/arp_announce
echo '0' > /proc/sys/net/ipv4/conf/all/arp_ignore
echo '0' > /proc/sys/net/ipv4/conf/all/arp_announce
;;
*)
echo 'Usage: $0 {start | stop}'
exit 1
esac
給腳本賦予執(zhí)行權(quán)限 chmod x /etc/init.d/lvsrs
并將這個腳本放到所有的realserver的/etc/init.d/下.下面開始測試: 先來確認(rèn)下我們做的變動:主從lvs分別安裝keepalived,并且在/etc/init.d/下添加了lvsdr腳本(不使用). 后端realserver分別在/etc/init.d/下添加了lvsrs腳本.我們先測試keepalived: 首先在主調(diào)度器上啟動keepalived: service keepalived start
查看日志文件: tail -50 /var/log/message
Mar 21 22:29:10 master kernel: device eth0 left promiscuous mode
Mar 21 22:29:10 master kernel: type=1700 audit(1332340150.598:12): dev=eth0 prom=0 old_prom=256 auid=4294967295 ses=4294967295
Apr 16 13:31:32 master Keepalived: Starting Keepalived v1.1.19 (04/16,2012)
Apr 16 13:31:32 master Keepalived_healthcheckers: Netlink reflector reports IP 192.168.3.101 added
Apr 16 13:31:32 master Keepalived_healthcheckers: Registering Kernel netlink reflector
Apr 16 13:31:32 master Keepalived_healthcheckers: Registering Kernel netlink command channel
Apr 16 13:31:32 master Keepalived_healthcheckers: Opening file '/etc/keepalived/keepalived.conf'.
Apr 16 13:31:32 master Keepalived_healthcheckers: Configuration is using : 8897 Bytes
Apr 16 13:31:32 master Keepalived_healthcheckers: Using LinkWatch kernel netlink reflector...
Apr 16 13:31:32 master Keepalived: Starting Healthcheck child process, pid=5369
Apr 16 13:31:32 master Keepalived: Starting VRRP child process, pid=5370
Apr 16 13:31:32 master Keepalived_vrrp: Netlink reflector reports IP 192.168.3.101 added
Apr 16 13:31:32 master Keepalived_vrrp: Registering Kernel netlink reflector
Apr 16 13:31:32 master Keepalived_vrrp: Registering Kernel netlink command channel
Apr 16 13:31:32 master Keepalived_vrrp: Registering gratutious ARP shared channel
Apr 16 13:31:32 master Keepalived_vrrp: Opening file '/etc/keepalived/keepalived.conf'.
Apr 16 13:31:32 master Keepalived_vrrp: Configuration is using : 36512 Bytes
Apr 16 13:31:32 master Keepalived_vrrp: Using LinkWatch kernel netlink reflector...
Apr 16 13:31:32 master Keepalived_vrrp: VRRP sockpool: [ifindex(2), proto(112), fd(10,11)]
Apr 16 13:31:33 master Keepalived_vrrp: VRRP_Instance(VI_1) Transition to MASTER STATE
Apr 16 13:31:34 master Keepalived_vrrp: VRRP_Instance(VI_1) Entering MASTER STATE
Apr 16 13:31:34 master Keepalived_vrrp: VRRP_Instance(VI_1) setting protocol VIPs.
Apr 16 13:31:34 master Keepalived_healthcheckers: Netlink reflector reports IP 192.168.3.100 added
Apr 16 13:31:34 master Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.3.100
Apr 16 13:31:34 master Keepalived_vrrp: Netlink reflector reports IP 192.168.3.100 added
Apr 16 13:31:39 master Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.3.100
然后在備用調(diào)度器上啟動keepalived然后查看日志: Apr 16 13:33:35 slave Keepalived_vrrp: VRRP_Instance(VI_1) Entering BACKUP STATE
Apr 16 13:33:35 slave Keepalived_vrrp: VRRP sockpool: [ifindex(2), proto(112), fd(11,12)]
Apr 16 13:33:35 slave Keepalived_healthcheckers: Netlink reflector reports IP 192.168.3.102 added
Apr 16 13:33:35 slave Keepalived_healthcheckers: Registering Kernel netlink reflector
Apr 16 13:33:35 slave Keepalived_healthcheckers: Registering Kernel netlink command channel
Apr 16 13:33:35 slave Keepalived_healthcheckers: Opening file '/etc/keepalived/keepalived.conf'.
Apr 16 13:33:35 slave Keepalived_healthcheckers: Configuration is using : 8895 Bytes
Apr 16 13:33:35 slave kernel: IPVS: [wlc] scheduler registered.
Apr 16 13:33:35 slave Keepalived_healthcheckers: Using LinkWatch kernel netlink reflector...
在主調(diào)度器上執(zhí)行 service keepalived stop
查看備用調(diào)度器日志: tail -20 /var/log/message
Apr 16 13:39:44 slave Keepalived_vrrp: VRRP_Instance(VI_1) Transition to MASTER STATE
Apr 16 13:39:45 slave Keepalived_vrrp: VRRP_Instance(VI_1) Entering MASTER STATE
Apr 16 13:39:45 slave Keepalived_vrrp: VRRP_Instance(VI_1) setting protocol VIPs.
Apr 16 13:39:45 slave Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.3.100
Apr 16 13:39:45 slave Keepalived_vrrp: Netlink reflector reports IP 192.168.3.100 added
Apr 16 13:39:45 slave Keepalived_healthcheckers: Netlink reflector reports IP 192.168.3.100 added
我們看到keepalived已經(jīng)成功切換. 然后我們使用ipvsadm命令查看(在此之前要確認(rèn)后端realserver已經(jīng)啟動了web服務(wù)): ipvsadm
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.3.100:http wlc
-> 192.168.3.3:http Route 3 0 0
-> 192.168.3.102:http Route 3 0 0
然后分別啟動后端realserver的lvsrs服務(wù): servie lvsrs start
然后瀏覽器訪問192.168.3.100,如果keepalived的persistence_timeout參數(shù)值為0,而且兩個后端realserver是不同的內(nèi)容,刷新就可以看到兩個不同的頁面交替. Category: 負(fù)載均衡 Tagged: 高可用 詳解 群集 lvs Linux keepalived Comments |
|
|
