緣起最近一直在研究日志收集系統(tǒng)的框架，之前在線游戲的數(shù)據(jù)發(fā)送都是由游戲服務(wù)器發(fā)送的，我來之前一直是rsync傳輸，也還算穩(wěn)定。但現(xiàn)在上了單機(jī)游戲，只能由手機(jī)客戶端直接發(fā)送，dau比較高，最近很火的<<貪吃蛇>>在海外上線，一個星期dau已經(jīng)達(dá)到千萬級別。初步方案，lvs做４層負(fù)載均衡，下掛nginx做7層轉(zhuǎn)發(fā)，數(shù)據(jù)直接入kafaka。一直不太喜歡通過reload修改nginx upstream，剛好看到微博的upsync覺得很有價值，于是拿來研究了一下。 感興趣的童鞋可以看下這篇文章http://mp.weixin.qq.com/s?__biz=MzAwMDU1MTE1OQ==&mid=404151075&idx=1&sn=5f3b8c007981a2d048766f808f8c8c98&scene=2&srcid=0223XScbJrOv7noogVX6T60Q&from=timeline&isappinstalled=0#wechat_redirect（Upsync：微博開源基于Nginx容器動態(tài)流量管理方案 ） consulupsync的upstream可以從consul(https://www./intro/getting-started/install.html)或etcd中獲取，本文使用的是consul這個用golang寫的服務(wù)注冊與發(fā)現(xiàn)框架，據(jù)我所知consul在國內(nèi)也用的非常廣，比如微博，今日頭條等。所以需要先安裝consul，consul的getting-started文檔用vagrant（https://www./docs/getting-started/）搭建集群的，vagrant是用ruby開發(fā)的用來構(gòu)建虛擬開發(fā)環(huán)境的工具，有點(diǎn)類似docker，不過vagrant是依賴與虛擬機(jī)的，比如virtualbox 下面是我ubuntu主機(jī)安裝操作： sudo apt-get install virtualbox
wget https://releases./vagrant/1.8.6/vagrant_1.8.6_x86_64.deb
sudo dpkg -i vagrant_1.8.6_x86_64.deb
2.下載vagrant鏡像 vagrant box add ubuntu ~/vagrant_start/ubuntu_trusty_64.box
把ubuntu_trusty_64.box加載到vagrant中，并命名為ubuntu 3.創(chuàng)建Vagrantfile # -*- mode: ruby -*-
# vi: set ft=ruby :
$script = <<SCRIPT
echo Installing dependencies...
sudo apt-get update
sudo apt-get install -y unzip curl
echo Fetching Consul...
cd /tmp/
curl https://releases.hashicorp.com/consul/0.6.4/consul_0.6.4_linux_amd64.zip -o consul.zip
echo Installing Consul...
unzip consul.zip
sudo chmod x consul
sudo mv consul /usr/bin/consul
sudo mkdir /etc/consul.d
sudo chmod a w /etc/consul.d
SCRIPT
# Vagrantfile API/syntax version. Don't touch unless you know what you're doing!
VAGRANTFILE_API_VERSION = '2'
Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
config.vm.box = 'ubuntu'
config.vm.provision 'shell', inline: $script
config.vm.define 'n1' do |n1|
n1.vm.hostname = 'n1'
n1.vm.network 'private_network', ip: '172.20.20.10'
end
config.vm.define 'n2' do |n2|
n2.vm.hostname = 'n2'
n2.vm.network 'private_network', ip: '172.20.20.11'
end
end
4.啟動虛擬機(jī)并登陸到n1 vagrant up
vagrant ssh n1
vagrent up將會讀取當(dāng)前目錄的Vagrantfile，創(chuàng)建兩個虛擬機(jī)n1,n2,并安裝consul到/usr/bin/consul目錄。 5.啟動consul consul agent -server -bootstrap-expect=1 -data-dir=/tmp/consul -node=agent-one -bind=172.20.20.10 -config-dir=/etc/consul.d -client 0.0.0.0 -ui
n1將會作為leader節(jié)點(diǎn)，并開啟ui界面，可通過http://172.20.20.10:8500訪問consul管理界面 curl -X PUT http://172.20.20.10:8500/v1/kv/upstreams/test/172.20.20.11:8080
將會把upsteam 172.20.20.11:8080存到consul的kv store中。 nginx-upsync-module nginx_upstream_check_moduleupsync(https://github.com/weibocom/nginx-upsync-module)是微博開源的，以及在微博廣泛使用了，下面將安裝上面這兩個nginx擴(kuò)展 1.下載對應(yīng)軟件包 需要注意版本，具體看upsync Readme，由于我使用時upsync master支持nginx1.9 ,所以我使用了nginx1.9.2 vagrant ssh n2
wget 'http:///download/nginx-1.9.2.tar.gz'
git clone https://github.com/weibocom/nginx-upsync-module
git clone https://github.com/xiaokai-wang/nginx_upstream_check_module
nginx_upstream_check_module需要根據(jù)nginx不同版本給nginx打patch，具體看Readme，upstream_check是tengine（http://tengine./documentation_cn.html）中的模塊，主要作用用于upstream的健康檢查。當(dāng)遇到不可用的upstream時，nginx雖然會自動使用下一個upstream，但下一次還是會重試該upstream，導(dǎo)致無效請求增多。upstream_check檢測到不可用會剔除該upsteam，該特性還是很有用的。 ２．編譯安裝nginx sudo apt-get install libreadline-dev libncurses5-dev libpcre3-dev libssl-dev perl make build-essential
wget 'http:///download/nginx-1.9.2.tar.gz'
git clone https://github.com/weibocom/nginx-upsync-module
git clone https://github.com/xiaokai-wang/nginx_upstream_check_module
patch -p0 < /vagrant/nginx_upstream_check_module/check_1.9.2 .patch
./configure --add-module=/vagrant/nginx_upstream_check_module --add-module=/vagrant/nginx-upsync-module
make && make install
3．配置nginx cd /usr/local/nginx
mkdir /usr/local/nginx/conf/conf.d
mv /usr/local/nginx/conf/nginx.conf /usr/local/nginx/conf/nginx.conf_back
useradd -M nginx
mkdir /var/log/nginx
mkdir /usr/local/nginx/conf/servers
將之前的nginx.conf備份，用下面內(nèi)容替換 user nginx;
worker_processes 5;
error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;
events {
worker_connections 1024;
}
http {
include /usr/local/nginx/conf/mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] '$request' '
'$status $body_bytes_sent '$http_referer' '
''$http_user_agent' '$http_x_forwarded_for'';
access_log /var/log/nginx/access.log main;
sendfile on;
#tcp_nopush on;
keepalive_timeout 65;
#gzip on;
include /usr/local/nginx/conf/conf.d/*.conf;
}
創(chuàng)建check.conf upstream test {
# fake server otherwise ngx_http_upstream will report error when startup
server 127.0.0.1:11111;
# all backend server will pull from consul when startup and will delete fake server
upsync 172.20.20.10:8500/v1/kv/upstreams/test upsync_timeout=6m upsync_interval=500ms upsync_type=consul strong_dependency=off;
upsync_dump_path /usr/local/nginx/conf/servers/servers_test.conf;
check interval=1000 rise=2 fall=2 timeout=3000 type=http default_down=false;
check_http_send 'HEAD / HTTP/1.0\r\n\r\n';
check_http_expect_alive http_2xx http_3xx;
}
upstream bar {
server 172.20.20.11:8080 weight=1 fail_timeout=10 max_fails=3;
}
server {
listen 80;
location = / {
proxy_pass http://test;
}
location = /upstream_show {
upstream_show;
}
location = /upstream_status {
check_status;
access_log off;
}
}
4.啟動nginx 我們在上面介紹consul的最好增加一個172.20.20.11:8080 upstream, 在n2上我們開一個python的http server /usr/local/nginx/sbin/nginx
python -m SimpleHTTPServer 8080
大功告成，看下成果 請將下面的172.20.20.13地址換成你們的172.20.20.11，我開了很多個虛擬機(jī)^^^^ 首頁就是用python開的simplehttpserver upstream列表 upstream狀態(tài) |
|
|
