当前位置：移动技术网 > IT编程>开发语言>Java > ceph部署rgw对象存储网关高可用集群

ceph部署rgw对象存储网关高可用集群

2020年07月23日 | 移动技术网IT编程 | 我要评论

部署rgw对象网关节点

要使用 Ceph Object Gateway对象网关组件，必须部署RGW的实例。执行以下操作以创建RGW新实例（已部署则忽略本节操作）：

官方参考：https://docs.ceph.com/docs/master/install/ceph-deploy/install-ceph-gateway/

安装radosgw包，默认已安装radosgw包，也可以手动安装相关包，以ceph-deploy部署的集群为例：

# ceph-deploy install --no-adjust-repos --rgw node1 node2 node3

# rpm -qa |grep radosgw
ceph-radosgw-14.2.9-0.el7.x86_64

创建rgw实例，这里在3个节点启用rgw对象存储网关：

ceph-deploy rgw create node1 node2 node3

默认情况下，RGW实例将侦听7480端口。为了方便访问，在运行RGW节点上编辑ceph.conf来更改此端口，如下所示：

#ceph-deploy节点
cat >> /etc/ceph/ceph.conf <<EOF
[client.rgw.node1]
rgw frontends = civetweb port=81

[client.rgw.node2]
rgw frontends = civetweb port=81

[client.rgw.node3]
rgw frontends = civetweb port=81
EOF

#更新到所有节点
ceph-deploy --overwrite-conf config push node1 node2 node3

#每个节点重启radosw服务
systemctl restart ceph-radosgw.target

验证rgb，浏览器访问http://192.168.93.40，输出以下内容

[root@node1 my-cluster]# curl http://192.168.93.40:81
<?xml version="1.0" encoding="UTF-8"?><ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>anonymous</ID><DisplayName></DisplayName></Owner><Buckets></Buckets></ListAllMyBucketsResult>

默认部署rgw后会自动创建以下4个存储池

[root@node1 my-cluster]# ceph osd lspools
1 .rgw.root
2 default.rgw.control
3 default.rgw.meta
4 default.rgw.log

部署rgw高可用集群

基本架构图
在这里插入图片描述
在3个rgw节点安装haproxy和keepalived

yum install -y haproxy keepalived

3个节点修改keepalived配置文件，以非抢占模式运行，3个节点配置相同。
只需修改interface以及virtual_ipaddress字段，需要提供一个与节点同网段的IP地址，其他默认即可。

cat > /etc/keepalived/keepalived.conf <<EOF
global_defs {
        router_id 10
        vrrp_version 2
        vrrp_garp_master_delay 1
        vrrp_garp_master_refresh 1
        vrrp_mcast_group4 224.0.0.18
    }   

    vrrp_script chk-haproxy {
        script       "killall -0 haproxy"
        timeout 1
        interval 1   # check every 1 second
        fall 2       # require 2 failures for KO
        rise 2       # require 2 successes for OK
    }

    vrrp_instance rgw {
        state BACKUP
        interface ens33
        virtual_router_id 100
        priority 1
        advert_int 1
        nopreempt
        track_script {
            chk-haproxy
        }
        authentication {
            auth_type PASS
            auth_pass haproxy
        }
        virtual_ipaddress {
            192.168.93.50/24 dev ens33
        }
    }
EOF

3个节点修改haproxy配置文件，3个节点配置全部相同。
只需修改server字段，地址为3个节点实际IP地址，其他保持默认即可。

cat > /etc/haproxy/haproxy.cfg << EOF
global
        #chroot  /var/lib/haproxy
        daemon
        #group haproxy
        #user haproxy
        log 127.0.0.1:514 local0 warning
        pidfile /var/lib/haproxy.pid
        maxconn 20000
        spread-checks 3
        nbproc 8

defaults
        log     global
        mode    http
        retries 3
        option redispatch

listen http-web
        bind *:80
        mode http
        balance roundrobin
        timeout server 15s
        timeout connect 15s

        server node1 192.168.93.40:81 check port 81 inter 5000 fall 5
        server node2 192.168.93.41:81 check port 81 inter 5000 fall 5
        server node3 192.168.93.42:81 check port 81 inter 5000 fall 5
EOF

启动keepalived及haproxy服务

systemctl enable --now keepalived haproxy

查看vip状态

[root@node1 ~]# ip a | grep ens33
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    inet 192.168.93.40/24 brd 192.168.93.255 scope global noprefixroute ens33
    inet 192.168.93.50/24 scope global secondary ens33

验证vip是否能够漂移，这里vip转移到node3节点

[root@node1 ~]# systemctl stop haproxy

[root@node3 ~]# ip a | grep  ens33
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    inet 192.168.93.42/24 brd 192.168.93.255 scope global noprefixroute ens33
    inet 192.168.93.50/24 scope global secondary ens33

验证vip转发后端是否正常

[root@node1 ~]# curl http://192.168.93.50:80
<?xml version="1.0" encoding="UTF-8"?><ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>anonymous</ID><DisplayName></DisplayName></Owner><Buckets></Buckets></ListAllMyBucketsResult>[root@node1 ~]#

修改s3cfg配置文件连接参数，改为vip地址：

[root@node1 my-cluster]# cat /root/.s3cfg |grep host_
host_base = 192.168.93.50:80
host_bucket = 192.168.93.50:80/%���(bucket)s

验证s3cmd命令访问对象存储

[root@node1 ~]# s3cmd ls
2020-07-03 06:49  s3://my-new-bucket
2020-07-03 07:03  s3://s3cmd-demo
2020-07-03 07:37  s3://swift-demo

[root@node1 ~]# s3cmd mb s3://test-1
Bucket 's3://test-1/' created

修改swift_openrc.sh文件，改为vip地址

[root@node1 my-cluster]# cat swift_openrc.sh  | grep ST_AUTH
export ST_AUTH=http://192.168.93.50:80/auth

验证swift命令访问对象存储

[root@node1 my-cluster]# source swift_openrc.sh 
[root@node1 my-cluster]# swift list
my-new-bucket
s3cmd-demo
swift-demo
test-1

至此高可用环境部署完成，后续可以通过多种方式对ceph对象存储进行操作。

s3风格接口操作

以python脚本方式操作对象存储。

创建用户

[root@node1 my-cluster]# radosgw-admin user create --uid="ceph-s3-user" --display-name="ceph s3 user demo"  
{
    "user_id": "ceph-s3-user",
    "display_name": "ceph s3 user demo",
    "email": "",
    "suspended": 0,
    "max_buckets": 1000,
    "subusers": [],
    "keys": [
        {
            "user": "ceph-s3-user",
            "access_key": "W4UQON1266AZX7H4R78A",
            "secret_key": "LjaAgGJOTZ0cLhVUHSlOZ45NuJtt2OElYF83el9r"
        }
    ],
    "swift_keys": [],
    "caps": [],
    "op_mask": "read, write, delete",
    "default_placement": "",
    "default_storage_class": "",
    "placement_tags": [],
    "bucket_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "user_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "temp_url_keys": [],
    "type": "rgw",
    "mfa_ids": []
}

查看创建的用户信息

[root@node1 my-cluster]# radosgw-admin user list
[root@node1 my-cluster]# radosgw-admin user info --uid ceph-s3-user

客户端连接

yum install -y python-boto

创建Python脚本:

cat > s3test.py <<EOF
import boto.s3.connection

access_key = 'W4UQON1266AZX7H4R78A'
secret_key = 'LjaAgGJOTZ0cLhVUHSlOZ45NuJtt2OElYF83el9r'
conn = boto.connect_s3(
        aws_access_key_id=access_key,
        aws_secret_access_key=secret_key,
        host='192.168.93.50', port=80,
        is_secure=False, calling_format=boto.s3.connection.OrdinaryCallingFormat(),
       )

bucket = conn.create_bucket('my-new-bucket')
for bucket in conn.get_all_buckets():
    print "{name} {created}".format(
        name=bucket.name,
        created=bucket.creation_date,
    )
EOF

注意替换hostname字段及port。

运行脚本：

[root@node1 my-cluster]# python s3test.py
my-new-bucket 2020-07-03T06:49:45.867Z

此时自动创建一个default.rgw.buckets.index的存储池

[root@node1 my-cluster]# ceph osd lspools
1 .rgw.root
2 default.rgw.control
3 default.rgw.meta
4 default.rgw.log
5 default.rgw.buckets.index

这种方式通常适用于开发人员以sdk方式操作。

s3cmd命令行操作

使用s3cm命令操作对象存储。

参考：https://github.com/s3tools/s3cmd

yum install -y s3cmd

配置

[root@node1 my-cluster]# s3cmd --configure

Enter new values or accept defaults in brackets with Enter.
Refer to user manual for detailed description of all options.

Access key and Secret key are your identifiers for Amazon S3. Leave them empty for using the env variables.
Access Key: W4UQON1266AZX7H4R78A
Secret Key: LjaAgGJOTZ0cLhVUHSlOZ45NuJtt2OElYF83el9r
Default Region [US]: 

Use "s3.amazonaws.com" for S3 Endpoint and not modify it to the target Amazon S3.
S3 Endpoint [s3.amazonaws.com]: 192.168.93.50:80

Use "%(bucket)s.s3.amazonaws.com" to the target Amazon S3. "%(bucket)s" and "%(location)s" vars can be used
if the target S3 system supports dns based buckets.
DNS-style bucket+hostname:port template for accessing a bucket [%(bucket)s.s3.amazonaws.com]: 192.168.93.40:80/%(bucket)s

Encryption password is used to protect your files from reading
by unauthorized persons while in transfer to S3
Encryption password: 
Path to GPG program [/usr/bin/gpg]: 

When using secure HTTPS protocol all communication with Amazon S3
servers is protected from 3rd party eavesdropping. This method is
slower than plain HTTP, and can only be proxied with Python 2.7 or newer
Use HTTPS protocol [Yes]: n

On some networks all internet access must go through a HTTP proxy.
Try setting it here if you can't connect to S3 directly
HTTP Proxy server name: 

New settings:
  Access Key: W4UQON1266AZX7H4R78A
  Secret Key: LjaAgGJOTZ0cLhVUHSlOZ45NuJtt2OElYF83el9r
  Default Region: US
  S3 Endpoint: 192.168.93.50:80
  DNS-style bucket+hostname:port template for accessing a bucket: 192.168.93.50:80/%���(bucket)s
  Encryption password: 
  Path to GPG program: /usr/bin/gpg
  Use HTTPS protocol: False
  HTTP Proxy server name: 
  HTTP Proxy server port: 0

Test access with supplied credentials? [Y/n] y
Please wait, attempting to list all buckets...
Success. Your access key and secret key worked fine :-)

Now verifying that encryption works...
Not configured. Never mind.

Save settings? [y/N] y
Configuration saved to '/root/.s3cfg'

查看bucket

[root@node1 my-cluster]# s3cmd ls
2020-07-03 06:49  s3://my-new-bucket

启用signature_v2

# vi /root/.s3cfg
signature_v2 = True

创建bucket

[root@node1 my-cluster]# s3cmd mb s3://s3cmd-demo
Bucket 's3://s3cmd-demo/' created

查看创建的bucket

[root@node1 my-cluster]# s3cmd ls
2020-07-03 06:49  s3://my-new-bucket
2020-07-03 07:03  s3://s3cmd-demo

上传文件或目录到bucket

s3cmd put /etc/fstab s3://s3cmd-demo/fstab-bak
s3cmd put /var/log/ --recursive s3://s3cmd-demo/log/

查看bucket中的文件

[root@node1 my-cluster]# s3cmd ls  s3://s3cmd-demo/
                          DIR  s3://s3cmd-demo/log/
2020-07-03 07:05          541  s3://s3cmd-demo/fstab-bak

下载文件

[root@node1 my-cluster]# s3cmd get s3://s3cmd-demo/fstab-bak fstab-bak

删除文件或目录

s3cmd rm s3://s3cmd-demo/fstab-bak
s3cmd rm --recursive s3://s3cmd-demo/log/

bucket上传文件后会自动新建一个default.rgw.buckets.data的存储池

[root@node1 my-cluster]# ceph osd lspools
1 .rgw.root
2 default.rgw.control
3 default.rgw.meta
4 default.rgw.log
5 default.rgw.buckets.index
6 default.rgw.buckets.data

查看存储池中的内容

[root@node1 my-cluster]# rados -p default.rgw.buckets.data ls
2f105b56-46fe-4230-a9ae-9bd0b0ac1f1d.4323.2_fstab-bak

文件前缀存储在default.rgw.buckets.index中

[root@node1 my-cluster]# rados -p default.rgw.buckets.index ls
.dir.2f105b56-46fe-4230-a9ae-9bd0b0ac1f1d.4323.2
.dir.2f105b56-46fe-4230-a9ae-9bd0b0ac1f1d.4323.1

swift风格接口操作

使用swift命令操作对象存储。

创建swift用户

[root@node1 my-cluster]# radosgw-admin user list
[
    "ceph-s3-user"
]

[root@node1 my-cluster]# radosgw-admin subuser create --uid=ceph-s3-user --subuser=ceph-s3-user:swift --access=full
{
    "user_id": "ceph-s3-user",
    "display_name": "ceph s3 user demo",
    "email": "",
    "suspended": 0,
    "max_buckets": 1000,
    "subusers": [
        {
            "id": "ceph-s3-user:swift",
            "permissions": "full-control"
        }
    ],
    "keys": [
        {
            "user": "ceph-s3-user",
            "access_key": "W4UQON1266AZX7H4R78A",
            "secret_key": "LjaAgGJOTZ0cLhVUHSlOZ45NuJtt2OElYF83el9r"
        }
    ],
    "swift_keys": [
        {
            "user": "ceph-s3-user:swift",
            "secret_key": "3zM2goJKoiRFUswG6MBoNEwTXwb3EaP4fU3SF4pA"
        }
    ],
    "caps": [],
    "op_mask": "read, write, delete",
    "default_placement": "",
    "default_storage_class": "",
    "placement_tags": [],
    "bucket_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "user_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "temp_url_keys": [],
    "type": "rgw",
    "mfa_ids": []
}

Create the secret key:

[root@node1 my-cluster]# radosgw-admin key create --subuser=ceph-s3-user:swift --key-type=swift --gen-secret
{
    "user_id": "ceph-s3-user",
    "display_name": "ceph s3 user demo",
    "email": "",
    "suspended": 0,
    "max_buckets": 1000,
    "subusers": [
        {
            "id": "ceph-s3-user:swift",
            "permissions": "full-control"
        }
    ],
    "keys": [
        {
            "user": "ceph-s3-user",
            "access_key": "W4UQON1266AZX7H4R78A",
            "secret_key": "LjaAgGJOTZ0cLhVUHSlOZ45NuJtt2OElYF83el9r"
        }
    ],
    "swift_keys": [
        {
            "user": "ceph-s3-user:swift",
            "secret_key": "ZLy2kCT1AJA6T2tKAhl1yjMKtMwYK9VJfZLAavJT"
        }
    ],
    "caps": [],
    "op_mask": "read, write, delete",
    "default_placement": "",
    "default_storage_class": "",
    "placement_tags": [],
    "bucket_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "user_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "temp_url_keys": [],
    "type": "rgw",
    "mfa_ids": []
}

安装swift客户端

sudo yum install python-setuptools
#sudo easy_install pip
yum install -y python-pip

mkdir ~/.pip
cat > ~/.pip/pip.conf << EOF 
[global]
trusted-host=mirrors.aliyun.com
index-url=https://mirrors.aliyun.com/pypi/simple/
EOF

pip install -U pip

sudo pip install --upgrade setuptools
sudo pip install --upgrade python-swiftclient

列出bucket

[root@node1 my-cluster]# swift -V 1 -A http://192.168.93.50:80/auth -U ceph-s3-user:swift -K 'ZLy2kCT1AJA6T2tKAhl1yjMKtMwYK9VJfZLAavJT' list
my-new-bucket
s3cmd-demo

注意替换IPADDRESS及key

定义环境变量

[root@node1 my-cluster]# swift list
Auth version 1.0 requires ST_AUTH, ST_USER, and ST_KEY environment variables
to be set or overridden with -A, -U, or -K.

Auth version 2.0 requires OS_AUTH_URL, OS_USERNAME, OS_PASSWORD, and
OS_TENANT_NAME OS_TENANT_ID to be set or overridden with --os-auth-url,
--os-username, --os-password, --os-tenant-name or os-tenant-id. Note:
adding "-V 2" is necessary for this.
[root@node1 my-cluster]#

定义环境变量

cat > swift_openrc.sh <<EOF
export ST_AUTH=http://192.168.93.50:80/auth
export ST_USER=ceph-s3-user:swift
export ST_KEY=ZLy2kCT1AJA6T2tKAhl1yjMKtMwYK9VJfZLAavJT
EOF

[root@node1 my-cluster]# set | grep ST_
ST_AUTH=http://192.168.93.50:80/auth
ST_KEY=ZLy2kCT1AJA6T2tKAhl1yjMKtMwYK9VJfZLAavJT
ST_USER=ceph-s3-user:swift

验证

[root@node1 my-cluster]# swift list
my-new-bucket
s3cmd-demo

创建bucket

[root@node1 my-cluster]# swift post swift-demo
[root@node1 my-cluster]# swift list
my-new-bucket
s3cmd-demo
swift-demo

上传文件或目录

swift upload swift-demo /etc/passwd
swift upload swift-demo /etc/

查看bucket中的文件

swift list swift-demo

下载bucket中的文件

swift download swift-demo etc/passwd

参考：https://edu.51cto.com/center/course/lesson/index?id=553461

本文地址：https://blog.csdn.net/networken/article/details/107488914

您可能感兴趣的文章:

如对本文有疑问，点击进行留言回复！！

springmvc之ResponseBody响应json数据遇到的错误及解决

使用@RequestBody注解把json的字符串转换成JavaBean的对象代码实践过程中遇到错误：[http... [阅读全文]
nuxt模板的创建及使用（koa、sass）

概念Nuxt.js 是一个基于 Vue.js 的服务端渲染应用框架。Nuxt.js 为客户端/服务端这种典型... [阅读全文]
今天学习了transform的几种变换属性

transform变换transform: translate(x,y);1、2D移动是2D转换里面的一个功能，... [阅读全文]
Docker部署Gitlab，ssh端口不用22的解决方法

虚拟机中的端口：22被占用了，所以Gitlab的端口：22需要重新映射解决办法：web: image: 'gi... [阅读全文]
uni-app 后台升级静默升级 uniapp 后台更新静默更新在线升级

uni-app开发到一段落了，最后的app更新好好搞一下，完美的收尾，哈哈哈在uni-app的文档中可以看到内置... [阅读全文]
VSCode使用码云(Gitee)简易教程

配置Git配置SSH公钥安装完Git以后，打开Git Bash。执行如下命令生成ssh公钥：ssh-keygen... [阅读全文]
JavaWeb 之Servlet 规律强化(三)

一.Servlet规律强化1.WEB项目Servlet服务器Java小程序。Jar包：2.... [阅读全文]
荐写好技术简历的一些建议

根据我求职与招聘的经验，结合网上相关资料，这里整理了一些写好技术简历的建议。这里提供了一份简历模版，可供参考。保... [阅读全文]
SpringBoot多Module启动报错Could not transfer metadata

原因：本地没找到项目依赖的jar去私服找解决办法：install本地项目 [阅读全文]
Hibernate项目报错：Cannot call sendError() after the response has been committed

问题描述：在之前的旧项目上开发一些新功能：其中一个功能是开发系统的操作日志，项目框架是：SSH，由于系统已经上... [阅读全文]