设为首页收藏本站

IDC免费资源网

 找回密码
 立即注册

QQ登录

只需一步,快速开始

搜索
IDC免费资源网 首页 操作系统 查看内容

openstack教程:Openstack 安装

2016-12-31 13:39| 发布者: idcFREE| 查看: 2924| 评论: 0

摘要: 目标 1)了解opentack的基本概念 2)手动安装allinone单机模式的openstack环境 Openstack 简介 openstack 是一整套资源管理软件的集合,也是当前最热的开源虚拟化管理软件之一,有一个全球139个国家将近两万开发者参 ...

目标

1)了解opentack的基本概念

2)手动安装allinone单机模式的openstack环境


Openstack 简介

openstack 是一整套资源管理软件的集合,也是当前最热的开源虚拟化管理软件之一,有一个全球139个国家将近两万开发者参与的开源社区(www.openstack.org)作为支持。openstack项目的目的是快速建设一个稳定可靠的公有云或私有云系统。整个项目涵盖了计算,存储,网络以及前端展现等关于云管理的全部方面,包含了众多子项目。

主要包含了一下几个子项目:

OpenStack Compute (code-name Nova) 计算服务

OpenStack Networking (code-name Neutron) 网络服务

OpenStack Object Storage (code-name Swift) 对象存储服务

OpenStack Block Storage (code-name Cinder) 块设备存储服务

OpenStack Identity (code-name Keystone) 认证服务

OpenStack Image Service (code-name Glance) 镜像文件服务

OpenStack Dashboard (code-name Horizon) 仪表盘服务

OpenStack Telemetry (code-name Ceilometer) 告警服务

OpenStack Orchestration (code-name Heat) 流程服务

OpenStack Database (code-name Trove) 数据库服务


VM硬件配置

硬件环境:

CPU:4 cores

内存:4G

磁盘:40G


软件环境

操作系统 centos7.2x86_64

宿主机: vmware workstation 11

安装OS及OS调整

安装OS

选择最小化安装

磁盘分区使用默认LVM


#!/bin/sh

passwd  root<<EOF

root123

root123

EOF

修改主机名

hostnamectl set-hostname stack

配置hosts文件

[root@stack ~]# cat /etc/hosts

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4 stack

::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

修改网卡地址

先使用dhclient获取到一个IP地址。然后ssh连接到主机进行修改。

查询网卡名称

[root@stack ~]# ip a

1: lo:mtu 65536 qdisc noqueue state UNKNOWN 

    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

    inet 127.0.0.1/8 scope host lo

       valid_lft forever preferred_lft forever

    inet6 ::1/128 scope host 

       valid_lft forever preferred_lft forever

2: eno16777728:mtu 1500 qdisc pfifo_fast state UP qlen 1000

    link/ether 00:0c:29:54:6a:68 brd ff:ff:ff:ff:ff:ff

    inet 192.168.18.169/24 brd 192.168.18.255 scope global dynamic eno16777728

       valid_lft 1610sec preferred_lft 1610sec

    inet6 fe80::20c:29ff:fe54:6a68/64 scope link 

       valid_lft forever preferred_lft forever


修改网卡名称。

/etc/udev/rules.d/70-net.rules 里添加如下内容:

SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:0c:29:54:6a:68", NAME="eth0"

重启主机

重启后,系统IP信息


[root@stack ~]# ip a

1: lo:mtu 65536 qdisc noqueue state UNKNOWN 

    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

    inet 127.0.0.1/8 scope host lo

       valid_lft forever preferred_lft forever

    inet6 ::1/128 scope host 

       valid_lft forever preferred_lft forever

2: eth0:mtu 1500 qdisc pfifo_fast state UP qlen 1000

    link/ether 00:0c:29:54:6a:68 brd ff:ff:ff:ff:ff:ff

    inet 192.168.18.169/24 brd 192.168.18.255 scope global dynamic eth0

       valid_lft 1368sec preferred_lft 1368sec

    inet6 fe80::20c:29ff:fe54:6a68/64 scope link 

       valid_lft forever preferred_lft forever

[root@stack ~]#

安装RPM包

yum install -y mlocate wget  erlang


安装RDO源

找到需要安装的openstack代号

https://wiki.openstack.org/wiki/Releases


这次实验使用发行版liberty,所有rpm包均使用YUM更新。


https://repos.fedorapeople.org/repos/openstack/EOL/

下面是部分发行版支持状态

 



wget https://repos.fedorapeople.org/repos/openstack/openstack-liberty/rdo-release-liberty-3.noarch.rpm

rpm -vih rdo-release-liberty-3.noarch.rpm


查看文件

[root@stack yum.repos.d]# cat rdo-release.repo 

[openstack-liberty]

name=OpenStack Liberty Repository

baseurl=http://mirror.centos.org/centos/7/cloud/$basearch/openstack-liberty/

gpgcheck=1

enabled=1

gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-SIG-Cloud


安装openstack-packstack相关包

yum install -y yum openstack-packstack  openstack-nova-compute puppet hiera                                openssh-clients     tar    nc     rubygem-json    openstack-nova-conductor             openstack-dashboard openstack-swift mongodb-server  openstack-ceilometer-central      mariadb  mariadb-galera-server    python-openstackclient openstack-cinder  openstack-liberty


关闭相应服务

systemctl disable NetworkManager-dispatcher.service

systemctl disable NetworkManager.service 

systemctl disable postfix.service


systemctl stop NetworkManager

systemctl status NetworkManager


systemctl list-unit-files |grep enable



这里必须停止 NetworkManager,不然一直卡起执行

Testing if puppet apply is finished: 192.168.18.167_glance.pp

Testing if puppet apply is finished: 192.168.18.167_glance.pp



关闭SELINUX

/etc/selinux/config里设置

SELINUX=disabled


设置DNS

[root@stack ~]# cat  /etc/resolv.conf 

nameserver 61.139.2.69



packstack安装配置

安装日志

[root@stack ~]# packstack --allinone  --provision-demo=n --nagios-install=n

Welcome to the Packstack setup utility


The installation log file is available at: /var/tmp/packstack/20160511-195846-0yNqkA/openstack-setup.log


Installing:

Clean Up                                             [ DONE ]

Discovering ip protocol version                      [ DONE ]

Setting up ssh keys                                  [ DONE ]

Preparing servers                                    [ DONE ]

Pre installing Puppet and discovering hosts' details [ DONE ]

Adding pre install manifest entries                  [ DONE ]

Setting up CACERT                                    [ DONE ]

Adding AMQP manifest entries                         [ DONE ]

Adding MariaDB manifest entries                      [ DONE ]

Fixing Keystone LDAP config parameters to be undef if empty[ DONE ]

Adding Keystone manifest entries                     [ DONE ]

Adding Glance Keystone manifest entries              [ DONE ]

Adding Glance manifest entries                       [ DONE ]

Adding Cinder Keystone manifest entries              [ DONE ]

Checking if the Cinder server has a cinder-volumes vg[ DONE ]

Adding Cinder manifest entries                       [ DONE ]

Adding Nova API manifest entries                     [ DONE ]

Adding Nova Keystone manifest entries                [ DONE ]

Adding Nova Cert manifest entries                    [ DONE ]

Adding Nova Conductor manifest entries               [ DONE ]

Creating ssh keys for Nova migration                 [ DONE ]

Gathering ssh host keys for Nova migration           [ DONE ]

Adding Nova Compute manifest entries                 [ DONE ]

Adding Nova Scheduler manifest entries               [ DONE ]

Adding Nova VNC Proxy manifest entries               [ DONE ]

Adding OpenStack Network-related Nova manifest entries[ DONE ]

Adding Nova Common manifest entries                  [ DONE ]

Adding Neutron VPNaaS Agent manifest entries         [ DONE ]

Adding Neutron FWaaS Agent manifest entries          [ DONE ]

Adding Neutron LBaaS Agent manifest entries          [ DONE ]

Adding Neutron API manifest entries                  [ DONE ]

Adding Neutron Keystone manifest entries             [ DONE ]

Adding Neutron L3 manifest entries                   [ DONE ]

Adding Neutron L2 Agent manifest entries             [ DONE ]

Adding Neutron DHCP Agent manifest entries           [ DONE ]

Adding Neutron Metering Agent manifest entries       [ DONE ]

Adding Neutron Metadata Agent manifest entries       [ DONE ]

Adding Neutron SR-IOV Switch Agent manifest entries  [ DONE ]

Checking if NetworkManager is enabled and running    [ DONE ]

Adding OpenStack Client manifest entries             [ DONE ]

Adding Horizon manifest entries                      [ DONE ]

Adding Swift Keystone manifest entries               [ DONE ]

Adding Swift builder manifest entries                [ DONE ]

Adding Swift proxy manifest entries                  [ DONE ]

Adding Swift storage manifest entries                [ DONE ]

Adding Swift common manifest entries                 [ DONE ]

Adding MongoDB manifest entries                      [ DONE ]

Adding Redis manifest entries                        [ DONE ]

Adding Ceilometer manifest entries                   [ DONE ]

Adding Ceilometer Keystone manifest entries          [ DONE ]

Adding post install manifest entries                 [ DONE ]

Copying Puppet modules and manifests                 [ DONE ]

Applying 192.168.18.169_prescript.pp

192.168.18.169_prescript.pp:                         [ DONE ]          

Applying 192.168.18.169_amqp.pp

Applying 192.168.18.169_mariadb.pp

192.168.18.169_amqp.pp:                              [ DONE ]        

192.168.18.169_mariadb.pp:                           [ DONE ]        

Applying 192.168.18.169_keystone.pp

Applying 192.168.18.169_glance.pp

Applying 192.168.18.169_cinder.pp

192.168.18.169_keystone.pp:                          [ DONE ]         

192.168.18.169_glance.pp:                            [ DONE ]         

192.168.18.169_cinder.pp:                            [ DONE ]         

Applying 192.168.18.169_api_nova.pp

192.168.18.169_api_nova.pp:                          [ DONE ]         

Applying 192.168.18.169_nova.pp

192.168.18.169_nova.pp:                              [ DONE ]     

Applying 192.168.18.169_neutron.pp

192.168.18.169_neutron.pp:                           [ DONE ]        

Applying 192.168.18.169_osclient.pp

Applying 192.168.18.169_horizon.pp

192.168.18.169_osclient.pp:                          [ DONE ]         

192.168.18.169_horizon.pp:                           [ DONE ]         

Applying 192.168.18.169_ring_swift.pp

192.168.18.169_ring_swift.pp:                        [ DONE ]           

Applying 192.168.18.169_swift.pp

192.168.18.169_swift.pp:                             [ DONE ]      

Applying 192.168.18.169_mongodb.pp

Applying 192.168.18.169_redis.pp

192.168.18.169_mongodb.pp:                           [ DONE ]        

192.168.18.169_redis.pp:                             [ DONE ]        

Applying 192.168.18.169_ceilometer.pp

192.168.18.169_ceilometer.pp:                        [ DONE ]           

Applying 192.168.18.169_postscript.pp

192.168.18.169_postscript.pp:                        [ DONE ]           

Applying Puppet manifests                            [ DONE ]

Finalizing                                           [ DONE ]


 **** Installation completed successfully ******


Additional information:

 * A new answerfile was created in: /root/packstack-answers-20160511-195847.txt

 * Time synchronization installation was skipped. Please note that unsynchronized time on server instances might be problem for some OpenStack components.

 * File /root/keystonerc_admin has been created on OpenStack client host 192.168.18.169. To use the command line tools you need to source the file.

 * To access the OpenStack Dashboard browse to http://192.168.18.169/dashboard .

Please, find your login credentials stored in the keystonerc_admin in your home directory.

 * The installation log file is available at: /var/tmp/packstack/20160511-195846-0yNqkA/openstack-setup.log

 * The generated manifests are available at: /var/tmp/packstack/20160511-195846-0yNqkA/manifests

[root@stack ~]# 

[root@stack ~]# 

[root@stack ~]#


Web登录网页地址是

http://192.168.18.169/dashboard

登录用户名是admin,密码在/root/keystonerc_admin 文件里。如下64338db216a24554就是密码。

[root@stack ~]# cat keystonerc_admin

unset OS_SERVICE_TOKEN

export OS_USERNAME=admin

export OS_PASSWORD=64338db216a24554

export OS_AUTH_URL=http://192.168.18.169:5000/v2.0

export PS1='[\u@\h \W(keystone_admin)]\$ '


export OS_TENANT_NAME=admin

export OS_REGION_NAME=RegionOne

web配置


创建镜像

网上有做好的镜像,下载后需要导入。

Cirros镜像是由 Openstack的开发,基本都使用这个image来测试,因为他比较小,只有10M。下载地址如下:

https://launchpad.net/cirros/trunk/0.3.0/+download/cirros-0.3.0-x86_64-disk.img

Cirros,是可以使用用户名和密码登陆,也可以使用密钥登陆


user:cirros

password:cubswin:)


这个密码比较特别,所以你可以使用上面的方法来设置这个镜像的root的密码,实现ssh登陆。

这里仅测试以密码登录。



创建卷

下面根据之前上传的镜像,新建一个卷。卷名称 VOL01

创建网络

创建实例

实例中openstack里可以认为就是VM的意思。下面使用刚创建的卷来创建一个实例。

点击实例名称,可以到控制台查看实例终端。


命令行

重启所有服务

 

[root@stack#

systemctl restart openstack-nova-api.service 

systemctl restart openstack-nova-cert.service 

systemctl restart openstack-nova-consoleauth.service 

systemctl restart openstack-nova-scheduler.service 

systemctl restart openstack-nova-conductor.service 

systemctl restart openstack-nova-novncproxy.service  

systemctl restart openstack-nova-compute.service   

systemctl restart neutron-server.service 

systemctl restart neutron-openvswitch-agent.service 

systemctl restart neutron-l3-agent.service

systemctl restart neutron-dhcp-agent.service 

systemctl restart neutron-metadata-agent.service

遇到问题记录

如下部分是安装遇到的一些问题,因YUM源在国外,可以在packstack –allinone之前先安装如下一些需要的包。

问题1: Testing if puppet apply is finished: 192.168.18.167_glance.pp一直运行,跑不完

Testing if puppet apply is finished: 192.168.18.167_glance.pp

Testing if puppet apply is finished: 192.168.18.167_glance.pp

 一直运行,跑不完

 

* A new answerfile was created in: /root/packstack-answers-20160508-121521.txt

 * Time synchronization installation was skipped. Please note that unsynchronized time on server instances might be problem for some OpenStack components.

 * Warning: NetworkManager is active on 192.168.18.167. OpenStack networking currently does not work on systems that have the Network Manager service enabled.

 * File /root/keystonerc_admin has been created on OpenStack client host 192.168.18.167. To use the command line tools you need to source the file.

 * To access the OpenStack Dashboard browse to http://192.168.18.167/dashboard .

Please, find your login credentials stored in the keystonerc_admin in your home directory.

 

 

解决方法:

禁用和停止NetworkManager服务

systemctl disable NetworkManager.service

systemctl stop NetworkManager

systemctl status NetworkManage

 

 

问题2/usr/bin/yum -d 0 -e 0 -y install openstack-nova-compute

192.168.18.167_nova.pp:                           [ ERROR ]      

Applying Puppet manifests                         [ ERROR ]

 

ERROR : Error appeared during Puppet run: 192.168.18.167_nova.pp

Error: Execution of '/usr/bin/yum -d 0 -e 0 -y install openstack-nova-compute' returned 1: Error downloading packages:

You will find full trace in log /var/tmp/packstack/20160508-124037-To1axn/manifests/192.168.18.167_nova.pp.log

Please check log file /var/tmp/packstack/20160508-124037-To1axn/openstack-setup.log for more information

Additional information:

 * A new answerfile was created in: /root/packstack-answers-20160508-124038.txt

 * Time synchronization installation was skipped. Please note that unsynchronized time on server instances might be problem for some OpenStack components.

 * File /root/keystonerc_admin has been created on OpenStack client host 192.168.18.167. To use the command line tools you need to source the file.

 * To access the OpenStack Dashboard browse to http://192.168.18.167/dashboard .

Please, find your login credentials stored in the keystonerc_admin in your home directory.

解决办法:/usr/bin/yum -d 0 -e 0 -y install openstack-nova-compute

问题三:yum install -y puppet hiera openssh-clients tar nc rubygem-json

 

Preparing servers                                    [ DONE ]

Pre installing Puppet and discovering hosts' details[ ERROR ]

 

ERROR : Failed to run remote script, stdout: Loaded plugins: fastestmirror

 

stderr: Warning: Permanently added '192.168.18.167' (ECDSA) to the list of known hosts.

+ trap t ERR

+ yum install -y puppet hiera openssh-clients tar nc rubygem-json

http://mirror.centos.org/centos/7/cloud/x86_64/openstack-liberty/repodata/repomd.xml: [Errno 14] curl#6 - "Could not resolve host: mirror.centos.org; Temporary failure in name resolution"

Trying other mirror.

解决办法:安装缺失的rpm

问题4 Timed out waiting for nova-conductor

May  8 13:29:49 localhost nova-compute: 2016-05-08 13:29:49.104 59005 WARNING nova.conductor.api [req-4adc7a55-3d19-40c1-876f-2c1e825a84a3 - - - - -] Timed out waiting for nova-conductor.  Is it running? Or did this service start before nova-conductor?  Reattempting establishment of nova-conductor connection...

 

解决:手动安装

yum install openstack-nova-conductor

systemctl restart openstack-nova-compute.service  

systemctl restart openstack-nova-conductor.service

 

 

问题5 /usr/bin/yum -d 0 -e 0 -y install openstack-dashboard

Applying 192.168.18.167_horizon.pp

192.168.18.167_osclient.pp:                          [ DONE ]        

192.168.18.167_horizon.pp:                        [ ERROR ]          

Applying Puppet manifests                         [ ERROR ]

 

ERROR : Error appeared during Puppet run: 192.168.18.167_horizon.pp

Error: Execution of '/usr/bin/yum -d 0 -e 0 -y install openstack-dashboard' returned 1: Error downloading packages:

You will find full trace in log /var/tmp/packstack/20160508-134040-omwtp7/manifests/192.168.18.167_horizon.pp.log

Please check log file /var/tmp/packstack/20160508-134040-omwtp7/openstack-setup.log for more information

Additional information:

 * A new answerfile was created in: /root/packstack-answers-20160508-134041.txt

 * Time synchronization installation was skipped. Please note that unsynchronized time on server instances might be problem for some OpenStack components.

 * File /root/keystonerc_admin has been created on OpenStack client host 192.168.18.167. To use the command line tools you need to source the file.

 * To access the OpenStack Dashboard browse to http://192.168.18.167/dashboard .

Please, find your login credentials stored in the keystonerc_admin in your home directory.

[root@stack ~]#

 

解决:

yum -y install openstack-dashboard

 

问题6Error: Could not update: Execution of '/usr/bin/yum -d 0 -e 0 -y install openstack-swift' returned 1: Error downloading packages:

 

 

192.168.18.167_ring_swift.pp:                     [ ERROR ]            

Applying Puppet manifests                         [ ERROR ]

 

ERROR : Error appeared during Puppet run: 192.168.18.167_ring_swift.pp

Error: Could not update: Execution of '/usr/bin/yum -d 0 -e 0 -y install openstack-swift' returned 1: Error downloading packages:

You will find full trace in log /var/tmp/packstack/20160508-143136-73Se6A/manifests/192.168.18.167_ring_swift.pp.log

Please check log file /var/tmp/packstack/20160508-143136-73Se6A/openstack-setup.log for more information

Additional information:

 * A new answerfile was created in: /root/packstack-answers-20160508-143137.txt

 * Time synchronization installation was skipped. Please note that unsynchronized time on server instances might be problem for some OpenStack components.

 * File /root/keystonerc_admin has been created on OpenStack client host 192.168.18.167. To use the command line tools you need to source the file.

 * To access the OpenStack Dashboard browse to http://192.168.18.167/dashboard .

Please, find your login credentials stored in the keystonerc_admin in your home directory.

[root@stack ~]#

[root@stack ~]#

 

解决:

yum -y install openstack-swift

 

问题7'/usr/bin/yum -d 0 -e 0 -y install mongodb-server'

192.168.18.167_mongodb.pp:                        [ ERROR ]         

Applying Puppet manifests                         [ ERROR ]

 

ERROR : Error appeared during Puppet run: 192.168.18.167_mongodb.pp

Error: Execution of '/usr/bin/yum -d 0 -e 0 -y install mongodb-server' returned 1: Error downloading packages:

You will find full trace in log /var/tmp/packstack/20160508-143930-TAwbuB/manifests/192.168.18.167_mongodb.pp.log

Please check log file /var/tmp/packstack/20160508-143930-TAwbuB/openstack-setup.log for more information

Additional information:

 * A new answerfile was created in: /root/packstack-answers-20160508-143930.txt

 * Time synchronization installation was skipped. Please note that unsynchronized time on server instances might be problem for some OpenStack components.

 * File /root/keystonerc_admin has been created on OpenStack client host 192.168.18.167. To use the command line tools you need to source the file.

 * To access the OpenStack Dashboard browse to http://192.168.18.167/dashboard .

Please, find your login credentials stored in the keystonerc_admin in your home directory.

[root@stack ~]#

解决办法:安装缺失的rpm

 

问题8: /usr/bin/yum -d 0 -e 0 -y install openstack-ceilometer-central'

 

ERROR : Error appeared during Puppet run: 192.168.18.167_ceilometer.pp

Error: Execution of '/usr/bin/yum -d 0 -e 0 -y install openstack-ceilometer-central' returned 1: Error downloading packages:

You will find full trace in log /var/tmp/packstack/20160508-160844-UQyxep/manifests/192.168.18.167_ceilometer.pp.log

Please check log file /var/tmp/packstack/20160508-160844-UQyxep/openstack-setup.log for more information

Additional information:

 * A new answerfile was created in: /root/packstack-answers-20160508-160844.txt

 * Time synchronization installation was skipped. Please note that unsynchronized time on server instances might be problem for some OpenStack components.

 * File /root/keystonerc_admin has been created on OpenStack client host 192.168.18.167. To use the command line tools you need to source the file.

 * To access the OpenStack Dashboard browse to http://192.168.18.167/dashboard .

Please, find your login credentials stored in the keystonerc_admin in your home directory.

[root@stack ~]#

 

解决办法:安装缺失的rpm

 

问题9错误:无法获取云硬盘限制信息

 

创建实例的时候出现这个信息,但是实例还是创建成功了。

Openstack网页管理

修改VM密码-无效

支持修改密码

默认OpenStackDashboard修改虚拟机的密码的功能是关闭的,你需要enable。我是使用RDO安装。

 

编辑文件

/usr/share/openstack-dashboard/openstack_dashboard/local/local_settings.py

 

# The OPENSTACK_HYPERVISOR_FEATURES settings can be used to enable optional

# services provided by hypervisors.

OPENSTACK_HYPERVISOR_FEATURES = {

    'can_set_mount_point': False,

    'can_set_password': True,

}

重启httpd服务

 

systemctl restart httpd.service

这时候你就可以看到设置管理员密码的地方


过这个时候,你可能还是不能修改密码。你需要修改计算节点的nova.conf文件

[libvirt]

 

# Inject the admin password at boot time, without an agent.

# (boolean value)

#inject_password=false

inject_password=True

这样重启服务,应该就可以。

 

openstack-service restart nova

 

修改镜像文件密码

yum install -y libguestfs-tools-c-1.28.1-1.55.el7.centos.x86_64

 

[root@stack ~(keystone_admin)]# nova network-list

+--------------------------------------+--------+------+

| ID                      | Label  | Cidr |

+--------------------------------------+--------+------+

| 3574f97e-11de-449c-926a-dd9fd1a8bd66 | public | -    |

+--------------------------------------+--------+------+

[root@stack ~(keystone_admin)]# nova keypair-list

+-------+-------------------------------------------------+

| Name  | Fingerprint                  |

+-------+-------------------------------------------------+

| key01 | 13:57:9b:76:af:c3:f6:18:dd:23:56:48:94:62:a1:92 |

+-------+-------------------------------------------------+

[root@stack ~(keystone_admin)]#


登录VM cirros

参考:http://www.tuicool.com/articles/QNZRJz

参考资料

https://en.wikipedia.org/wiki/OpenStack

http://wiki.openstack.org/Documentation


 

学习总结

1) 开源软件包之间的依赖性较强

2) 此次测试使用的系统是centos7.2,此系统目前比较新。


鲜花

握手

雷人

路过

鸡蛋

最新评论

小黑屋|Archiver|手机版|IDC免费资源网    

GMT+8, 2018-1-20 03:09 , Processed in 0.026001 second(s), 9 queries , Gzip On, Memcache On.

Powered by Discuz! X3.2

© 2001-2013 Comsenz Inc.

返回顶部