女主宣言

在對象存儲迭代研發(fā)、測試過程中，為了方便、準(zhǔn)確驗證 S3 接口協(xié)議兼容性，本文作者對Ceph官方采用的兼容性測試工具s3-tests進行了調(diào)研，并對其配置以及使用做出了詳細介紹，相信對于s3的使用者，會起到實質(zhì)性的幫助，下來就跟隨作者一探究竟吧。

PS：豐富的一線技術(shù)、多元化的表現(xiàn)形式，盡在“360云計算”，點關(guān)注哦！

1

簡介

在對象存儲迭代研發(fā)、測試過程中，為了方便、準(zhǔn)確驗證 S3 接口協(xié)議兼容性，我們內(nèi)部配置了 Ceph 官方采用的兼容性測試工具 s3-tests ，該工具基于 Python Nose 測試框架和官方的 Boto 庫(同時支持 boto 2 / boto 3)，并通過結(jié)合 HTML 擴展可將測試結(jié)果輸出到Web界面，通過頁面查詢相關(guān)測試統(tǒng)計、分析結(jié)果。

s3-tests

S3 compatibility tests - This is a set of unofficial Amazon AWS S3 compatibility tests, that can be useful to people implementing software that exposes an S3-like API. The tests use the Boto2 and Boto3 libraries.

倉庫地址：https://github.com/ceph/s3-tests

Nose 框架

Nose 是一種流行的 Python 單元測試框架擴展，它可以方便地自動運行一批測試和插件，比如度量代碼覆蓋率。Nose 能支持基于函數(shù)的測試，也能支持與 unittest 相類似的基于測試類的測試方式。

nose-html-reporting 擴展

Nose plugin that generates a nice html test report with ability of using template based on jinja2 templates from any folder.

源自: https://github.com/ionelmc/nose-htmloutput?，增加了對 jinja2 模板支持。

2

配置 s3-tests 實踐

下載源碼

# git clone https://github.com/ceph/s3-tests.git

安裝、部署

# sudo apt-get install python-virtualenv# cd s3-tests

啟動測試運行環(huán)境，安裝依賴軟件：

# ./bootstrap

創(chuàng)建配置文件

首先，需要通過 S3 管理接口提前創(chuàng)建好配置文件中的用戶、access_key，secret_key 等必要配置信息。

創(chuàng)建 config.yaml 增加如下內(nèi)容：

[DEFAULT]## this section is just used as default for all the "s3 *"## sections, you can place these variables also directly there## replace with e.g. "localhost" to run against local softwarehost = test-shyc2.s3.360.local # 也可用本機 S3 Gateway IP 地址## uncomment the port to use something other than 80port = 80## say "no" to disable TLSis_secure = no[fixtures]## all the buckets created will start with this prefix;## {random} will be filled with random characters to pad## the prefix to 30 characters long, and avoid collisionsbucket prefix = s3test-{random}-[s3 main]## the tests assume two accounts are defined, "main" and "alt".## user_id is a 64-character hexstringuser_id = infra-s3## display name typically looks more like a unix login, "jdoe" etcdisplay_name = infra-s3## replace these with your access keysaccess_key = infra-aksecret_key = infra-sk[s3 alt]## another user account, used for ACL-related testsuser_id = infra-altdisplay_name = infra-alt## the "alt" user needs to have email set, tooemail = infra-alt@qihoo.netaccess_key = infra-alt-aksecret_key = infra-alt-sk

查看測試項

~/s3-tests# S3TEST_CONF=./config.yaml ./virtualenv/bin/nosetests -v --collect-onlys3tests.functional.test_headers.test_object_create_bad_md5_invalid_short ... oks3tests.functional.test_headers.test_object_create_bad_md5_bad ... oks3tests.functional.test_headers.test_object_create_bad_md5_empty ... oks3tests.functional.test_headers.test_object_create_bad_md5_unreadable ... oks3tests.functional.test_headers.test_object_create_bad_md5_none ... oks3tests.functional.test_headers.test_object_create_bad_expect_mismatch ... oks3tests.functional.test_headers.test_object_create_bad_expect_empty ... oks3tests.functional.test_headers.test_object_create_bad_expect_none ... oks3tests.functional.test_headers.test_object_create_bad_expect_unreadable ... oks3tests.functional.test_headers.test_object_create_bad_contentlength_empty ... oks3tests.functional.test_headers.test_object_create_bad_contentlength_negative ... oks3tests.functional.test_headers.test_object_create_bad_contentlength_none ... oks3tests.functional.test_headers.test_object_create_bad_contentlength_unreadable ... oks3tests.functional.test_headers.test_object_create_bad_contentlength_mismatch_above ... oks3tests.functional.test_headers.test_object_create_bad_contenttype_invalid ... oks3tests.functional.test_headers.test_object_create_bad_contenttype_empty ... oks3tests.functional.test_headers.test_object_create_bad_contenttype_none ... oks3tests.functional.test_headers.test_object_create_bad_contenttype_unreadable ... oks3tests.functional.test_headers.test_object_create_bad_authorization_unreadable ... oks3tests.functional.test_headers.test_object_create_bad_authorization_empty ... ok

3

Nose Html 擴展

配置靜態(tài)資源服務(wù)器

搭建一個 HTTP 資源服務(wù)器用于展示 nose 輸出。

在線安裝 nginx / httpd 即可，下文中直接將輸出文件保存到默認(rèn)路徑?/var/www/html/?下。

安裝 nose-html-reporting

官方地址 :?https://pypi.org/project/nose-html-reporting/#description。

安裝 nose-html-reporting。

# ./virtualenv/bin/pip install nose-html-reporting

驗證，有如下輸出，代表擴展已安裝，支持輸出 html 文件。

~/s3-tests# ./virtualenv/bin/nosetests -h | grep html --with-html Enable plugin HtmlReport: Output test results as pretty html. [NOSE_WITH_HTML] --html-report=FILE Path to html file to store the report in. Default is nosetests.html in the working directory --html-report-template=FILE Path to html template file in with jinja2 format.Default is report.html in the lib --cover-html Produce HTML coverage information --cover-html-dir=DIR Produce HTML coverage information in dir~/s3-tests#

?執(zhí)行單個用例

單獨運行 test_bucket_list_empty 用例，指定 html 模板文件為 report2.jinja2。

執(zhí)行如下命令：

~/s3-tests# S3TEST_CONF=./config.yaml ./virtualenv/bin/nosetests s3tests.functional.test_s3:test_bucket_list_empty -v --with-html --html-report=/var/www/html/index.html --html-report-template=virtualenv/lib/python2.7/site-packages/nose_html_reporting/templates/report2.jinja2。

示例：

~/s3-tests# S3TEST_CONF=./config.yaml ./virtualenv/bin/nosetests s3tests.functional.test_s3:test_bucket_list_empty -v --with-html --html-report=/var/www/html/index.html --html-report-template=virtualenv/lib/python2.7/site-packages/nose_html_reporting/templates/report2.jinja2s3tests.functional.test_s3.test_bucket_list_empty ... ok----------------------------------------------------------------------HTML: /var/www/html/index.html----------------------------------------------------------------------Ran 1 test in 0.322sOK~/s3-tests#

打開 web 端，查看測試結(jié)果：

部分測試

運行?atomic read/write?單元測試用例:

#!/bin/bashcases="\ s3tests.functional.test_s3:test_atomic_read_1mb \ s3tests.functional.test_s3:test_atomic_read_4mb \ s3tests.functional.test_s3:test_atomic_read_8mb \ s3tests.functional.test_s3:test_atomic_write_1mb \ s3tests.functional.test_s3:test_atomic_write_4mb \ s3tests.functional.test_s3:test_atomic_write_8mb \ s3tests.functional.test_s3:test_atomic_dual_write_1mb \ s3tests.functional.test_s3:test_atomic_dual_write_4mb \ s3tests.functional.test_s3:test_atomic_dual_write_8mb \ s3tests.functional.test_s3:test_atomic_conditional_write_1mb"S3_USE_SIGV4=1 S3TEST_CONF=config.yaml virtualenv/bin/nosetests $cases -a \auth_aws4 -x --debug-log=./tests.log --cover-html --process-timeout=60 \--processes=10

如上通過指定特定的 case 進行測試，配置 ENV 中 S3_USE_SIGV4=1 和參數(shù)?-a auth_aws4 使能 AWS V4 鑒權(quán)(默認(rèn)為 V2), --debug-log 用于記錄測試過程中詳細日志。

執(zhí)行全量測試

執(zhí)行如下命令：

S3_USE_SIGV4=1 S3TEST_CONF=./config.yaml ./virtualenv/bin/nosetests -a auth_aws4 -v --with-html --html-report=/var/www/html/index.html

示例輸出：

s3tests.functional.test_headers.test_object_create_bad_md5_invalid_short ... oks3tests.functional.test_headers.test_object_create_bad_md5_bad ... oks3tests.functional.test_headers.test_object_create_bad_md5_empty ... oks3tests.functional.test_headers.test_object_create_bad_md5_unreadable ... oks3tests.functional.test_headers.test_object_create_bad_md5_none ... oks3tests.functional.test_headers.test_object_create_bad_expect_mismatch ... oks3tests.functional.test_headers.test_object_create_bad_expect_empty ... oks3tests.functional.test_headers.test_object_create_bad_expect_none ... oks3tests.functional.test_headers.test_object_create_bad_expect_unreadable ... oks3tests.functional.test_headers.test_object_create_bad_contentlength_empty ... oks3tests.functional.test_headers.test_object_create_bad_contentlength_negative ... oks3tests.functional.test_headers.test_object_create_bad_contentlength_none ... oks3tests.functional.test_headers.test_object_create_bad_contentlength_unreadable ... oks3tests.functional.test_headers.test_object_create_bad_contentlength_mismatch_above ... FAILs3tests.functional.test_headers.test_object_create_bad_contenttype_invalid ... oks3tests.functional.test_headers.test_object_create_bad_contenttype_empty ... oks3tests.functional.test_headers.test_object_create_bad_contenttype_none ... oks3tests.functional.test_headers.test_object_create_bad_contenttype_unreadable ... FAILs3tests.functional.test_headers.test_object_create_bad_authorization_unreadable ... FAILs3tests.functional.test_headers.test_object_create_bad_authorization_empty ... oks3tests.functional.test_headers.test_object_create_date_and_amz_date ... oks3tests.functional.test_headers.test_object_create_amz_date_and_no_date ... ok

最后見到如下統(tǒng)計輸出，并且可以看到 HTML 輸出到?/var/www/html/index.html 中了，代表任務(wù)成功運行，現(xiàn)在可以通過界面查看 HTML 端統(tǒng)計結(jié)果。

----------------------------------------------------------------------HTML: /var/www/html/index.html----------------------------------------------------------------------Ran 570 tests in 542.311sFAILED (SKIP=85, errors=26, failures=22)

4

統(tǒng)計分析

概覽統(tǒng)計

查詢測試結(jié)果，點擊 Detail 展開單元測試項運行狀態(tài)。

詳細信息

通過點擊失敗用例對應(yīng)的 Fail 按鈕，查詢具體錯誤信息。

通常我們根據(jù)斷言信息及請求、響應(yīng) header 就可以定位到具體問題了。

5

自定義測試項

對于我們自定義的類S3接口(如自定義Meta信息、私有管理接口等)，也可以很方便的通過擴展現(xiàn)有單元測試函數(shù)來實現(xiàn)，下面是 HTTP Range 請求下載功能的測試函數(shù)，函數(shù)位置：s3-tests/s3tests/functional/test_s3.py 中 Http Range 函數(shù)。

示例：

@attr(resource='object')@attr(method='get')@attr(operation='range')@attr(assertion='returns correct data, 206')def test_ranged_big_request_response_code(): bucket = get_new_bucket() key = bucket.new_key('testobj') string = os.urandom(8 * 1024 * 1024) # 上傳對象 key.set_contents_from_string(string) # 設(shè)置 header，指定 Range 數(shù)據(jù)段 key.open('r', headers={'Range': 'bytes=3145728-5242880'}) status = key.resp.status content_range = key.resp.getheader('Content-Range') fetched_content = '' for data in key: fetched_content += data; key.close() # 驗證 Gateway 響應(yīng)狀態(tài)碼，數(shù)據(jù)長度等 eq(fetched_content, string[3145728:5242881]) eq(status, 206) eq(content_range, 'bytes 3145728-5242880/8388608')

可以此為例，進行自定義用例擴展。

360云計算

由360云平臺團隊打造的技術(shù)分享公眾號，內(nèi)容涉及數(shù)據(jù)庫、大數(shù)據(jù)、微服務(wù)、容器、AIOps、IoT等眾多技術(shù)領(lǐng)域，通過夯實的技術(shù)積累和豐富的一線實戰(zhàn)經(jīng)驗，為你帶來最有料的技術(shù)分享

總結(jié)

以上是生活随笔為你收集整理的马逊s3云存储接口_利用 S3tests 测试 S3 接口兼容性的全部內(nèi)容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網(wǎng)站內(nèi)容還不錯，歡迎將生活随笔推薦給好友。