DAST引擎评价体系
市面上开源和商业化黑盒扫描器众多，蚂蚁内部除自有研发的黑盒扫描器，还使用了来自外部的商业黑盒扫描器，但蚂蚁的评价指标如检出率、准确率、遗漏率等，各类指标受限于无法外部复用或指标波动大等问题，并不能准确描述黑盒扫描器的客观能力，因此各类黑盒扫描器无法被统一的衡量各方面的优劣势，这实际上暴露出内部缺乏统一客观的黑盒扫描引擎评价体系。 黑盒扫描器一般由扫描引擎、展示后台和数据库等部分组成，对黑盒扫描器的扫描能力评估，重点在于对扫描引擎的评估。我们设计从漏洞覆盖度、测试点覆盖度、payload变形和响应识别覆盖度等维度，构建扫描引擎评价体系，并建设黑盒扫描靶场，利用黑盒扫描器对靶场的扫描结果综合评估黑盒扫描引擎的各项能力。
NIST在2008年发布了黑盒扫描器功能规范，提出黑盒扫描器需要具备以下能力：
- 识别OWASP top 10中描述的所有类型的漏洞。
- 输出报告证实漏洞存在
- 支持被动扫描，接受脚本/输入/上下文指导漏洞测试。
- 使用语义上与（OWASP）前10名相同的名称识别漏洞。
- 支持登录态的自定义和维持
- 具有足够低的假阳性率
由sectooladdict组织开源的WAVSEP项目，是一个对黑盒扫描器进行评估的靶场，通过对不同扫描器在多种漏洞上的测试，从不同漏洞的准确率上对黑盒扫描器进行评估，下图为WAVSEP的部分测评结果
Urbano等人基于WAVSEP提出的Reinforce WAVSEP，增加了非漏洞接口测试黑盒扫描器的真阴率，并使用混淆矩阵评价黑盒扫描器在不同漏洞上的扫描能力。
OWASP开源的BenchmarkJava项目也是评价黑盒扫描器的经典靶场，其中使用了评价卡的概念，描述每个接口的漏洞类型、CWE编号等信息，最后通过TPR和FPR评价黑盒扫描器
各类已有的评价体系基本上按照不同类型漏洞的检测能力（包括准确率，精确率和召回率等指标）维度展开，但对黑盒扫描引擎所具备的其他能力，如修改数据包、响应识别、登录态支持、爬虫等维度均不涉及，而这些能力也是黑盒扫描引擎的重要评价维度。
区别于已有的黑盒扫描引擎评价体系，除了对不同漏洞类型的检测能力外，我们将黑盒扫描器在数据包修改字段范围、payload变形支持、payload编码、响应判断和登录态支持等能力作为评价重要的部分，因此，拟定对黑盒扫描器的体系化评价标准如下
图中的每一个最小分支，即为一个评价点，对黑盒扫描引擎而言，满足的点越多，表明扫描引擎的能力越全面。这些评价点分布在蚂蚁黑盒靶场的不同接口中，当黑盒扫描器检测出某个接口，就可以证明黑盒扫描器拥有评价系统中的某项能力。
目前靶场已建设case和评价点的对应关系如下
| payload编码-base64 | BS00063,BS00064,BS00071,BS00077,BS00096 |
|---|---|
| payload编码-url | BS00016,BS00019,BS00020,BS00021,BS00022,BS00023,BS00024,BS00025,BS00026,BS00027,BS00041,BS00042,BS00045,BS00047,BS00087,BS00088,BS00092,BS00093,BS00094,BS00121,BS00123 |
| payload变形-大小写 | BS00119,BS00120 |
| payload变形-前后增加非字母符号 | BS00006,BS00025,BS00030,BS00031,BS00063,BS00064,BS00075,BS00084,BS00085,BS00087,BS00088,BS00092,BS00093,BS00094,BS00095,BS00096,BS00097,BS00098,BS00107,BS00108,BS00118,BS00124,BS00125,BS00126,BS00140,BS00142,BS00143,BS00144,BS00145,BS00147,BS00148,BS00149,BS00151 |
| payload变形-替换参数 | BS00001,BS00002,BS00003,BS00004,BS00005,BS00006,BS00007,BS00008,BS00009,BS00010,BS00011,BS00012,BS00013,BS00014,BS00015,BS00016,BS00017,BS00018,BS00019,BS00020,BS00021,BS00022,BS00023,BS00024,BS00025,BS00026,BS00027,BS00028,BS00029,BS00030,BS00031,BS00032,BS00033,BS00034,BS00035,BS00036,BS00037,BS00038,BS00039,BS00040,BS00041,BS00042,BS00045,BS00046,BS00047,BS00049,BS00050,BS00051,BS00052,BS00053,BS00054,BS00055,BS00056,BS00059,BS00063,BS00064,BS00065,BS00066,BS00067,BS00068,BS00071,BS00072,BS00073,BS00074,BS00075,BS00076,BS00079,BS00080,BS00081,BS00082,BS00083,BS00091,BS00092,BS00093,BS00094,BS00095,BS00106,BS00107,BS00108,BS00109,BS00110,BS00118,BS00119,BS00120,BS00121,BS00122,BS00123,BS00130,BS00131,BS00132,BS00133,BS00134,BS00135,BS00136,BS00137,BS00138,BS00139,BS00140,BS00141,BS00142,BS00143,BS00144,BS00145,BS00147,BS00148,BS00149,BS00150,BS00151,BS00154,BS00155,BS00156,BS00157 |
| payload变形-原型 | BS00006,BS00007,BS00008,BS00009,BS00010,BS00049,BS00050,BS00051,BS00052,BS00053,BS00054,BS00055,BS00056,BS00059,BS00067,BS00076,BS00079,BS00080,BS00099,BS00106,BS00113,BS00130 |
| payload变形-追加 | BS00084,BS00085 |
| 登录态-不支持 | 需要人工确认 |
| 登录态-自定义 | 需要人工确认 |
| 发包能力-并发数量控制 | 需要人工确认 |
| 发包能力-最大发包QPS(固定硬件/网络资源下) | 需要人工确认 |
| 改包能力-header-cookie中的key | 待建设 |
| 改包能力-header-cookie中的value | BS00001,BS00019,BS00020,BS00021,BS00022,BS00023 |
| 改包能力-header-key | BS00038 |
| 改包能力-header-value | BS00002,BS00003,BS00004,BS00005,BS00006,BS00019,BS00024,BS00025,BS00026,BS00027,BS00045,BS00087,BS00088,BS00111 |
| 改包能力-requestBody-json格式-key | |
| 改包能力-requestBody-json格式-value | BS00057,BS00067,BS00072,BS00078,BS00080,BS00095,BS00125,BS00131,BS00132,BS00133,BS00134,BS00135 |
| 改包能力-requestBody-json格式-列表的第n个值 | BS00124 |
| 改包能力-requestBody-xml格式-标签属性的key | BS00061,BS00062,BS00073 |
| 改包能力-requestBody-xml格式-标签属性的value | BS00081 |
| 改包能力-requestBody-xml格式-标签值 | BS00061,BS00062,BS00068,BS00073,BS00074,BS00079,BS00155 |
| 改包能力-requestBody-二进制流 | BS00096 |
| 改包能力-requestBody-文件上传格式 | BS00117 |
| 改包能力-requestBody中的key | BS00013 |
| 改包能力-requestBody中的value | BS00007,BS00008,BS00009,BS00010,BS00011,BS00012,BS00014,BS00016,BS00017,BS00028,BS00029,BS00030,BS00031,BS00032,BS00033,BS00034,BS00035,BS00036,BS00037,BS00039,BS00040,BS00041,BS00042,BS00046,BS00047,BS00049,BS00050,BS00051,BS00052,BS00053,BS00054,BS00055,BS00056,BS00059,BS00061,BS00062,BS00063,BS00064,BS00065,BS00066,BS00071,BS00075,BS00076,BS00077,BS00082,BS00083,BS00084,BS00085,BS00092,BS00106,BS00107,BS00108,BS00113,BS00115,BS00118,BS00119,BS00120,BS00121,BS00122,BS00123,BS00126,BS00138,BS00139,BS00140,BS00141,BS00142,BS00143,BS00144,BS00145,BS00147,BS00148,BS00149,BS00150,BS00151,BS00154,BS00155,BS00156,BS00157 |
| 改包能力-url参数-key | BS00013 |
| 改包能力-url参数-value | BS00007,BS00008,BS00009,BS00010,BS00011,BS00012,BS00014,BS00016,BS00017,BS00028,BS00029,BS00030,BS00031,BS00032,BS00033,BS00034,BS00035,BS00036,BS00037,BS00039,BS00040,BS00041,BS00042,BS00046,BS00047,BS00049,BS00050,BS00051,BS00052,BS00053,BS00054,BS00055,BS00056,BS00059,BS00061,BS00062,BS00063,BS00064,BS00065,BS00066,BS00071,BS00075,BS00076,BS00077,BS00082,BS00083,BS00084,BS00085,BS00091,BS00092,BS00093,BS00094,BS00097,BS00098,BS00099,BS00100,BS00101,BS00106,BS00107,BS00108,BS00109,BS00110,BS00113,BS00115,BS00118,BS00119,BS00120,BS00121,BS00122,BS00123,BS00126,BS00130,BS00138,BS00139,BS00140,BS00141,BS00142,BS00143,BS00144,BS00145,BS00147,BS00148,BS00149,BS00150,BS00151,BS00154,BS00156,BS00157 |
| 改包能力-url参数-value中的json | BS00136,BS00137 |
| 改包能力-url参数-value中的xml | 待建设 |
| 改包能力-url路径 | BS00116,BS00152 |
| 改包能力-请求方法-DELETE请求 | BS00085 |
| 改包能力-请求方法-GET请求 | BS00001,BS00003,BS00004,BS00005,BS00006,BS00007,BS00008,BS00009,BS00010,BS00011,BS00012,BS00013,BS00014,BS00015,BS00016,BS00017,BS00018,BS00019,BS00020,BS00021,BS00022,BS00023,BS00024,BS00025,BS00026,BS00027,BS00028,BS00029,BS00030,BS00031,BS00032,BS00033,BS00034,BS00035,BS00036,BS00037,BS00038,BS00039,BS00040,BS00041,BS00042,BS00043,BS00044,BS00045,BS00046,BS00047,BS00048,BS00049,BS00050,BS00051,BS00052,BS00053,BS00054,BS00055,BS00056,BS00057,BS00058,BS00059,BS00060,BS00061,BS00062,BS00063,BS00064,BS00065,BS00066,BS00067,BS00068,BS00069,BS00070,BS00071,BS00072,BS00073,BS00074,BS00075,BS00076,BS00077,BS00078,BS00079,BS00080,BS00081,BS00082,BS00083,BS00086,BS00087,BS00088,BS00089,BS00090,BS00091,BS00092,BS00093,BS00094,BS00096,BS00097,BS00098,BS00099,BS00100,BS00101,BS00102,BS00103,BS00104,BS00105,BS00106,BS00107,BS00108,BS00109,BS00110,BS00111,BS00112,BS00113,BS00114,BS00115,BS00116,BS00118,BS00119,BS00120,BS00121,BS00122,BS00123,BS00124,BS00125,BS00126,BS00127,BS00128,BS00129,BS00130,BS00131,BS00132,BS00133,BS00134,BS00135,BS00136,BS00137,BS00138,BS00139,BS00140,BS00141,BS00142,BS00143,BS00144,BS00145,BS00146,BS00147,BS00148,BS00149,BS00150,BS00151,BS00152,BS00153,BS00154,BS00155,BS00156,BS00157 |
| 改包能力-请求方法-POST请求 | BS00001,BS00002,BS00003,BS00004,BS00005,BS00006,BS00007,BS00008,BS00009,BS00010,BS00011,BS00012,BS00013,BS00014,BS00015,BS00016,BS00017,BS00018,BS00019,BS00020,BS00021,BS00022,BS00023,BS00024,BS00025,BS00026,BS00027,BS00028,BS00029,BS00030,BS00031,BS00032,BS00033,BS00034,BS00035,BS00036,BS00037,BS00038,BS00039,BS00040,BS00041,BS00042,BS00043,BS00044,BS00045,BS00046,BS00047,BS00048,BS00049,BS00050,BS00051,BS00052,BS00053,BS00054,BS00055,BS00056,BS00057,BS00058,BS00059,BS00060,BS00061,BS00062,BS00063,BS00064,BS00065,BS00066,BS00067,BS00068,BS00069,BS00070,BS00071,BS00072,BS00073,BS00074,BS00075,BS00076,BS00077,BS00078,BS00079,BS00080,BS00081,BS00082,BS00083,BS00087,BS00088,BS00092,BS00095,BS00096,BS00106,BS00107,BS00108,BS00111,BS00112,BS00113,BS00114,BS00115,BS00116,BS00117,BS00118,BS00119,BS00120,BS00121,BS00122,BS00123,BS00124,BS00125,BS00126,BS00127,BS00131,BS00132,BS00133,BS00134,BS00135,BS00136,BS00137,BS00138,BS00139,BS00140,BS00141,BS00142,BS00143,BS00144,BS00145,BS00146,BS00147,BS00148,BS00149,BS00150,BS00151,BS00152,BS00153,BS00154,BS00155,BS00156,BS00157 |
| 改包能力-请求方法-PUT请求 | BS00084 |
| 改包能力-设置多个值 | BS00110,BS00130,BS00139 |
| 监控-扫描对象状态监控 | 需要人工确认 |
| 监控-扫描任务状态监控 | 需要人工确认 |
| 监控-调试日志 | 需要人工确认 |
| 监控-运行日志 | 需要人工确认 |
| 爬虫-被动式 | 需要人工确认 |
| 爬虫-不支持 | 需要人工确认 |
| 爬虫-主动式-爬取完整度 | 靶场根据接口被访问情况自动计算 |
| 爬虫-主动式-支持ajax爬取 | BS00008外的所有case |
| 爬虫-主动式-支持html表单爬取 | BS00008 |
| 爬虫-主动式-支持禁爬 | 需要人工确认 |
| 爬虫-主动式-支持爬取不同源站点 | 需要人工确认 |
| 爬虫-主动式-支持爬取深度配置 | 需要人工确认 |
| 爬虫-主动式-支持跳转层数配置 | 需要人工确认 |
| 扫描控制-节点动态扩缩容 | 需要人工确认 |
| 扫描控制-支持扫描插件热更新 | 需要人工确认 |
| 扫描控制-支持扫描插件自定义 | 需要人工确认 |
| 扫描控制-支持扫描中止 | 需要人工确认 |
| 响应检测-body识别 | BS00001,BS00002,BS00003,BS00004,BS00005,BS00007,BS00008,BS00009,BS00010,BS00011,BS00012,BS00013,BS00014,BS00015,BS00016,BS00017,BS00018,BS00019,BS00020,BS00021,BS00022,BS00023,BS00024,BS00025,BS00026,BS00027,BS00028,BS00029,BS00030,BS00031,BS00032,BS00033,BS00034,BS00035,BS00036,BS00037,BS00038,BS00039,BS00040,BS00041,BS00042,BS00045,BS00046,BS00047,BS00049,BS00050,BS00054,BS00057,BS00061,BS00062,BS00063,BS00064,BS00065,BS00066,BS00067,BS00068,BS00069,BS00070,BS00071,BS00072,BS00073,BS00074,BS00075,BS00076,BS00077,BS00078,BS00079,BS00080,BS00081,BS00082,BS00083,BS00084,BS00085,BS00087,BS00088,BS00089,BS00090,BS00091,BS00092,BS00093,BS00094,BS00095,BS00096,BS00100,BS00101,BS00102,BS00103,BS00104,BS00105,BS00113,BS00114,BS00116,BS00118,BS00119,BS00120,BS00121,BS00122,BS00123,BS00124,BS00125,BS00126,BS00127,BS00128,BS00129,BS00130,BS00131,BS00132,BS00133,BS00134,BS00135,BS00136,BS00137,BS00138,BS00139,BS00141,BS00142,BS00147,BS00148,BS00149,BS00150,BS00151,BS00152,BS00153,BS00154,BS00155,BS00156,BS00157 |
| 响应检测-dnslog检测无回显 | BS00049,BS00050,BS00109,BS00110,BS00115,BS00134,BS00135,BS00136,BS00137,BS00138,BS00154 |
| 响应检测-header识别 | BS00051,BS00052,BS00053,BS00106,BS00111,BS00112 |
| 响应检测-前端渲染 | BS00054,BS00107,BS00108,BS00143,BS00144,BS00145 |
| 响应检测-实际body类型识别 | 待建设 |
| 响应检测-响应时长识别 | BS00098,BS00099 |
| 响应检测-状态码识别 | BS00051,BS00052,BS00053,BS00097,BS00106 |
区别于常规靶场在扫描后需要人工统计漏洞扫描结果，我们对靶场每个接口都设置了黑盒评价体系中的不同评价点，并提供自动化评价功能，输入黑盒扫描器结果后，靶场会根据黑盒扫描引擎评价体系，生成当前黑盒扫描引擎的评价卡
| 评价点 | 支持/数据 |
|---|---|
| payload变形-前后增加非字母符号 | 是 |
| payload变形-原型 | 是 |
| payload变形-大小写 | 是 |
| payload变形-替换参数 | 是 |
| payload变形-追加 | |
| payload编码-base64 | |
| payload编码-url | 是 |
| 发包能力-并发数量控制 | |
| 发包能力-最大发包QPS(固定硬件/网络资源下) | |
| 响应检测-body识别 | 是 |
| 响应检测-dnslog检测无回显 | |
| 响应检测-header识别 | 是 |
| 响应检测-前端渲染 | 是 |
| 响应检测-响应时长识别 | 是 |
| 响应检测-实际body类型识别 | |
| 响应检测-状态码识别 | |
| 扫描控制-支持扫描中止 | |
| 扫描控制-支持扫描插件热更新 | |
| 扫描控制-支持扫描插件自定义 | |
| 扫描控制-节点动态扩缩容 | |
| 改包能力-header-cookie中的key | |
| 改包能力-header-cookie中的value | |
| 改包能力-header-key | |
| 改包能力-header-value | 是 |
| 改包能力-requestBody-json格式-key | |
| 改包能力-requestBody-json格式-value | |
| 改包能力-requestBody-json格式-列表的第n个值 | |
| 改包能力-requestBody-xml格式-标签值 | 是 |
| 改包能力-requestBody-xml格式-标签属性的key | 是 |
| 改包能力-requestBody-xml格式-标签属性的value | |
| 改包能力-requestBody-二进制流 | |
| 改包能力-requestBody-文件上传格式 | |
| 改包能力-requestBody中的key | |
| 改包能力-requestBody中的value | 是 |
| 改包能力-url参数-key | |
| 改包能力-url参数-value | 是 |
| 改包能力-url参数-value中的json | |
| 改包能力-url参数-value中的xml | |
| 改包能力-url路径 | |
| 改包能力-设置多个值 | |
| 改包能力-请求方法-DELETE请求 | |
| 改包能力-请求方法-GET请求 | 是 |
| 改包能力-请求方法-POST请求 | 是 |
| 改包能力-请求方法-PUT请求 | |
| 爬虫-不支持 | |
| 爬虫-主动式-支持ajax爬取 | 是 |
| 爬虫-主动式-支持html表单爬取 | 是 |
| 爬虫-主动式-支持爬取不同源站点 | |
| 爬虫-主动式-支持爬取深度配置 | |
| 爬虫-主动式-支持禁爬 | |
| 爬虫-主动式-支持跳转层数配置 | |
| 爬虫-主动式-爬取完整度 | 95.54% |
| 爬虫-被动式 | |
| 登录态-不支持 | |
| 登录态-自定义 | |
| 引擎控制-扫描节点动态扩缩容 | |
| 引擎控制-是否支持扫描中止 | |
| 引擎控制-是否支持扫描规则自定义 | |
| 引擎控制-是否支持扫描规则热更新 |
需要注意的是，并非评价体系中所有的评价点均可通过靶场自动化检出，因此评价表中的部分选项，需要人工添加