Skip to content

Commit

Permalink
重大更新
Browse files Browse the repository at this point in the history 
根据blackhat议题新增perfix和suffix以探测公有云生产端点，新增多线程发起dns枚举，新增英文文档
  • Loading branch information
@shadowabi
shadowabi committed Jun 9, 2024
1 parent 64c7489 commit 39cf1cf
Show file tree
Hide file tree
Showing 9 changed files with 280 additions and 94 deletions.
5 changes: 5 additions & 0 deletions .idea/codeStyles/codeStyleConfig.xml
View file

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

100 changes: 61 additions & 39 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,10 @@
</a>
<h3 align="center">Endpoint Search</h3>
<p align="center">
一个探测云服务 endpoint 的扫描器，主要用于嗅探私有云的 endpoint 地址
<br />
<br />

Endpoint Search is a reconnaissance tool tailored for identifying and enumerating cloud service endpoints. Inspired by the Black Hat talk "Evading Logging in the Cloud: Bypassing AWS CloudTrail," it facilitates stealthy detection of potentially exposed services.
</p>

<a href="https://github.com/wgpsec/EndpointSearch/stargazers"><img alt="GitHub stars" src="https://img.shields.io/github/stars/wgpsec/EndpointSearch"/></a>
<a href="https://github.com/wgpsec/EndpointSearch/releases"><img alt="GitHub releases" src="https://img.shields.io/github/release/wgpsec/EndpointSearch"/></a>
<a href="https://github.com/wgpsec/EndpointSearch/blob/main/LICENSE"><img alt="License" src="https://img.shields.io/badge/License-Apache%202.0-blue.svg"/></a>
Expand All @@ -15,53 +16,78 @@
<a href="https://twitter.com/wgpsec"><img alt="Twitter" src="https://img.shields.io/twitter/follow/wgpsec?label=Followers&style=social" /></a>
<br>
<br>
<a href="https://github.com/wgpsec/EndpointSearch/discussions"><strong>探索更多Tricks »</strong></a>
<a href="https://github.com/wgpsec/EndpointSearch/discussions"><strong>More Tricks »</strong></a>
<br/>
<br />
<a href="https://github.com/wgpsec/EndpointSearch/releases">下载程序</a>
<a href="https://github.com/wgpsec/EndpointSearch/blob/main/README.md">中文文档</a>
.
<a href="https://github.com/wgpsec/EndpointSearch/releases">Download</a>
·
<a href="https://github.com/wgpsec/EndpointSearch/issues">反馈Bug</a>
<a href="https://github.com/wgpsec/EndpointSearch/issues">Issues</a>
·
<a href="https://github.com/wgpsec/EndpointSearch/discussions">提交需求</a>
<a href="https://github.com/wgpsec/EndpointSearch/discussions">Discussions</a>
</p>

## 安装
## Features
* **DNS Enumeration**: Constructs and queries DNS for probable endpoint URLs based on predefined patterns.
* **SRV Record Inspection**: Automatically detects SRV records to uncover associated ports.
* **HTTP/HTTPS Probing**: Tests endpoints with both HTTP and HTTPS protocols when SRV records are not present.
* **Endpoint Judgment**: Determines endpoints likelihood based on response content, currently focusing on XML format.
* **Automatic Domain Extraction**: Extracts domains from input URLs automatically.
* **Output Redundancy Removal**: Ensures unique results by deduplication.
* **Configurable Behavior**: Offers a flexible configuration file for customizing service names, connection modes, and more.

下载release中的二进制文件使用
## Installation

或使用Makefile进行编译二进制文件后使用
* Download precompiled binaries from the [releases page](https://github.com/wgpsec/EndpointSearch/releases).

## 配置
当首次运行 EndpointSearch 时，会检测 config.json 文件是否存在，不存在则会自动创建
* Alternatively, use the included Makefile to compile from source.

config.json的填写内容应该如下：
## Configuration
Upon first run, the tool checks for config.json. If missing, it generates one with default settings including:
* CloudService: Enumerated cloud services.
* Mode & Mode2: Patterns connecting services, prefixes/suffixes, and targets.
* PortList: Ports to scan if no SRV record is found.
* Prefix & Suffix: Common naming conventions for prefixing or suffixing service names.
```
{
"CloudEndpoint":"oss,ecs",
"Mode":",-,_,."，
"PortList":"80,443"
"CloudService":"oss,ecs",
"Mode":".",
"Mode2" :"-,.",
"PortList":"80,443",
"Prefix":"sonic,legacy,preprod,gamma,beta,staging",
"Suffix":"sonic,legacy,preprod,gamma,beta,staging",
}
```
CloudEndpoint 为枚举的端点，Mode是端点与域名的连接方式,PortList为扫描的端口

CloudEopint 可参考我的另一个字典项目: https://github.com/shadowabi/S-BlastingDictionary/blob/main/CloudEndpoint.txt
CloudService can refer to another of my dictionary projects : https://github.com/shadowabi/S-BlastingDictionary/blob/main/CloudService.txt

## 工作流程
1. 输入域名 example.com，首先会使用 DNS 去枚举 ossexample.com, oss-example.com... 即 CloudEndpoint + Mode + Domain 的方式
## Workflow
1. Enter the domain name example.com. DNS is used to enumerate example.com.
* Prefix + Mode2 + CloudService + Mode + Host
* CloudService + Mode2 + Suffix + Mode + Host
* CloudService + Mode +Host

2. 当域名存在时，会查询 dns 中的 srv 记录发现端口
For example, if Prefix is sonic, Suffix is legacy, CloudService is oss, Mode is., and Mode2 is -, the system will enumerate:
```
sonic-oss.examlpe.com
oss-legacy.example.com
oss.example.com
```

3. 若已经存在 srv 记录，则不会去枚举端口，而是直接用 HTTP / HTTPS 协议去请求这个URL
~~~~2. If the domain name exists, the system queries the srv records in the dns to discover the port
4. 否则将通过 HTTP 和 HTTPS 协议去尝试访问目标域名 + PortList 中的端口
3. If srv records already exist, HTTP/HTTPS is used to request the URL instead of enumerating the port
5. 最后通过 HTTP 的请求结果判断整个 URL 是否为 Endpoint，目前判断方式为：目标返回的数据是否为 xml 格式
4. Otherwise, the system attempts to access the port in the target domain name + PortList through HTTP or HTTPS
如果有其他特征，欢迎在 Issues 中提出，或者直接发起 PR。
5. Finally, determine whether the entire URL is an Endpoint based on the HTTP request result. At present, determine whether the destination data is returned in xml format
判断 Endpoint 的方法在 pkg 目录 data.go 的 JudgeEndpoint 函数中实现
If there are other characteristics, feel free to raise them in the Issues, or launch a PR directly.
## 用法
The method to determine the Endpoint is implemented in the JudgeEndpoint function in the pkg directory data.go
## Usage
```
Usage:
Expand All @@ -70,27 +96,23 @@ Usage:
Flags:
-e, --endpoint string 输入需要被枚举的 endpoint (Input [endpoint])
-f, --file string 从文件中读取目标地址 (Input filename)
-h, --help help for EndpointSearch
--logLevel string 设置日志等级 (Set log level) [trace|debug|info|warn|error|fatal|panic] (default "info")
-o, --output string 输入结果文件输出的位置 (Enter the location of the scan result output) (default "./result.txt")
-p, --port string 输入需要被扫描的端口，逗号分割 (Enter the port to be scanned, separated by commas (,))
-t, --timeout int 输入每个 http 请求的超时时间 (Enter the timeout period for every http request) (default 3)
--prefix string 输入需要被枚举的服务名称 (Input Service Name)
-s, --service string 输入需要被枚举的服务名称 (Input Service Name)
--suffix string 输入需要被枚举的服务名称 (Input Service Name)
-t, --timeout int 输入每个 http 请求的超时时间 (Enter the timeout period for every http request) (default 2)
-u, --url string 输入目标地址 (Input [domain|url])
```
EndpointSearch 同样支持手动覆盖配置参数，-e 参数默认为配置中的 CloudEndpoint，-p 参数为配置中的 PortList

当主动指定参数后，将不再使用配置文件中的默认值
EndpointSearch also supports manually overwriting configuration parameters. By default, the -e parameter is CloudEndpoint in the configuration, and the -p parameter is PortList in the configuration
## 功能列表
When parameters are actively specified, the default values in the configuration file are no longer used
1. 利用 dns 服务枚举端点，隐蔽侦查
2. 当域名存在时，自动探测 srv 服务发现端口
3. 自动去重
4. 输入的 url 将自动提取为域名
## TODO
1. 添加 socket5 代理的支持
2. 更多判断 Endpoint 的方法
1. Proxy Support: Implementation of SOCKS5 proxy support for enhanced anonymity.
2. Enhanced Endpoint Detection: Expanding endpoint validation criteria beyond XML responses.
114 changes: 114 additions & 0 deletions README_CN.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,114 @@
<p align="center">
<a href="https://github.com/wgpsec/ENScan_GO">
<img src="https://github.com/wgpsec/EndpointSearch/assets/16091665/9a26fcef-26fe-4f6b-8c8f-905cdd066296" alt="Logo" width="200" height="200">
</a>
<h3 align="center">Endpoint Search</h3>
<p align="center">
一个用于侦查云服务端点的工具，此工具参考了 Black Hat议题 Evading Logging in the Cloud: Bypassing AWS CloudTrail
</p>

<a href="https://github.com/wgpsec/EndpointSearch/stargazers"><img alt="GitHub stars" src="https://img.shields.io/github/stars/wgpsec/EndpointSearch"/></a>
<a href="https://github.com/wgpsec/EndpointSearch/releases"><img alt="GitHub releases" src="https://img.shields.io/github/release/wgpsec/EndpointSearch"/></a>
<a href="https://github.com/wgpsec/EndpointSearch/blob/main/LICENSE"><img alt="License" src="https://img.shields.io/badge/License-Apache%202.0-blue.svg"/></a>
<a href="https://github.com/wgpsec/EndpointSearch/releases"><img alt="Downloads" src="https://img.shields.io/github/downloads/wgpsec/EndpointSearch/total?color=brightgreen"/></a>
<a href="https://goreportcard.com/report/github.com/wgpsec/EndpointSearch"><img alt="Go Report Card" src="https://goreportcard.com/badge/github.com/wgpsec/EndpointSearch"/></a>
<a href="https://twitter.com/wgpsec"><img alt="Twitter" src="https://img.shields.io/twitter/follow/wgpsec?label=Followers&style=social" /></a>
<br>
<br>
<a href="https://github.com/wgpsec/EndpointSearch/discussions"><strong>探索更多Tricks »</strong></a>
<br/>
<br />
<a href="https://github.com/wgpsec/EndpointSearch/blob/main/README.md">英文文档</a>
.
<a href="https://github.com/wgpsec/EndpointSearch/releases">下载程序</a>
·
<a href="https://github.com/wgpsec/EndpointSearch/issues">反馈Bug</a>
·
<a href="https://github.com/wgpsec/EndpointSearch/discussions">提交需求</a>
</p>

## 安装

下载release中的二进制文件使用

或使用Makefile进行编译二进制文件后使用

## 配置
当首次运行 EndpointSearch 时，会检测 config.json 文件是否存在，不存在则会自动创建

config.json的填写内容应该如下：
```
{
"CloudService":"oss,ecs",
"Mode":".",
"Mode2" :"-,.",
"PortList":"80,443",
"Prefix":"sonic,legacy,preprod,gamma,beta,staging",
"Suffix":"sonic,legacy,preprod,gamma,beta,staging",
}
```
CloudService 为枚举的云服务名称，Mode 是连接 CloudService 与 target 的方式，Mode2 是连接前后缀与 CloudService 的方式, PortList 为扫描的端口，具体例子见工作流程


CloudService 可参考我的另一个字典项目: https://github.com/shadowabi/S-BlastingDictionary/blob/main/CloudService.txt

## 工作流程
1. 输入域名 example.com，首先会使用 DNS 去枚举 example.com，枚举方式遵循以下特点：
* Prefix + Mode2 + CloudService + Mode + Host
* CloudService + Mode2 + Suffix + Mode + Host
* CloudService + Mode +Host

例如 Prefix 为 sonic，Suffix 为 legacy，CloudService 为 oss，Mode 为 .，Mode2 为 -，则会枚举：
```
sonic-oss.examlpe.com
oss-legacy.example.com
oss.example.com
```

2. 当域名存在时，会查询 dns 中的 srv 记录发现端口

3. 若已经存在 srv 记录，则不会去枚举端口，而是直接用 HTTP / HTTPS 协议去请求这个URL

4. 否则将通过 HTTP 和 HTTPS 协议去尝试访问目标域名 + PortList 中的端口

5. 最后通过 HTTP 的请求结果判断整个 URL 是否为 Endpoint，目前判断方式为：目标返回的数据是否为 xml 格式

如果有其他特征，欢迎在 Issues 中提出，或者直接发起 PR。

判断 Endpoint 的方法在 pkg 目录 data.go 的 JudgeEndpoint 函数中实现

## 用法
```
Usage:
EndpointSearch [flags]
Flags:
-f, --file string 从文件中读取目标地址 (Input filename)
-h, --help help for EndpointSearch
--logLevel string 设置日志等级 (Set log level) [trace|debug|info|warn|error|fatal|panic] (default "info")
-o, --output string 输入结果文件输出的位置 (Enter the location of the scan result output) (default "./result.txt")
-p, --port string 输入需要被扫描的端口，逗号分割 (Enter the port to be scanned, separated by commas (,))
--prefix string 输入需要被枚举的服务名称 (Input Service Name)
-s, --service string 输入需要被枚举的服务名称 (Input Service Name)
--suffix string 输入需要被枚举的服务名称 (Input Service Name)
-t, --timeout int 输入每个 http 请求的超时时间 (Enter the timeout period for every http request) (default 2)
-u, --url string 输入目标地址 (Input [domain|url])
```
EndpointSearch 同样支持手动覆盖配置参数，-e 参数默认为配置中的 CloudEndpoint，-p 参数为配置中的 PortList

当主动指定参数后，将不再使用配置文件中的默认值

## 功能列表

1. 利用 dns 服务枚举端点，隐蔽侦查
2. 当域名存在时，自动探测 srv 服务发现端口
3. 自动去重
4. 输入的 url 将自动提取为域名

## TODO
1. 添加 socket5 代理的支持
2. 更多判断 Endpoint 的方法

25 changes: 18 additions & 7 deletions cmd/rootCmd.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -29,27 +29,33 @@ var RootCmd = &cobra.Command{
`
github.com/wgpsec/EndpointSearch
EndpointSearch 是一个探测云服务 endpoint 的扫描器
EndpointSearch 是一个探测云服务端点的扫描器
EndpointSearch is a scanner that probes the endpoint of a cloud service
`,
PersistentPreRun: func(cmd *cobra.Command, args []string) {
log.Init(logLevel)
if define.Url != "" && define.File != "" {
Error.HandleFatal(errors.New("参数不可以同时存在"))
Error.HandleFatal(errors.New("Url 参数和 File 参数不可以同时存在 (The Url parameter and File parameter cannot exist at the same time)"))
return
}
if define.Url == "" && define.File == "" {
Error.HandleFatal(errors.New("必选参数为空，请输入 -u 参数或 -f 参数"))
Error.HandleFatal(errors.New("必选参数为空，请输入 -u 参数或 -f 参数 (The mandatory parameter is empty. Enter the -u parameter or -f parameter)"))
return
}
},
Run: func(cmd *cobra.Command, args []string) {
if define.Endpoint == "" {
define.Endpoint = config.C.CloudEndpoint
if define.Service == "" {
define.Service = config.C.CloudService
}
if define.Port == "" {
define.Port = config.C.PortList
}
if define.Prefix == "" {
define.Prefix = config.C.Prefix
}
if define.Suffix == "" {
define.Suffix = config.C.Suffix
}
portList := strings.Split(define.Port, ",")

var hostList []string
Expand All @@ -60,14 +66,17 @@ EndpointSearch is a scanner that probes the endpoint of a cloud service
}
hostList = Compare.RemoveDuplicates(hostList)

reqList := pkg.ConvertToReqList(define.Endpoint, hostList...)
reqList := pkg.ConvertToReqList(define.Service, define.Prefix, define.Suffix, hostList...)
fmt.Println("Domain exist:")
ipRecordList := pkg.SearchDomain(reqList...)
fmt.Println("Domain srv exist:")
recordList := pkg.SearchSRVRecord(ipRecordList...)

client := pkg.GenerateHTTPClient(define.TimeOut)
respList := pkg.SearchEndpoint(client, portList, recordList...)

resultList := Compare.RemoveDuplicates(pkg.JudgeEndpoint(respList...))
fmt.Println("Service endpoint exist:")
pkg.WriteToFile(resultList, define.OutPut)
fmt.Printf("[+] The output is in %s\n", define.OutPut)
},
Expand All @@ -81,7 +90,9 @@ func init() {
RootCmd.SetHelpFunc(customHelpFunc)
RootCmd.Flags().StringVarP(&define.File, "file", "f", "", "从文件中读取目标地址 (Input filename)")
RootCmd.Flags().StringVarP(&define.Url, "url", "u", "", "输入目标地址 (Input [domain|url])")
RootCmd.Flags().StringVarP(&define.Endpoint, "endpoint", "e", "", "输入需要被枚举的 endpoint (Input [endpoint])")
RootCmd.Flags().StringVarP(&define.Service, "service", "s", "", "输入需要被枚举的服务名称 (Input Service Name)")
RootCmd.Flags().StringVarP(&define.Prefix, "prefix", "", "", "输入需要被枚举的服务名称 (Input Service Name)")
RootCmd.Flags().StringVarP(&define.Suffix, "suffix", "", "", "输入需要被枚举的服务名称 (Input Service Name)")
RootCmd.Flags().IntVarP(&define.TimeOut, "timeout", "t", 2, "输入每个 http 请求的超时时间 (Enter the timeout period for every http request)")
RootCmd.Flags().StringVarP(&define.OutPut, "output", "o", "./result.txt", "输入结果文件输出的位置 (Enter the location of the scan result output)")
RootCmd.Flags().StringVarP(&define.Port, "port", "p", "", "输入需要被扫描的端口，逗号分割 (Enter the port to be scanned, separated by commas (,))")
Expand Down
23 changes: 14 additions & 9 deletions define/var.go
View file
Original file line number Diff line number Diff line change
@@ -1,16 +1,21 @@
package define

type Configure struct {
Mode string `mapstructure:"Mode" json:"Mode" yaml:"Mode"`
CloudEndpoint string `mapstructure:"CloudEndpoint" json:"CloudEndpoint" yaml:"CloudEndpoint"`
PortList string `mapstructure:"PortList" json:"PortList" yaml:"PortList"`
Mode string `mapstructure:"Mode" json:"Mode" yaml:"Mode"`
Mode2 string `mapstructure:"Mode2" json:"Mode2" yaml:"Mode2"`
CloudService string `mapstructure:"CloudService" json:"CloudService" yaml:"CloudService"`
PortList string `mapstructure:"PortList" json:"PortList" yaml:"PortList"`
Prefix string `mapstructure:"Prefix" json:"Prefix" yaml:"Prefix"`
Suffix string `mapstructure:"Suffix" json:"Suffix" yaml:"Suffix"`
}

var (
File string
Url string
Endpoint string
OutPut string
TimeOut int
Port string
File string
Url string
Service string
OutPut string
TimeOut int
Port string
Prefix string
Suffix string
)
2 changes: 1 addition & 1 deletion main/main.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ func init() {
_, err := File.FileCreateIfNonExist(configFile)
Error.HandleFatal(err)
config.Init(configFile)
if config.C.CloudEndpoint == "" || config.C.Mode == "" || config.C.PortList == "" {
if config.C.CloudService == "" || config.C.Mode == "" || config.C.PortList == "" || config.C.Mode2 == "" {
Error.HandleFatal(errors.New("请配置config.json"))
return
}
Expand Down
Loading

0 comments on commit 39cf1cf

Please sign in to comment.