File tree 1 file changed +1
-46
lines changed
1 file changed +1
-46
lines changed Original file line number Diff line number Diff line change 1
1
# fff
2
2
3
- The Fairly Fast Fetcher. Requests a bunch of URLs provided on stdin fairly quickly.
4
-
5
- The main idea is to launch a new request every ` n ` milliseconds, without waiting
6
- for the last request to finish first. This makes for consistently fast fetching,
7
- but can be hard on system resources (e.g. you might run out of file descriptors).
8
- The advantage though, is that hitting a bunch of very slow URLs or URLs that
9
- result in timeouts doesn't slow the overall progress very much.
10
-
11
- ## Install
12
-
13
- ```
14
- ▶ go get -u github.com/tomnomnom/hacks/fff
15
- ```
16
-
17
- ## Usage
18
-
19
- Basic usage:
20
- ```
21
- ▶ cat urls.txt | fff
22
- ```
23
-
24
- Options:
25
-
26
- ```
27
- ▶ fff --help
28
- Request URLs provided on stdin fairly frickin' fast
29
-
30
- Options:
31
- -d, --delay <delay> Delay between issuing requests (ms)
32
- -H, --header <header> Add a header to the request (can be specified multiple times)
33
- -o, --output <dir> Directory to save responses in (will be created)
34
- -s, --save-status <code> Save responses with given status code (can be specified multiple times)
35
- -S, --save Save all responses
36
- ```
37
-
38
- ## Tuning
39
- You might want to increase your open file descriptor limit before doing anything crazy:
40
-
41
- ```
42
- ▶ ulimit -n 16384
43
- ```
44
-
45
- ## TODO
46
-
47
- * Different methods and request bodies
48
- * Create an index file in the output directory
3
+ Now at https://github.com/tomnomnom/fff
You can’t perform that action at this time.
0 commit comments