-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathREADME
125 lines (91 loc) · 5.26 KB
/
README
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
fb-crawl.pl is a script that crawls/scrapes Facebook friends and adds their information to a database.
It can be used for social graph analysis and refined Facebook searching.
FEATURES
- Multithreaded
- Aggregates information from multiple accounts
REQUIREMENTS
- Perl 5 or greater
- MySQL
INSTALLATION
$ cd fb-crawl
$ chmod +x fb-crawl.pl
$ ./fb-crawl.pl
fb-crawl.pl will set up all the required database tables.
All you have to do is provide it with the MySQL connection information and Facebook account.
$ ./fb-crawl.pl -u [email protected] -host mysql.host -user fb-crawl -pass mysqlPassword
OPTIONS
-u Facebook email address.
-p Facebook password.
-host MySQL server IP address or host name (default: localhost);
-port MySQL port (default: 3306)
-user MySQL user (default: root)
-pass MySQL password.
-db MySQL database (default: facebook)
-tables MySQL table names for info, wall, and friends in that order. Formatted in a colon separated list. (default: info:wall:friends)
-info User info save method (default: append)
append: This appends new comma separated information to the row and keeps the old information.
Useful when you want to save all user changes and don't care about when it was updated.
insert: This inserts new user information in a new row (degrages searchability).
Useful when you want to save all user changes and when that info was updated.
replace: This replaces all the current user information in the database with the new information.
Useful when you only care about the most recent user information.
-i Crawl user's information and add to info table.
-w Crawl user's wall posts and add to wall table.
-f Crawl user's friends and add to friends table.
-self Crawl your profile too.
-t Threads (default: 16)
-https Use SSL encryption.
-proxy Use an HTTP proxy. host[:port]
-timeout Timeout in seconds (default: 30)
-depth Crawl depth (default: 0)
0 - only your friends
1 - friends of friends
2 - friends of friends of friends
3 - friendception
-url Crawl these url(s) and also crawl the user's friends if -depth > 1
example: -url http://fb.com/profile.php?id=12345,profile.php?id=54321,john.smith.3
-name Search for and crawl these name(s) and also crawl the user's friends if -depth > 1.
This works by using Facebook's search by and using the first result. For more precision use -url.
example: -name "John Smith, Jane Smith"
-new Only crawl users that aren't in the database.
-old Only crawl users that are in the database.
-plugins Plug-ins to include.
example: birthday2date.pl,location2LatLon.pl
-h Help
EXAMPLES
Crawl your friends' Facebook information, wall, and friends:
$ ./fb-crawl.pl -u [email protected] -i -w -f
Crawl John Smith's Facebook information, wall, and friends:
$ ./fb-crawl.pl -u [email protected] -i -w -f -name 'John Smith'
Crawl Facebook information for friends of friends:
$ ./fb-crawl.pl -u [email protected] -depth 1 -i
Crawl Facebook information of John Smith's friends of friends:
$ ./fb-crawl.pl -u [email protected] -depth 1 -i -name 'John Smith'
Extreme: Crawl friends of friends of friends of friends with 200 threads:
$ ./fb-crawl.pl -u email@address -depth 4 -t 200 -i -w -f
MYSQL EXAMPLES
Find local singles:
SELECT `user_name`, `profile` FROM `info` WHERE `current_city` = 'My Current City, State' AND `sex` = 'Female' AND `relationship` = 'Single'
Find some Harvard singles:
SELECT `user_name`, `profile` FROM `info` WHERE `college` = 'Harvard University' AND `sex` = 'Female' AND `relationship` = 'Single'
How many Facebook employees have you crawled?
SELECT count(*) FROM `info` WHERE `company` = 'Facebook'
Find John Smith's friends:
SELECT `friends` FROM `friends` WHERE `name` = 'John Smith'
PLUG-INS
fb-crawl.pl will open a perl script that can analyze and modify user information before it goes into the database.
The script should contain a function with the same name as the file.
The function is passed a hash reference with the current user's information in it.
To load a plug-in use the -plugins option:
$ ./fb-crawl.pl -u email@address -i -plugins location2latlon.pl,birthday2date.pl
location2latlon.pl:
This plug-in adds the user's coordinates to the database using the Google Geocoding API.
birthday2date.pl:
This plug-in convert the user's birthday to MySQL date (YYYY-MM-DD) format.
See plugin files for implementation details.
FAQ
It's logging in but won't load my friends?
You probably have SSL enabled on your account. You need to use the -https option.
Can't locate object method "ssl_opts" via package "LWP::UserAgent"
You need to install LWP::Protocol::https.
$ sudo perl -MCPAN -e 'install LWP::Protocol::https'