Skip to content

Commit

Permalink
Fix furrynetwork
Browse files Browse the repository at this point in the history
Captcha broke. This new endpoint returns NSFW artwork but doesn't paginate.
On top of that, unauthenticated access to characters can break, so in
that case fallback to the same endpoint if neccessary. Luckily that
contains the needed information.

This isn't really that great of a solution, but it works.
On the plus-side, no more selenium.
  • Loading branch information
Earlopain committed Oct 2, 2023
1 parent a668a0e commit 3dedd75
Show file tree
Hide file tree
Showing 2 changed files with 11 additions and 34 deletions.
43 changes: 11 additions & 32 deletions app/logical/scraper/furrynetwork.rb
Original file line number Diff line number Diff line change
Expand Up @@ -11,24 +11,18 @@ def initialize(artist_url)
end

def self.required_config_keys
%i[furrynetwork_user furrynetwork_pass]
[]
end

def self.state
:offset
end

def fetch_next_batch
json = make_request("/search", {
size: PER_REQUEST,
from: @offset,
character: url_identifier,
types: ["artwork"],
sort: "published",
})
end_reached if @offset + PER_REQUEST >= json["total"]
@offset += PER_REQUEST
json["hits"].pluck("_source")
# This fetches EVERYTHING. Terribly sorry about that but I can't circumentvent the login captcha
json = make_request("/character/#{url_identifier}/artwork")
end_reached
json.sort_by { |entry| DateTime.parse(entry["published"]) }
end

def to_submission(submission)
Expand All @@ -48,31 +42,16 @@ def to_submission(submission)

def fetch_api_identifier
make_request("/character/#{url_identifier}")["id"]
rescue JSON::ParserError
# Some characters (the_secret_cave) return php error traces with status code 200 because why not
entries = make_request("/character/#{url_identifier}/artwork")
entries&.first&.dig("character_id")
end

private

def make_request(endpoint, query = {})
fetch_json("#{API_PREFIX}#{endpoint}",
query: query,
headers: {
Authorization: "Bearer #{bearer_token}",
},
)
end

# This whole thing is very brittle and may break at any moment
def bearer_token
SeleniumWrapper.driver do |driver|
driver.navigate.to "https://furrynetwork.com/login"
driver.manage.window.maximize
driver.find_element(id: "email").send_keys Config.furrynetwork_user
driver.find_element(id: "password").send_keys Config.furrynetwork_pass
driver.find_element(css: ".page--login__form button[type='submit']").click
driver.wait_for_element(class: "profile-switcher-menu")
driver.execute_script "return JSON.parse(window.localStorage.getItem('token')).access_token"
end
def make_request(endpoint)
fetch_json("#{API_PREFIX}#{endpoint}")
end
cache(:bearer_token, 55.minutes)
end
end
2 changes: 0 additions & 2 deletions config/reverser.yml
Original file line number Diff line number Diff line change
Expand Up @@ -55,8 +55,6 @@ e6_apikey:
furaffinity_user:
furaffinity_pass:
weasyl_apikey:
furrynetwork_user:
furrynetwork_pass:
sofurry_user:
sofurry_pass:
twitter_user:
Expand Down

0 comments on commit 3dedd75

Please sign in to comment.