Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add support for new list format, new dataset from 10/21, other fixes #285

Merged
merged 85 commits into from
Nov 12, 2024
Merged

Add support for new list format, new dataset from 10/21, other fixes #285

merged 85 commits into from
Nov 12, 2024

Conversation

perfectly-preserved-pie
Copy link
Owner

No description provided.

@perfectly-preserved-pie
Bump deps
2269821
@perfectly-preserved-pie
Initial commit
961219a
@perfectly-preserved-pie
Add a function to webscrape TLA
a27f3c8
@perfectly-preserved-pie
Add a function to use The Agency API to get property details
e7e98a6
@perfectly-preserved-pie
Fall back to fetching property data from The Agency if it doesn't exi…
3847346 
…st in BHHS
@perfectly-preserved-pie
Refactor fetch_the_agency_data to include row_index and total_rows pa…
72f260f 
…rameters
@perfectly-preserved-pie
Fetch the first property listing image from TA
fa8b0cd
@perfectly-preserved-pie
fetch and transform the first property listing image from The Agency API
ed34e4e
@perfectly-preserved-pie
Return an additional None
0e839e4
@perfectly-preserved-pie
Move webscraping logic into its own function
6193a17
@perfectly-preserved-pie
Add logging
ffbcf59
@perfectly-preserved-pie
Rename parking key
1e16c6b
@perfectly-preserved-pie
Clean up lease dataset. Rename columns to match the new lists'. Drop …
987ccbe 
…useless columns.
@perfectly-preserved-pie
Change column dtypes to the optimal dtype
1ea0421
@perfectly-preserved-pie
Change pet_policy and laundry and subtype to category dtype
f47d304
@perfectly-preserved-pie
Cast dtypes of new list to (hopefully) match those of the old dataset
718947b
@perfectly-preserved-pie
Rename "Garage Spaces" to "Parking Spaces" in popup.js
dae6a9b
@perfectly-preserved-pie
Convert senior_community into boolean
c873f51
@perfectly-preserved-pie
Normalize 'Sqft' column name to 'sqft' in LeaseFilters and BuyFilters
fc9ba9a
@perfectly-preserved-pie
Rename 'garage_spaces' to 'parking_spaces' in LeaseFilters for consis…
c8dece3 
…tency
@perfectly-preserved-pie
Rename 'PetsAllowed' to 'pet_policy' in LeaseFilters for consistency
4c48ccf
@perfectly-preserved-pie
Normalize 'Furnished' column name to 'furnished' in LeaseFilters for …
754d66e 
…consistency
@perfectly-preserved-pie
Normalize 'DepositSecurity' column name to 'security_deposit' in Leas…
b73dc94 
…eFilters for consistency
@perfectly-preserved-pie
Normalize 'DepositPets' column name to 'pets_deposit' in LeaseFilters…
e99ef62 
… for consistency
@perfectly-preserved-pie
Normalize 'DepositKey' column name to 'key_deposit' in LeaseFilters f…
f0aa22d 
…or consistency
@perfectly-preserved-pie
Normalize 'DepositOther' column name to 'other_deposit' in LeaseFilte…
d8dd2f2 
…rs for consistency
@perfectly-preserved-pie
Normalize 'Terms' column name to 'terms' in LeaseFilters for consistency
e3e18f9
@perfectly-preserved-pie
Normalize 'LaundryCategory' column name to 'laundry' in LeaseFilters …
39f42b7 
…for consistency
@perfectly-preserved-pie
Change latitude and longitude colum nmnames
cba3864
@perfectly-preserved-pie
Add .venv/ to .gitignore to exclude virtual environment files
caa5151
@perfectly-preserved-pie
Preliminary function to remove expired listings on The Agency
17b86e1
@perfectly-preserved-pie
Sort imports alphabetically
d83863a
@perfectly-preserved-pie
Consolidate expired listings check into one function
0d72883
@perfectly-preserved-pie
Remove unneeded imports
b55a18e
@perfectly-preserved-pie
Move The Agency removal check to webscraping_utils and change the log…
88231d5 
…ging message a bit
@perfectly-preserved-pie
Remove unneeded imports
d53892e
@perfectly-preserved-pie
Ensure listing_url and mls_number are strings in remove_inactive_list…
ab2c1c6 
…ings function
@perfectly-preserved-pie
Refactor check_expired_listing_bhhs to use synchronous requests and i…
0c1861b 
…mprove error handling
@perfectly-preserved-pie
Fix wrong expired listing check message on BHHS
42411cc
@perfectly-preserved-pie
Use specific domains for removing inactive listings
74ce223
@perfectly-preserved-pie
Refactor web scraping functions to use synchronous requests and impro…
75f1f6a 
…ve error handling; enhance listing expiration checks for BHHS and The Agency
@perfectly-preserved-pie
Fix categorize_laundry_features to handle NaN values using pd.isna
935910d
@perfectly-preserved-pie
New lease dataset for 10/21
3805db8
@perfectly-preserved-pie
Enhance rental terms checklist to handle 'Unknown' category and impro…
5a2e118 
…ve NaN value handling
@perfectly-preserved-pie
Drop old/redundant columns
2949f19
@perfectly-preserved-pie
Remove redundant bedrooms_bathrooms field from lease map data
48abd93
@perfectly-preserved-pie
Changing column dtypes
c14ae1d
@perfectly-preserved-pie
Drop a row with wild ass bedrooms/bathrooms. Fuck this i'm not dealin…
ded1f13 
…g with bad data
@perfectly-preserved-pie
Drop another row with fucked up sqft
b30e34e
@perfectly-preserved-pie
Remove trailing .0 in zipcode
6422e7c
@perfectly-preserved-pie
Remove trailing .0 in full_street_address
2ea2313
@perfectly-preserved-pie
Fix missing city and zipcode
4dea27a
@perfectly-preserved-pie
fix city
339618c
@perfectly-preserved-pie
Cast 'sqft' to UInt32 and update numeric columns to use UInt16Dtype
faebf07
@perfectly-preserved-pie
Copy missing values from their old column counterpart, set dtypes, ma…
64a86a8 
…n a bunch of other shit fuck this
@perfectly-preserved-pie perfectly-preserved-pie added enhancement New feature or request fix Fixing an issue or problem labels Nov 12, 2024
@perfectly-preserved-pie perfectly-preserved-pie merged commit 48287ef into dev Nov 12, 2024
1 check failed
@perfectly-preserved-pie perfectly-preserved-pie deleted the new-lists branch November 19, 2024 19:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@perfectly-preserved-pie perfectly-preserved-pie

Labels
enhancement New feature or request fix Fixing an issue or problem
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@perfectly-preserved-pie