New lists: Big ass change #287
Conversation
…eFilters for consistency
…rs for consistency
…to int for consistency
…mprove error handling
…ve error handling; enhance listing expiration checks for BHHS and The Agency
…ve NaN value handling
…n a bunch of other shit fuck this
Add support for new list format, new dataset from 10/21, other fixes
perfectly-preserved-pie
added
enhancement
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| mls_number = str(getattr(row, 'mls_number', '')) | ||
|
|
||
| # Check if the listing is expired on BHHS | ||
| if 'bhhscalifornia.com' in listing_url: |
Check failure
Code scanning / CodeQL
Incomplete URL substring sanitization High
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix AI about 2 months ago
To fix the problem, we should parse the URL and check the hostname instead of using a substring check. This ensures that the check is performed on the actual host part of the URL, preventing bypasses through embedding the allowed host in an unexpected location.
The best way to fix this is to use the urlparse function from the urllib.parse module to extract the hostname from the URL and then check if it matches the allowed host. This change should be made in the remove_inactive_listings function.
| @@ -3,2 +3,3 @@ | ||
| from loguru import logger | ||
| from urllib.parse import urlparse | ||
| import asyncio | ||
| @@ -29,3 +30,4 @@ | ||
| # Check if the listing is expired on BHHS | ||
| if 'bhhscalifornia.com' in listing_url: | ||
| parsed_url = urlparse(listing_url) | ||
| if parsed_url.hostname == 'bhhscalifornia.com': | ||
| is_expired = check_expired_listing_bhhs(listing_url, mls_number) | ||
| @@ -35,3 +37,3 @@ | ||
| # Check if the listing is expired on The Agency | ||
| elif 'theagencyre.com' in listing_url: | ||
| elif parsed_url.hostname == 'theagencyre.com': | ||
| is_sold = check_expired_listing_theagency(listing_url, mls_number) |
| indexes_to_drop.append(row.Index) | ||
| logger.success(f"Removed MLS {mls_number} (Index: {row.Index}) from the DataFrame because the listing has expired on BHHS.") | ||
| # Check if the listing is expired on The Agency | ||
| elif 'theagencyre.com' in listing_url: |
Check failure
Code scanning / CodeQL
Incomplete URL substring sanitization High
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix AI about 2 months ago
To fix the problem, we need to parse the URL and check the hostname to ensure it matches the expected domain. This can be done using the urlparse function from the urllib.parse module. Specifically, we will:
- Parse the
listing_urlto extract the hostname. - Check if the hostname matches the expected domain (
bhhscalifornia.comortheagencyre.com).
This approach ensures that the check is performed on the actual hostname, preventing bypasses through substring manipulation.
| @@ -29,3 +29,4 @@ | ||
| # Check if the listing is expired on BHHS | ||
| if 'bhhscalifornia.com' in listing_url: | ||
| parsed_url = urlparse(listing_url) | ||
| if parsed_url.hostname == 'bhhscalifornia.com': | ||
| is_expired = check_expired_listing_bhhs(listing_url, mls_number) | ||
| @@ -35,3 +36,3 @@ | ||
| # Check if the listing is expired on The Agency | ||
| elif 'theagencyre.com' in listing_url: | ||
| elif parsed_url.hostname == 'theagencyre.com': | ||
| is_sold = check_expired_listing_theagency(listing_url, mls_number) |
uvin the container