Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add list_and_lock_head, delete_request_lock, prolong_request_lock, batch_add_requests, batch_delete_requests and list_requests methods #129

Merged
merged 12 commits into from
Jul 24, 2023
126 changes: 126 additions & 0 deletions docs/docs.md
Original file line number Diff line number Diff line change
Expand Up @@ -3742,10 +3742,13 @@ Async sub-client for manipulating a single request queue.
* [async update()](#requestqueueclientasync-update)
* [async delete()](#requestqueueclientasync-delete)
* [async list\_head()](#requestqueueclientasync-list\_head)
* [async list\_and\_lock\_head()](#requestqueueclientasync-list\_and\_lock\_head)
* [async add\_request()](#requestqueueclientasync-add\_request)
* [async get\_request()](#requestqueueclientasync-get\_request)
* [async update\_request()](#requestqueueclientasync-update\_request)
* [async delete\_request()](#requestqueueclientasync-delete\_request)
* [async prolong\_request\_lock()](#requestqueueclientasync-prolong\_request\_lock)
* [async delete\_request\_lock()](#requestqueueclientasync-delete\_request\_lock)

***

Expand Down Expand Up @@ -3817,6 +3820,28 @@ Retrieve a given number of requests from the beginning of the queue.

***

#### [](#requestqueueclientasync-list_and_lock_head) `async RequestQueueClientAsync.list_and_lock_head(*, lock_secs, limit=None)`

Retrieve the given number of first requests from the queue and locks them for the given time.

[https://docs.apify.com/api/v2#/reference/request-queues/queue-head-with-locks/get-head-and-lock](https://docs.apify.com/api/v2#/reference/request-queues/queue-head-with-locks/get-head-and-lock)

* **Parameters**

* **lock_secs** (`int`) – How long the second request will be locked for

* **limit** (`int`, *optional*) – How many requests to retrieve

* **Returns**

The desired number of locked requests from the beginning of the queue.

* **Return type**

`dict`

***

#### [](#requestqueueclientasync-add_request) `async RequestQueueClientAsync.add_request(request, *, forefront=None)`

Add a request to the queue.
Expand Down Expand Up @@ -3897,6 +3922,44 @@ Delete a request from the queue.

***

#### [](#requestqueueclientasync-prolong_request_lock) `async RequestQueueClientAsync.prolong_request_lock(request_id, *, lock_secs, forefront=None)`

Prolong the lock on a request.

[https://docs.apify.com/api/v2#/reference/request-queues/request-lock/prolong-request-lock](https://docs.apify.com/api/v2#/reference/request-queues/request-lock/prolong-request-lock)

* **Parameters**

* **request_id** (`str`) – ID of the request to prolong the lock

* **lock_secs** (`int`) – How long to prolong the lock

* **forefront** (`bool`, *optional*) – Whether to put the request in the beginning or the end of the queue after lock exppires

* **Return type**

`Dict`

***

#### [](#requestqueueclientasync-delete_request_lock) `async RequestQueueClientAsync.delete_request_lock(request_id, *, forefront=None)`

Delete the lock on a request.

[https://docs.apify.com/api/v2#/reference/request-queues/request-lock/delete-request-lock](https://docs.apify.com/api/v2#/reference/request-queues/request-lock/delete-request-lock)

* **Parameters**

* **request_id** (`str`) – ID of the request to delete the lock

* **forefront** (`bool`, *optional*) – Whether to put the request in the beginning or the end of the queue after lock deleted

* **Return type**

`None`

***

### [](#requestqueueclient) RequestQueueClient

Sub-client for manipulating a single request queue.
Expand All @@ -3905,10 +3968,13 @@ Sub-client for manipulating a single request queue.
* [update()](#requestqueueclient-update)
* [delete()](#requestqueueclient-delete)
* [list\_head()](#requestqueueclient-list\_head)
* [list\_and\_lock\_head()](#requestqueueclient-list\_and\_lock\_head)
* [add\_request()](#requestqueueclient-add\_request)
* [get\_request()](#requestqueueclient-get\_request)
* [update\_request()](#requestqueueclient-update\_request)
* [delete\_request()](#requestqueueclient-delete\_request)
* [prolong\_request\_lock()](#requestqueueclient-prolong\_request\_lock)
* [delete\_request\_lock()](#requestqueueclient-delete\_request\_lock)

***

Expand Down Expand Up @@ -3980,6 +4046,28 @@ Retrieve a given number of requests from the beginning of the queue.

***

#### [](#requestqueueclient-list_and_lock_head) `RequestQueueClient.list_and_lock_head(*, lock_secs, limit=None)`

Retrieve the given number of first requests from the queue and locks them for the given time.

[https://docs.apify.com/api/v2#/reference/request-queues/queue-head-with-locks/get-head-and-lock](https://docs.apify.com/api/v2#/reference/request-queues/queue-head-with-locks/get-head-and-lock)

* **Parameters**

* **lock_secs** (`int`) – How long the second request will be locked for

* **limit** (`int`, *optional*) – How many requests to retrieve

* **Returns**

The desired number of locked requests from the beginning of the queue.

* **Return type**

`dict`

***

#### [](#requestqueueclient-add_request) `RequestQueueClient.add_request(request, *, forefront=None)`

Add a request to the queue.
Expand Down Expand Up @@ -4060,6 +4148,44 @@ Delete a request from the queue.

***

#### [](#requestqueueclient-prolong_request_lock) `RequestQueueClient.prolong_request_lock(request_id, *, lock_secs, forefront=None)`

Prolong the lock on a request.

[https://docs.apify.com/api/v2#/reference/request-queues/request-lock/prolong-request-lock](https://docs.apify.com/api/v2#/reference/request-queues/request-lock/prolong-request-lock)

* **Parameters**

* **request_id** (`str`) – ID of the request to prolong the lock

* **lock_secs** (`int`) – How long to prolong the lock

* **forefront** (`bool`, *optional*) – Whether to put the request in the beginning or the end of the queue after lock exppires

* **Return type**

`Dict`

***

#### [](#requestqueueclient-delete_request_lock) `RequestQueueClient.delete_request_lock(request_id, *, forefront=None)`

Delete the lock on a request.

[https://docs.apify.com/api/v2#/reference/request-queues/request-lock/delete-request-lock](https://docs.apify.com/api/v2#/reference/request-queues/request-lock/delete-request-lock)

* **Parameters**

* **request_id** (`str`) – ID of the request to delete the lock

* **forefront** (`bool`, *optional*) – Whether to put the request in the beginning or the end of the queue after lock deleted

* **Return type**

`None`

***

### [](#requestqueuecollectionclientasync) RequestQueueCollectionClientAsync

Async sub-client for manipulating request queues.
Expand Down
134 changes: 134 additions & 0 deletions src/apify_client/clients/resource_clients/request_queue.py
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,29 @@ def list_head(self, *, limit: Optional[int] = None) -> Dict:

return _parse_date_fields(_pluck_data(response.json()))

def list_and_lock_head(self, *, lock_secs: int, limit: Optional[int] = None) -> Dict:
"""Retrieve the given number of first requests from the queue and locks them for the given time.
drobnikj marked this conversation as resolved.
Show resolved Hide resolved

https://docs.apify.com/api/v2#/reference/request-queues/queue-head-with-locks/get-head-and-lock

Args:
lock_secs (int): How long the second request will be locked for
drobnikj marked this conversation as resolved.
Show resolved Hide resolved
limit (int, optional): How many requests to retrieve


Returns:
dict: The desired number of locked requests from the beginning of the queue.
"""
request_params = self._params(lockSecs=lock_secs, limit=limit, clientKey=self.client_key)

response = self.http_client.call(
url=self._url('head/lock'),
method='POST',
params=request_params,
)

return _parse_date_fields(_pluck_data(response.json()))

def add_request(self, request: Dict, *, forefront: Optional[bool] = None) -> Dict:
"""Add a request to the queue.

Expand Down Expand Up @@ -170,6 +193,50 @@ def delete_request(self, request_id: str) -> None:
params=request_params,
)

def prolong_request_lock(self, request_id: str, *, lock_secs: int, forefront: Optional[bool] = None) -> Dict:
drobnikj marked this conversation as resolved.
Show resolved Hide resolved
"""Prolong the lock on a request.

https://docs.apify.com/api/v2#/reference/request-queues/request-lock/prolong-request-lock

Args:
request_id (str): ID of the request to prolong the lock
lock_secs (int): How long to prolong the lock
drobnikj marked this conversation as resolved.
Show resolved Hide resolved
forefront (bool, optional): Whether to put the request in the beginning or the end of the queue after lock exppires
drobnikj marked this conversation as resolved.
Show resolved Hide resolved
"""
request_params = self._params(
drobnikj marked this conversation as resolved.
Show resolved Hide resolved
forefront=forefront,
lockSecs=lock_secs,
clientKey=self.client_key,
)

response = self.http_client.call(
url=self._url(f'requests/{request_id}/lock'),
method='PUT',
params=request_params,
)

return _parse_date_fields(_pluck_data(response.json()))

def delete_request_lock(self, request_id: str, *, forefront: Optional[bool] = None) -> None:
drobnikj marked this conversation as resolved.
Show resolved Hide resolved
"""Delete the lock on a request.

https://docs.apify.com/api/v2#/reference/request-queues/request-lock/delete-request-lock

Args:
request_id (str): ID of the request to delete the lock
forefront (bool, optional): Whether to put the request in the beginning or the end of the queue after lock deleted
drobnikj marked this conversation as resolved.
Show resolved Hide resolved
"""
request_params = self._params(
clientKey=self.client_key,
forefront=forefront,
)

self.http_client.call(
url=self._url(f'requests/{request_id}/lock'),
method='DELETE',
params=request_params,
)


class RequestQueueClientAsync(ResourceClientAsync):
"""Async sub-client for manipulating a single request queue."""
Expand Down Expand Up @@ -240,6 +307,29 @@ async def list_head(self, *, limit: Optional[int] = None) -> Dict:

return _parse_date_fields(_pluck_data(response.json()))

async def list_and_lock_head(self, *, lock_secs: int, limit: Optional[int] = None) -> Dict:
"""Retrieve the given number of first requests from the queue and locks them for the given time.

https://docs.apify.com/api/v2#/reference/request-queues/queue-head-with-locks/get-head-and-lock

Args:
lock_secs (int): How long the second request will be locked for
limit (int, optional): How many requests to retrieve


Returns:
dict: The desired number of locked requests from the beginning of the queue.
"""
request_params = self._params(lockSecs=lock_secs, limit=limit, clientKey=self.client_key)

response = await self.http_client.call(
url=self._url('head/lock'),
method='POST',
params=request_params,
)

return _parse_date_fields(_pluck_data(response.json()))

async def add_request(self, request: Dict, *, forefront: Optional[bool] = None) -> Dict:
"""Add a request to the queue.

Expand Down Expand Up @@ -335,3 +425,47 @@ async def delete_request(self, request_id: str) -> None:
method='DELETE',
params=request_params,
)

async def prolong_request_lock(self, request_id: str, *, lock_secs: int, forefront: Optional[bool] = None) -> Dict:
"""Prolong the lock on a request.

https://docs.apify.com/api/v2#/reference/request-queues/request-lock/prolong-request-lock

Args:
request_id (str): ID of the request to prolong the lock
lock_secs (int): How long to prolong the lock
forefront (bool, optional): Whether to put the request in the beginning or the end of the queue after lock exppires
"""
request_params = self._params(
forefront=forefront,
lockSecs=lock_secs,
clientKey=self.client_key,
)

response = await self.http_client.call(
url=self._url(f'requests/{request_id}/lock'),
method='PUT',
params=request_params,
)

return _parse_date_fields(_pluck_data(response.json()))

async def delete_request_lock(self, request_id: str, *, forefront: Optional[bool] = None) -> None:
"""Delete the lock on a request.

https://docs.apify.com/api/v2#/reference/request-queues/request-lock/delete-request-lock

Args:
request_id (str): ID of the request to delete the lock
forefront (bool, optional): Whether to put the request in the beginning or the end of the queue after lock deleted
"""
request_params = self._params(
clientKey=self.client_key,
forefront=forefront,
)

await self.http_client.call(
url=self._url(f'requests/{request_id}/lock'),
method='DELETE',
params=request_params,
)
55 changes: 55 additions & 0 deletions tests/integration/test_request_queue.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
import random
import string

from apify_client import ApifyClient, ApifyClientAsync


def random_string(length: int = 10) -> str:
return ''.join(random.choice(string.ascii_letters) for _ in range(length))


def random_queue_name() -> str:
return f'python-client-test-queue-{random_string(5)}'


class TestRequestQueueSync:
drobnikj marked this conversation as resolved.
Show resolved Hide resolved
def test_request_queue_lock(self, apify_client: ApifyClient) -> None:
created_queue = apify_client.request_queues().get_or_create(name=random_queue_name())
queue = apify_client.request_queue(created_queue['id'], client_key=random_string(10))
for i in range(15):
queue.add_request({'url': f'http://example.com3{i}', 'uniqueKey': f'http://example.com3{i}'})
drobnikj marked this conversation as resolved.
Show resolved Hide resolved
locked_requests_list = queue.list_and_lock_head(limit=10, lock_secs=10)
locked_requests = locked_requests_list['items']
for locked_request in locked_requests:
assert locked_request['lockExpiresAt'] is not None
queue.delete_request_lock(locked_requests[1])
delete_lock_request = queue.get_request(locked_requests[1]['id'])
assert delete_lock_request['lockExpiresAt'] is not None # type: ignore
queue.delete_request_lock(locked_requests[2]['id'], forefront=True)
delete_lock_request2 = queue.get_request(locked_requests[2]['id'])
assert delete_lock_request2['lockExpiresAt'] is not None # type: ignore
assert queue.prolong_request_lock(locked_requests[3]['id'], lock_secs=15)['lockExpiresAt'] is not None
queue.delete()
assert apify_client.request_queue(created_queue['id']).get() is None


class TestRequestQueueAsync:
async def test_request_queue_lock(self, apify_client_async: ApifyClientAsync) -> None:
created_queue = await apify_client_async.request_queues().get_or_create(name=random_queue_name())
queue = apify_client_async.request_queue(created_queue['id'], client_key=random_string(10))
for i in range(15):
await queue.add_request({'url': f'http://example.com3{i}', 'uniqueKey': f'http://example.com3{i}'})
locked_requests_list = await queue.list_and_lock_head(limit=10, lock_secs=10)
locked_requests = locked_requests_list['items']
for locked_request in locked_requests:
assert locked_request['lockExpiresAt'] is not None
await queue.delete_request_lock(locked_requests[1]['id'])
delete_lock_request = await queue.get_request(locked_requests[1]['id'])
assert delete_lock_request['lockExpiresAt'] is not None # type: ignore
await queue.delete_request_lock(locked_requests[2]['id'], forefront=True)
delete_lock_request2 = await queue.get_request(locked_requests[2]['id'])
assert delete_lock_request2['lockExpiresAt'] is not None # type: ignore
prolonged_request = await queue.prolong_request_lock(locked_requests[3]['id'], lock_secs=15)
assert prolonged_request['lockExpiresAt'] is not None
await queue.delete()
assert await apify_client_async.request_queue(created_queue['id']).get() is None
Loading