diff --git a/admin_manual/ai/ai_as_a_service.rst b/admin_manual/ai/ai_as_a_service.rst index b5aaed5e979..9612a724e98 100644 --- a/admin_manual/ai/ai_as_a_service.rst +++ b/admin_manual/ai/ai_as_a_service.rst @@ -17,6 +17,7 @@ In order to use these providers you will need to install the respective app from You can then add your API token and rate limits in the administration settings and set the providers live in the "Artificial intelligence" section of the admins settings. +Optionally but recommended, setup background workers for faster pickup of tasks. See :ref:`the relevant section in AI Overview` for more information. OpenAI integration ------------------ diff --git a/admin_manual/ai/app_assistant.rst b/admin_manual/ai/app_assistant.rst index ad790793227..21fe17f3076 100644 --- a/admin_manual/ai/app_assistant.rst +++ b/admin_manual/ai/app_assistant.rst @@ -84,7 +84,7 @@ Text-To-Image In order to make use of Text-To-Image features, you will need an app that provides an image generation backend: -* text2image_stablediffusion (Customer support available upon request) +* :ref:`tex2image_stablediffusion2` (Customer support available upon request) * *integration_openai* - Integrates with the OpenAI API to provide AI functionality from OpenAI servers (Customer support available upon request; see :ref:`AI as a Service`) * *integration_replicate* - Integrates with the replicate API to provide AI functionality from replicate servers (see :ref:`AI as a Service`) @@ -97,6 +97,17 @@ In order to make use of our special Context Chat feature, offering in-context in You will also need a text processing provider as specified above (ie. llm2 or integration_openai). +Context Agent +~~~~~~~~~~~~~ + +In order to make use of our AI agent feature, offering the execution of actions on behalf of the user based on the AI chat, you will need the following apps: + + +* :ref:`context_agent` - (Customer support available upon request) + + +You will also need a text processing provider as specified above (ie. *llm2* or *integration_openai*). + Configuration ------------- diff --git a/admin_manual/ai/app_context_agent.rst b/admin_manual/ai/app_context_agent.rst new file mode 100644 index 00000000000..e42460f0d67 --- /dev/null +++ b/admin_manual/ai/app_context_agent.rst @@ -0,0 +1,59 @@ +================================== +App: Context Agent (context_agent) +================================== + +.. _ai-app-context_agent: + +The *context_agent* app is the app that provides AI agent functionality in Nextcloud and acts as a backend for the :ref:`Nextcloud Assistant app`. Nextcloud can provide customer support upon request, please talk to your account manager for the possibilities. + +Requirements +------------ + +* This app is built as an External App and thus depends on AppAPI v3.1.0 or higher +* Nextcloud AIO is supported +* No GPU is necessary for Context Agent but one might be useful if you use it with a self-hosted provider like llm2 + +* CPU Sizing + + * At least 1GB of system RAM + +Installation +------------ + +0. Make sure the :ref:`Nextcloud Assistant app` is installed +1. :ref:`Install AppAPI and setup a Deploy Demon` +2. Install the "Context Agent" ExApp via the "Apps" page in the Nextcloud web admin user interface +3. Install a text generation backend like :ref:`llm2 ` or :ref:`integration_openai ` via the "Apps" page in Nextcloud + + +Model requirements +~~~~~~~~~~~~~~~~~~ + +This app requires underlying Large language models to support tool calling. The default model in *llm2* does *not* support tool calling. Instead we recommend: + +* Qwen 2.5 8B or higher +* Watt Tool 8B or higher + +See :ref:`llm2 documentation ` on how to configure alternate models. + +Scaling +------- + +It is currently not possible to scale this app, we are working on this. + +App store +--------- + +You can also find the app in our app store, where you can write a review: ``_ + +Repository +---------- + +You can find the app's code repository on GitHub where you can report bugs and contribute fixes and features: ``_ + +Nextcloud customers should file bugs directly with our Support system. + +Known Limitations +----------------- +* Make sure to test the language model you are using in concert with this app for whether they meet the use-case's quality requirements +* Customer support is available upon request, however we can't solve false or problematic output, most performance issues, or other problems caused by the underlying model. Support is thus limited only to bugs directly caused by the implementation of the app (connectors, API, front-end, AppAPI) \ No newline at end of file diff --git a/admin_manual/ai/app_llm2.rst b/admin_manual/ai/app_llm2.rst index 41a05b85e4e..4a7e55dd117 100644 --- a/admin_manual/ai/app_llm2.rst +++ b/admin_manual/ai/app_llm2.rst @@ -129,5 +129,4 @@ Known Limitations * Language models are likely to generate false information and should thus only be used in situations that are not critical. It's recommended to only use AI at the beginning of a creation process and not at the end, so that outputs of AI serve as a draft for example and not as final product. Always check the output of language models before using it. * Make sure to test the language model you are using it for whether it meets the use-case's quality requirements * Language models notoriously have a high energy consumption, if you want to reduce load on your server you can choose smaller models or quantized models in exchange for lower accuracy -* Customer support is available upon request, however we can't solve false or problematic output, most performance issues, or other problems caused by the underlying model. Support is thus limited only to bugs directly caused by the implementation of the app (connectors, API, front-end, AppAPI) -* Due to technical limitations that we are in the process of mitigating, each task currently incurs a time cost of between 0 and 5 minutes in addition to the actual processing time +* Customer support is available upon request, however we can't solve false or problematic output, most performance issues, or other problems caused by the underlying model. Support is thus limited only to bugs directly caused by the implementation of the app (connectors, API, front-end, AppAPI) \ No newline at end of file diff --git a/admin_manual/ai/app_text2image_stablediffusion2.rst b/admin_manual/ai/app_text2image_stablediffusion2.rst new file mode 100644 index 00000000000..d854d79703d --- /dev/null +++ b/admin_manual/ai/app_text2image_stablediffusion2.rst @@ -0,0 +1,60 @@ +========================================================= +App: Local Image Generation (text2image_stablediffusion2) +========================================================= + +.. _ai-app-text2image_stablediffusion2: + +The *text2image_stablediffusion2* app is one of the apps that provide image generation functionality in Nextcloud and act as an image generation backend for the :ref:`Nextcloud Assistant app` and other apps making use of the image generation functionality. The *text2image_stablediffusion2* app specifically runs only open source models and does so entirely on-premises. Nextcloud can provide customer support upon request, please talk to your account manager for the possibilities. + +Requirements +------------ + +* This app is built as an External App and thus depends on AppAPI v3.1.0 or higher +* Nextcloud AIO is supported +* We currently support NVIDIA GPUs and x86_64 CPUs +* CUDA >= v12.2 on your host system +* GPU Sizing + + * A NVIDIA GPU with at least 8GB VRAM + +* CPU Sizing + + * At least 8GB of system RAM + * The more cores you have and the more powerful the CPU the better, we recommend 10-20 cores + * The app will hog all cores by default, so it is usually better to run it on a separate machine + +Installation +------------ + +* Make sure the :ref:`Nextcloud Assistant app` is installed +* :ref:`Install AppAPI and setup a Deploy Demon` +* Install the "Local large language model" ExApp via the "Apps" page in the Nextcloud web admin user interface + + +Scaling +------- + +It is currently not possible to scale this app, we are working on this. Based on our calculations an instance has a rough capacity of 120 image requests per hour (each user request can be for multiple images). However, this number is based on theory and we do appreciate real-world feedback on this. + +App store +--------- + +You can also find the app in our app store, where you can write a review: ``_ + +Repository +---------- + +You can find the app's code repository on GitHub where you can report bugs and contribute fixes and features: ``_ + +Nextcloud customers should file bugs directly with our Support system. + +Known Limitations +----------------- + +* The generated images are of a fixed resolution (512x512 pix), and the model does not achieve perfect photorealism +* The model cannot render legible text +* Faces and people in general may not be generated properly +* The results for certain image generation requests can be biased and may enforce stereotypes +* We currently only support languages that the underlying model supports; correctness of language use in languages other than English may be poor depending on the language's coverage in the model's training data +* Make sure to test the app for whether it meets the use-case's quality requirements +* Customer support is available upon request, however we can't solve false or problematic output, most performance issues, or other problems caused by the underlying model. Support is thus limited only to bugs directly caused by the implementation of the app (connectors, API, front-end, AppAPI) diff --git a/admin_manual/ai/index.rst b/admin_manual/ai/index.rst index 802293c087d..26f0583ea7a 100644 --- a/admin_manual/ai/index.rst +++ b/admin_manual/ai/index.rst @@ -10,7 +10,9 @@ Artificial Intelligence app_translate2 app_llm2 app_stt_whisper2 + app_text2image_stablediffusion2 app_recognize app_context_chat + app_context_agent app_summary_bot ai_as_a_service diff --git a/developer_manual/client_apis/OCS/ocs-taskprocessing-api.rst b/developer_manual/client_apis/OCS/ocs-taskprocessing-api.rst index c8a485ac702..1c6ec37a6ad 100644 --- a/developer_manual/client_apis/OCS/ocs-taskprocessing-api.rst +++ b/developer_manual/client_apis/OCS/ocs-taskprocessing-api.rst @@ -8,182 +8,4 @@ OCS TaskProcessing API The OCS Task processing API allows you to run task processing tasks, like prompting large language models implemented by apps using :ref:`the backend Task Processing API`. -The base URL for all calls to this API is: ``/ocs/v2.php/taskprocessing/`` - -All calls to OCS endpoints require the ``OCS-APIRequest`` header to be set to ``true``. - - -Get available task types ------------------------- - -.. versionadded:: 30.0.0 - -* Method: ``GET`` -* Endpoint: ``/tasktypes`` -* Response: - - Status code: - + ``200 OK`` - - Data: - -+----------------------+--------+---------------------------------------------------------------------------------------------------------------+ -| field | type | Description | -+----------------------+--------+---------------------------------------------------------------------------------------------------------------+ -|``types`` | array | A map of supported task types. The keys are the task type IDs See below for the values. | -+----------------------+--------+---------------------------------------------------------------------------------------------------------------+ - -Task type fields: - -+-----------------------+--------+---------------------------------------------------------------------------------------------------------------+ -| field | type | Description | -+-----------------------+--------+---------------------------------------------------------------------------------------------------------------+ -|``name`` | string | The name of the task type in the user's language | -+-----------------------+--------+---------------------------------------------------------------------------------------------------------------+ -|``description`` | string | A description of the task type in the user's language | -+-----------------------+--------+---------------------------------------------------------------------------------------------------------------+ -|``inputShape`` | array | The input shape of this task type | -+-----------------------+--------+---------------------------------------------------------------------------------------------------------------+ -|``outputShape`` | array | The output shape of this task type | -+-----------------------+--------+---------------------------------------------------------------------------------------------------------------+ - -Input and output shape fields are maps from slot key to slot metadata: - -+----------------------+--------+---------------------------------------------------------------------------------------------------------------+ -| field | type | Description | -+----------------------+--------+---------------------------------------------------------------------------------------------------------------+ -|``name`` | string | The name of the I/O slot | -+----------------------+--------+---------------------------------------------------------------------------------------------------------------+ -|``description`` | string | A description of the I/O slot | -+----------------------+--------+---------------------------------------------------------------------------------------------------------------+ -|``type`` | int | The I/O slot type (See backend API for the available types) | -+----------------------+--------+---------------------------------------------------------------------------------------------------------------+ -|``mandatory`` | bool | Whether this slot is mandatory or not | -+----------------------+--------+---------------------------------------------------------------------------------------------------------------+ - -Schedule a task ---------------- - -.. versionadded:: 30.0.0 - -.. note:: The endpoint is rate limited as it can be quite resource intensive. Users can make 20 requests in 2 minutes, guests only 5 - -* Method: ``POST`` -* Endpoint: ``/schedule`` -* Data: - -+-----------------+-------------+--------------------------------------------------------------------------------+ -| field | type | Description | -+-----------------+-------------+--------------------------------------------------------------------------------+ -|``input`` | array | The input text for the task | -+-----------------+-------------+--------------------------------------------------------------------------------+ -|``type`` | string | Id of this task's type. | -+-----------------+-------------+--------------------------------------------------------------------------------+ -|``appId`` | string | The id of the requesting app | -+-----------------+-------------+--------------------------------------------------------------------------------+ -|``customId`` | string | An custom app-defined identifier for the task (optional) | -+-----------------+-------------+--------------------------------------------------------------------------------+ - -* Response: - - Status code: - + ``200 OK`` - + ``400 Bad Request`` - When the task type is invalid - + ``412 Precondition Failed`` - When the task type is not available currently - + ``401 Unauthenticated`` - When the input references a file that the user doesn't have access to - + ``429 Too Many Requests`` - When the rate limiting was exceeded - - - Data: - + ``input`` - Only provided in case of ``200 OK``, the task input, array - + ``type`` - Only provided in case of ``200 OK``, the task type, string - + ``id`` - Only provided in case of ``200 OK``, the assigned task id, int - + ``status`` - Only provided in case of ``200 OK``, the current task status, int, see backend API - + ``userId`` - Only provided in case of ``200 OK``, the originating userId of the task, string - + ``appId`` - Only provided in case of ``200 OK``, the originating appId of the task, string - + ``customId`` - Only provided in case of ``200 OK``, the custom id of the task, string - + ``output`` - Only provided in case of ``200 OK``, null - + ``message`` - Only provided when not ``200 OK``, an error message in the user's language, ready to be displayed - -Fetch a task by ID ------------------- - -.. versionadded:: 30.0.0 - -.. note:: The endpoint is rate limited as it can be quite resource intensive. Users can make 20 requests in 2 minutes, guests only 5 - -* Method: ``POST`` -* Endpoint: ``/task/{id}`` - -* Response: - - Status code: - + ``200 OK`` - + ``404 Not Found`` - When the task could not be found - - - Data: - + ``input`` - Only provided in case of ``200 OK``, the task input, array - + ``type`` - Only provided in case of ``200 OK``, the task type, string - + ``id`` - Only provided in case of ``200 OK``, the assigned task id, int - + ``status`` - Only provided in case of ``200 OK``, the current task status, int, see backend API - + ``userId`` - Only provided in case of ``200 OK``, the originating userId of the task, string - + ``appId`` - Only provided in case of ``200 OK``, the originating appId of the task, string - + ``customId`` - Only provided in case of ``200 OK``, the custom id of the task, string - + ``output`` - Only provided in case of ``200 OK``, the output from the model, array or null - + ``message`` - Only provided when not ``200 OK``, an error message in the user's language, ready to be displayed - - -Cancel a task -------------- - -.. versionadded:: 30.0.0 - -* Method: ``POST`` -* Endpoint: ``/task/{id}/cancel`` - -* Response: - - Status code: - + ``200 OK`` - + ``404 Not Found`` - When the task could not be found - - - Data: - - Data: - + ``input`` - Only provided in case of ``200 OK``, the task input, array - + ``type`` - Only provided in case of ``200 OK``, the task type, string - + ``id`` - Only provided in case of ``200 OK``, the assigned task id, int - + ``status`` - Only provided in case of ``200 OK``, the current task status, int, see backend API - + ``userId`` - Only provided in case of ``200 OK``, the originating userId of the task, string - + ``appId`` - Only provided in case of ``200 OK``, the originating appId of the task, string - + ``customId`` - Only provided in case of ``200 OK``, the custom id of the task, string - + ``output`` - Only provided in case of ``200 OK``, the output from the model, array or null - + ``message`` - Only provided when not ``200 OK``, an error message in the user's language, ready to be displayed - - -Delete a task -------------- - -.. versionadded:: 30.0.0 - -* Method: ``DELETE`` -* Endpoint: ``/task/{id}`` - -* Response: - - Status code: - + ``200 OK`` - + ``404 Not Found`` - When the task could not be found - - - Data: - + ``message`` - Only provided when not ``200 OK``, an error message in the user's language, ready to be displayed - - -Get task file contents ----------------------- - -.. versionadded:: 30.0.0 - -* Method: ``GET`` -* Endpoint: ``/tasks/{id}/file/{fileId}`` - -* Response: - - Status code: - + ``200 OK`` - + ``404 Not Found`` - When the task could not be found - - - Data: - + If ``200 OK`` this endpoint returns the raw data of the file - + ``message`` - Only provided when not ``200 OK``, an error message in the user's language, ready to be displayed +See the relevant `OpenAPI docs `_. \ No newline at end of file diff --git a/developer_manual/digging_deeper/task_processing.rst b/developer_manual/digging_deeper/task_processing.rst index 280c14735f5..d6267dcba96 100644 --- a/developer_manual/digging_deeper/task_processing.rst +++ b/developer_manual/digging_deeper/task_processing.rst @@ -42,11 +42,12 @@ The following built-in task types are available: * Input shape: * ``system_prompt``: ``Text`` * ``input``: ``Text`` - * ``history``: ``ListOfTexts`` - * ``tools``: ``Text`` The tools parameter should be a JSON object formatted according to the OpenAI API specifications: https://platform.openai.com/docs/api-reference/chat/create#chat-create-tools + * ``tool_message``: ``Text`` A string containing a JSON array of ``{"name": string, "content": string, "tool_call_id": string}`` + * ``history``: ``ListOfTexts`` Each list item is a JSON string with ``{"role": "human", "content": string}`` or ``{"role": "assistant", "content": string, (optional: "tool_calls": array<{"name": string, "type": "tool_call", "id": string, "args": object}>)}`` or ``{"role": "tool", "content": string, "name": string, "tool_call_id": string}`` + * ``tools``: ``Text`` The tools parameter should be a JSON array formatted according to the OpenAI API specifications: https://platform.openai.com/docs/api-reference/chat/create#chat-create-tools * Output shape: * ``output``: ``Text`` - * ``tool_calls``: ``Text`` + * ``tool_calls``: ``Text`` A string containing a JSON array with ``{"name": string, "type": "tool_call", "id": string, "args": object}`` * ``'core:contextagent:interaction'``: This task allows chatting with an agent. It is implemented by ``\OCP\TaskProcessing\TaskTypes\ContextAgentInteraction`` * Input shape: * ``input``: ``Text``