Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Collections Datastore #2620

Draft
wants to merge 8 commits into
base: main
Choose a base branch
from
Draft

Collections Datastore #2620

wants to merge 8 commits into from

Commits on Oct 22, 2024

  1. Add workflow collections (#2569)

    * Add workflow collections
    
    These are named and shared key-value storages
    
    * Increase timestamp precision
    
    * Use single return patter on same get function
    
    * Optimize all ops once the names are stable
    
    There is no user update on collection name
    
    * Changelog and formatting
    
    * Add stream_all allowing equivalent queries to HGETALL and HSCAN
    
    * Add stream_match for wildcard prefix queries
    
    * Add pg_trigram and GIN index on the key to allow multi wildcard
    jyeshe committed Oct 22, 2024
    Configuration menu
    Copy the full SHA
    fd14f8f View commit details
    Browse the repository at this point in the history
  2. Sort by ascending upsert timestamp

    jyeshe authored and stuartc committed Oct 22, 2024
    Configuration menu
    Copy the full SHA
    30e08b5 View commit details
    Browse the repository at this point in the history
  3. Use only unique constraint (required and non-relational to save a BTREE)

    Also changes timestamp type
    jyeshe authored and stuartc committed Oct 22, 2024
    Configuration menu
    Copy the full SHA
    6b22d9a View commit details
    Browse the repository at this point in the history

Commits on Oct 23, 2024

  1. Configuration menu
    Copy the full SHA
    9c7cf60 View commit details
    Browse the repository at this point in the history

Commits on Oct 27, 2024

  1. Configuration menu
    Copy the full SHA
    31775b0 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    98f23b4 View commit details
    Browse the repository at this point in the history
  3. Collections CRUD UI (#2587)

    * Collections CRUD UI
    
    * Mix verify
    
    * Code readability issue
    
    * Render raw_name errors
    
    * Sorting by name
    
    * Tests for copy_error/4
    
    * Tests for url_safe_name/1
    
    * Tests for list_collections/1, create_collection/1, and update_collection/2
    
    * Format aliases
    
    * Tests for collection live
    
    * Test cancel collection creation modal
    
    * Collection tests
    
    * Update CHANGELOG
    elias-ba authored Oct 27, 2024
    Configuration menu
    Copy the full SHA
    6edc03e View commit details
    Browse the repository at this point in the history

Commits on Oct 30, 2024

  1. Collections API (#2603)

    * streaming collections api WIP
    
    * Refactoring UserTokens
    
    - Add sub claim to UserToken
    - Abstract RSA key generation
    
    This change reduces the requirement to shell out to `openssl` to
    generate RSA key pairs, in addition the functions have been moved into
    a common `Utils.Crypto` module so the functions can be used elsewhere.
    
    - Add common token_signer for new api tokens
    - Move UserToken for API auth to Tokens.PersonalAccessToken
    - Workers.Token -> Workers.WorkerToken
    
    * Conform RunToken generation to use RunOptions
    
    * Authentication and authorization for Collections API
    
    * Streaming response for stream_all
    
    - Added `has_many :items` to Collection model allowing easy creation of
      items during tests.
    - Added Jason.Encoder protocol to Collections.Item for serializing.
    - Collections controller sends through a streaming response for "all".
    - Conform FallbackControlly and templates to all (and optionally)
      support `@error`
    
    **NOTE** The controller has a few things to note:
    
    - It does not conform to the entire spec, as only one route so far.
    - The shape of the response is currently _only_ an array and not an
      object with `values` and `metadata`.
    - There is some cleanup and refactoring required with the authentication
      etc, for now there is a `action/2` callback where all the resulting
      plugs and pipelines will be built from.
    
    * Collections API with cursor (#2616)
    
    * Implements collections REST api (including streaming)
    
    commit 810228e
    Author: Rogerio Pontual <[email protected]>
    Date:   Tue Oct 29 17:07:47 2024 +0100
    
        Fix POST to work with multiple {key: key, value: value}
    
    commit 4321f4f
    Author: Rogerio Pontual <[email protected]>
    Date:   Tue Oct 29 15:55:45 2024 +0100
    
        Implements remaining http verbs
    
        - post, put, get and delete
    
    commit de1923a
    Author: Rogerio Pontual <[email protected]>
    Date:   Tue Oct 29 08:25:20 2024 +0100
    
        Streaming to network (local) while streaming from database
    
        - Additionally uses cursor and limit
    
    commit 2bc0d94
    Author: Stuart Corbishley <[email protected]>
    Date:   Sun Oct 27 13:16:33 2024 +0200
    
        Streaming response for stream_all
    
        - Added `has_many :items` to Collection model allowing easy creation of
          items during tests.
        - Added Jason.Encoder protocol to Collections.Item for serializing.
        - Collections controller sends through a streaming response for "all".
        - Conform FallbackControlly and templates to all (and optionally)
          support `@error`
    
        **NOTE** The controller has a few things to note:
    
        - It does not conform to the entire spec, as only one route so far.
        - The shape of the response is currently _only_ an array and not an
          object with `values` and `metadata`.
        - There is some cleanup and refactoring required with the authentication
          etc, for now there is a `action/2` callback where all the resulting
          plugs and pipelines will be built from.
    
    commit cab0565
    Author: Stuart Corbishley <[email protected]>
    Date:   Thu Oct 24 12:00:29 2024 +0200
    
        Authentication and authorization for Collections API
    
    commit 3d7f511
    Author: Stuart Corbishley <[email protected]>
    Date:   Tue Oct 22 14:28:32 2024 +0200
    
        Conform RunToken generation to use RunOptions
    
    commit 9eae9e6
    Author: Stuart Corbishley <[email protected]>
    Date:   Tue Oct 15 13:01:19 2024 +0200
    
        Refactoring UserTokens
    
        - Add sub claim to UserToken
        - Abstract RSA key generation
    
        This change reduces the requirement to shell out to `openssl` to
        generate RSA key pairs, in addition the functions have been moved into
        a common `Utils.Crypto` module so the functions can be used elsewhere.
    
        - Add common token_signer for new api tokens
        - Move UserToken for API auth to Tokens.PersonalAccessToken
        - Workers.Token -> Workers.WorkerToken
    
    commit 03e6a32
    Author: Stuart Corbishley <[email protected]>
    Date:   Wed Oct 9 08:26:59 2024 +0200
    
        streaming collections api WIP
    
    commit 6edc03e
    Author: Elias W. BA <[email protected]>
    Date:   Sun Oct 27 12:17:08 2024 +0000
    
        Collections CRUD UI (#2587)
    
        * Collections CRUD UI
    
        * Mix verify
    
        * Code readability issue
    
        * Render raw_name errors
    
        * Sorting by name
    
        * Tests for copy_error/4
    
        * Tests for url_safe_name/1
    
        * Tests for list_collections/1, create_collection/1, and update_collection/2
    
        * Format aliases
    
        * Tests for collection live
    
        * Test cancel collection creation modal
    
        * Collection tests
    
        * Update CHANGELOG
    
    * Add records with created_at timestamps and unique updated_at
    
    * Formatting responses
    
    ---------
    
    Co-authored-by: Rogerio Pontual <[email protected]>
    stuartc and jyeshe authored Oct 30, 2024
    Configuration menu
    Copy the full SHA
    8255a5e View commit details
    Browse the repository at this point in the history