Clean up the sdk a bit #51
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jsun-m
pushed a commit
that referenced
this pull request
Jan 26, 2024
* clean up sdk a bit * wip * add go tests to ci * add tests badge * wip * move default dir * remove github tokens * remove github tokens --------- Co-authored-by: Luke Lombardi <[email protected]>
jsun-m
added a commit
that referenced
this pull request
Jan 29, 2024
* layout (#1) * start laying out adapter interface * move to map * started implementing infinifs paper * remove old old * refactor storage * add gateway * rename all references to activator * remove backwards compatibility * add in queue * rename types * more renaming * add app/task * test out golang plugins * add worker exec * testing dynamic code execution * more complex function * cleanup --------- Co-authored-by: Luke Lombardi <[email protected]> Co-authored-by: luke lombardi <[email protected]> * rename pkg to internal * wip * GRPC refactor pt1 (#2) * refactor grpc organization for abstractions * separate map svc endpoints out * remove old code * simplify organization * remove more code * connect redis client * add more service methods * add storage folder * clean up interface * forgot keys * push up full map implementation --------- Co-authored-by: Luke Lombardi <[email protected]> * rename var * fix typo * cleanup, add clang formatting to proto files (#3) * add some more primitives * remove imageservice rpc * configure autoformatting for proto files --------- Co-authored-by: Luke Lombardi <[email protected]> * storage (#4) * add juicefs to dockerfile * add juicefs config * add juicefs format method * integrate storage * remove comment --------- Co-authored-by: Luke Lombardi <[email protected]> * image service v2 (#5) * pull in imageservice code. unsure of what to do with it * move shit around * rename volume * remove runner type * remove plugins * remove plugins * refactor image service rpc * register image service * update protobuf version * remove mode from container request * remove identity quota tracking * remove identity field * add back worker build target * separate worker and beam images * fix keys * remove print * code removal * prototyping something for interactivity * add execute method * add runner dockerfiles * add runner target * fix build * remove build service logic * consolidate image clients * rename server * add container server addr * clean up error, add runc_client * start setting up runc client * implement more runc methods * move unpacking over into image client * move client, allow server to communicate with gateway * hardcode image * implement safemap * rename var * pass container instances around w/ safemap; * sending and aggregating logs! * try actually execing things * move archive * remove container mode * make filesystem r/w * comment out tmp dirs * add log buffer * implement log streaming endpoint * streaming logs all the way to client * logs streaming!! * pass correct image tag * fix error handling * wip * remove more old code * stop hardcoding * fix skopeo build * messing with args * wip * remove some more code * remove an arg * fix up skopeo build some more * refactor image building scheme * fix tests * lots more cleanup * remove code * keep cleaning up * remove pointer * feeling very very dumb right now * fix secrets template * fix tag * remove more code * actually archive the image * add archive service * remove build container * add pip install * temporarily remove identity * actually archive * remove sleeps, add images volume * cleanup * check packages * rename a couple things * more renames --------- Co-authored-by: Luke Lombardi <[email protected]> * more cleanup (#6) * remove some old code * remove request bucket repo * cleanup * add l * turn off reflection * remove more shit --------- Co-authored-by: Luke Lombardi <[email protected]> * Add sdk (#7) * add beam sdk * remove deprecated code * fix proto gen * remove bucket * remove license file --------- Co-authored-by: Luke Lombardi <[email protected]> * object syncing pt 1 (#8) * add object sync rpc endpoint * remove bucket abstraction temporarily * add gateway service * add more endpoints * add object path * remove fields * wip --------- Co-authored-by: Luke Lombardi <[email protected]> * Object sync pt 2 (#9) * remove put and extract object * finish sync endpoints * wip * return sync result * fix type --------- Co-authored-by: Luke Lombardi <[email protected]> * function abstraction (#10) * use sync result in a more realistic way * add callable wrapper; * add map placeholder * moving things around * remove arg * wip * wip * wip * trying to get invoke working * more cleanup * add grpc server side * invoking the shit * store args * store invocation id * update args * return real args * better build config * wip * sort of lay out scheduling * found a bug in image archiving * override root; * deal with rootfs in a hacky way * fix tmp bug * add sdk source into worker * update base image reqs * add rich * try defining external entry point * dont copy src every time * stream logs * args work! * wip * wip * try mounting storage * install juicefs into worker * push channel into abc; * add aio module * fix types * remove more code * cleanup types * add context * add a context * unzip * something wrong with the args * fix args * remove comments * clean up configs * always set container exit code * add set result * set result * add compute config * we know when it ends now * get function result * get it back * remove arg * dont print empty strings * make pythonunbuffered global * log err * fix error * clean up mounts * handle every exception * add map * k3d, updates, and fixes * rename init and add setup make targets * tidy * update port to 6000 * Revert "update port to 6000" This reverts commit f9e240a. * fix lock filename * add base requirements, remove lockfile * switch from arch64 -> amd64 * switch back beam and beam-worker to x86 * fix mod? --------- Co-authored-by: Luke Lombardi <[email protected]> Co-authored-by: Nick Petrovic <[email protected]> Co-authored-by: Nick Petrovic <[email protected]> * fix function map (#13) * fix function map * pull out function --------- Co-authored-by: Luke Lombardi <[email protected]> * turn on postgres in k3d (#14) Co-authored-by: Luke Lombardi <[email protected]> * Queue abstraction * Queue abstraction and renamed map * add custom exception * Rename simple queue to queue on import in __init__.py * flake and fixing few more things * integrate postgres db and migrations using sqlx (#16) * add backend repository * try using the new sqlx connection * renaming things * add migration process * remove call * wip * remove comment * remove old structs * forward postgres service to local * add models * add deployment status enum * update protobuf version * update models * make hash and name unique * add more realistic models * update names * fix fks * fix migration order --------- Co-authored-by: Luke Lombardi <[email protected]> * refactor queue function names (#18) * refactor * rebuild proto * fix merge issues * GRPC Authentication (#20) * add configure endpoint * working on auth * add docs placeholder * create authenticated channel class * got a token * handle streams too * add authinterceptor struct * wip * add auth package * wip * cleaning things up * locking shit down * rename config file * wip * create new tokens * update msg * create token * pass tokens * remove build files * remove egg --------- Co-authored-by: Luke Lombardi <[email protected]> * rename url -> host * fix token storage * remove check for auth since its redundant * Misc. cleanup (#21) * remove some old identity related code * remove proto file * add event bus back to gateway, move http auth * remove beam repo * more cleanup * remove paths --------- Co-authored-by: Luke Lombardi <[email protected]> * Function task tracking and refactoring (#22) * rename invocation_id to task_id * remove unused env vars * cleaning some things up * add container id to task * tasks curd * track objects in db * remove some debug code * fix task id * move object to seprate module * wip * add stub config version field * fix field * add a stub creation func * refactor storage instantiation * pass repo instead of whole struct * move task services over to gateway * remove constants file * get or create the stub * able to create and update tasks * more cleanup * remove mock data * mostly track the tasks, still needs some work * comments * fix unpacking race conditions, create stub once * add task id to invoke response * handle some errors * remove types * pickle exceptions * store task status * remove comment --------- Co-authored-by: Luke Lombardi <[email protected]> * update contexts * add real container hostname to runner env * add cli framework and configure command (#19) * add configure endpoint * working on auth * add docs placeholder * create authenticated channel class * got a token * handle streams too * add cli framework and configure command * add authinterceptor struct * wip * add auth package * Add gateway url andport * git url * wip * cleaning things up * locking shit down * save to file * clean * refactored config code * remove something/ * test * fixed --------- Co-authored-by: Luke Lombardi <[email protected]> * fix config process * rename context to workspace (#23) * rename context to workspace * missed a few spots * fix few missing spots --------- Co-authored-by: Luke Lombardi <[email protected]> * Add worker request requirements override, update k8s, remove k3d agent (#24) * SDK: Add tasks subcommand (#25) * Return empty string when date is zero (#27) * SDK: Add filter option to beam tasks list command (#29) * volumes abstraction (#28) * add volumes * cleanup * Volume grpc class * spacE * update for pr * initialize volumes with [] ' * prints removal and name updates * global volume service * remove commented out code * reraise error more elegantly * task queues and deployment (#26) * scaffolding deployment * remove deployment status * structuring the task queue, kinda trash * messing with some struct fields * still doesnt even build... * fix proto files * wip * move container methods to repo * make it build again * remove task repo * more cleanup * refactor keys * laying out sdk * refactoring task queue server * add backend repo * wip; * pushing something... * no args... * remove args * create a real task * hook up an autoscaler... * make a little more realistic...? * still messed up * remove unnecessary args * print state for debugging * refactor runtime/stub logic to be consistent * more cleanup * fix instance lock key * make taskqueues unload themselves automatically when not in use * stupid json serialization * wip * cleanup * more and more cleanup * wip * remove commented out code * rename non-public vars * add stoppable containers back * wip * start pulling over task retry logic * more cleanup, task monitoring * wip * pass all the stub data * add scale down delay * pass context properly * make config a classmethod * use real failed containers val * rename scale down delay to keep warm seconds * remove unused func * real task policy * remove comment * fixing stub config * fix join * remove last print * try to fix merge confllicts * more cleanup * add back print format exc * one more bugfix * wip * Fix volumes (#30) * merge * fixed volumes * cleanup? --------- Co-authored-by: Luke Lombardi <[email protected]> Co-authored-by: jsun-m <[email protected]> * push load handler functionality into runner abstraction (#31) * push load handler functionality into runner abstraction * fix the func --------- Co-authored-by: Luke Lombardi <[email protected]> * add container id and more task logs (#32) Co-authored-by: Luke Lombardi <[email protected]> * Remove deprecated code (#33) * move base abstractions into a subdir * remove unused event bus * remove old code * comment out broken code * remove container config response * create queue instance if not exists --------- Co-authored-by: Luke Lombardi <[email protected]> * Setup echo http server (#34) * remove gorm * set up echo health check group * add additional service port * setup basic routes * fix comment * tidy go.mod * wip --------- Co-authored-by: Luke Lombardi <[email protected]> * add auth middleware to http server (#35) * add auth middleware * fix context value * updated pr to show usage * fix route --------- Co-authored-by: Luke Lombardi <[email protected]> * added license and readme (#37) * added license and readme * Update README.md * Abstraction http interface (#39) * wip * connect a route * sort of implement task queue http endpoints * cleanup * rename volume svc * wip * wip * return invalid stub id err * remove fvolume type --------- Co-authored-by: Luke Lombardi <[email protected]> * enforce service interfaces (#40) Co-authored-by: Luke Lombardi <[email protected]> * sdk test cases (#36) * sdk test cases 1 * add task queue tests * Add tests for function * Add test for function map * add comment explanation * change function to use mock_coroutine_with_result * updated for review * add github actions * patch actions * poetry * poetry * forgot ot checkut * fix ruff * fix ruff * fix ruff * last step * ignore auth for ci * ignore auth for ci * Basic taskqueue deployment (#42) * wip deployment mods * fix stub types * hide echo banner * hide port * invoke the deployment grpc service * actually creating deployment objects now * deployment works e2e * wip * cleanup * cleanup * more cleanup * make it an err --------- Co-authored-by: Luke Lombardi <[email protected]> * update deployment model to include stub_type (#43) Co-authored-by: Luke Lombardi <[email protected]> * New app config (#41) * remove beam repo, gorm, and old db models (#44) Co-authored-by: Luke Lombardi <[email protected]> * Add function deployments (#45) * refactor invoke to support function deployments * works except for arg passing * fix task id * change order of operations * wip * support both json and cloudpickle for functions * remove cache files * add containers banner --------- Co-authored-by: Luke Lombardi <[email protected]> * Namespace more stuff by workspaceName (#46) * namespace more stuff by workspace name * namespace everything by workspace --------- Co-authored-by: Luke Lombardi <[email protected]> * fix some bugs with simplequeue (that I created); add lock for mapset (#47) * fix some bugs; add set lock * fix tests --------- Co-authored-by: Luke Lombardi <[email protected]> * SDK docstrings (#48) * start adding docstrings * add docstring for map * wip * fix tests * wip * moving things around * add function docstrings * wip * validate cpu and parse it * update cpu values * fix cpu value * wip * fix some stuff * wip * create factories for function/taskqueue * remove extra symbol * add test case for cpu parsing --------- Co-authored-by: Luke Lombardi <[email protected]> * clean up docstrings (#49) * start adding docstrings * wip * moving things around * add function docstrings * wip * validate cpu and parse it * wip * create factories for function/taskqueue * clean up docstrings * remove extra symbol * update more docstrings * wip * fix import --------- Co-authored-by: Luke Lombardi <[email protected]> * More docs changes (#50) * more README changes * update intro --------- Co-authored-by: Luke Lombardi <[email protected]> * Clean up the sdk a bit (#51) * clean up sdk a bit * wip * add go tests to ci * add tests badge * wip * move default dir * remove github tokens * remove github tokens --------- Co-authored-by: Luke Lombardi <[email protected]> * fix ci test badge (#52) Co-authored-by: Luke Lombardi <[email protected]> * Change registry host port to 5001, add label to worker (#53) * Add scaffold for web endpoint & fix some bugs (#54) * layout webendpoint * wip * dont error out if we cant find the exit code * add proxy * remove proxy --------- Co-authored-by: Luke Lombardi <[email protected]> * Add metrics serve * remove charts * cleanup * remove charts * prototype logs sscraping * Fix GPU support with k3d (#59) * Build GPU docker image on linux if nvidia-smi exists (#61) * AWS K3S Deployment (#56) * setup headscale in k3d * remove messed up svc config * fix svc * fix ports * actually fix port * expose headscale too * update server url * tidy * add some terraform stuff * refactor subnets * move domain and hosted zone to vars * wip * fix nginx-ingress-controller * force nginx class to be default * add redis and postgres back in * remove old stuff * wip * more cleanup * configure wireguard/update ami * update security group replacement, install nginx ingress controller * refactor into modules * remove lb controller json * tag subnets * change provider id * use correct chart * fix indent * pin eks chart version * update lb name * remove comment * got ssl working * wip * wip * move to aws-dev * rename buckets * manage root password * add backend repo back in * remove data volume * refactor variables * add redis secret in * populate config, create secret * wip * fix auth, bump redis chart version * fix juicefs bucket * fix rds security group * remove some redundant names * remove spaces * fix redundant gitignore * add depends * make instance type and worker count configurable * pin k3s version * make bucket perms more restrictive * fix vare * less permissive policies --------- Co-authored-by: Luke Lombardi <[email protected]> * Rename project to beta9 (#62) * rename to beta9 * remove todo * fix gateway okteto --------- Co-authored-by: Luke Lombardi <[email protected]> * setup victoria metrics * remove some defaults (#63) Co-authored-by: Luke Lombardi <[email protected]> * verified * Add gRPC msg size, better image name parsing, update a label (#64) * Update readme (#65) * Init ignore file when one doesn't exist (#66) * Improve request concurrency (#67) * increased max postgres connections (#68) * increased max postgres connections * remove period * add comment --------- Co-authored-by: Luke Lombardi <[email protected]> * Add metrics server (#57) * Add metrics serve * remove charts * cleanup * remove charts * setup victoria metrics * verified * finished setup * cleanup * cleanup * make async * cleanup for pr * cleanup for pr * clean up comments and changed to TODOs * finished k3d setup * add more examples * add an examples folder * added in loki host to fluentbit * Add namespace to secrets * remove kinesis from localstack * merged fluentbit examples to 1 file --------- Co-authored-by: luke-lombardi <[email protected]> Co-authored-by: Luke Lombardi <[email protected]> Co-authored-by: luke lombardi <[email protected]> Co-authored-by: Luke Lombardi <[email protected]> Co-authored-by: Nick Petrovic <[email protected]> Co-authored-by: Nick Petrovic <[email protected]> Co-authored-by: Eli Mernit <[email protected]>
jsun-m
added a commit
that referenced
this pull request
Jan 30, 2024
* layout (#1) * start laying out adapter interface * move to map * started implementing infinifs paper * remove old old * refactor storage * add gateway * rename all references to activator * remove backwards compatibility * add in queue * rename types * more renaming * add app/task * test out golang plugins * add worker exec * testing dynamic code execution * more complex function * cleanup --------- Co-authored-by: Luke Lombardi <[email protected]> Co-authored-by: luke lombardi <[email protected]> * rename pkg to internal * wip * GRPC refactor pt1 (#2) * refactor grpc organization for abstractions * separate map svc endpoints out * remove old code * simplify organization * remove more code * connect redis client * add more service methods * add storage folder * clean up interface * forgot keys * push up full map implementation --------- Co-authored-by: Luke Lombardi <[email protected]> * rename var * fix typo * cleanup, add clang formatting to proto files (#3) * add some more primitives * remove imageservice rpc * configure autoformatting for proto files --------- Co-authored-by: Luke Lombardi <[email protected]> * storage (#4) * add juicefs to dockerfile * add juicefs config * add juicefs format method * integrate storage * remove comment --------- Co-authored-by: Luke Lombardi <[email protected]> * image service v2 (#5) * pull in imageservice code. unsure of what to do with it * move shit around * rename volume * remove runner type * remove plugins * remove plugins * refactor image service rpc * register image service * update protobuf version * remove mode from container request * remove identity quota tracking * remove identity field * add back worker build target * separate worker and beam images * fix keys * remove print * code removal * prototyping something for interactivity * add execute method * add runner dockerfiles * add runner target * fix build * remove build service logic * consolidate image clients * rename server * add container server addr * clean up error, add runc_client * start setting up runc client * implement more runc methods * move unpacking over into image client * move client, allow server to communicate with gateway * hardcode image * implement safemap * rename var * pass container instances around w/ safemap; * sending and aggregating logs! * try actually execing things * move archive * remove container mode * make filesystem r/w * comment out tmp dirs * add log buffer * implement log streaming endpoint * streaming logs all the way to client * logs streaming!! * pass correct image tag * fix error handling * wip * remove more old code * stop hardcoding * fix skopeo build * messing with args * wip * remove some more code * remove an arg * fix up skopeo build some more * refactor image building scheme * fix tests * lots more cleanup * remove code * keep cleaning up * remove pointer * feeling very very dumb right now * fix secrets template * fix tag * remove more code * actually archive the image * add archive service * remove build container * add pip install * temporarily remove identity * actually archive * remove sleeps, add images volume * cleanup * check packages * rename a couple things * more renames --------- Co-authored-by: Luke Lombardi <[email protected]> * more cleanup (#6) * remove some old code * remove request bucket repo * cleanup * add l * turn off reflection * remove more shit --------- Co-authored-by: Luke Lombardi <[email protected]> * Add sdk (#7) * add beam sdk * remove deprecated code * fix proto gen * remove bucket * remove license file --------- Co-authored-by: Luke Lombardi <[email protected]> * object syncing pt 1 (#8) * add object sync rpc endpoint * remove bucket abstraction temporarily * add gateway service * add more endpoints * add object path * remove fields * wip --------- Co-authored-by: Luke Lombardi <[email protected]> * Object sync pt 2 (#9) * remove put and extract object * finish sync endpoints * wip * return sync result * fix type --------- Co-authored-by: Luke Lombardi <[email protected]> * function abstraction (#10) * use sync result in a more realistic way * add callable wrapper; * add map placeholder * moving things around * remove arg * wip * wip * wip * trying to get invoke working * more cleanup * add grpc server side * invoking the shit * store args * store invocation id * update args * return real args * better build config * wip * sort of lay out scheduling * found a bug in image archiving * override root; * deal with rootfs in a hacky way * fix tmp bug * add sdk source into worker * update base image reqs * add rich * try defining external entry point * dont copy src every time * stream logs * args work! * wip * wip * try mounting storage * install juicefs into worker * push channel into abc; * add aio module * fix types * remove more code * cleanup types * add context * add a context * unzip * something wrong with the args * fix args * remove comments * clean up configs * always set container exit code * add set result * set result * add compute config * we know when it ends now * get function result * get it back * remove arg * dont print empty strings * make pythonunbuffered global * log err * fix error * clean up mounts * handle every exception * add map * k3d, updates, and fixes * rename init and add setup make targets * tidy * update port to 6000 * Revert "update port to 6000" This reverts commit f9e240a. * fix lock filename * add base requirements, remove lockfile * switch from arch64 -> amd64 * switch back beam and beam-worker to x86 * fix mod? --------- Co-authored-by: Luke Lombardi <[email protected]> Co-authored-by: Nick Petrovic <[email protected]> Co-authored-by: Nick Petrovic <[email protected]> * fix function map (#13) * fix function map * pull out function --------- Co-authored-by: Luke Lombardi <[email protected]> * turn on postgres in k3d (#14) Co-authored-by: Luke Lombardi <[email protected]> * Queue abstraction * Queue abstraction and renamed map * add custom exception * Rename simple queue to queue on import in __init__.py * flake and fixing few more things * integrate postgres db and migrations using sqlx (#16) * add backend repository * try using the new sqlx connection * renaming things * add migration process * remove call * wip * remove comment * remove old structs * forward postgres service to local * add models * add deployment status enum * update protobuf version * update models * make hash and name unique * add more realistic models * update names * fix fks * fix migration order --------- Co-authored-by: Luke Lombardi <[email protected]> * refactor queue function names (#18) * refactor * rebuild proto * fix merge issues * GRPC Authentication (#20) * add configure endpoint * working on auth * add docs placeholder * create authenticated channel class * got a token * handle streams too * add authinterceptor struct * wip * add auth package * wip * cleaning things up * locking shit down * rename config file * wip * create new tokens * update msg * create token * pass tokens * remove build files * remove egg --------- Co-authored-by: Luke Lombardi <[email protected]> * rename url -> host * fix token storage * remove check for auth since its redundant * Misc. cleanup (#21) * remove some old identity related code * remove proto file * add event bus back to gateway, move http auth * remove beam repo * more cleanup * remove paths --------- Co-authored-by: Luke Lombardi <[email protected]> * Function task tracking and refactoring (#22) * rename invocation_id to task_id * remove unused env vars * cleaning some things up * add container id to task * tasks curd * track objects in db * remove some debug code * fix task id * move object to seprate module * wip * add stub config version field * fix field * add a stub creation func * refactor storage instantiation * pass repo instead of whole struct * move task services over to gateway * remove constants file * get or create the stub * able to create and update tasks * more cleanup * remove mock data * mostly track the tasks, still needs some work * comments * fix unpacking race conditions, create stub once * add task id to invoke response * handle some errors * remove types * pickle exceptions * store task status * remove comment --------- Co-authored-by: Luke Lombardi <[email protected]> * update contexts * add real container hostname to runner env * add cli framework and configure command (#19) * add configure endpoint * working on auth * add docs placeholder * create authenticated channel class * got a token * handle streams too * add cli framework and configure command * add authinterceptor struct * wip * add auth package * Add gateway url andport * git url * wip * cleaning things up * locking shit down * save to file * clean * refactored config code * remove something/ * test * fixed --------- Co-authored-by: Luke Lombardi <[email protected]> * fix config process * rename context to workspace (#23) * rename context to workspace * missed a few spots * fix few missing spots --------- Co-authored-by: Luke Lombardi <[email protected]> * Add worker request requirements override, update k8s, remove k3d agent (#24) * SDK: Add tasks subcommand (#25) * Return empty string when date is zero (#27) * SDK: Add filter option to beam tasks list command (#29) * volumes abstraction (#28) * add volumes * cleanup * Volume grpc class * spacE * update for pr * initialize volumes with [] ' * prints removal and name updates * global volume service * remove commented out code * reraise error more elegantly * task queues and deployment (#26) * scaffolding deployment * remove deployment status * structuring the task queue, kinda trash * messing with some struct fields * still doesnt even build... * fix proto files * wip * move container methods to repo * make it build again * remove task repo * more cleanup * refactor keys * laying out sdk * refactoring task queue server * add backend repo * wip; * pushing something... * no args... * remove args * create a real task * hook up an autoscaler... * make a little more realistic...? * still messed up * remove unnecessary args * print state for debugging * refactor runtime/stub logic to be consistent * more cleanup * fix instance lock key * make taskqueues unload themselves automatically when not in use * stupid json serialization * wip * cleanup * more and more cleanup * wip * remove commented out code * rename non-public vars * add stoppable containers back * wip * start pulling over task retry logic * more cleanup, task monitoring * wip * pass all the stub data * add scale down delay * pass context properly * make config a classmethod * use real failed containers val * rename scale down delay to keep warm seconds * remove unused func * real task policy * remove comment * fixing stub config * fix join * remove last print * try to fix merge confllicts * more cleanup * add back print format exc * one more bugfix * wip * Fix volumes (#30) * merge * fixed volumes * cleanup? --------- Co-authored-by: Luke Lombardi <[email protected]> Co-authored-by: jsun-m <[email protected]> * push load handler functionality into runner abstraction (#31) * push load handler functionality into runner abstraction * fix the func --------- Co-authored-by: Luke Lombardi <[email protected]> * add container id and more task logs (#32) Co-authored-by: Luke Lombardi <[email protected]> * Remove deprecated code (#33) * move base abstractions into a subdir * remove unused event bus * remove old code * comment out broken code * remove container config response * create queue instance if not exists --------- Co-authored-by: Luke Lombardi <[email protected]> * Setup echo http server (#34) * remove gorm * set up echo health check group * add additional service port * setup basic routes * fix comment * tidy go.mod * wip --------- Co-authored-by: Luke Lombardi <[email protected]> * add auth middleware to http server (#35) * add auth middleware * fix context value * updated pr to show usage * fix route --------- Co-authored-by: Luke Lombardi <[email protected]> * added license and readme (#37) * added license and readme * Update README.md * Abstraction http interface (#39) * wip * connect a route * sort of implement task queue http endpoints * cleanup * rename volume svc * wip * wip * return invalid stub id err * remove fvolume type --------- Co-authored-by: Luke Lombardi <[email protected]> * enforce service interfaces (#40) Co-authored-by: Luke Lombardi <[email protected]> * sdk test cases (#36) * sdk test cases 1 * add task queue tests * Add tests for function * Add test for function map * add comment explanation * change function to use mock_coroutine_with_result * updated for review * add github actions * patch actions * poetry * poetry * forgot ot checkut * fix ruff * fix ruff * fix ruff * last step * ignore auth for ci * ignore auth for ci * Basic taskqueue deployment (#42) * wip deployment mods * fix stub types * hide echo banner * hide port * invoke the deployment grpc service * actually creating deployment objects now * deployment works e2e * wip * cleanup * cleanup * more cleanup * make it an err --------- Co-authored-by: Luke Lombardi <[email protected]> * update deployment model to include stub_type (#43) Co-authored-by: Luke Lombardi <[email protected]> * New app config (#41) * remove beam repo, gorm, and old db models (#44) Co-authored-by: Luke Lombardi <[email protected]> * Add function deployments (#45) * refactor invoke to support function deployments * works except for arg passing * fix task id * change order of operations * wip * support both json and cloudpickle for functions * remove cache files * add containers banner --------- Co-authored-by: Luke Lombardi <[email protected]> * Namespace more stuff by workspaceName (#46) * namespace more stuff by workspace name * namespace everything by workspace --------- Co-authored-by: Luke Lombardi <[email protected]> * fix some bugs with simplequeue (that I created); add lock for mapset (#47) * fix some bugs; add set lock * fix tests --------- Co-authored-by: Luke Lombardi <[email protected]> * SDK docstrings (#48) * start adding docstrings * add docstring for map * wip * fix tests * wip * moving things around * add function docstrings * wip * validate cpu and parse it * update cpu values * fix cpu value * wip * fix some stuff * wip * create factories for function/taskqueue * remove extra symbol * add test case for cpu parsing --------- Co-authored-by: Luke Lombardi <[email protected]> * clean up docstrings (#49) * start adding docstrings * wip * moving things around * add function docstrings * wip * validate cpu and parse it * wip * create factories for function/taskqueue * clean up docstrings * remove extra symbol * update more docstrings * wip * fix import --------- Co-authored-by: Luke Lombardi <[email protected]> * More docs changes (#50) * more README changes * update intro --------- Co-authored-by: Luke Lombardi <[email protected]> * Clean up the sdk a bit (#51) * clean up sdk a bit * wip * add go tests to ci * add tests badge * wip * move default dir * remove github tokens * remove github tokens --------- Co-authored-by: Luke Lombardi <[email protected]> * fix ci test badge (#52) Co-authored-by: Luke Lombardi <[email protected]> * Change registry host port to 5001, add label to worker (#53) * Add scaffold for web endpoint & fix some bugs (#54) * layout webendpoint * wip * dont error out if we cant find the exit code * add proxy * remove proxy --------- Co-authored-by: Luke Lombardi <[email protected]> * Add metrics serve * Add metrics serve * remove charts * remove charts * cleanup * cleanup * remove charts * remove charts * prototype logs sscraping * prototype logs sscraping * take a break from eventing * Fix GPU support with k3d (#59) * Build GPU docker image on linux if nvidia-smi exists (#61) * AWS K3S Deployment (#56) * setup headscale in k3d * remove messed up svc config * fix svc * fix ports * actually fix port * expose headscale too * update server url * tidy * add some terraform stuff * refactor subnets * move domain and hosted zone to vars * wip * fix nginx-ingress-controller * force nginx class to be default * add redis and postgres back in * remove old stuff * wip * more cleanup * configure wireguard/update ami * update security group replacement, install nginx ingress controller * refactor into modules * remove lb controller json * tag subnets * change provider id * use correct chart * fix indent * pin eks chart version * update lb name * remove comment * got ssl working * wip * wip * move to aws-dev * rename buckets * manage root password * add backend repo back in * remove data volume * refactor variables * add redis secret in * populate config, create secret * wip * fix auth, bump redis chart version * fix juicefs bucket * fix rds security group * remove some redundant names * remove spaces * fix redundant gitignore * add depends * make instance type and worker count configurable * pin k3s version * make bucket perms more restrictive * fix vare * less permissive policies --------- Co-authored-by: Luke Lombardi <[email protected]> * Rename project to beta9 (#62) * rename to beta9 * remove todo * fix gateway okteto --------- Co-authored-by: Luke Lombardi <[email protected]> * setup victoria metrics * remove some defaults (#63) Co-authored-by: Luke Lombardi <[email protected]> * verified * Add gRPC msg size, better image name parsing, update a label (#64) * Update readme (#65) * Init ignore file when one doesn't exist (#66) * Improve request concurrency (#67) * increased max postgres connections (#68) * increased max postgres connections * remove period * add comment --------- Co-authored-by: Luke Lombardi <[email protected]> * Add metrics server (#57) * Add metrics serve * remove charts * cleanup * remove charts * setup victoria metrics * verified * finished setup * cleanup * cleanup * make async * cleanup for pr * cleanup for pr * clean up comments and changed to TODOs * finished k3d setup * add more examples * add an examples folder * added in loki host to fluentbit * Add namespace to secrets * remove kinesis from localstack * merged fluentbit examples to 1 file * clean up events client * remove prometheus * fixed tests * cleanup pr * update month standard * update PR: reduce scope of config for fluentbit * handle error * handle error * fixed tests --------- Co-authored-by: luke-lombardi <[email protected]> Co-authored-by: Luke Lombardi <[email protected]> Co-authored-by: luke lombardi <[email protected]> Co-authored-by: Luke Lombardi <[email protected]> Co-authored-by: Nick Petrovic <[email protected]> Co-authored-by: Nick Petrovic <[email protected]> Co-authored-by: Eli Mernit <[email protected]>
luke-lombardi
added a commit
that referenced
this pull request
May 6, 2024
* clean up sdk a bit * wip * add go tests to ci * add tests badge * wip * move default dir * remove github tokens * remove github tokens --------- Co-authored-by: Luke Lombardi <[email protected]>
luke-lombardi
added a commit
that referenced
this pull request
May 6, 2024
* layout (#1) * start laying out adapter interface * move to map * started implementing infinifs paper * remove old old * refactor storage * add gateway * rename all references to activator * remove backwards compatibility * add in queue * rename types * more renaming * add app/task * test out golang plugins * add worker exec * testing dynamic code execution * more complex function * cleanup --------- Co-authored-by: Luke Lombardi <[email protected]> Co-authored-by: luke lombardi <[email protected]> * rename pkg to internal * wip * GRPC refactor pt1 (#2) * refactor grpc organization for abstractions * separate map svc endpoints out * remove old code * simplify organization * remove more code * connect redis client * add more service methods * add storage folder * clean up interface * forgot keys * push up full map implementation --------- Co-authored-by: Luke Lombardi <[email protected]> * rename var * fix typo * cleanup, add clang formatting to proto files (#3) * add some more primitives * remove imageservice rpc * configure autoformatting for proto files --------- Co-authored-by: Luke Lombardi <[email protected]> * storage (#4) * add juicefs to dockerfile * add juicefs config * add juicefs format method * integrate storage * remove comment --------- Co-authored-by: Luke Lombardi <[email protected]> * image service v2 (#5) * pull in imageservice code. unsure of what to do with it * move shit around * rename volume * remove runner type * remove plugins * remove plugins * refactor image service rpc * register image service * update protobuf version * remove mode from container request * remove identity quota tracking * remove identity field * add back worker build target * separate worker and beam images * fix keys * remove print * code removal * prototyping something for interactivity * add execute method * add runner dockerfiles * add runner target * fix build * remove build service logic * consolidate image clients * rename server * add container server addr * clean up error, add runc_client * start setting up runc client * implement more runc methods * move unpacking over into image client * move client, allow server to communicate with gateway * hardcode image * implement safemap * rename var * pass container instances around w/ safemap; * sending and aggregating logs! * try actually execing things * move archive * remove container mode * make filesystem r/w * comment out tmp dirs * add log buffer * implement log streaming endpoint * streaming logs all the way to client * logs streaming!! * pass correct image tag * fix error handling * wip * remove more old code * stop hardcoding * fix skopeo build * messing with args * wip * remove some more code * remove an arg * fix up skopeo build some more * refactor image building scheme * fix tests * lots more cleanup * remove code * keep cleaning up * remove pointer * feeling very very dumb right now * fix secrets template * fix tag * remove more code * actually archive the image * add archive service * remove build container * add pip install * temporarily remove identity * actually archive * remove sleeps, add images volume * cleanup * check packages * rename a couple things * more renames --------- Co-authored-by: Luke Lombardi <[email protected]> * more cleanup (#6) * remove some old code * remove request bucket repo * cleanup * add l * turn off reflection * remove more shit --------- Co-authored-by: Luke Lombardi <[email protected]> * Add sdk (#7) * add beam sdk * remove deprecated code * fix proto gen * remove bucket * remove license file --------- Co-authored-by: Luke Lombardi <[email protected]> * object syncing pt 1 (#8) * add object sync rpc endpoint * remove bucket abstraction temporarily * add gateway service * add more endpoints * add object path * remove fields * wip --------- Co-authored-by: Luke Lombardi <[email protected]> * Object sync pt 2 (#9) * remove put and extract object * finish sync endpoints * wip * return sync result * fix type --------- Co-authored-by: Luke Lombardi <[email protected]> * function abstraction (#10) * use sync result in a more realistic way * add callable wrapper; * add map placeholder * moving things around * remove arg * wip * wip * wip * trying to get invoke working * more cleanup * add grpc server side * invoking the shit * store args * store invocation id * update args * return real args * better build config * wip * sort of lay out scheduling * found a bug in image archiving * override root; * deal with rootfs in a hacky way * fix tmp bug * add sdk source into worker * update base image reqs * add rich * try defining external entry point * dont copy src every time * stream logs * args work! * wip * wip * try mounting storage * install juicefs into worker * push channel into abc; * add aio module * fix types * remove more code * cleanup types * add context * add a context * unzip * something wrong with the args * fix args * remove comments * clean up configs * always set container exit code * add set result * set result * add compute config * we know when it ends now * get function result * get it back * remove arg * dont print empty strings * make pythonunbuffered global * log err * fix error * clean up mounts * handle every exception * add map * k3d, updates, and fixes * rename init and add setup make targets * tidy * update port to 6000 * Revert "update port to 6000" This reverts commit f9e240a. * fix lock filename * add base requirements, remove lockfile * switch from arch64 -> amd64 * switch back beam and beam-worker to x86 * fix mod? --------- Co-authored-by: Luke Lombardi <[email protected]> Co-authored-by: Nick Petrovic <[email protected]> Co-authored-by: Nick Petrovic <[email protected]> * fix function map (#13) * fix function map * pull out function --------- Co-authored-by: Luke Lombardi <[email protected]> * turn on postgres in k3d (#14) Co-authored-by: Luke Lombardi <[email protected]> * Queue abstraction * Queue abstraction and renamed map * add custom exception * Rename simple queue to queue on import in __init__.py * flake and fixing few more things * integrate postgres db and migrations using sqlx (#16) * add backend repository * try using the new sqlx connection * renaming things * add migration process * remove call * wip * remove comment * remove old structs * forward postgres service to local * add models * add deployment status enum * update protobuf version * update models * make hash and name unique * add more realistic models * update names * fix fks * fix migration order --------- Co-authored-by: Luke Lombardi <[email protected]> * refactor queue function names (#18) * refactor * rebuild proto * fix merge issues * GRPC Authentication (#20) * add configure endpoint * working on auth * add docs placeholder * create authenticated channel class * got a token * handle streams too * add authinterceptor struct * wip * add auth package * wip * cleaning things up * locking shit down * rename config file * wip * create new tokens * update msg * create token * pass tokens * remove build files * remove egg --------- Co-authored-by: Luke Lombardi <[email protected]> * rename url -> host * fix token storage * remove check for auth since its redundant * Misc. cleanup (#21) * remove some old identity related code * remove proto file * add event bus back to gateway, move http auth * remove beam repo * more cleanup * remove paths --------- Co-authored-by: Luke Lombardi <[email protected]> * Function task tracking and refactoring (#22) * rename invocation_id to task_id * remove unused env vars * cleaning some things up * add container id to task * tasks curd * track objects in db * remove some debug code * fix task id * move object to seprate module * wip * add stub config version field * fix field * add a stub creation func * refactor storage instantiation * pass repo instead of whole struct * move task services over to gateway * remove constants file * get or create the stub * able to create and update tasks * more cleanup * remove mock data * mostly track the tasks, still needs some work * comments * fix unpacking race conditions, create stub once * add task id to invoke response * handle some errors * remove types * pickle exceptions * store task status * remove comment --------- Co-authored-by: Luke Lombardi <[email protected]> * update contexts * add real container hostname to runner env * add cli framework and configure command (#19) * add configure endpoint * working on auth * add docs placeholder * create authenticated channel class * got a token * handle streams too * add cli framework and configure command * add authinterceptor struct * wip * add auth package * Add gateway url andport * git url * wip * cleaning things up * locking shit down * save to file * clean * refactored config code * remove something/ * test * fixed --------- Co-authored-by: Luke Lombardi <[email protected]> * fix config process * rename context to workspace (#23) * rename context to workspace * missed a few spots * fix few missing spots --------- Co-authored-by: Luke Lombardi <[email protected]> * Add worker request requirements override, update k8s, remove k3d agent (#24) * SDK: Add tasks subcommand (#25) * Return empty string when date is zero (#27) * SDK: Add filter option to beam tasks list command (#29) * volumes abstraction (#28) * add volumes * cleanup * Volume grpc class * spacE * update for pr * initialize volumes with [] ' * prints removal and name updates * global volume service * remove commented out code * reraise error more elegantly * task queues and deployment (#26) * scaffolding deployment * remove deployment status * structuring the task queue, kinda trash * messing with some struct fields * still doesnt even build... * fix proto files * wip * move container methods to repo * make it build again * remove task repo * more cleanup * refactor keys * laying out sdk * refactoring task queue server * add backend repo * wip; * pushing something... * no args... * remove args * create a real task * hook up an autoscaler... * make a little more realistic...? * still messed up * remove unnecessary args * print state for debugging * refactor runtime/stub logic to be consistent * more cleanup * fix instance lock key * make taskqueues unload themselves automatically when not in use * stupid json serialization * wip * cleanup * more and more cleanup * wip * remove commented out code * rename non-public vars * add stoppable containers back * wip * start pulling over task retry logic * more cleanup, task monitoring * wip * pass all the stub data * add scale down delay * pass context properly * make config a classmethod * use real failed containers val * rename scale down delay to keep warm seconds * remove unused func * real task policy * remove comment * fixing stub config * fix join * remove last print * try to fix merge confllicts * more cleanup * add back print format exc * one more bugfix * wip * Fix volumes (#30) * merge * fixed volumes * cleanup? --------- Co-authored-by: Luke Lombardi <[email protected]> Co-authored-by: jsun-m <[email protected]> * push load handler functionality into runner abstraction (#31) * push load handler functionality into runner abstraction * fix the func --------- Co-authored-by: Luke Lombardi <[email protected]> * add container id and more task logs (#32) Co-authored-by: Luke Lombardi <[email protected]> * Remove deprecated code (#33) * move base abstractions into a subdir * remove unused event bus * remove old code * comment out broken code * remove container config response * create queue instance if not exists --------- Co-authored-by: Luke Lombardi <[email protected]> * Setup echo http server (#34) * remove gorm * set up echo health check group * add additional service port * setup basic routes * fix comment * tidy go.mod * wip --------- Co-authored-by: Luke Lombardi <[email protected]> * add auth middleware to http server (#35) * add auth middleware * fix context value * updated pr to show usage * fix route --------- Co-authored-by: Luke Lombardi <[email protected]> * added license and readme (#37) * added license and readme * Update README.md * Abstraction http interface (#39) * wip * connect a route * sort of implement task queue http endpoints * cleanup * rename volume svc * wip * wip * return invalid stub id err * remove fvolume type --------- Co-authored-by: Luke Lombardi <[email protected]> * enforce service interfaces (#40) Co-authored-by: Luke Lombardi <[email protected]> * sdk test cases (#36) * sdk test cases 1 * add task queue tests * Add tests for function * Add test for function map * add comment explanation * change function to use mock_coroutine_with_result * updated for review * add github actions * patch actions * poetry * poetry * forgot ot checkut * fix ruff * fix ruff * fix ruff * last step * ignore auth for ci * ignore auth for ci * Basic taskqueue deployment (#42) * wip deployment mods * fix stub types * hide echo banner * hide port * invoke the deployment grpc service * actually creating deployment objects now * deployment works e2e * wip * cleanup * cleanup * more cleanup * make it an err --------- Co-authored-by: Luke Lombardi <[email protected]> * update deployment model to include stub_type (#43) Co-authored-by: Luke Lombardi <[email protected]> * New app config (#41) * remove beam repo, gorm, and old db models (#44) Co-authored-by: Luke Lombardi <[email protected]> * Add function deployments (#45) * refactor invoke to support function deployments * works except for arg passing * fix task id * change order of operations * wip * support both json and cloudpickle for functions * remove cache files * add containers banner --------- Co-authored-by: Luke Lombardi <[email protected]> * Namespace more stuff by workspaceName (#46) * namespace more stuff by workspace name * namespace everything by workspace --------- Co-authored-by: Luke Lombardi <[email protected]> * fix some bugs with simplequeue (that I created); add lock for mapset (#47) * fix some bugs; add set lock * fix tests --------- Co-authored-by: Luke Lombardi <[email protected]> * SDK docstrings (#48) * start adding docstrings * add docstring for map * wip * fix tests * wip * moving things around * add function docstrings * wip * validate cpu and parse it * update cpu values * fix cpu value * wip * fix some stuff * wip * create factories for function/taskqueue * remove extra symbol * add test case for cpu parsing --------- Co-authored-by: Luke Lombardi <[email protected]> * clean up docstrings (#49) * start adding docstrings * wip * moving things around * add function docstrings * wip * validate cpu and parse it * wip * create factories for function/taskqueue * clean up docstrings * remove extra symbol * update more docstrings * wip * fix import --------- Co-authored-by: Luke Lombardi <[email protected]> * More docs changes (#50) * more README changes * update intro --------- Co-authored-by: Luke Lombardi <[email protected]> * Clean up the sdk a bit (#51) * clean up sdk a bit * wip * add go tests to ci * add tests badge * wip * move default dir * remove github tokens * remove github tokens --------- Co-authored-by: Luke Lombardi <[email protected]> * fix ci test badge (#52) Co-authored-by: Luke Lombardi <[email protected]> * Change registry host port to 5001, add label to worker (#53) * Add scaffold for web endpoint & fix some bugs (#54) * layout webendpoint * wip * dont error out if we cant find the exit code * add proxy * remove proxy --------- Co-authored-by: Luke Lombardi <[email protected]> * Add metrics serve * remove charts * cleanup * remove charts * prototype logs sscraping * Fix GPU support with k3d (#59) * Build GPU docker image on linux if nvidia-smi exists (#61) * AWS K3S Deployment (#56) * setup headscale in k3d * remove messed up svc config * fix svc * fix ports * actually fix port * expose headscale too * update server url * tidy * add some terraform stuff * refactor subnets * move domain and hosted zone to vars * wip * fix nginx-ingress-controller * force nginx class to be default * add redis and postgres back in * remove old stuff * wip * more cleanup * configure wireguard/update ami * update security group replacement, install nginx ingress controller * refactor into modules * remove lb controller json * tag subnets * change provider id * use correct chart * fix indent * pin eks chart version * update lb name * remove comment * got ssl working * wip * wip * move to aws-dev * rename buckets * manage root password * add backend repo back in * remove data volume * refactor variables * add redis secret in * populate config, create secret * wip * fix auth, bump redis chart version * fix juicefs bucket * fix rds security group * remove some redundant names * remove spaces * fix redundant gitignore * add depends * make instance type and worker count configurable * pin k3s version * make bucket perms more restrictive * fix vare * less permissive policies --------- Co-authored-by: Luke Lombardi <[email protected]> * Rename project to beta9 (#62) * rename to beta9 * remove todo * fix gateway okteto --------- Co-authored-by: Luke Lombardi <[email protected]> * setup victoria metrics * remove some defaults (#63) Co-authored-by: Luke Lombardi <[email protected]> * verified * Add gRPC msg size, better image name parsing, update a label (#64) * Update readme (#65) * Init ignore file when one doesn't exist (#66) * Improve request concurrency (#67) * increased max postgres connections (#68) * increased max postgres connections * remove period * add comment --------- Co-authored-by: Luke Lombardi <[email protected]> * Add metrics server (#57) * Add metrics serve * remove charts * cleanup * remove charts * setup victoria metrics * verified * finished setup * cleanup * cleanup * make async * cleanup for pr * cleanup for pr * clean up comments and changed to TODOs * finished k3d setup * add more examples * add an examples folder * added in loki host to fluentbit * Add namespace to secrets * remove kinesis from localstack * merged fluentbit examples to 1 file --------- Co-authored-by: luke-lombardi <[email protected]> Co-authored-by: Luke Lombardi <[email protected]> Co-authored-by: luke lombardi <[email protected]> Co-authored-by: Luke Lombardi <[email protected]> Co-authored-by: Nick Petrovic <[email protected]> Co-authored-by: Nick Petrovic <[email protected]> Co-authored-by: Eli Mernit <[email protected]>
luke-lombardi
added a commit
that referenced
this pull request
May 6, 2024
* layout (#1) * start laying out adapter interface * move to map * started implementing infinifs paper * remove old old * refactor storage * add gateway * rename all references to activator * remove backwards compatibility * add in queue * rename types * more renaming * add app/task * test out golang plugins * add worker exec * testing dynamic code execution * more complex function * cleanup --------- Co-authored-by: Luke Lombardi <[email protected]> Co-authored-by: luke lombardi <[email protected]> * rename pkg to internal * wip * GRPC refactor pt1 (#2) * refactor grpc organization for abstractions * separate map svc endpoints out * remove old code * simplify organization * remove more code * connect redis client * add more service methods * add storage folder * clean up interface * forgot keys * push up full map implementation --------- Co-authored-by: Luke Lombardi <[email protected]> * rename var * fix typo * cleanup, add clang formatting to proto files (#3) * add some more primitives * remove imageservice rpc * configure autoformatting for proto files --------- Co-authored-by: Luke Lombardi <[email protected]> * storage (#4) * add juicefs to dockerfile * add juicefs config * add juicefs format method * integrate storage * remove comment --------- Co-authored-by: Luke Lombardi <[email protected]> * image service v2 (#5) * pull in imageservice code. unsure of what to do with it * move shit around * rename volume * remove runner type * remove plugins * remove plugins * refactor image service rpc * register image service * update protobuf version * remove mode from container request * remove identity quota tracking * remove identity field * add back worker build target * separate worker and beam images * fix keys * remove print * code removal * prototyping something for interactivity * add execute method * add runner dockerfiles * add runner target * fix build * remove build service logic * consolidate image clients * rename server * add container server addr * clean up error, add runc_client * start setting up runc client * implement more runc methods * move unpacking over into image client * move client, allow server to communicate with gateway * hardcode image * implement safemap * rename var * pass container instances around w/ safemap; * sending and aggregating logs! * try actually execing things * move archive * remove container mode * make filesystem r/w * comment out tmp dirs * add log buffer * implement log streaming endpoint * streaming logs all the way to client * logs streaming!! * pass correct image tag * fix error handling * wip * remove more old code * stop hardcoding * fix skopeo build * messing with args * wip * remove some more code * remove an arg * fix up skopeo build some more * refactor image building scheme * fix tests * lots more cleanup * remove code * keep cleaning up * remove pointer * feeling very very dumb right now * fix secrets template * fix tag * remove more code * actually archive the image * add archive service * remove build container * add pip install * temporarily remove identity * actually archive * remove sleeps, add images volume * cleanup * check packages * rename a couple things * more renames --------- Co-authored-by: Luke Lombardi <[email protected]> * more cleanup (#6) * remove some old code * remove request bucket repo * cleanup * add l * turn off reflection * remove more shit --------- Co-authored-by: Luke Lombardi <[email protected]> * Add sdk (#7) * add beam sdk * remove deprecated code * fix proto gen * remove bucket * remove license file --------- Co-authored-by: Luke Lombardi <[email protected]> * object syncing pt 1 (#8) * add object sync rpc endpoint * remove bucket abstraction temporarily * add gateway service * add more endpoints * add object path * remove fields * wip --------- Co-authored-by: Luke Lombardi <[email protected]> * Object sync pt 2 (#9) * remove put and extract object * finish sync endpoints * wip * return sync result * fix type --------- Co-authored-by: Luke Lombardi <[email protected]> * function abstraction (#10) * use sync result in a more realistic way * add callable wrapper; * add map placeholder * moving things around * remove arg * wip * wip * wip * trying to get invoke working * more cleanup * add grpc server side * invoking the shit * store args * store invocation id * update args * return real args * better build config * wip * sort of lay out scheduling * found a bug in image archiving * override root; * deal with rootfs in a hacky way * fix tmp bug * add sdk source into worker * update base image reqs * add rich * try defining external entry point * dont copy src every time * stream logs * args work! * wip * wip * try mounting storage * install juicefs into worker * push channel into abc; * add aio module * fix types * remove more code * cleanup types * add context * add a context * unzip * something wrong with the args * fix args * remove comments * clean up configs * always set container exit code * add set result * set result * add compute config * we know when it ends now * get function result * get it back * remove arg * dont print empty strings * make pythonunbuffered global * log err * fix error * clean up mounts * handle every exception * add map * k3d, updates, and fixes * rename init and add setup make targets * tidy * update port to 6000 * Revert "update port to 6000" This reverts commit f9e240a. * fix lock filename * add base requirements, remove lockfile * switch from arch64 -> amd64 * switch back beam and beam-worker to x86 * fix mod? --------- Co-authored-by: Luke Lombardi <[email protected]> Co-authored-by: Nick Petrovic <[email protected]> Co-authored-by: Nick Petrovic <[email protected]> * fix function map (#13) * fix function map * pull out function --------- Co-authored-by: Luke Lombardi <[email protected]> * turn on postgres in k3d (#14) Co-authored-by: Luke Lombardi <[email protected]> * Queue abstraction * Queue abstraction and renamed map * add custom exception * Rename simple queue to queue on import in __init__.py * flake and fixing few more things * integrate postgres db and migrations using sqlx (#16) * add backend repository * try using the new sqlx connection * renaming things * add migration process * remove call * wip * remove comment * remove old structs * forward postgres service to local * add models * add deployment status enum * update protobuf version * update models * make hash and name unique * add more realistic models * update names * fix fks * fix migration order --------- Co-authored-by: Luke Lombardi <[email protected]> * refactor queue function names (#18) * refactor * rebuild proto * fix merge issues * GRPC Authentication (#20) * add configure endpoint * working on auth * add docs placeholder * create authenticated channel class * got a token * handle streams too * add authinterceptor struct * wip * add auth package * wip * cleaning things up * locking shit down * rename config file * wip * create new tokens * update msg * create token * pass tokens * remove build files * remove egg --------- Co-authored-by: Luke Lombardi <[email protected]> * rename url -> host * fix token storage * remove check for auth since its redundant * Misc. cleanup (#21) * remove some old identity related code * remove proto file * add event bus back to gateway, move http auth * remove beam repo * more cleanup * remove paths --------- Co-authored-by: Luke Lombardi <[email protected]> * Function task tracking and refactoring (#22) * rename invocation_id to task_id * remove unused env vars * cleaning some things up * add container id to task * tasks curd * track objects in db * remove some debug code * fix task id * move object to seprate module * wip * add stub config version field * fix field * add a stub creation func * refactor storage instantiation * pass repo instead of whole struct * move task services over to gateway * remove constants file * get or create the stub * able to create and update tasks * more cleanup * remove mock data * mostly track the tasks, still needs some work * comments * fix unpacking race conditions, create stub once * add task id to invoke response * handle some errors * remove types * pickle exceptions * store task status * remove comment --------- Co-authored-by: Luke Lombardi <[email protected]> * update contexts * add real container hostname to runner env * add cli framework and configure command (#19) * add configure endpoint * working on auth * add docs placeholder * create authenticated channel class * got a token * handle streams too * add cli framework and configure command * add authinterceptor struct * wip * add auth package * Add gateway url andport * git url * wip * cleaning things up * locking shit down * save to file * clean * refactored config code * remove something/ * test * fixed --------- Co-authored-by: Luke Lombardi <[email protected]> * fix config process * rename context to workspace (#23) * rename context to workspace * missed a few spots * fix few missing spots --------- Co-authored-by: Luke Lombardi <[email protected]> * Add worker request requirements override, update k8s, remove k3d agent (#24) * SDK: Add tasks subcommand (#25) * Return empty string when date is zero (#27) * SDK: Add filter option to beam tasks list command (#29) * volumes abstraction (#28) * add volumes * cleanup * Volume grpc class * spacE * update for pr * initialize volumes with [] ' * prints removal and name updates * global volume service * remove commented out code * reraise error more elegantly * task queues and deployment (#26) * scaffolding deployment * remove deployment status * structuring the task queue, kinda trash * messing with some struct fields * still doesnt even build... * fix proto files * wip * move container methods to repo * make it build again * remove task repo * more cleanup * refactor keys * laying out sdk * refactoring task queue server * add backend repo * wip; * pushing something... * no args... * remove args * create a real task * hook up an autoscaler... * make a little more realistic...? * still messed up * remove unnecessary args * print state for debugging * refactor runtime/stub logic to be consistent * more cleanup * fix instance lock key * make taskqueues unload themselves automatically when not in use * stupid json serialization * wip * cleanup * more and more cleanup * wip * remove commented out code * rename non-public vars * add stoppable containers back * wip * start pulling over task retry logic * more cleanup, task monitoring * wip * pass all the stub data * add scale down delay * pass context properly * make config a classmethod * use real failed containers val * rename scale down delay to keep warm seconds * remove unused func * real task policy * remove comment * fixing stub config * fix join * remove last print * try to fix merge confllicts * more cleanup * add back print format exc * one more bugfix * wip * Fix volumes (#30) * merge * fixed volumes * cleanup? --------- Co-authored-by: Luke Lombardi <[email protected]> Co-authored-by: jsun-m <[email protected]> * push load handler functionality into runner abstraction (#31) * push load handler functionality into runner abstraction * fix the func --------- Co-authored-by: Luke Lombardi <[email protected]> * add container id and more task logs (#32) Co-authored-by: Luke Lombardi <[email protected]> * Remove deprecated code (#33) * move base abstractions into a subdir * remove unused event bus * remove old code * comment out broken code * remove container config response * create queue instance if not exists --------- Co-authored-by: Luke Lombardi <[email protected]> * Setup echo http server (#34) * remove gorm * set up echo health check group * add additional service port * setup basic routes * fix comment * tidy go.mod * wip --------- Co-authored-by: Luke Lombardi <[email protected]> * add auth middleware to http server (#35) * add auth middleware * fix context value * updated pr to show usage * fix route --------- Co-authored-by: Luke Lombardi <[email protected]> * added license and readme (#37) * added license and readme * Update README.md * Abstraction http interface (#39) * wip * connect a route * sort of implement task queue http endpoints * cleanup * rename volume svc * wip * wip * return invalid stub id err * remove fvolume type --------- Co-authored-by: Luke Lombardi <[email protected]> * enforce service interfaces (#40) Co-authored-by: Luke Lombardi <[email protected]> * sdk test cases (#36) * sdk test cases 1 * add task queue tests * Add tests for function * Add test for function map * add comment explanation * change function to use mock_coroutine_with_result * updated for review * add github actions * patch actions * poetry * poetry * forgot ot checkut * fix ruff * fix ruff * fix ruff * last step * ignore auth for ci * ignore auth for ci * Basic taskqueue deployment (#42) * wip deployment mods * fix stub types * hide echo banner * hide port * invoke the deployment grpc service * actually creating deployment objects now * deployment works e2e * wip * cleanup * cleanup * more cleanup * make it an err --------- Co-authored-by: Luke Lombardi <[email protected]> * update deployment model to include stub_type (#43) Co-authored-by: Luke Lombardi <[email protected]> * New app config (#41) * remove beam repo, gorm, and old db models (#44) Co-authored-by: Luke Lombardi <[email protected]> * Add function deployments (#45) * refactor invoke to support function deployments * works except for arg passing * fix task id * change order of operations * wip * support both json and cloudpickle for functions * remove cache files * add containers banner --------- Co-authored-by: Luke Lombardi <[email protected]> * Namespace more stuff by workspaceName (#46) * namespace more stuff by workspace name * namespace everything by workspace --------- Co-authored-by: Luke Lombardi <[email protected]> * fix some bugs with simplequeue (that I created); add lock for mapset (#47) * fix some bugs; add set lock * fix tests --------- Co-authored-by: Luke Lombardi <[email protected]> * SDK docstrings (#48) * start adding docstrings * add docstring for map * wip * fix tests * wip * moving things around * add function docstrings * wip * validate cpu and parse it * update cpu values * fix cpu value * wip * fix some stuff * wip * create factories for function/taskqueue * remove extra symbol * add test case for cpu parsing --------- Co-authored-by: Luke Lombardi <[email protected]> * clean up docstrings (#49) * start adding docstrings * wip * moving things around * add function docstrings * wip * validate cpu and parse it * wip * create factories for function/taskqueue * clean up docstrings * remove extra symbol * update more docstrings * wip * fix import --------- Co-authored-by: Luke Lombardi <[email protected]> * More docs changes (#50) * more README changes * update intro --------- Co-authored-by: Luke Lombardi <[email protected]> * Clean up the sdk a bit (#51) * clean up sdk a bit * wip * add go tests to ci * add tests badge * wip * move default dir * remove github tokens * remove github tokens --------- Co-authored-by: Luke Lombardi <[email protected]> * fix ci test badge (#52) Co-authored-by: Luke Lombardi <[email protected]> * Change registry host port to 5001, add label to worker (#53) * Add scaffold for web endpoint & fix some bugs (#54) * layout webendpoint * wip * dont error out if we cant find the exit code * add proxy * remove proxy --------- Co-authored-by: Luke Lombardi <[email protected]> * Add metrics serve * Add metrics serve * remove charts * remove charts * cleanup * cleanup * remove charts * remove charts * prototype logs sscraping * prototype logs sscraping * take a break from eventing * Fix GPU support with k3d (#59) * Build GPU docker image on linux if nvidia-smi exists (#61) * AWS K3S Deployment (#56) * setup headscale in k3d * remove messed up svc config * fix svc * fix ports * actually fix port * expose headscale too * update server url * tidy * add some terraform stuff * refactor subnets * move domain and hosted zone to vars * wip * fix nginx-ingress-controller * force nginx class to be default * add redis and postgres back in * remove old stuff * wip * more cleanup * configure wireguard/update ami * update security group replacement, install nginx ingress controller * refactor into modules * remove lb controller json * tag subnets * change provider id * use correct chart * fix indent * pin eks chart version * update lb name * remove comment * got ssl working * wip * wip * move to aws-dev * rename buckets * manage root password * add backend repo back in * remove data volume * refactor variables * add redis secret in * populate config, create secret * wip * fix auth, bump redis chart version * fix juicefs bucket * fix rds security group * remove some redundant names * remove spaces * fix redundant gitignore * add depends * make instance type and worker count configurable * pin k3s version * make bucket perms more restrictive * fix vare * less permissive policies --------- Co-authored-by: Luke Lombardi <[email protected]> * Rename project to beta9 (#62) * rename to beta9 * remove todo * fix gateway okteto --------- Co-authored-by: Luke Lombardi <[email protected]> * setup victoria metrics * remove some defaults (#63) Co-authored-by: Luke Lombardi <[email protected]> * verified * Add gRPC msg size, better image name parsing, update a label (#64) * Update readme (#65) * Init ignore file when one doesn't exist (#66) * Improve request concurrency (#67) * increased max postgres connections (#68) * increased max postgres connections * remove period * add comment --------- Co-authored-by: Luke Lombardi <[email protected]> * Add metrics server (#57) * Add metrics serve * remove charts * cleanup * remove charts * setup victoria metrics * verified * finished setup * cleanup * cleanup * make async * cleanup for pr * cleanup for pr * clean up comments and changed to TODOs * finished k3d setup * add more examples * add an examples folder * added in loki host to fluentbit * Add namespace to secrets * remove kinesis from localstack * merged fluentbit examples to 1 file * clean up events client * remove prometheus * fixed tests * cleanup pr * update month standard * update PR: reduce scope of config for fluentbit * handle error * handle error * fixed tests --------- Co-authored-by: luke-lombardi <[email protected]> Co-authored-by: Luke Lombardi <[email protected]> Co-authored-by: luke lombardi <[email protected]> Co-authored-by: Luke Lombardi <[email protected]> Co-authored-by: Nick Petrovic <[email protected]> Co-authored-by: Nick Petrovic <[email protected]> Co-authored-by: Eli Mernit <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.