Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[pull] main from bacalhau-project:main #3

Merged
merged 71 commits into from
Feb 14, 2024

Conversation

pull[bot]
Copy link

@pull pull bot commented Jan 26, 2024

See Commits and Changes for more details.


Created by pull[bot]

Can you help keep this open source service alive? 💖 Please sponsor : )

This commit adds `ask` mode authentication that allows node operators
to configure Bacalhau to ask the user for arbitrary information to be
used as a credential.

This method can be used to implement basic usernames and passwords,
shared secrets, security questions and even 2FA.

The associated policy additionally returns a JSON Schema to show what
information is required. The CLI uses the schema to ask the user for
the right information. In the future, the Web UI will do this as well.
@pull pull bot added the ⤵️ pull label Jan 26, 2024
frrist and others added 28 commits January 26, 2024 11:12
- Job Duration
- Number of Jobs Submitted
- Active Docker Executions
- Active WASM Executions
- Basic Node Info (metric isn't valuable, labels are)
According to CircleCI we are using a deprecated image (see
https://discuss.circleci.com/t/linux-image-deprecations-and-eol-for-2024/50177)
and so this PR switches to using the new 'default' image which is the
latest stable version

Includes formatting fixes

Fixes #3271
- options are the functions that modify a config.
This change allows finer control over each HTTP request by accepting
a context.Context in methods like GET, PUT, POST, and DELETE, aligning
with Go's idiomatic use of context for request-scoped settings, such as
timeouts and cancellation. This approach offers greater flexibility and
is consistent with the standard library's design.
- Enforce base URL as mandatory in constructor; removes the illusion
of it being an 'optional' setting when it's fundamentally required for operation
the context is passed into each method
Currently the headers in the BaseRequest structure are excluded for
query parameters, but as still included when encoded as json. This
commit changes that to exclude the headers in BaseRequest any time they
are marshalled as json.

Fixes #3340
Currently we run golangci-lint run with the `--fix` flag that allows
individual linters to make changes to the files where it finds problems.
Ideally the linter should just flag the issue for the developer to fix,
using --fix if they wish, but not making that choice on their behalf.
Bumps
[github.com/invopop/jsonschema](https://github.com/invopop/jsonschema)
from 0.11.0 to 0.12.0.
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/invopop/jsonschema/commit/9b6bb6e8f15aa557c1df053de86d4e5043365ca9"><code>9b6bb6e</code></a>
Merge pull request <a
href="https://redirect.github.com/invopop/jsonschema/issues/108">#108</a>
from invopop/schema-property-alias</li>
<li><a
href="https://github.com/invopop/jsonschema/commit/f45517cf07169cf4e35eae0a4e47ad8086098583"><code>f45517c</code></a>
Typo</li>
<li><a
href="https://github.com/invopop/jsonschema/commit/ab81e2aa6405fefd82b6958075d64d2de1844bcd"><code>ab81e2a</code></a>
Fixing linting</li>
<li><a
href="https://github.com/invopop/jsonschema/commit/933814a23a0b8401d90625df47f4be2a0c9a3039"><code>933814a</code></a>
support for JSONSchemaAlias y JSONSchemaProperty methods</li>
<li><a
href="https://github.com/invopop/jsonschema/commit/0108689b9f4e04f64a5986ea21650f07673c21cf"><code>0108689</code></a>
Support for JSONSchemaAlias method</li>
<li><a
href="https://github.com/invopop/jsonschema/commit/12cbc49539c8deac9baeae7811fdcab42fd5def0"><code>12cbc49</code></a>
Updating test output with new number parsing</li>
<li><a
href="https://github.com/invopop/jsonschema/commit/cada51da5649ccf919c51225dd8e2260a388b883"><code>cada51d</code></a>
Merge pull request <a
href="https://redirect.github.com/invopop/jsonschema/issues/96">#96</a>
from Hotdawg/regex_with_equals</li>
<li><a
href="https://github.com/invopop/jsonschema/commit/9d905a25553837495e11ecb6d61f5348110dc49f"><code>9d905a2</code></a>
Limit split on genericKeywords</li>
<li><a
href="https://github.com/invopop/jsonschema/commit/080d97dda979ca9544cc362b9f1c7c02cfc03697"><code>080d97d</code></a>
Merge branch 'invopop:main' into regex_with_equals</li>
<li><a
href="https://github.com/invopop/jsonschema/commit/8a098cbcb66558b5c55fccd24c7cc42825d906d6"><code>8a098cb</code></a>
Fix linter issue</li>
<li>See full diff in <a
href="https://github.com/invopop/jsonschema/compare/v0.11.0...v0.12.0">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=github.com/invopop/jsonschema&package-manager=go_modules&previous-version=0.11.0&new-version=0.12.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
tls_test will test the CLI flags "tls", "insecure" and "cacert".
Currently, the generated self-signed certificates are stored in
/testdata. They will most likely need to be generated every time the
test is run, and this is
addressed in a new issue (#3337)

fixes #3283 , #3284 , #3285 , #3286
…d skipping draft PRs (#3280)

…d skipping draft PRs on CI
This should **JUST** be linting/whitespace - except for two files in the
middle:

webui/src/images/__tests___/jobsIcon.test.tsx
webui/src/images/svg-image.tsx
LogStream is tightly coupled to libp2p to stream logs from the compute
node to the requester, which makes supporting NATS more challenging
bacalhau-project/expanso-planning#413

This PR decouples LogStream and makes fetching logs from the executor
and compute node agnostic to how they are published to the requester.

The way this is achieved by utilizing channels for async vending of
logs. The compute nodes return a channel for
`AsyncResult[models.ExecutionLog]`, which is consumed by the transport
layer (`bprotocol` in current implementation) and sent to the requester.
A similar implementation is also introduced from the client to the
requester node where instead of exposing websockets to the client user,
the new SDK just returns a channel of `AsyncResult[models.ExecutionLog]`

This PR also migrates `logs` endpoint from the old APIs to the new
version, implements logs SDK, and introduces helper methods to expose
and consume additional websockets in the future.
Rebased doc branch. Should be able to merge now.

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: n.garanovich <[email protected]>
Co-authored-by: Ross Jones <[email protected]>
Co-authored-by: frrist <[email protected]>
Co-authored-by: Simon Worthington <[email protected]>
Co-authored-by: Simon Worthington <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Forrest <[email protected]>
Co-authored-by: Al Norman <[email protected]>
Co-authored-by: David Aronchick <[email protected]>
Co-authored-by: axel <[email protected]>
Co-authored-by: AlekseyKorzik <[email protected]>
most of the code is just s/npm/yarn/g and yarn files for building/etc
(in config/ and scripts/).

there is also - webui/src/images/svg-image.tsx - for rendering SVGs
properly.
Adds initial docs for the exec command.
Includes the currently supported job types in the help text so that
currently it displays

```
Allows for the execution of a job type with the given code,
without the need to create a container, or webassembly module.
By specifying the code with the '--code' flag you can ship
the code to the cluster for execution, specified by the
remainder of the command line.  See examples below.

 Supported job types:

  *  duckdb
  *  python

```

Resolves #3390
simonwo and others added 29 commits February 6, 2024 03:53
CircleCI has been failing because of python builds and tests expecting a
python version less than 3.12. Even though we were installing python3.10
in our config using `apt get`, circleci appears to be using `pyenv` and
was ignoring the version we installed.

```
ERROR: Package 'flytekit' requires a different Python: 3.12.1 not in '<3.12,>=3.8'
```
This PR introduces a NATS `stream` package that enables response streams
patterns. NATS currently supports request/response patterns which is
great when expecting a single response per request. However, this is not
sufficient to serve job logs where a user initiates a request to stream
logs, and the compute node start serving those logs.

## Overview of `stream` package

This package implements a client leveraging NATS for sending and
receiving streaming data records. It abstracts NATS connections,
subscriptions, and message handling complexities, offering a simplified
interface for data streaming. The client supports multiplexing multiple
streams over a single NATS subscription, handling responses from
different streams using a unique token-based mechanism. Additionally,
the package introduces a Writer component, designed to abstract the
complexities of data encoding and NATS publishing into a simple,
intuitive interface.

## How It Works

- **The Client** part of the package manages dynamic inboxes for each
streaming session, facilitating the sending of data and listening for
responses on dedicated subjects. It leverages the NATS publish-subscribe
model for asynchronous communication, efficiently routing and
correlating messages to their respective streams.
  
- **The Writer** component allows for easy publishing of structured data
to any NATS subject. It integrates tightly with the Client, utilizing
the same connection for streamlined data streaming. The Writer
simplifies the publication process, automatically handling data
serialization and supporting graceful stream closure with custom codes.

## Multiplexing Streams

To efficiently handle multiple streams, the client uses a single
wildcard subscription for all responses. Each request is sent with a
unique response subject (derived from a base inbox prefix), with
responses routed back to this subject. The client demultiplexes incoming
messages by extracting a token from the response subject, identifying
the correct stream (or "bucket") for the message. This approach allows
managing multiple concurrent streams with minimal overhead, leveraging
NATS's lightweight subjects and messaging capabilities.

## Key Features

- **Asynchronous Streaming**: Supports asynchronous data streaming,
allowing clients to send data and receive responses without blocking.

- **Context Support**: Integrates with Go's context package for
timeouts, cancellation, and deadlines for streaming requests.

- **Multiplexing**: Efficiently multiplexes multiple streams over a
single NATS subscription, using unique response subjects for message
correlation.

- **Error Handling**: Provides robust error handling, including custom
error codes for stream-related errors (e.g., bad data, normal closure).

- **Data Publication**: The Writer simplifies structured data publishing
to NATS, with support for automatic serialization and stream closure
signals.

## Usage

Initialize the client with a NATS connection and use the provided
methods to send streaming requests and handle responses. The client
manages the NATS subscription and response routing, simplifying the
process of working with streaming data.

### Example

```go
params := stream.ClientParams{Conn: natsConn}
client, err := stream.NewClient(params)
if err != nil {
    log.Fatal(err)
}

ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()

responseChan, err := client.OpenStream(ctx, "subject", []byte("data"))
if err != nil {
    log.Fatal(err)
}

for asyncResult := range responseChan {
    if asyncResult.Err != nil {
        log.Printf("Received an error: %v", asyncResult.Err)
        continue
    }
    log.Printf("Received data: %s", asyncResult.Value)
}

### Remaining Work
- The compute node has no knowledge when the requester is no longer interested to stream logs. We need to implement a heartbeat or a feedback loop to let the compute node stop publishing logs when there are no active consumers
- The requester node has no knowledge if the compute node is still alive and publishing logs. We do have a graceful handling when the compute node reaches the end of the stream where it informs the requester node that no more logs are coming, but we don't handle ungraceful termination of compute nodes

---------

Co-authored-by: Ross Jones <[email protected]>
Currently we allow users to define limit query params when listing jobs,
but we always return an empty next_token (offset), which prevents users
from actual pagination and querying the next set of jobs.

This PR moves label filtering into the jobstore, and introduces a
default sort order so that the offset/limit make sense. This allows for
proper pagination from the CLI. The next-token is now used by the
jobstore, but it only encodes the offset (e.g. it is the offset). The
next-token should really encode more information so that it can be
re-constituted with the correct parameters. Without this (as currently)
it relies on the same sort being applied for each request.
Currently we default to IPFS for the publisher if one is not specified.
This commit changes this to allow no publisher to be specified. It is
expected that this will allow us to run jobs with no publisher, and also
to have the requester node choose which publisher to use.
Bumps [cryptography](https://github.com/pyca/cryptography) from 41.0.6
to 42.0.0.
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/pyca/cryptography/blob/main/CHANGELOG.rst">cryptography's
changelog</a>.</em></p>
<blockquote>
<p>42.0.0 - 2024-01-22</p>
<pre><code>
* **BACKWARDS INCOMPATIBLE:** Dropped support for LibreSSL &lt; 3.7.
* **BACKWARDS INCOMPATIBLE:** Loading a PKCS7 with no content field
using

:func:`~cryptography.hazmat.primitives.serialization.pkcs7.load_pem_pkcs7_certificates`
  or

:func:`~cryptography.hazmat.primitives.serialization.pkcs7.load_der_pkcs7_certificates`
  will now raise a ``ValueError`` rather than return an empty list.
* Parsing SSH certificates no longer permits malformed critical options
with
  values, as documented in the 41.0.2 release notes.
* Updated Windows, macOS, and Linux wheels to be compiled with OpenSSL
3.2.0.
* Updated the minimum supported Rust version (MSRV) to 1.63.0, from
1.56.0.
* We now publish both ``py37`` and ``py39`` ``abi3`` wheels. This should
resolve some errors relating to initializing a module multiple times per
  process.
* Support
:class:`~cryptography.hazmat.primitives.asymmetric.padding.PSS` for
X.509 certificate signing requests and certificate revocation lists with
the
  keyword-only argument ``rsa_padding`` on the ``sign`` methods for
  :class:`~cryptography.x509.CertificateSigningRequestBuilder` and
  :class:`~cryptography.x509.CertificateRevocationListBuilder`.
* Added support for obtaining X.509 certificate signing request
signature
  algorithm parameters (including PSS) via

:meth:`~cryptography.x509.CertificateSigningRequest.signature_algorithm_parameters`.
* Added support for obtaining X.509 certificate revocation list
signature
  algorithm parameters (including PSS) via

:meth:`~cryptography.x509.CertificateRevocationList.signature_algorithm_parameters`.
* Added ``mgf`` property to
  :class:`~cryptography.hazmat.primitives.asymmetric.padding.PSS`.
* Added ``algorithm`` and ``mgf`` properties to
  :class:`~cryptography.hazmat.primitives.asymmetric.padding.OAEP`.
* Added the following properties that return timezone-aware ``datetime``
objects:
  :meth:`~cryptography.x509.Certificate.not_valid_before_utc`,
  :meth:`~cryptography.x509.Certificate.not_valid_after_utc`,
  :meth:`~cryptography.x509.RevokedCertificate.revocation_date_utc`,
  :meth:`~cryptography.x509.CertificateRevocationList.next_update_utc`,
  :meth:`~cryptography.x509.CertificateRevocationList.last_update_utc`.
These are timezone-aware variants of existing properties that return
naïve
  ``datetime`` objects.
* Deprecated the following properties that return naïve ``datetime``
objects:
  :meth:`~cryptography.x509.Certificate.not_valid_before`,
  :meth:`~cryptography.x509.Certificate.not_valid_after`,
  :meth:`~cryptography.x509.RevokedCertificate.revocation_date`,
  :meth:`~cryptography.x509.CertificateRevocationList.next_update`,
  :meth:`~cryptography.x509.CertificateRevocationList.last_update`
  in favor of the new timezone-aware variants mentioned above.
* Added support for
  :class:`~cryptography.hazmat.primitives.ciphers.algorithms.ChaCha20`
  on LibreSSL.
* Added support for RSA PSS signatures in PKCS7 with
&lt;/tr&gt;&lt;/table&gt; 
</code></pre>
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/pyca/cryptography/commit/4e64baf360a3a89bd92582f59344c12b5c0bd3fd"><code>4e64baf</code></a>
42.0.0 version bump (<a
href="https://redirect.github.com/pyca/cryptography/issues/10232">#10232</a>)</li>
<li><a
href="https://github.com/pyca/cryptography/commit/7cb13a3bc91b7537c6231674fb5b0d2132a7edbe"><code>7cb13a3</code></a>
we'll ship 3.2.0 for 42 (<a
href="https://redirect.github.com/pyca/cryptography/issues/9951">#9951</a>)</li>
<li><a
href="https://github.com/pyca/cryptography/commit/605c74e41c75edc717f21afaa5e6a0eee9863a10"><code>605c74e</code></a>
Bump x509-limbo and/or wycheproof in CI (<a
href="https://redirect.github.com/pyca/cryptography/issues/10231">#10231</a>)</li>
<li><a
href="https://github.com/pyca/cryptography/commit/97578b98ffc417864e07d0ff9b76c02d2cb4e6da"><code>97578b9</code></a>
Bump BoringSSL and/or OpenSSL in CI (<a
href="https://redirect.github.com/pyca/cryptography/issues/10230">#10230</a>)</li>
<li><a
href="https://github.com/pyca/cryptography/commit/972a7b5896a6047ea43a86db87820ab474d898ff"><code>972a7b5</code></a>
verification: add test_verify_tz_aware (<a
href="https://redirect.github.com/pyca/cryptography/issues/10229">#10229</a>)</li>
<li><a
href="https://github.com/pyca/cryptography/commit/41daf2d86dd9bf18081802fa5d851a7953810786"><code>41daf2d</code></a>
Migrate PKCS7 backend to Rust (<a
href="https://redirect.github.com/pyca/cryptography/issues/10228">#10228</a>)</li>
<li><a
href="https://github.com/pyca/cryptography/commit/d54093e62e7e68c02efbb4d6a09162ddb39bf72f"><code>d54093e</code></a>
Remove some skips in tests that aren't needed anymore (<a
href="https://redirect.github.com/pyca/cryptography/issues/10223">#10223</a>)</li>
<li><a
href="https://github.com/pyca/cryptography/commit/71929bd91f34213b9f4a3a0a493c218c35fa25eb"><code>71929bd</code></a>
Remove binding that's not used anymore (<a
href="https://redirect.github.com/pyca/cryptography/issues/10224">#10224</a>)</li>
<li><a
href="https://github.com/pyca/cryptography/commit/7ea4b89cea553ce0f641ed29e1ce2e3e34278f1d"><code>7ea4b89</code></a>
fixed formatting in changelog (<a
href="https://redirect.github.com/pyca/cryptography/issues/10225">#10225</a>)</li>
<li><a
href="https://github.com/pyca/cryptography/commit/410f4a1ee4cbf46fe7e969bb48fccf261f74bbcd"><code>410f4a1</code></a>
Allow brainpool on libressl (<a
href="https://redirect.github.com/pyca/cryptography/issues/10222">#10222</a>)</li>
<li>Additional commits viewable in <a
href="https://github.com/pyca/cryptography/compare/41.0.6...42.0.0">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=cryptography&package-manager=pip&previous-version=41.0.6&new-version=42.0.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
You can disable automated security fix PRs for this repo from the
[Security Alerts
page](https://github.com/bacalhau-project/bacalhau/network/alerts).

</details>

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: David Aronchick <[email protected]>
We no longer need the inmemory version of the jobstore, and so this
commit removes it from the codebase.

This should resolve #3418
We are reasonably comfortable with the compute node's bolt store and
there inmemory store was only getting use in tests and not in use.

The inmemory store and inlocal store are included as separate commits,
so if we don't want to remove the inlocal store then we can just remove
the second commit
[92d7c39](92d7c39)

Resolves #3423
Storyboard is now working great. In the process, I had to finalize
swapping out the remainder of the (out of date) create react app for
Webpack, which is also now working properly.
Previously the v1 API client did not send auth headers. It now does.

This commit updates the example anon mode policy to support v1 APIs in
a similar way to v2, i.e. to allow people with auth tokens to submit
jobs and allow people without auth tokens to access any other endpoint
This automatically takes account of any necessary authorization.
This is intended to allow eslint to correctly run linting workflow in
github actions which is currently complaining that `vite-env.d.ts` is
not included.
Currently the makefile does not work on a clean checkout because of the
missing webui/node_modules directory.
This commit adds simple authentication to our NATS based networking.
Now, NATS servers will self-generate an auth token and print this as
part of their output config, and clients will need to connect with
this auth token. Clients that connect with no or wrong tokens will be
denied access (and will fail on boot).

Note that this doesn't add any *authz*, so connected clients can still
read and write the full contents of the pubsub as desired.
Currently the main() func and the devstack are sharing a cancellable
context, and when it is cancelled by SIGINT, the main function exits
before the deferred tasks in devstack can complete. This resulted in
devstack not running a second time because of the existing pid or port
(or both) files.

Now main has its own context which is cancelled only when the job is
complete (or cancelled from SIGINT). This gives devstack enough time to
shutdown.
As it is no longer mandatory to have a publisher defined for a job
(either by the user, or using the default) there is a possibility of
confusion when stuff doesn't continue to default to IPFS.

To allow for the old behaviour to continue as a short-term measure, this
commit introduces the `--default-publisher` option for the devstack and
requester/compute node (launched by serve).

Using the `--default-publisher` flag is similar to specifying a
publisher, and so to continue the same behaviour as we currently have
where the CLI has a default, we would start the node with
`--default-publisher ipfs`.

This has been implemented for both the requester and orchestrator (v1
and v2) job submissions.
This should avoid them being regenerated everytime we run `make
generate` to add new config items.
Fixing a bug in channel transformer where it tries to unmarshal
`models.ExecutionLog` where the object is
`concurrency.AsyncResult[models.ExecutionLog]`
TLS certificates were being generated by a bash script and stored in the
testdata directory.

Certificates/keys are now generated during the test suite build and
stored in a temp directory, so private keys and certs are always removed
once testing is completed

fixes #3337
- when main changes this action triggers a deployment to the development
  environment
Bumps
[github.com/aws/aws-sdk-go-v2](https://github.com/aws/aws-sdk-go-v2)
from 1.21.1 to 1.24.1.
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/aws/aws-sdk-go-v2/commit/47dd1b1bcbde244357a82ef00fa6a61a9bfb9b39"><code>47dd1b1</code></a>
Release 2024-01-04</li>
<li><a
href="https://github.com/aws/aws-sdk-go-v2/commit/23145e3e605a93582020facfe13350b4153714e1"><code>23145e3</code></a>
Regenerated Clients</li>
<li><a
href="https://github.com/aws/aws-sdk-go-v2/commit/00e57bb7feb2d104387103aa4fbcd55dfff3a6a7"><code>00e57bb</code></a>
Update API model</li>
<li><a
href="https://github.com/aws/aws-sdk-go-v2/commit/73e1a99f2fa858ca56627779852768a9198ba057"><code>73e1a99</code></a>
Merge pull request <a
href="https://redirect.github.com/aws/aws-sdk-go-v2/issues/2441">#2441</a>
from RanVaknin/fix-documentation-config</li>
<li><a
href="https://github.com/aws/aws-sdk-go-v2/commit/0f8ad11593c219c52ad7fb12998c75ade39dc7ad"><code>0f8ad11</code></a>
Fix SRA auth trailing checksum retry bug (<a
href="https://redirect.github.com/aws/aws-sdk-go-v2/issues/2438">#2438</a>)</li>
<li><a
href="https://github.com/aws/aws-sdk-go-v2/commit/efbc5aa622a882167129e69a88aa50c730cd1904"><code>efbc5aa</code></a>
Release 2024-01-03</li>
<li><a
href="https://github.com/aws/aws-sdk-go-v2/commit/78357bb87682230e24b15c01e807d7375a9474e4"><code>78357bb</code></a>
Regenerated Clients</li>
<li><a
href="https://github.com/aws/aws-sdk-go-v2/commit/e465ddd60d18e91b34de5917534cfa1542323027"><code>e465ddd</code></a>
Update endpoints model</li>
<li><a
href="https://github.com/aws/aws-sdk-go-v2/commit/384ad3b7ec21eecb3c6c38b69f86fb6342906b11"><code>384ad3b</code></a>
Update API model</li>
<li><a
href="https://github.com/aws/aws-sdk-go-v2/commit/1126a91e53a20b18bc1db74225a8417bfb63f427"><code>1126a91</code></a>
changelog added</li>
<li>Additional commits viewable in <a
href="https://github.com/aws/aws-sdk-go-v2/compare/v1.21.1...v1.24.1">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=github.com/aws/aws-sdk-go-v2&package-manager=go_modules&previous-version=1.21.1&new-version=1.24.1)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: David Aronchick <[email protected]>
This commit upgrades the authoriser to pass all the metadata necessary
to verify received JWTs, assuming that they were signed by this node.

(Policy authors are free to verify tokens issued from other sources
and ignore the material passed to the policy.)

Policies can now call `io.jwt.decode_verify` as in the ns_anon policy
to verify submitted JWTs.
The full job spec is persisted alongside each execution, and retrieved
in our `bacalhau job executions <jobid> --output yaml` and
`api/v1/orchestrator/jobs/:jobid/executions` API. This is is unnecessary
and prints a lot of information in the cli.

This PR makes does the following changes:
1. don't persist the job spec with each execution
2. introduce an optional `IncludeJob` option to `GetExecutions` to
populate the execution with the job spec

Note that `IncludeJob` is not exposed in `bacalhau job executions
<jobid>` cli as it doesn't make sense to print the job spec here. Though
it will make sense once we introduce `bacalhau execution ...` CLIs
@Nasfame Nasfame merged commit 56d57f1 into DeCenter-AI:main Feb 14, 2024
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants