Production Test & Sandbox Data/Requests

[travisgosselin]

Historically at SPS Commerce customer configurations and connection testing for onboarding and setting up new customers was done in a separate environment called preprod with distinct deployable services and matching persistent data only used for testing and sandboxing purposes. This preprod environment matched prod application versions as part of the deployment pipelines. This enabled a sandbox environment to validate runtime and dynamic data used by customer implementation teams.

However, a separate environment and deployment to solve this problem are more of a stopgap that provides a quick win without modification of the platform capabilities. In reality, over time this extra environment breaks down as a strategy:

• Additional regular deployment overhead and speed to production, including extra configuration to be maintained (i.e. Database and persistence configuration, feature flag configs, etc).
• Additional financial cost on a growing environment that is questionable on its ability to provide tangible value.
• More importantly, data interoperability becomes a concern:
  • Do all services require a preprod environment and are isolated against the entire platform?
  • If some services do not require a preprod environment for customer testing, do they still have to create one to support the wider initiative?
  • If the platform of services has some service not in preprod then data is intermixed with identifiers between preprod and prod environments.
  • Services not using preprod that interact with services that do, will need to code decision statements with two different API URLs.

To sum it up: preprod is a use case, not an environment (from @acchristensen ). It is a feature, requirement and/or capability of the system. Assuming this stance moving forward then, it is fair to ask the question with regard to API Standards at SPS Commerce, how do we effectively indicate test or sandbox data has part of our requests, that is obvious, convenient, and consistent across the platform.

As a point of clarification, the API Standards already indicate the "environment" names should not appear within the URL Path (https://spscommerce.github.io/sps-api-standards/standards/url-structure.html#host). An argument could be made that this is no longer an environment and it would be appropriate to represent the URLs as such:

https://api.spscommerce.com/items/attributes			# access item attributes fully published in production
https://api.spscommerce.com/items/preprod/attributes	# access item attributes that are considered `preprod`.

I think the preprod term carries too much baggage at SPS to continue in its usage and has many other implications to the external industry that should not be exposed as shown above in any form. Additionally, having duplicate URLs with slightly augmented paths for preprod or "test" data is usually undesirable for API implementors and should be taken into consideration. This also doesn't represent a valid hierarchy, as the type of data is more an attribute itself and goes against our existing design guidelines.

Thinking of preprod or sandbox data appropriately as an attribute of the resource request seems to be ideal. It should be addressed in a consistent way across the API, and not be required, defaulting of course to "production" data.

For example:

# Query String - Makes it easy to make a GET request from a browser and apply sandbox mode
curl -X GET https://api.spscommerce.com/items/attributes  				
curl -X GET https://api.spscommerce.com/items/attributes?sandbox=true 

# Header - API Consumers can easily apply headers to switch all methods between production and sandbox mode...
curl --header -X GET https://api.spscommerce.com/items/attributes
curl --header "SPS-Sandbox: true" -X GET https://api.spscommerce.com/items/attributes

Thoughts?

• What method is best for indicating sandbox data?
• Do we move away from preprod terminology to distinguish the pattern? Do we call it sandbox? other?
• It is likely there may be some endpoints in your pattern that are agnostic of sandbox data, if the query string or header is provided do we issue a 400 response with details?
• How does this integration with the SPS Authorization JWT tokens provide information about if it's a test token? Should this be done through authorization only?
• Are there considerations for "action" style endpoints that would be used to migrate or promote data from sandbox to production? (a benefit of having one app, not different environments).
• Should there be a method for indicating this type of sandbox data or should teams be using promotable and versioned configurations with domain-specific switches on what is in use? (its not sandbox data it is an untested, non-live version of the configuration).

I believe both Rules Management API (@eggilbert ) and Inventory API (@shifr) have both token approaches to this scenario. Please if you are able to provide your approaches and thoughts, that would be much appreciated.

[eggilbert]

Our service was brand new and didn't have dependencies on any existing services in "preprod", so we were largely able to ignore most of the interoperability considerations mentioned above.

Are there considerations for "action" style endpoints that would be used to migrate or promote data from sandbox to production? (a benefit of having one app, not different environments).

This is probably the closest thing to what we did end up doing?

One of our design goals was to avoid the historical patterns of having multiple environmental databases in production. We didn't want to burden users with creating Rule artifacts, test them, change statuses, etc. and then do it all over again in the "real" production environment. Instead, we built our system around having a single set of artifacts, and just managing the state of which things are currently active/live, so we were effectively moving pointers around and recording a log of changes, instead of duplicating everything and leaving it to users to keep equivalent things in sync.

That said, it our solution isn't a sandbox vs production segmentation of data. It's more a "currently active" vs "not active/everything else" distinction if that makes sense? There's still just one set of data, but with some lifecycle statuses. Users are still able to test changes to Rules because of other product features to allow that happen in a safe way via optional request body parameters. That isn't switching between data Segment A and data Segment B (i.e. sandbox vs production), but instead enables a sort of testing mode where we return non-active results as instructed to simulate what would happen should they be made active - before any changes are actually performed or persisted.

I'm not sure this is necessarily a general pattern that would work for everything though. But, it made sense for our service and our other design goals of having (mostly) immutable artifacts.

[travisgosselin]

Yep that makes a lot of sense to me. I think of that as the "data versioning" model or promotable configurations as the last bullet point mentioned. You described it way better, and I'm a big fan of that solution over anything else discussed prior. I'm not sure I know enough across our systems to understand if such a model would work well universally for SPS or worked well only in your specific use cases.

[eggilbert]

Yeah, for us this is a core product feature. We very intentionally only have one copy of data. But there is a first-class citizen entity in the domain used to describe these "overrides" of current state, which get their own identifiers and management. And on the main processing endpoint you can optionally give us a reference to one and we'll use its contents in the processing. It seems like it would be kind of weird to bury that in a header.

[acchristensen]

Nice @travisgosselin, I think this sums things up really well. A couple other thoughts, here to build on this conversation.

Regarding @eggilbert, response, the Rules Mgmt service he's discussing is really built to help other services handle this problem. In terms of thinking of the data (rules) as active or not, that distinction is to support other services that want to know what their configuration should be if there request is production (active) or "preprod" (not active).

Taking a little bit of a turn, if we think about AWS's services, those are all production (from AWS' perspective), even if their customer is using an instance of some service for testing/integration/staging/whatever. But API Gateway has an interesting functionality where a "stage" can be set. It's somewhat open to how a stage would be used, but stages do let you use the production gateway instance to serve things through a development lifecycle. It's flexible in the since that you can decide what your stage is named.

Tying that back to the Rules Mgmt use case, it let's you create a temporary, named "override" of rules. So, a service using Rules Mgmt to get its config can also give it a named override in order to act as if the config is in production. You could also think of these as test scenarios.

So, when we talk about "preprod" as an environment there's also a limitation that there is only one state/override/test-scenario. Which, is probably okay sometimes, but there will be times where you need more than one active test scenario at a time. Not to mention that once something is in production there's a chance that preprod could become stale. So, thinking in terms of "preprod" as a use case, maybe it's really that preprod is a way to think about naming multiple, temporary configs for test scenarios we want to run.

So, bringing it back to API standards. I wonder if rather than saying SPS-TEST=true|false, or having the JWT have a claim for test: yes or test: no, or having a query param ?sandbox=true, I wonder if we actually want something more like testScenario=customer-update-123.

[travisgosselin]

Ya I'm really digging this more dynamic approach for sure - with named instances. I'm struggling with what a good name would be to represent this kind of thing - but we can keep iterating on that a bit. If this is the case, how would it feel to have rules mgmt API apply a standard header or query parameter with this approach? Correct me if I'm wrong but rules mgmt has different endpoints for working with different "versions" or "configurations" as you describe, and you pass these in as primary inputs/parameters? Can you break down some real examples of rules mgmt API to represent what you are describing?

[acchristensen]

In Rules Mgmt the way you put the override in to effect is to pass in the body to the /evaluate endpoint. That was done because we couldn't really make that endpoint strictly RESTful, and didn't want to mix-and-match query params with a body to a POST. So... that's some information.
Passing along which override/test-scenario should be used needs to take in to account the different protocols that we use, and so we'd need at least a handful of standards since queues are up there with apis for how we pass data from one stage of processing to the next. Some thoughts on this below in response to Alex's comment (regarding JWTs).
The real world example is... complex. Our implementation teams can use the UI create new instances of a Rule, then create the override. Then, when they set up data flow, they have to set which override is used for the one particular component (WREX extension) that currently uses Rules Mgmt and overrides. After that, for now, it's a request to the team to "manually" promote the override instance to be the active instance once testing is done and we're satisfied the new instance will have the desired effect.

[acchristensen]

Another thing regarding how we do this.

• The JWT does support a test claim. Can we formalize what it's meant for? Probably a longer term cutover, but it might be the "simplest" thing in the long run to always support the claim in the JWT along with another standardized method.
• Is there any reason we could support both headers and a query param? Maybe something like always use the Header as the primary way to indicate a test scenario. But consider supporting the query parameter for GETs if a consumer may find it useful to see data from the API in a browser.
• I think at one point for another API we pretty much considered calling "preprod" "test" because, from an external customer's perspective it's testing.
• I wonder if we need to always allow the test indicator (i.e. header or whatever). It kind of only matters if the API is doing something stateful and needs to look up or store data based on whether it's a test or not... AND if a service is calling another upstream service and needs to pass the test indicator on. Should probably remember that one (it just underlines the need for standardization).

[alexander-ivakhnenko]

I like the idea of using test claim and maintaining test context in a downstream chain. Overall "preprod" use case is about the isolation of the configuration and data context that need to be respected by APIs and data processing services. Like in the example of AWS services from @acchristensen, customer has option with tags, naming conventions, etc. to create an isolated context for data. I'm wondering if we should tie this together with versioning of the configuration data and if we have a use case beyond "preprod" (like "preprod" might be just a tag or label)? Obviously, "preprod" is already a use case today.

[acchristensen]

Regarding JWTs... if all of our services (API or Queue-based) needed to pass on a JWT (already true for APIs) then this would be a great way to propagate which override/test-scenario is to be used (or not) without a service really needing to think about it. If we went a step further and had configuration for processing always pulled out of Rules Mgmt... we're kinda done.

[acchristensen]

Regarding preprod already being a use case... I mean sure. I don't think the idea of "many preprods" is "we already do this, so what would it mean to do a preprod 2". I think this is more a reframing of customer-focused testing that we do. And maybe an explicit acknowledgement that we do do things like this. It's just that right now you need to create a whole new setup, or new elements of a setup, or overwrite an existing setup to be able to test more than one "preprod". So, make it explicit rather than implicit and something where an implementor needs to be clever about how to handle this use case.

[greylurk]

I think the best name for Preprod would actually have been "Integration", since that's what it's used for, but that ship has sailed unfortunately, and it's gone the wrong direction :)

I think that propagating the "test" claim on the JWT in M2M transactions could be a challenging thing to validate and rely on. It would require us to establish two tokens with nearly identical permissions in each environment. This could be automated of course, but M2M APIs will need to recognize the test claim, and then select the appropriate token for use with downstream services, which is a vaguely dangerous failure point.

[travisgosselin]

I think the best name for Preprod would actually have been "Integration", since that's what it's used for, but that ship has sailed...

haha - totally.

"test" claim token

Everything you say is true, but what Aaron is discussing is the reality of what Atlas provides today, including OPA that recognizes this claim already. JWT tokens are already generated for new apps for both scenarios by default. But, most apps do not use that claim or respect it - likely because we haven't figured out how to leverage it consistently outside of DevCenter APIs yet.

[jbongaarts]

An important feature of a sandbox is the box. Isolation is critical. Its true of any app, but especially in a microservice workflow-based application like Fulfillment that the application code having conditionals for test data and being responsible for propagating it though the entire workflow has countless points of failure that are easy to miss or invert with network-wide consequences that could be especially challenging to remediate. Additionally, code that has conditionals for testing are not actually testing with the code that will be used with production data, which reduces the value of the test, sometimes to zero.

While I prefer a separate environment because it is simply not possible to cross the boundary between the sandbox and production, if we are not willing to go that far then the next level of isolation would be the Organization.

Rather than putting the burden on applications to have any awareness of whether data or a transaction is "test data", we can instead issue customers "Sandbox" organizations and provide tooling to easily create or "refresh" the sandbox org from another existing org. While a sandbox org could have some indicator that it is a sandbox (for billing or metrics for example) services involved in data flow would not have a reason to be aware of it. The data would be routed and processed like any other organization. If we go a step further and create "well-known" conventions for test profiles (presumably requiring them for v4 retailers), we could have the organization creation/reset wire up trading partner relationships automatically. If we auto-provision SFTP, AS2 or ICA accounts to mirror the origin organization, we could potentially reach a point where customers could self-service their sandboxes.

[travisgosselin]

The implication of this discussion though would be to envision the idea of a service like NCS having only one Environment (i.e. PROD) and how the API would be augmented to support "preprod" or integration like data in one instance - where you could request request map paths for the scenario of testing you need. Obviously it's mostly hypothetical for short or medium-term for service like NCS that are not changing any time soon, but more of an immediate reality as we build new services potentially.

[greylurk]

Currently the network configuration information is one the primary differences between "preprod" and "prod" though. In order to merge the two, we'd have to combine the two databases, which would result in key collisions, so the key lookups would have to include the "integration" or "sandbox" as a parameter (possibly using the JWT as the vessel for that parameter). NCS could be a primary gatekeeper of the division between the integration mode and production mode, but as I said, there's still foundational work to be done decoupling the primary identifiers.

[travisgosselin]

Agreed - it won't just work today, but envisioning a future, that seems reasonable.

[acchristensen]

I'd rather avoid re-re-re-litigating the namings we've settled on, if we can. I think we've committed (as an organization) at least to what integration and production are. Continuing to use the term preprod is probably just fine, as well... if we decide to have a dynamic name for customer test scenarios, then we can still "classify" it with the term preprod. This is part of the "re-framing" exercise.

And like Travis points out, I'm thinking in terms of how we set our architectural guidepost, not immediate next steps.

Around propagating JWTs, whether M2M or not, yeah, this would make it tricky to encode in the JWT when the test name is dynamic. We'd have to think of a way to start a chain of requests from the sender and have the same JWT passed through the entire request. But, this is just context that has to get passed through, and the JWT is just a way to house that context. There probably would be non-JWT things that would have to get passed through, so maybe it's not that big of a deal to put the test scenario name in the JWT. The nice thing, though, is we'd have a secure token that we're forcing ourselves to use the whole way through a request, making each step auditable back to the "user" that initiated the request/processing chain. We would need to have a way to create, on behalf of the initiator, the actual JWT with the test name as a claim for service-initiated actions (most of what we do). That is doable, and there probably are benefits to authorizing on setting that claim and passing it through the system.

Regarding NCS and duplicating/merging/migrating/etc. This is definitely going to be a big hurdle we'll have to figure out. Regardless of how we indicate which test scenario is in play, the second we want to have more than one test scenario (i.e. beyond the one preprod test scenario) we need to duplicate NCS if we used a parity-based solution (i.e using the current model). And the duplication would happen over and over again for each test scenario. Add in the non-Rule Mgmt config dbs (as you point out, Adam) and we're looking at something that isn't really feasible. We could find a better model for each service/config-db individually... but, longer term I think we need to find our common pattern for config if we want to support multiple test scenarios. Considering how we'd adapt NCS to this is something we'd need to think about. We'd also have to consider the various other services and their config dbs. Which is why I'm thinking about Rule Mgmt, since it's built for the access patterns that seem to align to the multi-test-scenarios use case, as well as providing purpose-built, schema-verified, audited, granularly authorized (some day), referentially integrable, context-sensitive, change impact tracked, config storage.

[nicholas-s-perkins]

Rather than putting the burden on applications to have any awareness of whether data or a transaction is "test data", we can instead issue customers "Sandbox" organizations

I would second that we don't really want to try to maintain that every app in the company has some if block that's like "if sandbox token use this other database". That comes across to me as preprod accelerationism.

Sandbox Orgs maybe even solve the current mess we have with UIs, since there wouldn't be this prod/preprod toggle dance that every UI would have to independently solve. You could however update something like nexus to be able to bind an org with it's sandbox org in some way so you could compare them easily.

Thinking about it more, if you allowed arbitrary amounts of sandbox orgs, you could even build a setup versioning system this way.

NCS is probably the single most important place where those pieces of data should be separated in the primary data workflow.

Yeah, though abstractly we can think of NCS as just being some "routing" data and it has a logical translation between prod and preprod. In fact, there is already one (or three?) tools that can migrate a setup from prod to preprod meaningfully. If you went with @jbongaarts idea around an Organization sandbox, you would basically just be syncing a setup between two orgs where the migrator in this case would be the key thing that defines the difference between a prod and sandbox org.

Though I might note that some of this is also complicated because what a "setup" is at the company is still a very organic thing, and definitely extends well beyond NCS and also maybe Orgs? But that thought starts getting deeper into why preprod-as-env is a sticky strategy. I could see that maybe Org is one of the only existing concepts that could be rallied around outside of envs.

Could also rally around salesforce projects, but I think they are already tied to orgs?

[travisgosselin]

Header based solution following discussions on SPS Execution Context:
#39