Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

partial implementation of new lightbeam create functionality #56

Draft
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

tomreitz
Copy link
Collaborator

@tomreitz tomreitz commented Oct 7, 2024

This is a draft PR with a partial implementation of a new function, lightbeam create (typically used with a -s selector), which, in the current dir, derived from the Ed-Fi API's Swagger:

  • creates (if it doesn't already exist, otherwise adds/overwrites) a partial earthmover.yml with empty sources and transformations but destinations for each selected endpoint, plus comments indicating what column names and data types/values are required, and the required grain of the table (based on Swagger isIdentity fields)
  • creates (or overwrites) templates/*.jsont for each selected endpoint, with skeleton JSON that includes all the required fields (including nested ones), optional fields wrapped in conditionals, and comments with some of the property metadata from Swagger, like type, description, isIdentity, etc.
  • outputs a message that points to docs somewhere that explain how to install earthmover and finish out the earthmover.yml's sources and transformations

(The idea for this feature arose from discussions with a developer who figured out how to use earthmover but was struggling to learn how to construct the Ed-Fi payloads. lightbeam create will help scaffold developers by automatically creating some of the boilerplate code required for an earthmover Ed-Fi project.)

I want to wait for PR 54 to be reviewed and merged so I can incorporate its refactor of api.get_params_for_endpoint() and addition of util.get_swagger_ref_for_endpoint() here. I'd like to implement api.get_params_for_endpoint(..., type='all') to build the sample payload for each endpoint, such as

{
  "property_bool": true,
  "property_int": 1,
  "property_float": 1.0,
  "property_string": "string",
  "property_date": "date",
  "property_string_optional": "string",
  "property_descriptor": "uri://ed-fi.org/SomeDescriptor#SomeValue",
  "property_object": {
    "property_object_1": "string",
    "property_object_2": "string"
  },
  "property_array": [
    {
      "property_array_1": "string",
      "property_array_2": "string"
    }
  ]
}

which can then be turned into a JSON template such as

{
  "property_bool": {{property_bool}},
  "property_int": {{property_int}},
  "property_float": {{property_float}},
  "property_string": "{{property_string}}",
  "property_date": "{{property_date}}",
  {% if property_string_optional %}
  "property_string_optional": "{{property_string_optional}}",
  {% endif %}
  "property_descriptor": "uri://ed-fi.org/SomeDescriptor#{{property_descriptor}}",
  "property_object": {
    "property_object_1": "{{property_object_1}}",
    "property_object_2": "{{property_object_2}}"
  },
  "property_array": [
    {% for item in property_array %}
    {
      "property_array_1": "{{item.property_array_1}}",
      "property_array_2": "{{item.property_array_2}}"
    } {% if not loop.last %},{% endif %}
    {% endfor %}
  ]
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant