Skip to content

Commit

Permalink
DASB-234 - Implement report counts data projection (#58)
Browse files Browse the repository at this point in the history 
* Code cleanup

* Start of hl7v2 counts implementation

* More detailed counts by stage name

* Optimized counts query and logic

* Renamed endpoint to be generic for getting report counts

* Send hl7 messages in a batch

* Updated service bus library to latest

* Add retry logic for persisting reports to cosmosdb

* Create report retries rework

* Added constants for default retry interval and max retries

* Added more logging to creating reports

* Fixes to creating reports logic

* Added uploadId to creating report log messages

* Added calculated backoff interval based on retry number

* Fixed bug in retry interval when status code unknown

* Added more useful links to comments surrounding cosmosdb 429s

* Updated azure cosmos library to 4.55.0

* Name changes in counts endpoint to be more generic

* Create initial container with 50000 R/Us

* Added new endpoint for getting counts with query params

* Set R/Us back to 1000, the minimum

* Updated upload status to use new content format in reports

* Code cleanup

* Fixed timestamp and double to long number policy with json conversions

* Don't show time uploading when metadata verify failed

* Fixed bug with wrong stage report id returned when creating a report

* Moved aggregate summary class to its own file, added class comment header

* Special counting based on schema name in progress

* Code cleanup

* Special counting based on schema name working for aggregate counts as well

* Code cleanup

* Get counts for structure validation summary results

* Code cleanup

* Updated postman and openapi spec for new counts endpoints

---------

Co-authored-by: Matt B Krystof <[email protected]>
  • Loading branch information
@mkrystof
mkrystof and Matt B Krystof authored Feb 18, 2024
1 parent 9ae2599 commit dfd4952
Show file tree
Hide file tree
Showing 23 changed files with 1,096 additions and 121 deletions.
171 changes: 158 additions & 13 deletions docs/processing-status-openapi-spec.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ tags:
description: Find out more
url: 'https://github.com/CDCgov/data-exchange-csv'
- name: DEX Status
description: CDC Data Exchange Status
description: CDC Data Exchange Status and Data Projections
- name: DEX Info
description: CDC Data Exchange Info
- name: Health
Expand Down Expand Up @@ -220,9 +220,9 @@ paths:
get:
tags:
- DEX Tracing
summary: Gets a processing status trace from a report ID.(Not yet implemented)
summary: Gets a processing status trace from a report ID. (Not yet implemented)
description: >-
Retrieves a distributed tracing trace.(Not yet implemented)
Retrieves a distributed tracing trace. **(Not yet implemented)**
parameters:
- name: reportId
in: path
Expand Down Expand Up @@ -300,7 +300,7 @@ paths:
required: true
schema:
type: string
description: Report Stage Name.
description: Report Stage Name.
responses:
'200':
description: Trace for the given upload dentifier
Expand Down Expand Up @@ -340,7 +340,7 @@ paths:
required: true
schema:
type: string
description: Report Stage Name.
description: Report Stage Name.
responses:
'200':
description: Trace for the given upload dentifier
Expand Down Expand Up @@ -382,7 +382,9 @@ paths:
put:
tags:
- DEX Reports
summary: Ammend an existing report using the provided trace identifier to lookup the report.(Not yet implemented)
summary: Replace an existing report. (Not yet implemented)
description: >-
Replace an existing report using the provided trace identifier to lookup the report. **(Not yet implemented)**
parameters:
- name: traceId
in: path
Expand Down Expand Up @@ -431,15 +433,16 @@ paths:
get:
tags:
- DEX Reports
summary: Provides list of report identifiers for given criteria(Not yet implemented)
summary: Provides list of report identifiers for given criteria. (Not yet implemented)
description: >-
Provides list of report identifiers for the given criteria. **(Not yet implemented)**
parameters:
- name: ext_event
in: query
schema:
type: string
description: >-
If specified, also filter by meta_ext_event that was provided with
the upload.(Not yet implemented)
If specified, also filter by meta_ext_event that was provided with the upload.
example: RoutineImmunization
- name: destinationId
in: path
Expand Down Expand Up @@ -492,10 +495,12 @@ paths:
tags:
- DEX Info
summary: >-
Provides a list of all known destination IDs.(Not yet implemented)
Provides a list of all known destination IDs. (Not yet implemented)
description: >-
Provides a list of all known destination IDs. **(Not yet implemented)**
responses:
'200':
description: All of the known destination identifiers(Not yet implemented)
description: All of the known destination identifiers.
content:
application/json:
schema:
Expand All @@ -509,7 +514,9 @@ paths:
tags:
- DEX Info
summary: >-
Provides a list of all known event types for the given destination identifier.(Not yet implemented)
Provides a list of all known event types. (Not yet implemented)
description: >-
Provides a list of all known event types for the given destination identifier. **(Not yet implemented)**
parameters:
- name: destinationId
in: path
Expand All @@ -534,7 +541,9 @@ paths:
tags:
- DEX Info
summary: >-
Provides a list of all known report schema identifiers.(Not yet implemented)
Provides a list of all known report schema identifiers. (Not yet implemented)
description: >-
Provides a list of all known report schema identifiers. **(Not yet implemented)**
responses:
'200':
description: All of the known report schema identifiers
Expand Down Expand Up @@ -651,6 +660,81 @@ paths:
description: Bad request. The destination identifier provided was not found.
'401':
description: Authorization information is missing or invalid.
'/report/counts/{uploadId}':
get:
tags:
- DEX Status
summary: Report counts for the given upload identifier.
description: >-
Provides report counts for the given upload identifier.
parameters:
- name: uploadId
in: path
required: true
schema:
type: string
format: uuid
description: >-
Upload identifier to retrieve the report counts for.
responses:
'200':
description: Report counts for the given upload identifier
content:
application/json:
schema:
$ref: '#/components/schemas/ReportCountsForUploadId'
'400':
description: Bad request. The destination identifier provided was not found.
'401':
description: Authorization information is missing or invalid.
'/report/counts':
get:
tags:
- DEX Status
summary: Report counts for the given criteria.
description: >-
Provides report counts for the given criteria.
parameters:
- name: destination_id
in: query
required: true
schema:
type: string
description: Filter by the destination identifier to retrieve the report counts for.
- name: ext_event
in: query
required: true
schema:
type: string
description: Filter by the ext event that was provided with the upload.
example: routineImmunization
- name: date_start
in: query
schema:
type: string
description: >-
Start date of filter. Format is YYYYMMDDTHHMMSSZ (UTC format,
inclusive). Defaults to 3 months prior to today.
example: 20231020T14103000Z
- name: date_end
in: query
schema:
type: string
description: >-
End date of filter. Format is YYYYMMDDTHHMMSSZ (UTC format,
exclusive). Defaults to today.
example: 20231030T14103000Z
responses:
'200':
description: Report counts for the given upload identifier
content:
application/json:
schema:
$ref: '#/components/schemas/ReportCountsForUploads'
'400':
description: Bad request. The destination identifier provided was not found.
'401':
description: Authorization information is missing or invalid.
'/health':
get:
tags:
Expand Down Expand Up @@ -904,6 +988,67 @@ components:
format: uuid
stage_name:
type: string
ReportCountsForUploadId:
type: object
properties:
upload_id:
type: string
format: uuid
destination_id:
type: string
example: ndlp
event_type:
type: string
example: routineImmunization
stages:
type: object
properties:
dex-metadata-verify:
type: integer
example: 1
dex-upload-status:
type: integer
example: 1
dex-routing:
type: integer
example: 1
dex-hl7-receiver:
type: object
properties:
counts:
type: integer
example: 100
number_of_messages:
type: integer
example: 96
number_of_messages_not_propogated:
type: integer
example: 4
dex-hl7-structure:
type: object
properties:
counts:
type: integer
example: 96
valid:
type: integer
example: 90
invalid:
type: integer
example: 6
ReportCountsForUploads:
type: object
properties:
summary:
type: object
properties:
num_uploads:
type: integer
example: 14
uploads:
type: array
items:
$ref: '#/components/schemas/ReportCountsForUploadId'
HL7Report:
type: object
properties:
Expand Down
4 changes: 2 additions & 2 deletions processing-status-api-function-app/build.gradle
View file
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ configurations {

dependencies {
implementation "com.microsoft.azure.functions:azure-functions-java-library:3.0.0"
implementation 'com.azure:azure-cosmos:4.42.0'
implementation 'com.azure:azure-cosmos:4.55.0'
implementation "com.fasterxml.jackson.module:jackson-module-kotlin:2.14.2"
implementation 'io.opentelemetry:opentelemetry-api:1.29.0'
implementation 'io.opentelemetry:opentelemetry-sdk:1.29.0'
Expand All @@ -44,7 +44,7 @@ dependencies {
implementation group: 'ch.qos.logback.contrib', name: 'logback-json-classic', version: '0.1.5'
implementation group: 'ch.qos.logback.contrib', name: 'logback-jackson', version: '0.1.5'

implementation 'com.azure:azure-messaging-servicebus:7.13.3'
implementation 'com.azure:azure-messaging-servicebus:7.15.0'
implementation 'com.microsoft.azure:azure-servicebus:3.6.7'
implementation 'com.azure:azure-identity:1.8.0'

Expand Down
24 changes: 24 additions & 0 deletions ...api-function-app/src/main/java/gov/cdc/ocio/processingstatusapi/FunctionJavaWrappers.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@
import gov.cdc.ocio.processingstatusapi.functions.HealthCheckFunction;
import gov.cdc.ocio.processingstatusapi.functions.reports.CreateReportFunction;
import gov.cdc.ocio.processingstatusapi.functions.reports.GetReportFunction;
import gov.cdc.ocio.processingstatusapi.functions.status.GetReportCountsFunction;
import gov.cdc.ocio.processingstatusapi.functions.status.GetUploadStatusFunction;
import gov.cdc.ocio.processingstatusapi.functions.reports.ServiceBusProcessor;
import gov.cdc.ocio.processingstatusapi.functions.status.GetStatusFunction;
Expand Down Expand Up @@ -196,6 +197,29 @@ public HttpResponseMessage getReportByStage(
return new GetReportFunction(request).withDestinationId(destinationId, stageName);
}

@FunctionName("GetReportCountsByUploadId")
public HttpResponseMessage getReportCountsByUploadId(
@HttpTrigger(
name = "req",
methods = {HttpMethod.GET},
route = "report/counts/{uploadId}",
authLevel = AuthorizationLevel.ANONYMOUS
) HttpRequestMessage<Optional<String>> request,
@BindingName("uploadId") String uploadId) {
return new GetReportCountsFunction(request).withUploadId(uploadId);
}

@FunctionName("GetReportCountsWithQueryParams")
public HttpResponseMessage getReportCountsWithQueryParams(
@HttpTrigger(
name = "req",
methods = {HttpMethod.GET},
route = "report/counts",
authLevel = AuthorizationLevel.ANONYMOUS
) HttpRequestMessage<Optional<String>> request) {
return new GetReportCountsFunction(request).withQueryParams();
}

@FunctionName("GetStatusByUploadId")
public HttpResponseMessage getStatusByUploadId(
@HttpTrigger(
Expand Down
4 changes: 2 additions & 2 deletions ...ion-app/src/main/kotlin/gov/cdc/ocio/processingstatusapi/cosmos/CosmosContainerManager.kt
View file
Original file line number Diff line number Diff line change
Expand Up @@ -34,9 +34,9 @@ class CosmosContainerManager {
val containerProperties = CosmosContainerProperties(containerName, partitionKey)

// Provision throughput
val throughputProperties = ThroughputProperties.createManualThroughput(400)
val throughputProperties = ThroughputProperties.createAutoscaledThroughput(1000)

// Create container with 400 RU/s
// Create container with 1000 RU/s
logger.info("calling createContainerIfNotExists...")
val databaseResponse = db.createContainerIfNotExists(containerProperties, throughputProperties)

Expand Down
14 changes: 10 additions & 4 deletions ...p/src/main/kotlin/gov/cdc/ocio/processingstatusapi/functions/reports/GetReportFunction.kt
View file
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ import gov.cdc.ocio.processingstatusapi.model.*
import gov.cdc.ocio.processingstatusapi.model.reports.Report
import gov.cdc.ocio.processingstatusapi.model.reports.ReportDao
import gov.cdc.ocio.processingstatusapi.model.reports.ReportSerializer
import gov.cdc.ocio.processingstatusapi.utils.JsonUtils
import mu.KotlinLogging
import java.util.*

Expand All @@ -34,10 +35,15 @@ class GetReportFunction(
}

private val gson = GsonBuilder()
.registerTypeAdapter(
Report::class.java,
ReportSerializer()
).create()
.registerTypeAdapter(
Report::class.java,
ReportSerializer()
)
.registerTypeAdapter(
Date::class.java,
JsonUtils.GsonUTCDateAdapter()
)
.create()

/**
* Retrieve a report with the provided upload ID.
Expand Down
Loading

0 comments on commit dfd4952

Please sign in to comment.