Merge pull request #25 from SumoLogic/hpal_release_v1.0.11

Releasing v.1.0.11
SumoLogic · Nov 8, 2024 · 79e5fd8 · 79e5fd8
2 parents 60b82a8 + bae07d2
commit 79e5fd8
Show file tree

Hide file tree

Showing 21 changed files with 469 additions and 269 deletions.
diff --git a/.bandit b/.bandit
diff --git a/.flake8 b/.flake8
diff --git a/.github/workflows/runtest.yaml b/.github/workflows/runtest.yaml
@@ -9,17 +9,17 @@ jobs:
     steps:
       - uses: actions/checkout@v4
 
-      - name: Set up Python 3.10
+      - name: Set up Python 3.12
         uses: actions/setup-python@v5
         with:
-          python-version: '3.10'
+          python-version: '3.12'
 
       - name: Install dev dependencies
         run: python -m pip install -U pip flake8 bandit mypy
 
       - name: Run Security Tests
         run: |
-          bandit -c .bandit -r sumomongodbatlascollector/ -f custom
+          bandit -r sumomongodbatlascollector/ --skip B101,B105,B108,B113,B311,B314,B405,B404,B501,B506,B602,B604,B605,B607,B608
 
       - name: Run Linting Tests
         run: |
@@ -34,24 +34,49 @@ jobs:
       #     mypy --install-types ./api.py
       #     mypy ./api.py --disable-error-code=import-untyped
 
+  build-test:
+    name: "Run Build Tests"
+    runs-on: "ubuntu-latest"
+
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Set up Python 3.12
+        uses: actions/setup-python@v5
+        with:
+          python-version: 3.12
+
+      - name: Install build dependencies
+        run: |
+          python -m pip install -U pip wheel setuptools
+      - name: "Build"
+        run: |
+          python setup.py sdist bdist_wheel
+
+      - name: "Install"
+        run: |
+          pip install dist/sumologic_mongodb_atlas*.whl
+
+
   run-test:
-    name: "Run Build and Install Tests"
+    name: "Run Tests"
     runs-on: "ubuntu-latest"
 
     steps:
       - uses: actions/checkout@v4
 
-      - name: Set up Python 3.10
+      - name: Set up Python 3.12
         uses: actions/setup-python@v5
         with:
-          python-version: '3.10'
+          python-version: '3.12'
 
       - name: Install build and test dependencies
         run: |
           python -m pip install --upgrade pip
           pip install -e .
           pip install pytest
+          pip install setuptools
 
       - name: "Run Pytest"
         run: |
-          pytest -vvv sumomongodbatlascollector/test_mongo_collector.py
+          pytest -vvv tests/test_mongo_collector.py
diff --git a/INTERNAL_README.md b/INTERNAL_README.md
@@ -1,46 +1,59 @@
-We use sumoappclient for testing the lambda paackage. We can install it by: 
-```
-pip install sumologic-sdk
-```
-Description:
-```
-sumoappclient [-h] [-e {onprem,aws,gcp,azure}] [-d {prod,test,local}] [-c CONFIGPATH] [-pc PYTHON_CMD] [-g TARGET_FOLDER]
-```
+### OnPrem
 
-Deployment testing for PyPi:
-1.  For onprem pypi testing of this package, we perform these steps:
-    * Upgrade the version in the Version file, eg 1.0.10 -> 1.0.11
+1. For generating the build and running locally, we perform these steps:
+
+    * Upgrade the version in the `VERSION` file, eg 1.0.10 -> 1.0.11
+    * Run the `build.sh`.
+    * Create a `mongodbatlas.yaml` in home folder (for ex ~/sumo/mongodbatlas.yaml), refer the `sumomongodbatlascollector/mongodbatlas.yaml` file for instructions on each of the parameter. You can override any parameter in this file
+        ```
+        SumoLogic:
+          HTTP_LOGS_ENDPOINT: <Paste the HTTP Logs source URL from step 2.>
+          HTTP_METRICS_ENDPOINT: <Paste the HTTP Metrics source URL from step 2.>
+
+        MongoDBAtlas:
+          ORGANIZATION_ID: <Paste the Organization ID from step 1.>
+          PROJECT_ID: <Paste the Project ID from step 1.>
+          PRIVATE_API_KEY: <Paste the Private Key from step 1.>
+          PUBLIC_API_KEY: <Paste the Public Key from step 1.>
+        ```
+    * Run the below command to start the collector
+        ```
+            python -m sumomongodbatlascollector.main ~/sumo/mongodbatlas.yaml
+        ```
+
+2.  For deploying on test pypi account we perform these steps:
     * Run the following command:
         ```
-        sumoappclient -d sumotest -c mongodbatlas.yaml -e onprem
+        python -m twine upload dist/* --repository sumotestpypi
         ```
     * This deploys the package in the testing org of pypi via the credentials stored in the .pypirc file for the sumotestpypi section. You can find the file in the shared vault.
-2.  For onprem pypi production deployment, we perform these steps:
-    * Upgrade the version in the Version file, eg 1.0.10 -> 1.0.11
+3.  For deploying on prod pypi account we perform these steps:
     * Run the following command:
         ```
-        sumoappclient -d sumopypi -c mongodbatlas.yaml -e onprem
+        python -m twine upload dist/* --repository sumopypi
         ```
     * This deploys the package in the production org of pypi via the credentials stored in the .pypirc file for the sumopypi section. You can find the file in the shared vault.
 
-Deployment testing for AWS:
-1.  For testing of this package, we perform these steps:
-    * Update .aws file to use a personal aws account credentials
-    * Create a S3 bucket in the personal aws account
-    * Set two environment variables:
-        * export SAM_S3_BUCKET = 'name of the S3 Bucket created'
-        * export AWS_REGION = 'region of the deployment'
-    * Upgrade the SemanticVersion in the template.yaml file, eg 1.0.16 -> 1.0.17
-    * Run the following command:
-        ```
-        sumoappclient -d test -c mongodbatlas.yaml -e aws
+### AWS
+
+1.  For testing and deploying the lambda function, we perform these steps:
+    * Update .aws file to use a personal aws account credentials.
+    * Create a S3 bucket in the personal aws account.
+    * In samconfig.toml file update s3_bucket, region parameters.
+    * Generate credentials in Mongodb atlas portal and update parameter_overrides in samconfig.toml file.
+    * Upgrade the SemanticVersion in the template.yaml file and s3_prefix in samconfig.toml file
+    * Run the following commands:
+        ```
+            sam build
+            sam package
+            sam deploy
         ```
     * This deploys the package via a personal aws account onto AWS Serverless Application Repository
-2.  For production deployment, we perform these steps:
-    * Update .aws file to use sumocontent aws account credentials
-    * Upgrade the SemanticVersion in the template.yaml file, eg 1.0.16 -> 1.0.17
+    * After deploying go the lambda function created by above command and run the function by clicking on test button.
+2.  For publishing the sam application, we perform these steps:
+    * Update the s3_bucket parameter to appdevstore bucket
     * Run the following command:
         ```
-        sumoappclient -d prod -c mongodbatlas.yaml -e aws
+        sam publish
         ```
     * This deploys the package via the sumocontent aws account onto AWS Serverless Application Repository
diff --git a/MANIFEST.in b/MANIFEST.in
@@ -1,5 +1,6 @@
 include requirements.txt
 include LICENSE
 include VERSION
-include README
+include README.md
 include sumomongodbatlascollector/mongodbatlas.yaml
+recursive-exclude tests *
diff --git a/build.sh b/build.sh
@@ -0,0 +1,3 @@
+python setup.py bdist_wheel
+pip uninstall -y sumologic_mongodb_atlas
+pip install dist/sumologic_mongodb_atlas*.whl
diff --git a/packaged.yaml b/packaged.yaml
@@ -15,10 +15,10 @@ Metadata:
     - atlas
     - serverless
     - MongoDBAtlas
-    LicenseUrl: s3://appdevstore/MongoDBAtlas/v1.0.17/5122657d5b9a0d3713e24d3a33eae431
+    LicenseUrl: s3://appdevstore/MongoDBAtlas/v1.0.18/5122657d5b9a0d3713e24d3a33eae431
     Name: sumologic-mongodb-atlas
-    ReadmeUrl: s3://appdevstore/MongoDBAtlas/v1.0.17/3088978c83fe207a640a2584aff2c79d
-    SemanticVersion: 1.0.17
+    ReadmeUrl: s3://appdevstore/MongoDBAtlas/v1.0.18/3088978c83fe207a640a2584aff2c79d
+    SemanticVersion: 1.0.18
     SourceCodeUrl: https://github.com/SumoLogic/sumologic-mongodb-atlas
     SpdxLicenseId: Apache-2.0
 Outputs:
@@ -28,6 +28,10 @@ Outputs:
       Fn::GetAtt:
       - MongoDBAtlasFunction
       - Arn
+  MongoDBAtlasTable:
+    Description: MongoDBAtlasTable DynamoDB Table Name
+    Value:
+      Ref: MongoDBAtlasTableResource
 Parameters:
   HttpLogsEndpoint:
     Type: String
@@ -44,9 +48,11 @@ Parameters:
 Resources:
   MongoDBAtlasFunction:
     Properties:
-      CodeUri: s3://appdevstore/MongoDBAtlas/v1.0.17/56c4530782ed537044f71da21d57b605
+      CodeUri: s3://appdevstore/MongoDBAtlas/v1.0.18/5c5c3a5bfa0831572f5452a56ebc0b20
       Environment:
         Variables:
+          DBNAME:
+            Ref: MongoDBAtlasTableResource
           ENABLE_CONSOLE_LOG: 'false'
           ENVIRONMENT: aws
           HTTP_LOGS_ENDPOINT:
@@ -69,10 +75,37 @@ Resources:
       Handler: main.main
       MemorySize: 256
       Policies:
-      - AmazonDynamoDBFullAccess
+      - Statement:
+        - Action:
+          - dynamodb:PutItem
+          - dynamodb:UpdateItem
+          - dynamodb:GetItem
+          - dynamodb:Scan
+          - dynamodb:Query
+          - dynamodb:DescribeTable
+          Effect: Allow
+          Resource:
+            Fn::GetAtt:
+            - MongoDBAtlasTableResource
+            - Arn
+        Version: '2012-10-17'
       Runtime: python3.11
       Timeout: 900
     Type: AWS::Serverless::Function
     Metadata:
       SamResourceId: MongoDBAtlasFunction
+  MongoDBAtlasTableResource:
+    Properties:
+      AttributeDefinitions:
+      - AttributeName: key_col
+        AttributeType: S
+      KeySchema:
+      - AttributeName: key_col
+        KeyType: HASH
+      ProvisionedThroughput:
+        ReadCapacityUnits: 30
+        WriteCapacityUnits: 20
+    Type: AWS::DynamoDB::Table
+    Metadata:
+      SamResourceId: MongoDBAtlasTableResource
 Transform: AWS::Serverless-2016-10-31
diff --git a/requirements.txt b/requirements.txt
@@ -1,3 +1,3 @@
-sumologic-appclient-sdk==2.0.10
+sumologic-appclient-sdk>=2.0.10
 requests>=2.32.3
-future-fstrings
+future_fstrings==1.2.0
diff --git a/samconfig.toml b/samconfig.toml
@@ -0,0 +1,45 @@
+version = 0.1
+
+[default.build]
+[default.build.parameters]
+cached = true
+parallel = true
+debug=true
+use_container=true
+
+[default.package]
+[default.package.parameters]
+output_template_file="packaged.yaml"
+s3_prefix="MongoDBAtlas/v1.0.18"
+region="us-east-1"
+s3_bucket="appdevstore-test"
+
+[default.deploy.parameters]
+capabilities = "CAPABILITY_IAM"
+confirm_changeset = true
+stack_name="testingMongoDBAtlas"
+parameter_overrides="HttpLogsEndpoint=<http logs source endpoint> HttpMetricsEndpoint=<http metrics source endpoint> OrganizationId=<mongodbatlas org id> PrivateApiKey=<mongodbatlas private api key> ProjectId=<mongodb atlas cluster project id> PublicApiKey=<mongodbatlas public api key>"
+s3_prefix="MongoDBAtlas/v1.0.18"
+region="us-east-1"
+s3_bucket="appdevstore-test"
+
+[default.publish.parameters]
+template="packaged.yaml"
+region="us-east-1"
+
+[default.sync.parameters]
+watch = true
+
+[default.local_start_api.parameters]
+warm_containers = "EAGER"
+
+[default.local_invoke.parameters]
+parameter_overrides="ENVIRONMENT=onprem HttpLogsEndpoint=<http logs source endpoint> HttpMetricsEndpoint=<http metrics source endpoint> OrganizationId=<mongodbatlas org id> PrivateApiKey=<mongodbatlas private api key> ProjectId=<mongodb atlas cluster project id> PublicApiKey=<mongodbatlas public api key>"
+
+# https://github.com/aws/aws-sam-cli/issues/5492
+# Allow specification of parameters that should apply to all (or multiple) environments in AWS SAM CLI configuration file (samconfig.toml)
+
+[prod]
+[prod.sync]
+[prod.sync.parameters]
+watch = false
diff --git a/setup.py b/setup.py
@@ -21,7 +21,8 @@
 setup(
     name="sumologic-mongodb-atlas",
     version=__versionstr__,
-    packages=find_packages(),
+    packages=find_packages(exclude=["tests*"]),
+    package_data={"sumomongodbatlascollector": ["mongodbatlas.yaml"]},
     install_requires=INSTALL_REQUIRES,
     extras_require={
         "aws": ["boto3>=1.34.149", "botocore>=1.34.149"],
@@ -42,7 +43,6 @@
     keywords="sumologic python rest api log management analytics logreduce mongodb atlas agent security siem collector forwarder",
     url="https://github.com/SumoLogic/sumologic-mongodb-atlas",
     zip_safe=True,
-    include_package_data=True,
     classifiers=[
         "Development Status :: 4 - Beta",
         "Intended Audience :: Developers",
@@ -52,6 +52,7 @@
         "Programming Language :: Python :: 3.9",
         "Programming Language :: Python :: 3.10",
         "Programming Language :: Python :: 3.11",
+        "Programming Language :: Python :: 3.12",
         "Operating System :: OS Independent",
     ],
     entry_points={