Skip to content

Commit

Permalink
fix: resolve conflicts and merge with main
Browse files Browse the repository at this point in the history
  • Loading branch information
AndreiAlexandruParaschiv committed Sep 5, 2024
1 parent 415ada4 commit 09a113b
Show file tree
Hide file tree
Showing 18 changed files with 1,320 additions and 739 deletions.
96 changes: 96 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,99 @@
## [1.30.4](https://github.com/adobe/spacecat-audit-worker/compare/v1.30.3...v1.30.4) (2024-09-05)


### Bug Fixes

* **broken-backlinks:** introduce auto-suggest flag for broken backlinks suggestions ([#396](https://github.com/adobe/spacecat-audit-worker/issues/396)) ([4601b00](https://github.com/adobe/spacecat-audit-worker/commit/4601b0084c8373bfda1af58f93d1bdecb382c986))

## [1.30.3](https://github.com/adobe/spacecat-audit-worker/compare/v1.30.2...v1.30.3) (2024-09-04)


### Bug Fixes

* **deps:** update dependency @adobe/spacecat-shared-google-client to v1.2.3 ([#395](https://github.com/adobe/spacecat-audit-worker/issues/395)) ([e92d4cd](https://github.com/adobe/spacecat-audit-worker/commit/e92d4cd9223d35f959a000134e1660c6b2d15a21))

## [1.30.2](https://github.com/adobe/spacecat-audit-worker/compare/v1.30.1...v1.30.2) (2024-09-04)


### Bug Fixes

* **experimentation-ess:** audit url calculating for rum bundler API ([#389](https://github.com/adobe/spacecat-audit-worker/issues/389)) ([488a9a7](https://github.com/adobe/spacecat-audit-worker/commit/488a9a7fdcc248a8f49e30b211f6c36d3aa68091))

## [1.30.1](https://github.com/adobe/spacecat-audit-worker/compare/v1.30.0...v1.30.1) (2024-09-04)


### Bug Fixes

* **deps:** update dependency @adobe/spacecat-shared-ahrefs-client to v1.5.2 ([#394](https://github.com/adobe/spacecat-audit-worker/issues/394)) ([6e684ff](https://github.com/adobe/spacecat-audit-worker/commit/6e684ffebf30ab4fdd633f2a4419c646675e80e7))

# [1.30.0](https://github.com/adobe/spacecat-audit-worker/compare/v1.29.2...v1.30.0) (2024-09-03)


### Bug Fixes

* **deps:** update dependency @adobe/spacecat-shared-rum-api-client to v2.9.0 ([#388](https://github.com/adobe/spacecat-audit-worker/issues/388)) ([b55e4fb](https://github.com/adobe/spacecat-audit-worker/commit/b55e4fb993a7d3f782e4e2109ce63c46e2df74bf))


### Features

* high bounce rate and low ctr opportunities ([#384](https://github.com/adobe/spacecat-audit-worker/issues/384)) ([33aa025](https://github.com/adobe/spacecat-audit-worker/commit/33aa0257031ddc92c47718aaf100a30cf7662a61))

## [1.29.3](https://github.com/adobe/spacecat-audit-worker/compare/v1.29.2...v1.29.3) (2024-09-03)


### Bug Fixes

* **deps:** update dependency @adobe/spacecat-shared-rum-api-client to v2.9.0 ([#388](https://github.com/adobe/spacecat-audit-worker/issues/388)) ([b55e4fb](https://github.com/adobe/spacecat-audit-worker/commit/b55e4fb993a7d3f782e4e2109ce63c46e2df74bf))

## [1.29.2](https://github.com/adobe/spacecat-audit-worker/compare/v1.29.1...v1.29.2) (2024-09-02)


### Bug Fixes

* **cwv:** audit url calculating for rum bundler API ([#386](https://github.com/adobe/spacecat-audit-worker/issues/386)) ([1ac8123](https://github.com/adobe/spacecat-audit-worker/commit/1ac8123b08d8e2fbb62c10addf9893ba3124019c))

## [1.29.1](https://github.com/adobe/spacecat-audit-worker/compare/v1.29.0...v1.29.1) (2024-08-28)


### Bug Fixes

* audit context includes interval info ([#383](https://github.com/adobe/spacecat-audit-worker/issues/383)) ([c0add98](https://github.com/adobe/spacecat-audit-worker/commit/c0add98e5b1a794284bcb72624c284f0aed6593d))

# [1.29.0](https://github.com/adobe/spacecat-audit-worker/compare/v1.28.1...v1.29.0) (2024-08-27)


### Features

* introduce structured data audit ([#367](https://github.com/adobe/spacecat-audit-worker/issues/367)) ([3e6901d](https://github.com/adobe/spacecat-audit-worker/commit/3e6901d5c51c2ade763aeae7607ba8f476d55dce))

## [1.28.1](https://github.com/adobe/spacecat-audit-worker/compare/v1.28.0...v1.28.1) (2024-08-27)


### Bug Fixes

* **deps:** update dependency jsdom to v25 ([#379](https://github.com/adobe/spacecat-audit-worker/issues/379)) ([d785019](https://github.com/adobe/spacecat-audit-worker/commit/d7850193c94d6f6bc03134c1cfb00d28dd16f807))

# [1.28.0](https://github.com/adobe/spacecat-audit-worker/compare/v1.27.0...v1.28.0) (2024-08-26)


### Features

* opportunities audit ([#360](https://github.com/adobe/spacecat-audit-worker/issues/360)) ([22128fd](https://github.com/adobe/spacecat-audit-worker/commit/22128fd0c311d65e0a5b60e4d229205feb2bf219))

# [1.27.0](https://github.com/adobe/spacecat-audit-worker/compare/v1.26.5...v1.27.0) (2024-08-26)


### Features

* cwv audit revival ([#378](https://github.com/adobe/spacecat-audit-worker/issues/378)) ([3a27bde](https://github.com/adobe/spacecat-audit-worker/commit/3a27bded17a6efe7354b5533a217e47fc14c2285))

## [1.26.5](https://github.com/adobe/spacecat-audit-worker/compare/v1.26.4...v1.26.5) (2024-08-24)


### Bug Fixes

* **deps:** update external fixes ([#376](https://github.com/adobe/spacecat-audit-worker/issues/376)) ([923ad80](https://github.com/adobe/spacecat-audit-worker/commit/923ad80f45b2ea2462857c997c377260349629f7))

## [1.26.4](https://github.com/adobe/spacecat-audit-worker/compare/v1.26.3...v1.26.4) (2024-08-23)


Expand Down
5 changes: 1 addition & 4 deletions default.json
Original file line number Diff line number Diff line change
Expand Up @@ -46,8 +46,5 @@
"updateTypes": ["patch", "minor"],
"automerge": true
}
],
"encrypted": {
"npmToken": "wcFMA/xDdHCJBTolAQ//cBnUY2GbbFUjxEW/JyLas0ZYKaS9XCOrP4FA7r43SwRV44ynLM0XIWB+ceGUFrTN8YWfvkbbMVFcxqKYFnjl2l1S/8G9jvvS7bJLZCd1WNk7oVkIXnpb76FE6FQe0giudOiaWo21QVORVEoN/G6du3EOEiCghLMSCMtjg9+0M9Qg9dFJFoUeAHxLhGMjU3kQFO89ljYjaEYA4dvY2Id4pQOx2l0K2T70hSvGhrDGnDyGzOdA3rtz5as+/LgASvOOsTvn9JRndN9G1vnWqHR8xJBibPOqRyJ82IH+pe/+fRex/4Dr1HxlVa9BPhLslAFgUZqTrmwVhZutwURd5sRYNbYiMCBNNjmuVSfIIv+3/y4yEimaQWoh5RVGNxfYddryOSL/FKGeIybuKpskhzTzjtohvu71EjBinyx0qw/O92gZ+UHIZqhOWfmIoGV/pLPjGqkboXYjeQmSauVl8e9by1QXDiGmKe0T16UblNAhU+sQJdneFE4UBflK61jnReAUOsRVFvn9vuVrOuF8zwdXPs8xb7vuE77wUg9M/2N7I+y3tzVtb4uHvoVjtzIlkBe37xkzunKCDbk/Hdgk4lGdlFeTpQYL8Q/rU0itnYHZwfUThgPq1BhP+rWxj6Qkv1DR1983yowG9Zv54bO42oGO6apVAMZQ4q8Bz15BK0TCQFzSdAHRSJq9bALOc/HK5GbEQQ7sZ1JU76jjSFXqeBMZs7A2jrm1yNJAJ7WW+zLgXLjm9s1e1epn4PXerD50ypBKZnKfhmnnBB67IlO4IVfOLIllq3N1dJT2q5zqGdfgMN1qbyCg2JlgVSM8k83wwYtdLXC+mfl+"
}
]
}
27 changes: 15 additions & 12 deletions src/backlinks/handler.js
Original file line number Diff line number Diff line change
Expand Up @@ -64,18 +64,21 @@ export default async function auditBrokenBacklinks(message, context) {
(backlink) => !excludedURLs?.includes(backlink.url_to),
);
let brokenBacklinks = await filterOutValidBacklinks(filteredBacklinks, log);
try {
const topPages = await dataAccess.getTopPagesForSite(siteId, 'ahrefs', 'global');
const keywords = topPages.map(
(page) => ({
url: page.getURL(),
keyword: page.getTopKeyword(),
traffic: page.getTraffic(),
}),
);
brokenBacklinks = enhanceBacklinksWithFixes(brokenBacklinks, keywords, log);
} catch (e) {
log.error(`Enhancing backlinks with fixes for siteId ${siteId} failed with error: ${e.message}`, e);

if (configuration.isHandlerEnabledForSite(`${type}-auto-suggest`, site)) {
try {
const topPages = await dataAccess.getTopPagesForSite(siteId, 'ahrefs', 'global');
const keywords = topPages.map(
(page) => ({
url: page.getURL(),
keyword: page.getTopKeyword(),
traffic: page.getTraffic(),
}),
);
brokenBacklinks = enhanceBacklinksWithFixes(brokenBacklinks, keywords, log);
} catch (e) {
log.error(`Enhancing backlinks with fixes for siteId ${siteId} failed with error: ${e.message}`, e);
}
}

auditResult = {
Expand Down
9 changes: 8 additions & 1 deletion src/common/audit.js
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,9 @@
* governing permissions and limitations under the License.
*/

import { composeAuditURL } from '@adobe/spacecat-shared-utils';
import { composeAuditURL, hasText } from '@adobe/spacecat-shared-utils';
import { ok } from '@adobe/spacecat-shared-http-utils';
import URI from 'urijs';
import { retrieveSiteBySiteId } from '../utils/data-access.js';

export async function defaultMessageSender(resultMessage, context) {
Expand Down Expand Up @@ -52,6 +53,12 @@ export async function defaultUrlResolver(site) {
return composeAuditURL(site.getBaseURL());
}

export function wwwUrlResolver(site) {
const baseURL = site.getBaseURL();
const uri = new URI(baseURL);
return hasText(uri.subdomain()) ? baseURL.replace(/https?:\/\//, '') : baseURL.replace(/https?:\/\//, 'www.');
}

export async function noopUrlResolver(site) {
return site.getBaseURL();
}
Expand Down
113 changes: 34 additions & 79 deletions src/cwv/handler.js
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
/*
* Copyright 2023 Adobe. All rights reserved.
* Copyright 2024 Adobe. All rights reserved.
* This file is licensed to you under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License. You may obtain a copy
* of the License at http://www.apache.org/licenses/LICENSE-2.0
Expand All @@ -10,83 +10,38 @@
* governing permissions and limitations under the License.
*/

import RUMAPIClient, { createRUMURL } from '@adobe/spacecat-shared-rum-api-client-v1';
import { internalServerError, noContent } from '@adobe/spacecat-shared-http-utils';
import { composeAuditURL } from '@adobe/spacecat-shared-utils';
import { retrieveSiteBySiteId } from '../utils/data-access.js';

const PAGEVIEW_THRESHOLD = 35000;

export function filterRUMData(data) {
return data.pageviews > PAGEVIEW_THRESHOLD // ignore the pages with low pageviews
&& data.url.toLowerCase() !== 'other'; // ignore the combined result
import RUMAPIClient from '@adobe/spacecat-shared-rum-api-client';
import { getRUMDomainkey } from '../support/utils.js';
import { AuditBuilder } from '../common/audit-builder.js';
import { wwwUrlResolver } from '../common/audit.js';

const DAILY_THRESHOLD = 1000;
const INTERVAL = 7; // days

export async function CWVRunner(auditUrl, context, site) {
const rumAPIClient = RUMAPIClient.createFrom(context);
const domainkey = await getRUMDomainkey(site.getBaseURL(), context);
const options = {
domain: auditUrl,
domainkey,
interval: INTERVAL,
granularity: 'hourly',
};
const cwvData = await rumAPIClient.query('cwv', options);
const auditResult = {
cwv: cwvData.filter((data) => data.pageviews >= DAILY_THRESHOLD * INTERVAL),
auditContext: {
interval: INTERVAL,
},
};

return {
auditResult,
fullAuditRef: auditUrl,
};
}

/**
* url param in run-query@v3/rum-dashboard works in a 'startsWith' fashion. url=domain.com returns
* an empty result whereas url=www.domain.com/ returns the desired result. To catch the redirects
* to subdomains we issue a GET call to the domain, then use the final url after redirects
* @param url
* @returns finalUrl {Promise<string>}
*/

function processRUMResponse(data) {
return data
.filter(filterRUMData)
.map((row) => ({
url: row.url,
pageviews: row.pageviews,
CLS: row.avgcls,
INP: row.avginp,
LCP: row.avglcp,
}));
}
export default async function auditCWV(message, context) {
const { type, url: siteId, auditContext = {} } = message;
const { dataAccess, log, sqs } = context;
const {
AUDIT_RESULTS_QUEUE_URL: queueUrl,
} = context.env;
try {
const site = await retrieveSiteBySiteId(dataAccess, siteId, log);
const url = site.getBaseURL();

log.info(`Received audit req for domain: ${url}`);

const rumAPIClient = RUMAPIClient.createFrom(context);
const finalUrl = await composeAuditURL(url);
auditContext.finalUrl = finalUrl;

const params = {
url: finalUrl,
};

const data = await rumAPIClient.getRUMDashboard(params);
const auditResult = processRUMResponse(data);
const fullAuditRef = createRUMURL({ ...params, domainkey: '' });

const auditData = {
siteId: site.getId(),
isLive: site.isLive(),
auditedAt: new Date().toISOString(),
auditType: type,
fullAuditRef,
auditResult,
};

await dataAccess.addAudit(auditData);

await sqs.sendMessage(queueUrl, {
type,
url,
auditContext,
auditResult,
});

log.info(`Successfully audited ${url} for ${type} type audit`);
return noContent();
} catch (e) {
log.info(`CWV audit failed for ${siteId} failed due to ${e.message}`);
return internalServerError(`Internal server error: ${e.message}`);
}
}
export default new AuditBuilder()
.withUrlResolver(wwwUrlResolver)
.withRunner(CWVRunner)
.build();
2 changes: 2 additions & 0 deletions src/experimentation-ess/all.js
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@
/* c8 ignore start */
import { AuditBuilder } from '../common/audit-builder.js';
import { processAudit, postProcessor } from './common.js';
import { wwwUrlResolver } from '../common/audit.js';

const DAYS = 180;

Expand Down Expand Up @@ -79,6 +80,7 @@ export async function essExperimentationAllAuditRunner(auditUrl, context, site)

export default new AuditBuilder()
.withRunner(essExperimentationAllAuditRunner)
.withUrlResolver(wwwUrlResolver)
.withPostProcessors([postProcessor])
.withPersister(persistOnlyMetadata)
.withMessageSender(() => true)
Expand Down
2 changes: 2 additions & 0 deletions src/experimentation-ess/daily.js
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@
/* c8 ignore start */
import { AuditBuilder } from '../common/audit-builder.js';
import { processAudit } from './common.js';
import { wwwUrlResolver } from '../common/audit.js';

const DAYS = 1;

Expand Down Expand Up @@ -44,5 +45,6 @@ export async function essExperimentationDailyAuditRunner(auditUrl, context, site

export default new AuditBuilder()
.withRunner(essExperimentationDailyAuditRunner)
.withUrlResolver(wwwUrlResolver)
.build();
/* c8 ignore stop */
61 changes: 61 additions & 0 deletions src/experimentation-opportunities/experimentation-opportunities.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
/*
* Copyright 2024 Adobe. All rights reserved.
* This file is licensed to you under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License. You may obtain a copy
* of the License at http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software distributed under
* the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR REPRESENTATIONS
* OF ANY KIND, either express or implied. See the License for the specific language
* governing permissions and limitations under the License.
*/

import RUMAPIClient from '@adobe/spacecat-shared-rum-api-client';
import { AuditBuilder } from '../common/audit-builder.js';
import { getRUMDomainkey } from '../support/utils.js';
import { wwwUrlResolver } from '../common/audit.js';

const DAYS = 30;
const OPPTY_QUERIES = [
'rageclick',
'high-inorganic-high-bounce-rate',
'high-organic-low-ctr',
];

/**
* Audit handler container for all the opportunities
* @param {*} auditUrl
* @param {*} context
* @param {*} site
* @returns
*/

export async function handler(auditUrl, context, site) {
const { log } = context;

const rumAPIClient = RUMAPIClient.createFrom(context);
const domainkey = await getRUMDomainkey(site.getBaseURL(), context);
const options = {
domain: auditUrl,
domainkey,
interval: DAYS,
granularity: 'hourly',
};

const queryResults = await rumAPIClient.queryMulti(OPPTY_QUERIES, options);
const experimentationOpportunities = Object.values(queryResults).flatMap((oppty) => oppty);

log.info(`Found ${experimentationOpportunities.length} many experimentation opportunites for ${auditUrl}`);

return {
auditResult: {
experimentationOpportunities,
},
fullAuditRef: auditUrl,
};
}

export default new AuditBuilder()
.withRunner(handler)
.withUrlResolver(wwwUrlResolver)
.build();
Loading

0 comments on commit 09a113b

Please sign in to comment.