Skip to content
This repository was archived by the owner on Mar 4, 2025. It is now read-only.

Commit

Permalink
Upstream sync part1 (#441)
Browse files Browse the repository at this point in the history 
* Upstream synbc afb9638..bfc8b54

* Add missing `volatile` in Tracer creation (#2508)

This was missed in the change from `LazyInitializer.EnsureInitialized()`. This could lead to a race condition during initialization

* Fixes for the build related to hotfix releases (#2511)

* Allow forcing releases from a specific commit sha (#2509) (#2510)

This should not be the default approach, but it allows us to create releases without waiting for a full rebuild to occur, or where we have issues with temporary flake/capacity in CI.

This does some minimal verification of the provided sha (that it's on the branch provided) and checks that the artifacts are available, but does not require the build to have succeeded

* Allow running the GitLab build for hotfix and release branches

* Reverse to protobuf 3.7.0 to avoid unexpected new dependency (System.Memory.dll) (#2513)

* [CI App] - Agentless Writer (#2402)

* Initial Commit.

* Add truncator class

* Changes

* Reduce allocations

* Fixes and unit test suite.

* Move previous CIAgentWriter logic to new OriginTagTraceProcessor.

* Initial event model structure.

* Initial writer work

* Changes

* changes

* Custom Meta enumerator for tagslist.

* Optimize TagsList enumerators.

* Changes to use the AgentPayload format.

* CompactWhiteSpace method port.

* ReplaceDigits method port.

* fix merge

* Fix from merge

* add CI App MessagePack support

* Formatter changes

* Fix messagepack formatters

* Revert source generator changes and json support.

* Update the writer.

* multiple file in the file writer.

* Add SpanEvent message.

* Changes.

* Add a new EventsBuffer class

* Changes and fixes.

* Improve formatter.

* changes.

* changes

* Fixes

* Fixes and documentation.

* Refactor

* Changes

* Initial tests

* Use ThrowHelper

* refactor test

* Changes and new Tests

* Fix tests

* Fix Tests

* Fix tests.

* Remove JSON serialization patch.

* Changes.

* Changes.

* Fixes

* test by replacing Ping

* Changes

* fixes

* Revert some changes.

* Update to latest specs.

* fixes.

* Fix tests

* revert one unnescesary change.

* ConfigurationKeys refactor and Initial support to webproxies

* Not supported proxy schema message.

* fix build

* Update payload specification.

* Fix test

* Change DD_SITE default value

* Add support for agentless in the runner.

* Rename ICIVisibilityWriter to IEventWriter

* Update all github links with permanent SHA1 (latest commit on master)

* Ensure NormalizeTag doesn't return null.

* Add null case to normalizer test.

* Move version value to the CIVisibilityEvent object.

* Remove unused arguments.

* Add comment for ParentId validation.

* Change LogLevel from Information to Debug

* Update tracer/src/Datadog.Trace/TraceProcessors/NormalizerTraceProcessor.cs

Co-authored-by: Zach Montoya <[email protected]>

* Update tracer/src/Datadog.Trace/TraceProcessors/NormalizerTraceProcessor.cs

Co-authored-by: Zach Montoya <[email protected]>

* Change Log.Information with Log.Debug

* Change the default url for UriBuilder with a generic host.

* Change DelegatingHandler to HttpClientHandler.

* fix build

* Remove unused string extensions.

* Remove unused method parameter.

* Remove unnescessary Ping method from the ICIAgentlessWriterSender interface.

* Try to add an event to the buffer again if is not processed (buffer full).

* Remove unnescessary items.

* Ensure there's always a sampler instance.

* Remove #if DEBUG preprocessor with a runtime check.

* Move ArraySlice to Datadog.Trace.Util namespace

* Apply suggested optimization.

* Update tracer/src/Datadog.Trace.Tools.Runner/LegacyCommand.cs

Co-authored-by: Andrew Lock <[email protected]>

* Update tracer/src/Datadog.Trace.Tools.Runner/RunCommand.cs

Co-authored-by: Andrew Lock <[email protected]>

* Add DefaultMaxBufferSize const.

* Use #nullable enable and remove some comments.

* Remove some comments and add readonly to some fields.

* Update tracer/src/Datadog.Trace/Ci/CITracerManagerFactory.cs

Co-authored-by: Andrew Lock <[email protected]>

* remove comment.

* Change the algorithm to find the root span.

* Rename methods.

* fixes.

* Processors refactor

* Change NormalizerTraceProcessorTests to avoid VSTest serializer bug.

* Fix SourceGenerators tests

Co-authored-by: Zach Montoya <[email protected]>
Co-authored-by: Andrew Lock <[email protected]>

* build fixes

* update en variables DD->SIGNALFX

* update internal config

Co-authored-by: Andrew Lock <[email protected]>
Co-authored-by: chrisnas <[email protected]>
Co-authored-by: Tony Redondo <[email protected]>
Co-authored-by: Zach Montoya <[email protected]>
  • Loading branch information
@Kielek @andrewlock
@chrisnas @tonyredondo @zacharycmontoya
5 people authored Mar 28, 2022
1 parent af10433 commit cf04d52
Show file tree
Hide file tree
Showing 147 changed files with 3,980 additions and 827 deletions.
3 changes: 3 additions & 0 deletions docs/internal/internal-config.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -34,13 +34,16 @@ These settings should be never used by the users.
| `SIGNALFX_APPSEC_IPHEADER` | Optional name of the custom header to take into account for the ip address. | |
| `SIGNALFX_APPSEC_KEEP_TRACES` | Specifies if the AppSec traces should be explicitly kept or droped. | `true` |
| `SIGNALFX_APPSEC_RULES` | Overrides the default rules file provided. Must be a path to a valid JSON rules file. | |
| `SIGNALFX_CIVISIBILITY_AGENTLESS_ENABLED` | Enable to activate Agentless in CI Visibility. | `false` |
| `SIGNALFX_CIVISIBILITY_ENABLED` | Enable to activate CI Visibility. | `false` |
| `SIGNALFX_DOGSTATSD_ARGS` | Comma-separated list of arguments to be passed to the DogStatsD process. | |
| `SIGNALFX_DOGSTATSD_PATH` | The DogStatsD path for when a standalone instance needs to be started. | |
| `SIGNALFX_DOGSTATSD_PIPE_NAME` | The named pipe that DogStatsD binds to. | |
| `SIGNALFX_DOGSTATSD_PORT` | The port of the targeted StatsD server. | `8125` |
| `SIGNALFX_INTERNAL_TRACE_VERSION_COMPATIBILITY` | Enables the compatibility with other versions of tracer. | `false` |
| `SIGNALFX_MAX_TRACES_PER_SECOND` | The number of traces allowed to be submitted per second. | `100` |
| `SIGNALFX_PROXY_HTTPS` | TConfiguration key to set a proxy server for https requests. | |
| `SIGNALFX_PROXY_NO_PROXY` | Configuration key to set a list of hosts that should bypass the proxy. The list is space-separated| |
| `SIGNALFX_TRACE_{0}_ANALYTICS_ENABLED` | Enable to activate analytics for specific integration. | `false` |
| `SIGNALFX_TRACE_{0}_ANALYTICS_SAMPLE_RATE` | Set sample rate for analytics in specific integration. | |
| `SIGNALFX_TRACE_AGENT_ARGS` | Comma-separated list of arguments to be passed to the Trace Agent process. | |
Expand Down
2 changes: 1 addition & 1 deletion profiler/src/ProfilerEngine/Datadog.Profiler.Managed/Datadog.Profiler.Managed.csproj
View file
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@
</PropertyGroup>

<ItemGroup>
<PackageReference Include="Google.Protobuf" Version="3.19.4" />
<PackageReference Include="Google.Protobuf" Version="3.7.0" />
</ItemGroup>

<ItemGroup Condition=" $(TargetFramework.StartsWith('net4')) ">
Expand Down
135 changes: 95 additions & 40 deletions tracer/build/_build/Build.GitHub.cs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,9 @@ partial class Build
[Parameter("The Pull Request number for GitHub Actions")]
readonly int? PullRequestNumber;

[Parameter("The specific commit sha to use", List = false)]
readonly string CommitSha;

[Parameter("The git branch to use", List = false)]
readonly string TargetBranch;

Expand Down Expand Up @@ -491,57 +494,109 @@ await client.Issue.Milestone.Update(

BuildArtifact artifact = null;
var artifactName = $"{FullVersion}-release-artifacts";
string commitSha = CommitSha;

Logger.Info($"Checking builds for artifact called: {artifactName}");
string commitSha = String.Empty;

// start from the current commit, and keep looking backwards until we find a commit that has a build
// that has successful artifacts. Should only be called from branches with a linear history (i.e. single parent)
// This solves a potential issue where we previously selecting a build by start order, not by the actual
// git commit order. Generally that shouldn't be an issue, but if we manually trigger builds on master
// (which we sometimes do e.g. trying to bisect and issue, or retrying flaky test for coverage reasons),
// then we could end up selecting the wrong build.
const int maxCommitsBack = 20;
for (var i = 0; i < maxCommitsBack; i++)
if (!string.IsNullOrEmpty(CommitSha))
{
commitSha = GitTasks.Git($"log {TargetBranch}~{i} -1 --pretty=%H")
.FirstOrDefault(x => x.Type == OutputType.Std)
.Text;
var foundSha = false;
var maxCommitsBack = 20;
// basic verification, to ensure that the provided commitsha is actually on this branch
for (var i = 0; i < maxCommitsBack; i++)
{
var sha = GitTasks.Git($"log {TargetBranch}~{i} -1 --pretty=%H")
.FirstOrDefault(x => x.Type == OutputType.Std)
.Text;

Logger.Info($"Looking for builds for {commitSha}");
if (string.Equals(CommitSha, sha, StringComparison.OrdinalIgnoreCase))
{
// OK, this SHA is definitely on this branch
foundSha = true;
break;
}
}

foreach (var build in builds)
if (!foundSha)
{
if (string.Equals(build.SourceVersion, commitSha, StringComparison.OrdinalIgnoreCase))
{
// Found a build for the commit, so should be successful and have an artifact
if (build.Result != BuildResult.Succeeded && build.Result != BuildResult.PartiallySucceeded)
{
Logger.Error($"::error::The build for commit {commitSha} was not successful. Please retry any failed stages for the build before creating a release");
throw new Exception("Latest build for branch was not successful. Please retry the build before creating a release");
}
Logger.Error($"Error: The commit {CommitSha} could not be found in the last {maxCommitsBack} of the branch {TargetBranch}" +
$"Ensure that the commit sha you have provided is correct, and you are running the create_release action from the correct branch");
throw new Exception($"The commit {CommitSha} could not found in the latest {maxCommitsBack} of target branch {TargetBranch}");
}

try
{
artifact = await buildHttpClient.GetArtifactAsync(
project: AzureDevopsProjectId,
buildId: build.Id,
artifactName: artifactName);

break;
}
catch (ArtifactNotFoundException)
Logger.Info($"Finding build for commit sha: {CommitSha}");
var build = builds
.FirstOrDefault(b => string.Equals(b.SourceVersion, CommitSha, StringComparison.OrdinalIgnoreCase));
if (build is null)
{
throw new Exception($"No builds for commit {CommitSha} found. Please check you have provided the correct SHA, and that there is a build in AzureDevops for the commit");
}

try
{
artifact = await buildHttpClient.GetArtifactAsync(
project: AzureDevopsProjectId,
buildId: build.Id,
artifactName: artifactName);
}
catch (ArtifactNotFoundException)
{
Logger.Error($"Error: The build {build.Id} for commit could not find {artifactName} artifact for build {build.Id} for commit {commitSha}. " +
$"Ensure the build has successfully generated artifacts for this commit before creating a release");
throw;
}
}
else
{
Logger.Info($"Checking builds for artifact called: {artifactName}");

// start from the current commit, and keep looking backwards until we find a commit that has a build
// that has successful artifacts. Should only be called from branches with a linear history (i.e. single parent)
// This solves a potential issue where we previously selecting a build by start order, not by the actual
// git commit order. Generally that shouldn't be an issue, but if we manually trigger builds on master
// (which we sometimes do e.g. trying to bisect and issue, or retrying flaky test for coverage reasons),
// then we could end up selecting the wrong build.
const int maxCommitsBack = 20;
for (var i = 0; i < maxCommitsBack; i++)
{
commitSha = GitTasks.Git($"log {TargetBranch}~{i} -1 --pretty=%H")
.FirstOrDefault(x => x.Type == OutputType.Std)
.Text;

Logger.Info($"Looking for builds for {commitSha}");

foreach (var build in builds)
{
if (string.Equals(build.SourceVersion, commitSha, StringComparison.OrdinalIgnoreCase))
{
Logger.Error($"Error: could not find {artifactName} artifact for build {build.Id} for commit {commitSha}. " +
$"Ensure the build has completed successfully for this commit before creating a release");
throw;
// Found a build for the commit, so should be successful and have an artifact
if (build.Result != BuildResult.Succeeded && build.Result != BuildResult.PartiallySucceeded)
{
Logger.Error($"::error::The build for commit {commitSha} was not successful. Please retry any failed stages for the build before creating a release");
throw new Exception("Latest build for branch was not successful. Please retry the build before creating a release");
}

try
{
artifact = await buildHttpClient.GetArtifactAsync(
project: AzureDevopsProjectId,
buildId: build.Id,
artifactName: artifactName);

break;
}
catch (ArtifactNotFoundException)
{
Logger.Error($"Error: could not find {artifactName} artifact for build {build.Id} for commit {commitSha}. " +
$"Ensure the build has completed successfully for this commit before creating a release");
throw;
}
}
}
}

if (artifact is not null)
{
break;
if (artifact is not null)
{
break;
}
}
}

Expand Down
7 changes: 4 additions & 3 deletions tracer/src/Datadog.Trace.BenchmarkDotNet/DatadogExporter.cs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ static DatadogExporter()
{
try
{
Environment.SetEnvironmentVariable(Configuration.ConfigurationKeys.CIVisibilityEnabled, "1", EnvironmentVariableTarget.Process);
Environment.SetEnvironmentVariable(Configuration.ConfigurationKeys.CIVisibility.Enabled, "1", EnvironmentVariableTarget.Process);
}
catch
{
Expand Down Expand Up @@ -71,13 +71,14 @@ public IEnumerable<string> ExportToFiles(Summary summary, ILogger consoleLogger)
span.ResourceName = $"{report.BenchmarkCase.Descriptor.Type.FullName}.{report.BenchmarkCase.Descriptor.WorkloadMethod.Name}";
CIEnvironmentValues.Instance.DecorateSpan(span);

span.SetTag(Tags.Origin, TestTags.CIAppTestOriginName);
span.SetTag(Tags.Language, TracerConstants.Language);
span.SetTag(TestTags.Name, report.BenchmarkCase.Descriptor.WorkloadMethodDisplayInfo);
span.SetTag(TestTags.Type, TestTags.TypeBenchmark);
span.SetTag(TestTags.Suite, report.BenchmarkCase.Descriptor.Type.FullName);
span.SetTag(TestTags.Framework, $"BenchmarkDotNet {summary.HostEnvironmentInfo.BenchmarkDotNetVersion}");
span.SetTag(TestTags.Status, report.Success ? TestTags.StatusPass : TestTags.StatusFail);
span.SetTag(TestTags.Language, TracerConstants.Language);
span.SetTag(TestTags.CILibraryVersion, TracerConstants.AssemblyVersion);
span.SetTag(CommonTags.LibraryVersion, TracerConstants.AssemblyVersion);

if (summary.HostEnvironmentInfo != null)
{
Expand Down
7 changes: 4 additions & 3 deletions tracer/src/Datadog.Trace.MSBuild/DatadogLogger.cs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ static DatadogLogger()
{
try
{
Environment.SetEnvironmentVariable(Configuration.ConfigurationKeys.CIVisibilityEnabled, "1", EnvironmentVariableTarget.Process);
Environment.SetEnvironmentVariable(Configuration.ConfigurationKeys.CIVisibility.Enabled, "1", EnvironmentVariableTarget.Process);
}
catch
{
Expand Down Expand Up @@ -107,6 +107,8 @@ private void EventSource_BuildStarted(object sender, BuildStartedEventArgs e)
_buildSpan.SetTraceSamplingPriority(SamplingPriorityValues.AutoKeep);

_buildSpan.Type = SpanTypes.Build;
_buildSpan.SetTag(Tags.Origin, TestTags.CIAppTestOriginName);
_buildSpan.SetTag(Tags.Language, TracerConstants.Language);
_buildSpan.SetTag(BuildTags.BuildName, e.SenderName);
foreach (KeyValuePair<string, string> envValue in e.BuildEnvironment)
{
Expand All @@ -120,8 +122,7 @@ private void EventSource_BuildStarted(object sender, BuildStartedEventArgs e)
_buildSpan.SetTag(CommonTags.OSArchitecture, Environment.Is64BitOperatingSystem ? "x64" : "x86");
_buildSpan.SetTag(CommonTags.OSVersion, Environment.OSVersion.VersionString);
_buildSpan.SetTag(CommonTags.RuntimeArchitecture, Environment.Is64BitProcess ? "x64" : "x86");
_buildSpan.SetTag(TestTags.Language, TracerConstants.Language);
_buildSpan.SetTag(TestTags.CILibraryVersion, TracerConstants.AssemblyVersion);
_buildSpan.SetTag(CommonTags.LibraryVersion, TracerConstants.AssemblyVersion);
CIEnvironmentValues.Instance.DecorateSpan(_buildSpan);
}
catch (Exception ex)
Expand Down
15 changes: 8 additions & 7 deletions tracer/src/Datadog.Trace.SourceGenerators/TagsListGenerator/Sources.cs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,6 @@

// Modified by Splunk Inc.

using System.Collections.Generic;
using System.Text;

namespace Datadog.Trace.SourceGenerators.TagsListGenerator
Expand Down Expand Up @@ -66,6 +65,8 @@ public static string CreateTagsList(StringBuilder sb, TagListGenerator.TagList t
sb.Append(@"// <auto-generated/>
#nullable enable
using Datadog.Trace.Processors;
namespace ");
sb.Append(tagList.Namespace)
.Append(
Expand Down Expand Up @@ -201,7 +202,7 @@ public override void SetTag(string key, string value)
sb.Append(@"Properties;
}
protected override int WriteAdditionalTags(ref byte[] bytes, ref int offset)
protected override int WriteAdditionalTags(ref byte[] bytes, ref int offset, ITagProcessor[] tagProcessors)
{
var count = 0;
");
Expand All @@ -216,14 +217,14 @@ protected override int WriteAdditionalTags(ref byte[] bytes, ref int offset)
.Append(property.PropertyName)
.Append(@"Bytes, ")
.Append(property.PropertyName)
.Append(@");
.Append(@", tagProcessors);
}
");
}

sb.Append(
@"return count + base.WriteAdditionalTags(ref bytes, ref offset);
@"return count + base.WriteAdditionalTags(ref bytes, ref offset, tagProcessors);
}
protected override void WriteAdditionalTags(System.Text.StringBuilder sb)
Expand Down Expand Up @@ -330,7 +331,7 @@ public override void SetMetric(string key, double? value)
}
}
protected override int WriteAdditionalMetrics(ref byte[] bytes, ref int offset)
protected override int WriteAdditionalMetrics(ref byte[] bytes, ref int offset, ITagProcessor[] tagProcessors)
{
var count = 0;
");
Expand All @@ -345,14 +346,14 @@ protected override int WriteAdditionalMetrics(ref byte[] bytes, ref int offset)
.Append(property.PropertyName)
.Append(@"Bytes, ")
.Append(property.PropertyName)
.Append(@".Value);
.Append(@".Value, tagProcessors);
}
");
}

sb.Append(
@"return count + base.WriteAdditionalMetrics(ref bytes, ref offset);
@"return count + base.WriteAdditionalMetrics(ref bytes, ref offset, tagProcessors);
}
protected override void WriteAdditionalMetrics(System.Text.StringBuilder sb)
Expand Down
3 changes: 3 additions & 0 deletions tracer/src/Datadog.Trace/Agent/IApiRequestFactory.cs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
// </copyright>

using System;
using System.Net;

namespace Datadog.Trace.Agent
{
Expand All @@ -12,5 +13,7 @@ internal interface IApiRequestFactory
string Info(Uri endpoint);

IApiRequest Create(Uri endpoint);

void SetProxy(WebProxy proxy, NetworkCredential credential);
}
}
9 changes: 8 additions & 1 deletion tracer/src/Datadog.Trace/Agent/MessagePack/SpanMessagePackFormatter.cs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@

using System;
using Datadog.Trace.ExtensionMethods;
using Datadog.Trace.Processors;
using Datadog.Trace.Vendors.MessagePack;
using Datadog.Trace.Vendors.MessagePack.Formatters;

Expand Down Expand Up @@ -95,7 +96,13 @@ public int Serialize(ref byte[] bytes, int offset, Span value, IFormatterResolve
offset += MessagePackBinary.WriteByte(ref bytes, offset, 1);
}

offset += value.Tags.SerializeTo(ref bytes, offset, value);
ITagProcessor[] tagProcessors = null;
if (value.Context.TraceContext?.Tracer is Tracer tracer)
{
tagProcessors = tracer.TracerManager?.TagProcessors;
}

offset += value.Tags.SerializeTo(ref bytes, offset, value, tagProcessors);

return offset - originalOffset;
}
Expand Down
17 changes: 17 additions & 0 deletions tracer/src/Datadog.Trace/Agent/Transports/ApiWebRequestFactory.cs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,8 @@ namespace Datadog.Trace.Agent.Transports
internal class ApiWebRequestFactory : IApiRequestFactory
{
private readonly KeyValuePair<string, string>[] _defaultHeaders;
private WebProxy _proxy;
private NetworkCredential _credential;

public ApiWebRequestFactory(KeyValuePair<string, string>[] defaultHeaders)
{
Expand All @@ -26,6 +28,15 @@ public string Info(Uri endpoint)
public IApiRequest Create(Uri endpoint)
{
var request = WebRequest.CreateHttp(endpoint);
if (_proxy is not null)
{
request.Proxy = _proxy;
}

if (_credential is not null)
{
request.Credentials = _credential;
}

foreach (var pair in _defaultHeaders)
{
Expand All @@ -34,5 +45,11 @@ public IApiRequest Create(Uri endpoint)

return new ApiWebRequest(request);
}

public void SetProxy(WebProxy proxy, NetworkCredential credential)
{
_proxy = proxy;
_credential = credential;
}
}
}
Loading

0 comments on commit cf04d52

Please sign in to comment.