Skip to content

Fix for sending duplicate logs when using Agentless Logging in Azure Function host #7383

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 9 commits into from
Oct 3, 2025
Merged

Fix for sending duplicate logs when using Agentless Logging in Azure Function host #7383

merged 9 commits into from
Oct 3, 2025

Conversation

@bouwkast
Copy link
Collaborator

@bouwkast bouwkast commented Aug 12, 2025 •
edited
Loading

Summary of changes

Disables agentless logging (for the Azure Functions Host Process) if we detect that we are instrumenting an Azure Functions host process. Previously this would cause us to duplicate every log sent from the worker process.

This bug fix can be reverted by setting the new configuration key DD_LOGS_DIRECT_SUBMISSION_AZURE_FUNCTIONS_HOST_ENABLED to true.

Reason for change

If customers are running with the Isolated Azure Function model we will be instrumenting two applications, one is the function host process the other is the worker process. If they have direct log submission enabled the function host ends up duplicating the logs from the function process which results in us shipping two nearly identical logs.

This behavior isn't ideal as the duplicate log is likely not valuable so we've disabled agentless logging in the azure functions host process.

Implementation details

Added IsRunningInAzureFunctionsHost() to EnvironmentHelpers.cs which allows for a rough detection of whether we are running on the function host using the following logic:

  • Is FUNCTIONS_WORKER_RUNTIME present AND set to dotnet-isolated?
  • Are both --functions-worker-id or --workerId NOT present in the command line?

If both are true we treat that scenario as being running in the function host - otherwise we are likely the worker process

I wasn't able to find a more robust way of checking, but when looking at the various log output that I had the --functions-worker-id or --workerId seemed to always be called by the function host.

[2025-10-03T16:01:42.901Z] Reading functions metadata (Worker)
[2025-10-03T16:01:47.176Z] {
[2025-10-03T16:01:47.177Z]   "ProcessId": 71080,
[2025-10-03T16:01:47.178Z]   "RuntimeIdentifier": "win-x64",
[2025-10-03T16:01:47.179Z]   "WorkerVersion": "2.0.0.0",
[2025-10-03T16:01:47.180Z]   "ProductVersion": "2.0.0\u002Bd8b5fe998a8c92819b8ee41d2569d2525413e9c5",
[2025-10-03T16:01:47.181Z]   "FrameworkDescription": ".NET 9.0.9",
[2025-10-03T16:01:47.182Z]   "OSDescription": "Microsoft Windows 10.0.26100",
[2025-10-03T16:01:47.183Z]   "OSArchitecture": "X64",
[2025-10-03T16:01:47.184Z]   "CommandLine": "C:\\Users\\steven.bouwkamp\\source\\repos\\dd-trace-dotnet\\artifacts\\bin\\Samples.AzureFunctions.V4Isolated.AspNetCore\\debug_net9.0\\Samples.AzureFunctions.V4Isolated.AspNetCore.dll --host 127.0.0.1 --port 65401 --workerId e94d23fd-cd3c-4780-a3e3-4980d7b0f644 --requestId 6dba68ac-1954-466a-aeb4-9570cc9b12c2 --grpcMaxMessageLength 2147483647 --functions-uri http://127.0.0.1:65401/ --functions-worker-id e94d23fd-cd3c-4780-a3e3-4980d7b0f644 --functions-request-id 6dba68ac-1954-466a-aeb4-9570cc9b12c2 --functions-grpc-max-message-length 2147483647"
[2025-10-03T16:01:47.185Z] }
  • Added DD_LOGS_DIRECT_SUBMISSION_AZURE_FUNCTIONS_HOST_ENABLED that defaults to false to disable the duplicate logs from being sent.

Test coverage

  • Added a new test project and test for having host logs enabled / disabled.
    • The reason why I added a new test project instead of re-using an existing one was because when I re-ran the Function application in our tests multiple times the func.exe would fail to obtain a lock and would need to wait some period of time to for recovery after each subsequent run of the same application. I think this is because we end the func.exe process with a Kill(). Making a new project wasn't ideal but was a quick and simple workaround.

Other details

Fixes SLES-2364

@bouwkast bouwkast added the status:work-in-progress Actively worked on. If this is a PR, no review needed yet. WIP. label Aug 12, 2025
@dd-trace-dotnet-ci-bot
Copy link

dd-trace-dotnet-ci-bot bot commented Aug 12, 2025 •
edited
Loading

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

  • Welch test with statistical test for significance of 5%
  • Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.8) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Bailout
    This PR (7383) - mean (79ms)  : 76, 83
     .   : milestone, 79,
    master - mean (80ms)  : 77, 84
     .   : milestone, 80,

    section Baseline
    This PR (7383) - mean (75ms)  : 71, 79
     .   : milestone, 75,
    master - mean (75ms)  : 71, 79
     .   : milestone, 75,

    section CallTarget+Inlining+NGEN
    This PR (7383) - mean (1,112ms)  : 1048, 1176
     .   : milestone, 1112,
    master - mean (1,122ms)  : 1058, 1186
     .   : milestone, 1122,
Loading 
gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Bailout
    This PR (7383) - mean (118ms)  : 114, 122
     .   : milestone, 118,
    master - mean (120ms)  : 114, 125
     .   : milestone, 120,

    section Baseline
    This PR (7383) - mean (116ms)  : 112, 121
     .   : milestone, 116,
    master - mean (119ms)  : 115, 123
     .   : milestone, 119,

    section CallTarget+Inlining+NGEN
    This PR (7383) - mean (785ms)  : 763, 806
     .   : milestone, 785,
    master - mean (802ms)  : 782, 821
     .   : milestone, 802,
Loading 
gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Bailout
    This PR (7383) - mean (112ms)  : 108, 116
     .   : milestone, 112,
    master - mean (114ms)  : 109, 119
     .   : milestone, 114,

    section Baseline
    This PR (7383) - mean (111ms)  : 106, 115
     .   : milestone, 111,
    master - mean (113ms)  : 107, 118
     .   : milestone, 113,

    section CallTarget+Inlining+NGEN
    This PR (7383) - mean (829ms)  : 782, 875
     .   : milestone, 829,
    master - mean (838ms)  : 778, 898
     .   : milestone, 838,
Loading 
gantt
    title Execution time (ms) FakeDbCommand (.NET 8) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Bailout
    This PR (7383) - mean (103ms)  : 98, 108
     .   : milestone, 103,
    master - mean (104ms)  : 99, 109
     .   : milestone, 104,

    section Baseline
    This PR (7383) - mean (102ms)  : 98, 106
     .   : milestone, 102,
    master - mean (103ms)  : 98, 108
     .   : milestone, 103,

    section CallTarget+Inlining+NGEN
    This PR (7383) - mean (705ms)  : 685, 725
     .   : milestone, 705,
    master - mean (722ms)  : 698, 745
     .   : milestone, 722,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.8) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Bailout
    This PR (7383) - mean (204ms)  : 199, 209
     .   : milestone, 204,
    master - mean (202ms)  : 198, 207
     .   : milestone, 202,

    section Baseline
    This PR (7383) - mean (198ms)  : 195, 202
     .   : milestone, 198,
    master - mean (200ms)  : 193, 206
     .   : milestone, 200,

    section CallTarget+Inlining+NGEN
    This PR (7383) - mean (1,178ms)  : 1117, 1240
     .   : milestone, 1178,
    master - mean (1,213ms)  : 1145, 1282
     .   : milestone, 1213,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Bailout
    This PR (7383) - mean (286ms)  : 281, 290
     .   : milestone, 286,
    master - mean (296ms)  : 284, 308
     .   : milestone, 296,

    section Baseline
    This PR (7383) - mean (285ms)  : 280, 290
     .   : milestone, 285,
    master - mean (288ms)  : 281, 295
     .   : milestone, 288,

    section CallTarget+Inlining+NGEN
    This PR (7383) - mean (952ms)  : 915, 989
     .   : milestone, 952,
    master - mean (981ms)  : 938, 1024
     .   : milestone, 981,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Bailout
    This PR (7383) - mean (289ms)  : 283, 295
     .   : milestone, 289,
    master - mean (299ms)  : 288, 310
     .   : milestone, 299,

    section Baseline
    This PR (7383) - mean (288ms)  : 281, 294
     .   : milestone, 288,
    master - mean (301ms)  : 288, 314
     .   : milestone, 301,

    section CallTarget+Inlining+NGEN
    This PR (7383) - mean (1,002ms)  : 965, 1038
     .   : milestone, 1002,
    master - mean (1,053ms)  : 993, 1114
     .   : milestone, 1053,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET 8) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Bailout
    This PR (7383) - mean (278ms)  : 272, 284
     .   : milestone, 278,
    master - mean (295ms)  : 285, 304
     .   : milestone, 295,

    section Baseline
    This PR (7383) - mean (276ms)  : 269, 282
     .   : milestone, 276,
    master - mean (291ms)  : 284, 299
     .   : milestone, 291,

    section CallTarget+Inlining+NGEN
    This PR (7383) - mean (862ms)  : 839, 884
     .   : milestone, 862,
    master - mean (888ms)  : 850, 926
     .   : milestone, 888,
Loading

@pr-commenter
Copy link

pr-commenter bot commented Aug 15, 2025 •
edited
Loading

Benchmarks

Benchmarks Report for benchmark platform 🐌

Benchmarks for #7383 compared to master:

  • 1 benchmarks are faster, with geometric mean 1.128
  • 1 benchmarks are slower, with geometric mean 1.123
  • 7 benchmarks have fewer allocations
  • 8 benchmarks have more allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartStopWithChild net6.0 10.6μs 60ns 384ns 0 0 0 5.49 KB
master StartStopWithChild netcoreapp3.1 13.5μs 69.9ns 342ns 0 0 0 5.71 KB
master StartStopWithChild net472 22.7μs 121ns 641ns 0.955 0.318 0.106 6 KB
#7383 StartStopWithChild net6.0 10.3μs 54.6ns 289ns 0 0 0 5.48 KB
#7383 StartStopWithChild netcoreapp3.1 13.4μs 69.8ns 342ns 0 0 0 5.7 KB
#7383 StartStopWithChild net472 21.7μs 106ns 541ns 0.972 0.324 0.108 6.01 KB
Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 932μs 76.3ns 264ns 0 0 0 2.71 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 1.03ms 96.1ns 360ns 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces net472 1.2ms 2.36μs 9.15μs 0 0 0 3.31 KB
#7383 WriteAndFlushEnrichedTraces net6.0 935μs 190ns 713ns 0 0 0 2.71 KB
#7383 WriteAndFlushEnrichedTraces netcoreapp3.1 1.02ms 201ns 754ns 0 0 0 2.7 KB
#7383 WriteAndFlushEnrichedTraces net472 1.19ms 76ns 294ns 0 0 0 3.31 KB
Benchmarks.Trace.Asm.AppSecBodyBenchmark - Same speed ✔️ More allocations ⚠️

More allocations ⚠️ in #7383

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleSimpleBody‑net472 194.24 KB 199.88 KB 5.64 KB 2.90%
Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleMoreComplexBody‑net472 197.76 KB 203.39 KB 5.63 KB 2.85%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master AllCycleSimpleBody net6.0 328μs 312ns 1.21μs 0 0 0 172.08 KB
master AllCycleSimpleBody netcoreapp3.1 457μs 1.16μs 4.51μs 0 0 0 174.18 KB
master AllCycleSimpleBody net472 428μs 144ns 558ns 29.2 0 0 194.24 KB
master AllCycleMoreComplexBody net6.0 335μs 912ns 3.41μs 0 0 0 175.58 KB
master AllCycleMoreComplexBody netcoreapp3.1 503μs 872ns 3.38μs 0 0 0 177.6 KB
master AllCycleMoreComplexBody net472 440μs 145ns 560ns 30.2 0 0 197.76 KB
master ObjectExtractorSimpleBody net6.0 329ns 1.58ns 6.7ns 0 0 0 280 B
master ObjectExtractorSimpleBody netcoreapp3.1 398ns 2.08ns 10.8ns 0 0 0 272 B
master ObjectExtractorSimpleBody net472 300ns 0.188ns 0.726ns 0.0437 0 0 281 B
master ObjectExtractorMoreComplexBody net6.0 6.29μs 30ns 116ns 0 0 0 3.78 KB
master ObjectExtractorMoreComplexBody netcoreapp3.1 7.76μs 35.1ns 136ns 0 0 0 3.69 KB
master ObjectExtractorMoreComplexBody net472 6.74μs 5.78ns 22.4ns 0.57 0 0 3.8 KB
#7383 AllCycleSimpleBody net6.0 329μs 1.25μs 4.83μs 0 0 0 172.25 KB
#7383 AllCycleSimpleBody netcoreapp3.1 488μs 2.27μs 9.38μs 0 0 0 174.36 KB
#7383 AllCycleSimpleBody net472 430μs 171ns 663ns 30.2 0 0 199.88 KB
#7383 AllCycleMoreComplexBody net6.0 331μs 1.63μs 6.93μs 0 0 0 175.75 KB
#7383 AllCycleMoreComplexBody netcoreapp3.1 476μs 171ns 640ns 0 0 0 177.78 KB
#7383 AllCycleMoreComplexBody net472 442μs 163ns 633ns 30.2 0 0 203.39 KB
#7383 ObjectExtractorSimpleBody net6.0 322ns 0.165ns 0.572ns 0 0 0 280 B
#7383 ObjectExtractorSimpleBody netcoreapp3.1 398ns 1.91ns 7.86ns 0 0 0 272 B
#7383 ObjectExtractorSimpleBody net472 303ns 0.0312ns 0.121ns 0.0442 0 0 281 B
#7383 ObjectExtractorMoreComplexBody net6.0 6.26μs 3.03ns 11.3ns 0 0 0 3.78 KB
#7383 ObjectExtractorMoreComplexBody netcoreapp3.1 7.74μs 16.7ns 64.6ns 0 0 0 3.69 KB
#7383 ObjectExtractorMoreComplexBody net472 6.83μs 4.87ns 18.8ns 0.58 0 0 3.8 KB
Benchmarks.Trace.Asm.AppSecEncoderBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EncodeArgs net6.0 77.1μs 53.4ns 207ns 0 0 0 32.4 KB
master EncodeArgs netcoreapp3.1 97μs 245ns 948ns 0 0 0 32.4 KB
master EncodeArgs net472 109μs 48.1ns 180ns 4.93 0 0 32.5 KB
master EncodeLegacyArgs net6.0 145μs 103ns 401ns 0 0 0 2.15 KB
master EncodeLegacyArgs netcoreapp3.1 197μs 203ns 787ns 0 0 0 2.14 KB
master EncodeLegacyArgs net472 262μs 66.4ns 257ns 0 0 0 2.16 KB
#7383 EncodeArgs net6.0 77.5μs 262ns 981ns 0 0 0 32.4 KB
#7383 EncodeArgs netcoreapp3.1 98.2μs 135ns 522ns 0 0 0 32.4 KB
#7383 EncodeArgs net472 111μs 10.4ns 36.1ns 5 0 0 32.5 KB
#7383 EncodeLegacyArgs net6.0 145μs 45.7ns 177ns 0 0 0 2.15 KB
#7383 EncodeLegacyArgs netcoreapp3.1 198μs 26.4ns 95.1ns 0 0 0 2.14 KB
#7383 EncodeLegacyArgs net472 261μs 50.6ns 189ns 0 0 0 2.16 KB
Benchmarks.Trace.Asm.AppSecWafBenchmark - Same speed ✔️ More allocations ⚠️

More allocations ⚠️ in #7383

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.Asm.AppSecWafBenchmark.RunWafRealisticBenchmarkWithAttack‑net472 2.29 KB 2.3 KB 13 B 0.57%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunWafRealisticBenchmark net6.0 398μs 161ns 622ns 0 0 0 4.55 KB
master RunWafRealisticBenchmark netcoreapp3.1 817μs 14.7μs 144μs 0 0 0 4.48 KB
master RunWafRealisticBenchmark net472 437μs 80.5ns 301ns 0 0 0 4.66 KB
master RunWafRealisticBenchmarkWithAttack net6.0 289μs 106ns 367ns 0 0 0 2.24 KB
master RunWafRealisticBenchmarkWithAttack netcoreapp3.1 490μs 6.83μs 67.9μs 0 0 0 2.22 KB
master RunWafRealisticBenchmarkWithAttack net472 313μs 58.8ns 212ns 0 0 0 2.29 KB
#7383 RunWafRealisticBenchmark net6.0 397μs 106ns 396ns 0 0 0 4.56 KB
#7383 RunWafRealisticBenchmark netcoreapp3.1 821μs 11.6μs 116μs 0 0 0 4.48 KB
#7383 RunWafRealisticBenchmark net472 435μs 65.8ns 255ns 0 0 0 4.66 KB
#7383 RunWafRealisticBenchmarkWithAttack net6.0 289μs 79.9ns 309ns 0 0 0 2.24 KB
#7383 RunWafRealisticBenchmarkWithAttack netcoreapp3.1 511μs 79.7ns 276ns 0 0 0 2.22 KB
#7383 RunWafRealisticBenchmarkWithAttack net472 312μs 68ns 254ns 0 0 0 2.3 KB
Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendRequest net6.0 61μs 51.2ns 177ns 0 0 0 14.52 KB
master SendRequest netcoreapp3.1 71.2μs 67.2ns 260ns 0 0 0 17.42 KB
master SendRequest net472 0.00966ns 0.00336ns 0.013ns 0 0 0 0 b
#7383 SendRequest net6.0 61.5μs 154ns 576ns 0 0 0 14.52 KB
#7383 SendRequest netcoreapp3.1 71.1μs 283ns 1.13μs 0 0 0 17.42 KB
#7383 SendRequest net472 0.00541ns 0.00201ns 0.0078ns 0 0 0 0 b
Benchmarks.Trace.CharSliceBenchmark - Same speed ✔️ Fewer allocations 🎉

Fewer allocations 🎉 in #7383

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSlice‑net6.0 4 B 2 B -2 B -50.00%
Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSliceWithPool‑net6.0 5 B 1 B -4 B -80.00%
Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSlice‑net472 73 B 0 b -73 B -100.00%
Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSliceWithPool‑net472 49 B 0 b -49 B -100.00%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master OriginalCharSlice net6.0 1.9ms 4.67μs 18.1μs 0 0 0 640.01 KB
master OriginalCharSlice netcoreapp3.1 2.08ms 7.43μs 26.8μs 0 0 0 640 KB
master OriginalCharSlice net472 2.64ms 578ns 2.24μs 100 0 0 641.95 KB
master OptimizedCharSlice net6.0 1.49ms 99.8ns 386ns 0 0 0 4 B
master OptimizedCharSlice netcoreapp3.1 1.64ms 579ns 2.24μs 0 0 0 1 B
master OptimizedCharSlice net472 1.96ms 324ns 1.21μs 0 0 0 73 B
master OptimizedCharSliceWithPool net6.0 796μs 30ns 116ns 0 0 0 5 B
master OptimizedCharSliceWithPool netcoreapp3.1 832μs 72.6ns 262ns 0 0 0 1 B
master OptimizedCharSliceWithPool net472 1.18ms 241ns 902ns 0 0 0 49 B
#7383 OriginalCharSlice net6.0 1.93ms 741ns 2.77μs 0 0 0 640 KB
#7383 OriginalCharSlice netcoreapp3.1 2.08ms 7.11μs 26.6μs 0 0 0 640 KB
#7383 OriginalCharSlice net472 2.59ms 939ns 3.39μs 100 0 0 641.95 KB
#7383 OptimizedCharSlice net6.0 1.36ms 203ns 786ns 0 0 0 2 B
#7383 OptimizedCharSlice netcoreapp3.1 1.68ms 315ns 1.22μs 0 0 0 1 B
#7383 OptimizedCharSlice net472 2.01ms 65.9ns 237ns 0 0 0 0 b
#7383 OptimizedCharSliceWithPool net6.0 821μs 31ns 120ns 0 0 0 1 B
#7383 OptimizedCharSliceWithPool netcoreapp3.1 812μs 77ns 288ns 0 0 0 1 B
#7383 OptimizedCharSliceWithPool net472 1.13ms 86.4ns 335ns 0 0 0 0 b
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Slower ⚠️ More allocations ⚠️

Slower ⚠️ in #7383

Benchmark diff/base Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces‑netcoreapp3.1 1.123 679,349.04 763,232.34

Faster 🎉 in #7383

Benchmark base/diff Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces‑net472 1.128 965,697.08 855,741.25

More allocations ⚠️ in #7383

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces‑netcoreapp3.1 41.87 KB 42.88 KB 1.01 KB 2.42%

Fewer allocations 🎉 in #7383

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces‑net472 56.36 KB 55.84 KB -512 B -0.91%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 683μs 1.75μs 6.07μs 0 0 0 41.7 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 680μs 745ns 2.88μs 0 0 0 41.87 KB
master WriteAndFlushEnrichedTraces net472 968μs 3.99μs 14.4μs 8.33 0 0 56.36 KB
#7383 WriteAndFlushEnrichedTraces net6.0 709μs 1.04μs 4.02μs 0 0 0 41.86 KB
#7383 WriteAndFlushEnrichedTraces netcoreapp3.1 759μs 3.75μs 15.5μs 0 0 0 42.88 KB
#7383 WriteAndFlushEnrichedTraces net472 860μs 4.3μs 18.8μs 8.33 0 0 55.84 KB
Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteNonQuery net6.0 1.92μs 6.88ns 26.6ns 0 0 0 1.02 KB
master ExecuteNonQuery netcoreapp3.1 2.74μs 7.4ns 28.7ns 0 0 0 1.02 KB
master ExecuteNonQuery net472 2.82μs 2.35ns 8.48ns 0.156 0.0141 0 987 B
#7383 ExecuteNonQuery net6.0 1.83μs 2.41ns 9.32ns 0 0 0 1.02 KB
#7383 ExecuteNonQuery netcoreapp3.1 2.5μs 1.15ns 4.32ns 0 0 0 1.02 KB
#7383 ExecuteNonQuery net472 2.8μs 2.24ns 8.07ns 0.154 0.014 0 987 B
Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master CallElasticsearch net6.0 1.78μs 6.16ns 21.4ns 0 0 0 1.03 KB
master CallElasticsearch netcoreapp3.1 2.26μs 9.56ns 37ns 0 0 0 1.03 KB
master CallElasticsearch net472 3.56μs 4.71ns 18.2ns 0.16 0 0 1.04 KB
master CallElasticsearchAsync net6.0 1.85μs 9.52ns 45.6ns 0 0 0 1.01 KB
master CallElasticsearchAsync netcoreapp3.1 2.33μs 11.1ns 41.4ns 0 0 0 1.08 KB
master CallElasticsearchAsync net472 3.79μs 2.38ns 9.23ns 0.17 0 0 1.1 KB
#7383 CallElasticsearch net6.0 1.68μs 1.1ns 3.96ns 0 0 0 1.03 KB
#7383 CallElasticsearch netcoreapp3.1 2.28μs 11.1ns 44.3ns 0 0 0 1.03 KB
#7383 CallElasticsearch net472 3.56μs 1.65ns 6.16ns 0.16 0 0 1.04 KB
#7383 CallElasticsearchAsync net6.0 1.81μs 8.89ns 39.7ns 0 0 0 1.01 KB
#7383 CallElasticsearchAsync netcoreapp3.1 2.4μs 11.8ns 51.7ns 0 0 0 1.08 KB
#7383 CallElasticsearchAsync net472 3.82μs 1.84ns 7.12ns 0.172 0 0 1.1 KB
Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteAsync net6.0 1.98μs 6.54ns 25.3ns 0 0 0 952 B
master ExecuteAsync netcoreapp3.1 2.3μs 1.89ns 7.32ns 0 0 0 952 B
master ExecuteAsync net472 2.42μs 2.32ns 8.69ns 0.145 0 0 915 B
#7383 ExecuteAsync net6.0 1.83μs 0.758ns 2.73ns 0 0 0 952 B
#7383 ExecuteAsync netcoreapp3.1 2.34μs 10.8ns 43.2ns 0 0 0 952 B
#7383 ExecuteAsync net472 2.47μs 2.17ns 7.82ns 0.136 0 0 915 B
Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendAsync net6.0 6.91μs 4.85ns 17.5ns 0 0 0 2.36 KB
master SendAsync netcoreapp3.1 8.38μs 22.4ns 86.9ns 0 0 0 2.9 KB
master SendAsync net472 12.5μs 7.53ns 29.2ns 0.498 0 0 3.18 KB
#7383 SendAsync net6.0 7.03μs 4.21ns 14.6ns 0 0 0 2.36 KB
#7383 SendAsync netcoreapp3.1 8.31μs 30.5ns 118ns 0 0 0 2.9 KB
#7383 SendAsync net472 12.4μs 6.16ns 22.2ns 0.497 0 0 3.18 KB
Benchmarks.Trace.Iast.StringAspectsBenchmark - Same speed ✔️ More allocations ⚠️

More allocations ⚠️ in #7383

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatBenchmark‑net472 57.34 KB 65.54 KB 8.19 KB 14.29%
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark‑net6.0 258.26 KB 277.2 KB 18.94 KB 7.33%
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark‑netcoreapp3.1 257.73 KB 274.42 KB 16.7 KB 6.48%
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatBenchmark‑netcoreapp3.1 42.64 KB 43.31 KB 672 B 1.58%

Fewer allocations 🎉 in #7383

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatBenchmark‑net6.0 43.77 KB 43.49 KB -280 B -0.64%
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark‑net472 348.55 KB 278.53 KB -70.02 KB -20.09%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StringConcatBenchmark net6.0 47.6μs 215ns 803ns 0 0 0 43.77 KB
master StringConcatBenchmark netcoreapp3.1 49.5μs 280ns 1.81μs 0 0 0 42.64 KB
master StringConcatBenchmark net472 57.2μs 241ns 868ns 0 0 0 57.34 KB
master StringConcatAspectBenchmark net6.0 465μs 1.42μs 5.11μs 0 0 0 258.26 KB
master StringConcatAspectBenchmark netcoreapp3.1 541μs 2.5μs 10μs 0 0 0 257.73 KB
master StringConcatAspectBenchmark net472 407μs 2.14μs 10.9μs 0 0 0 348.55 KB
#7383 StringConcatBenchmark net6.0 48.7μs 212ns 794ns 0 0 0 43.49 KB
#7383 StringConcatBenchmark netcoreapp3.1 50.7μs 406ns 3.81μs 0 0 0 43.31 KB
#7383 StringConcatBenchmark net472 57.6μs 134ns 483ns 0 0 0 65.54 KB
#7383 StringConcatAspectBenchmark net6.0 477μs 1.35μs 4.69μs 0 0 0 277.2 KB
#7383 StringConcatAspectBenchmark netcoreapp3.1 536μs 2.17μs 7.84μs 0 0 0 274.42 KB
#7383 StringConcatAspectBenchmark net472 404μs 2.16μs 11.2μs 0 0 0 278.53 KB
Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 2.62μs 4.41ns 17.1ns 0 0 0 1.7 KB
master EnrichedLog netcoreapp3.1 3.29μs 9.02ns 35ns 0 0 0 1.7 KB
master EnrichedLog net472 4μs 4.26ns 16.5ns 0.258 0 0 1.64 KB
#7383 EnrichedLog net6.0 2.56μs 13.7ns 76ns 0 0 0 1.7 KB
#7383 EnrichedLog netcoreapp3.1 3.45μs 15.6ns 60.3ns 0 0 0 1.7 KB
#7383 EnrichedLog net472 4μs 1.96ns 6.78ns 0.24 0 0 1.64 KB
Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 123μs 110ns 398ns 0 0 0 4.31 KB
master EnrichedLog netcoreapp3.1 128μs 203ns 787ns 0 0 0 4.31 KB
master EnrichedLog net472 167μs 38.1ns 148ns 0 0 0 4.52 KB
#7383 EnrichedLog net6.0 124μs 119ns 461ns 0 0 0 4.31 KB
#7383 EnrichedLog netcoreapp3.1 127μs 93.9ns 339ns 0 0 0 4.31 KB
#7383 EnrichedLog net472 168μs 48.9ns 183ns 0 0 0 4.52 KB
Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 4.99μs 19.9ns 77.1ns 0 0 0 2.26 KB
master EnrichedLog netcoreapp3.1 6.71μs 20.4ns 78.9ns 0 0 0 2.26 KB
master EnrichedLog net472 7.46μs 9.81ns 38ns 0.3 0 0 2.08 KB
#7383 EnrichedLog net6.0 5.03μs 1.92ns 7.45ns 0 0 0 2.26 KB
#7383 EnrichedLog netcoreapp3.1 6.65μs 10.2ns 39.4ns 0 0 0 2.26 KB
#7383 EnrichedLog net472 7.25μs 4.3ns 16.1ns 0.326 0 0 2.08 KB
Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendReceive net6.0 1.99μs 10.1ns 44.2ns 0 0 0 1.2 KB
master SendReceive netcoreapp3.1 2.71μs 13.2ns 52.8ns 0 0 0 1.2 KB
master SendReceive net472 3.12μs 7.59ns 29.4ns 0.189 0 0 1.2 KB
#7383 SendReceive net6.0 1.96μs 9.69ns 42.2ns 0 0 0 1.2 KB
#7383 SendReceive netcoreapp3.1 2.57μs 10.9ns 40.7ns 0 0 0 1.2 KB
#7383 SendReceive net472 3.07μs 5.9ns 22.1ns 0.183 0 0 1.2 KB
Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 4.13μs 16.6ns 64.2ns 0 0 0 1.58 KB
master EnrichedLog netcoreapp3.1 5.6μs 12.3ns 47.7ns 0 0 0 1.63 KB
master EnrichedLog net472 6.65μs 11.3ns 43.9ns 0.3 0 0 2.03 KB
#7383 EnrichedLog net6.0 4.17μs 19.1ns 74.2ns 0 0 0 1.58 KB
#7383 EnrichedLog netcoreapp3.1 5.71μs 5.21ns 20.2ns 0 0 0 1.63 KB
#7383 EnrichedLog net472 6.64μs 11.5ns 44.7ns 0.296 0 0 2.03 KB
Benchmarks.Trace.SpanBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartFinishSpan net6.0 752ns 3.61ns 14ns 0 0 0 576 B
master StartFinishSpan netcoreapp3.1 916ns 4.94ns 29.6ns 0 0 0 576 B
master StartFinishSpan net472 898ns 0.0935ns 0.35ns 0.0903 0 0 578 B
master StartFinishScope net6.0 907ns 4.31ns 16.7ns 0 0 0 696 B
master StartFinishScope netcoreapp3.1 1.15μs 1.32ns 5.1ns 0 0 0 696 B
master StartFinishScope net472 1.09μs 0.242ns 0.937ns 0.104 0 0 658 B
#7383 StartFinishSpan net6.0 741ns 0.31ns 1.2ns 0 0 0 576 B
#7383 StartFinishSpan netcoreapp3.1 946ns 4.41ns 17.1ns 0 0 0 576 B
#7383 StartFinishSpan net472 890ns 0.0938ns 0.351ns 0.0894 0 0 578 B
#7383 StartFinishScope net6.0 895ns 2.06ns 7.98ns 0 0 0 696 B
#7383 StartFinishScope netcoreapp3.1 1.12μs 2.41ns 9.32ns 0 0 0 696 B
#7383 StartFinishScope net472 1.11μs 0.328ns 1.27ns 0.0995 0 0 658 B
Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunOnMethodBegin net6.0 1.04μs 5.36ns 25.7ns 0 0 0 696 B
master RunOnMethodBegin netcoreapp3.1 1.33μs 6.48ns 29ns 0 0 0 696 B
master RunOnMethodBegin net472 1.4μs 1.54ns 5.76ns 0.105 0 0 658 B
#7383 RunOnMethodBegin net6.0 1.03μs 0.837ns 3.24ns 0 0 0 696 B
#7383 RunOnMethodBegin netcoreapp3.1 1.35μs 6.64ns 28.9ns 0 0 0 696 B
#7383 RunOnMethodBegin net472 1.39μs 0.728ns 2.82ns 0.104 0 0 658 B

@bouwkast bouwkast force-pushed the steven/detect-azure-func-host-disable-direct-log branch from fd9864f to 9dd3a22 Compare August 19, 2025 20:00
@github-actions
Copy link
Contributor

Snapshots difference summary

The following differences have been observed in committed snapshots. It is meant to help the reviewer.
The diff is simplistic, so please check some files anyway while we improve it.

@datadog-official

This comment has been minimized.

Sign in to view
bouwkast
bouwkast commented Aug 20, 2025
View reviewed changes
...plications/azure-functions/Samples.AzureFunctions.V4Isolated.HostLogsDisabled/AllTriggers.cs
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I copied the files because I was running into frequent "File in use" when attempting to build / run the samples and tests and the only thing I can think of is that there is some form of race condition between linking the various files with the worker extension projects that get generated.

Not really sure what to do or if that was the cause/solution, but it went away after I did this.

@bouwkast bouwkast force-pushed the steven/detect-azure-func-host-disable-direct-log branch from 1454cb9 to e35949b Compare August 21, 2025 13:17
@bouwkast bouwkast changed the title [WIP] Allow for disabling agentless logging when running in an Azure Function host Allow for disabling agentless logging when running in an Azure Function host Sep 4, 2025
@bouwkast bouwkast force-pushed the steven/detect-azure-func-host-disable-direct-log branch from 9c5e4cb to 73bc40f Compare September 4, 2025 14:06
@bouwkast bouwkast marked this pull request as ready for review September 4, 2025 14:06
@bouwkast bouwkast requested review from a team as code owners September 4, 2025 14:06
andrewlock
andrewlock approved these changes Sep 5, 2025
View reviewed changes
Copy link
Member

@andrewlock andrewlock left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks!

tracer/src/Datadog.Trace/Logging/DirectSubmission/DirectLogSubmissionSettings.cs Outdated
BatchPeriod = TimeSpan.FromSeconds(seconds);

AzureFunctionsHostEnabled = config.WithKeys(ConfigurationKeys.DirectLogSubmission.AzureFunctionsHostEnabled)
.AsBool(true);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if this should be false (effectively making this a bug-fix that you can revert)?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lucaspimentel thoughts on this?

I don't think these host logs are necessarily useful, but it could be a breaking change if someone is using them.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My thoughts exactly.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Every bug fix is technically a breaking change 😉

image

IMO the current behaviour is "wrong" because it causes duplicate logs 🤷‍♂️ But my level of care is low, so will def defer to @lucaspimentel as code ower😄

Copy link
Member

@lucaspimentel lucaspimentel Sep 16, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the current behaviour is "wrong" because it causes duplicate logs

Hmm. Before this PR, we duplicate logs from the worker process, not logs from the host process. This PR fix that duplication, but in the process we also loose all logs from the host process.

Maybe the better solution is to disable agentless logging in the worker process instead of the host process, so we can capture all logs (once) in the host process. This is assuming all the logs from the worker are sent to the host so we can capture them there.

@lucaspimentel
Copy link
Member

@codex review

@chatgpt-codex-connector
Copy link

Codex Review: Didn't find any major issues. Swish!

About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you open a pull request for review, mark a draft as ready, or comment "@codex review". If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex fix this CI failure" or "@codex address that feedback".

chatgpt-codex-connector[bot]
chatgpt-codex-connector bot reviewed Sep 16, 2025
View reviewed changes
Copy link

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Codex Review: Here are some suggestions.

Reply with @codex fix comments to fix any unresolved comments.

About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you open a pull request for review, mark a draft as ready, or comment "@codex review". If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex fix this CI failure" or "@codex address that feedback".

tracer/src/Datadog.Trace/Util/EnvironmentHelpers.cs Outdated
Comment on lines 142 to 147
var cmd = Environment.CommandLine ?? string.Empty;
var hasWorkerId = cmd.IndexOf("--functions-workerid", StringComparison.OrdinalIgnoreCase) >= 0 ||
cmd.IndexOf("--workerId", StringComparison.OrdinalIgnoreCase) >= 0;

return !hasWorkerId
&& IsAzureFunctions()
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Invert workerId check in Azure Functions host detection

The documentation for IsRunningInAzureFunctionsHost() says the host process is identified when FUNCTIONS_WORKER_RUNTIME=dotnet-isolated and the command line contains --functions-workerid/--workerId, but the code currently returns !hasWorkerId && …. That makes the method true only when the worker-id flags are absent, so DirectLogSubmissionSettings will keep host logging enabled even when DD_LOGS_DIRECT_SUBMISSION_AZURE_FUNCTIONS_HOST_ENABLED=false, causing the duplicate log scenario this change was meant to avoid. Flip the condition to check hasWorkerId instead of its negation.

Useful? React with 👍 / 👎.

Copy link
Member

@lucaspimentel lucaspimentel Sep 18, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does hasWorkerId mean we are running on the host or the worker? I think robot has it backwards. Might want to clarify that in the xml comments above.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clarified, my documentation comment was incorrect (as was the PR description), but the code was correct.

The worker process will have those flags (actually mis-spelled one 😢 )
The host process will not have those flags

At least per what I see locally

[2025-10-03T16:01:42.901Z] Reading functions metadata (Worker)
[2025-10-03T16:01:47.176Z] {
[2025-10-03T16:01:47.177Z]   "ProcessId": 71080,
[2025-10-03T16:01:47.178Z]   "RuntimeIdentifier": "win-x64",
[2025-10-03T16:01:47.179Z]   "WorkerVersion": "2.0.0.0",
[2025-10-03T16:01:47.180Z]   "ProductVersion": "2.0.0\u002Bd8b5fe998a8c92819b8ee41d2569d2525413e9c5",
[2025-10-03T16:01:47.181Z]   "FrameworkDescription": ".NET 9.0.9",
[2025-10-03T16:01:47.182Z]   "OSDescription": "Microsoft Windows 10.0.26100",
[2025-10-03T16:01:47.183Z]   "OSArchitecture": "X64",
[2025-10-03T16:01:47.184Z]   "CommandLine": "C:\\Users\\steven.bouwkamp\\source\\repos\\dd-trace-dotnet\\artifacts\\bin\\Samples.AzureFunctions.V4Isolated.AspNetCore\\debug_net9.0\\Samples.AzureFunctions.V4Isolated.AspNetCore.dll --host 127.0.0.1 --port 65401 --workerId e94d23fd-cd3c-4780-a3e3-4980d7b0f644 --requestId 6dba68ac-1954-466a-aeb4-9570cc9b12c2 --grpcMaxMessageLength 2147483647 --functions-uri http://127.0.0.1:65401/ --functions-worker-id e94d23fd-cd3c-4780-a3e3-4980d7b0f644 --functions-request-id 6dba68ac-1954-466a-aeb4-9570cc9b12c2 --functions-grpc-max-message-length 2147483647"
[2025-10-03T16:01:47.185Z] }

@chatgpt-codex-connector
Copy link

Codex Review: Here are some suggestions.

dd-trace-dotnet/tracer/src/Datadog.Trace/Util/EnvironmentHelpers.cs

Lines 142 to 147 in f228f6c

var cmd = Environment.CommandLine ?? string.Empty;
var hasWorkerId = cmd.IndexOf("--functions-workerid", StringComparison.OrdinalIgnoreCase) >= 0 ||
cmd.IndexOf("--workerId", StringComparison.OrdinalIgnoreCase) >= 0;
return !hasWorkerId
&& IsAzureFunctions()

[P1] Invert workerId check in Azure Functions host detection

The documentation for IsRunningInAzureFunctionsHost() says the host process is identified when FUNCTIONS_WORKER_RUNTIME=dotnet-isolated and the command line contains --functions-workerid/--workerId, but the code currently returns !hasWorkerId && …. That makes the method true only when the worker-id flags are absent, so DirectLogSubmissionSettings will keep host logging enabled even when DD_LOGS_DIRECT_SUBMISSION_AZURE_FUNCTIONS_HOST_ENABLED=false, causing the duplicate log scenario this change was meant to avoid. Flip the condition to check hasWorkerId instead of its negation.

Reply with @codex fix comments to fix any unresolved comments.

About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you open a pull request for review, mark a draft as ready, or comment "@codex review". If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex fix this CI failure" or "@codex address that feedback".

@bouwkast bouwkast force-pushed the steven/detect-azure-func-host-disable-direct-log branch from f228f6c to a57dd43 Compare September 22, 2025 16:07
@bouwkast bouwkast added type:bug area:serverless and removed status:work-in-progress Actively worked on. If this is a PR, no review needed yet. WIP. labels Sep 22, 2025
@bouwkast bouwkast changed the title Allow for disabling agentless logging when running in an Azure Function host Fix for sending duplicate logs when using Agentless Logging in Azure Function host Sep 22, 2025
@bouwkast
Add DD_LOGS_DIRECT_SUBMISSION_AZURE_FUNCTIONS_HOST_ENABLED
18d55a5 
Allows for the ability to disable direct log
submission when we detect that we are running on
the function host.

When using direct log submission in a dotnet-isolated
Azure Functions application we end up getting the
worker logs re-logged by the host process which
makes it appear that there are duplicated logs.

This allows for the ability to entirely disable
direct log submission on that function host.
@bouwkast
Add new Azure Functions sample for disabled logging
6981092 
Ran into an issue where re-running the same
Azure Function application in our tests would cause
the func.exe to fail to acquire a lock for 60s and
cause the tests to take longer.

This was the simplest workaround.
@bouwkast
Add tests for disabling host logging
a5cabf7
@bouwkast
Use a range for expected logs
6b6877e 
For some reason we get a different amount of logs
on the different frameworks.

It is usually 13, but sometimes 14, sometimes 16

Unsure really, but if we were still sending host
logs we'd expect many more than 20, so seems fine
@bouwkast
Change --functions-workerid to --functions-worker-id
d139ef4
@bouwkast
Fix incorrect XML comment
46593f8
@bouwkast bouwkast force-pushed the steven/detect-azure-func-host-disable-direct-log branch from 6b6a444 to 46593f8 Compare October 3, 2025 16:15
@bouwkast
Copy link
Collaborator Author

bouwkast commented Oct 3, 2025

@codex review

chatgpt-codex-connector[bot]
chatgpt-codex-connector bot reviewed Oct 3, 2025
View reviewed changes
Copy link

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting

@bouwkast bouwkast merged commit 6d68aad into master Oct 3, 2025
70 of 72 checks passed
@bouwkast bouwkast deleted the steven/detect-azure-func-host-disable-direct-log branch October 3, 2025 20:38
@github-actions github-actions bot added this to the vNext-v3 milestone Oct 3, 2025
@bouwkast
Copy link
Collaborator Author

bouwkast commented Oct 3, 2025

Tested this out locally and in real Azure Functions

igoragoli pushed a commit that referenced this pull request Oct 6, 2025
@bouwkast @igoragoli
Fix for sending duplicate logs when using Agentless Logging in Azure …
0a62e80 
…Function host (#7383)

## Summary of changes

Disables agentless logging (for the Azure Functions Host Process) if we
detect that we are instrumenting an Azure Functions host process.
Previously this would cause us to duplicate every log sent from the
worker process.

This bug fix can be reverted by setting the new configuration key
`DD_LOGS_DIRECT_SUBMISSION_AZURE_FUNCTIONS_HOST_ENABLED` to `true`.

## Reason for change

If customers are running with the Isolated Azure Function model we will
be instrumenting two applications, one is the function host process the
other is the worker process. If they have direct log submission enabled
the function host ends up duplicating the logs from the function process
which results in us shipping two nearly identical logs.

This behavior isn't ideal as the duplicate log is likely not valuable so
we've disabled agentless logging in the azure functions host process.

## Implementation details

Added `IsRunningInAzureFunctionsHost()` to `EnvironmentHelpers.cs` which
allows for a rough detection of whether we are running on the function
host using the following logic:

- Is `FUNCTIONS_WORKER_RUNTIME` present AND set to `dotnet-isolated`?
- Are both `--functions-worker-id` or `--workerId` NOT present in the
command line?

If both are true we treat that scenario as being running in the function
host - otherwise we are likely the worker process

I wasn't able to find a more robust way of checking, but when looking at
the various log output that I had the `--functions-worker-id` or
`--workerId` seemed to always be called by the function host.

```
[2025-10-03T16:01:42.901Z] Reading functions metadata (Worker)
[2025-10-03T16:01:47.176Z] {
[2025-10-03T16:01:47.177Z]   "ProcessId": 71080,
[2025-10-03T16:01:47.178Z]   "RuntimeIdentifier": "win-x64",
[2025-10-03T16:01:47.179Z]   "WorkerVersion": "2.0.0.0",
[2025-10-03T16:01:47.180Z]   "ProductVersion": "2.0.0\u002Bd8b5fe998a8c92819b8ee41d2569d2525413e9c5",
[2025-10-03T16:01:47.181Z]   "FrameworkDescription": ".NET 9.0.9",
[2025-10-03T16:01:47.182Z]   "OSDescription": "Microsoft Windows 10.0.26100",
[2025-10-03T16:01:47.183Z]   "OSArchitecture": "X64",
[2025-10-03T16:01:47.184Z]   "CommandLine": "C:\\Users\\steven.bouwkamp\\source\\repos\\dd-trace-dotnet\\artifacts\\bin\\Samples.AzureFunctions.V4Isolated.AspNetCore\\debug_net9.0\\Samples.AzureFunctions.V4Isolated.AspNetCore.dll --host 127.0.0.1 --port 65401 --workerId e94d23fd-cd3c-4780-a3e3-4980d7b0f644 --requestId 6dba68ac-1954-466a-aeb4-9570cc9b12c2 --grpcMaxMessageLength 2147483647 --functions-uri http://127.0.0.1:65401/ --functions-worker-id e94d23fd-cd3c-4780-a3e3-4980d7b0f644 --functions-request-id 6dba68ac-1954-466a-aeb4-9570cc9b12c2 --functions-grpc-max-message-length 2147483647"
[2025-10-03T16:01:47.185Z] }
```

- Added `DD_LOGS_DIRECT_SUBMISSION_AZURE_FUNCTIONS_HOST_ENABLED` that
defaults to `false` to disable the duplicate logs from being sent.

## Test coverage

- Added a new test project and test for having host logs enabled /
disabled.
- The reason why I added a new test project instead of re-using an
existing one was because when I re-ran the Function application in our
tests multiple times the `func.exe` would fail to obtain a lock and
would need to wait some period of time to for recovery after each
subsequent run of the same application. I think this is because we end
the `func.exe` process with a `Kill()`. Making a new project wasn't
ideal but was a quick and simple workaround.

## Other details
Fixes SLES-2364

<!--  ⚠️ Note:

Where possible, please obtain 2 approvals prior to merging. Unless
CODEOWNERS specifies otherwise, for external teams it is typically best
to have one review from a team member, and one review from apm-dotnet.
Trivial changes do not require 2 reviews.

MergeQueue is NOT enabled in this repository. If you have write access
to the repo, the PR has 1-2 approvals (see above), and all of the
required checks have passed, you can use the Squash and Merge button to
merge the PR. If you don't have write access, or you need help, reach
out in the #apm-dotnet channel in Slack.
-->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@lucaspimentel lucaspimentel lucaspimentel left review comments

@andrewlock andrewlock andrewlock approved these changes

@chatgpt-codex-connector chatgpt-codex-connector[bot] chatgpt-codex-connector[bot] left review comments

Assignees

No one assigned

Labels

type:bug

Projects

None yet

Milestone

3.28.0

Development

Successfully merging this pull request may close these issues.

3 participants

@bouwkast @lucaspimentel @andrewlock