Allow empty fields in the exchange integration redux (#12846)

* Updated the yaml and the test log file. The expected output still needs to be generated.

* Added ignore field and added new expected output.

* Used elastic-package to generate a new changelog entry and manifest file.

* Update packages/microsoft_exchange_server/data_stream/messagetracking/elasticsearch/ingest_pipeline/default.yml

Co-authored-by: Andrew Kroh <[email protected]>

* Update packages/microsoft_exchange_server/data_stream/messagetracking/elasticsearch/ingest_pipeline/default.yml

Co-authored-by: Andrew Kroh <[email protected]>

* Updated the expected data.

* Changed the sender address back to a list append.

* Added a test for missing networkmessageid.

---------

Co-authored-by: Andrew Kroh <[email protected]>

packages/microsoft_exchange_server/changelog.yml

@@ -1,4 +1,9 @@
 # newer versions go on top
+- version: "1.3.1"
+  changes:
+    - description: Handle events where `networkmessageid` or `senderaddress` are not present.
+      type: bugfix
+      link: https://github.com/elastic/integrations/pull/12846
 - version: "1.3.0"
   changes:
     - description: ECS version updated to 8.17.0.

packages/microsoft_exchange_server/data_stream/messagetracking/_dev/test/pipeline/test-messagetracking.log

@@ -2,3 +2,6 @@
 2024-01-25T15:16:09.949Z,10.11.12.14,exchange-mail.my.domain.com,10.11.12.14,exchange-mail,08DC1DB12C345BE5;2024-01-25T15:16:09.544Z;0,exchange-mail\Default exchange-mail,SMTP,RECEIVE,70912345566403,<[email protected]>,1e6eb197-c6b4-1234-1b69-56dc1db88f50,[email protected],,7229,1,,,vzdump backup status (host01.my.domain.com): backup successful,[email protected],[email protected],0cA: ,Incoming,,10.11.12.13,10.11.12.14,S:ProxyHop1=exchange-mail.my.domain.com(10.11.12.14);S:MessageValue=MediumHigh;S:Replication=Failed;S:FirstForestHop=exchange-mail.my.domain.com;S:FromEntity=Internet;S:ProxiedClientIPAddress=10.11.12.13;S:ProxiedClientHostname=host01.my.domain.com;S:DeliveryPriority=Normal;S:AccountForest=my.domain.com,Email,05503123-c5b9-46fe-1234-56dc1db88f8f,15.02.0330.005
 2024-01-25T15:16:14.415Z,10.11.12.14,exchange-mail.my.domain.com,10.11.12.14,exchange-mail,08DC1DB12C345BE9;2024-01-25T15:16:12.885Z;0,exchange-mail\Default exchange-mail,SMTP,RECEIVE,70912345566407,<[email protected]>,c95b5dd1-f520-1234-e6dc-56dc1db8914d,[email protected],,8251,1,,,vzdump backup status (pve-vhost01.my.domain.com): backup successful,[email protected],[email protected],0cA: ,Incoming,,10.11.12.15,10.11.12.14,S:ProxyHop1=exchange-mail.my.domain.com(10.11.12.14);S:MessageValue=MediumHigh;S:Replication=Failed;S:FirstForestHop=exchange-mail.my.domain.com;S:FromEntity=Internet;S:ProxiedClientIPAddress=10.11.12.15;S:ProxiedClientHostname=pve-vhost01.my.domain.com;S:DeliveryPriority=Normal;S:AccountForest=my.domain.com,Email,d6aef52d-0e05-1234-e29b-56dc1db89238,15.02.0330.005
 2024-01-07T00:00:07.463Z,192.168.0.1,exchange,192.168.0.2,exchange.example.com,;250 [email protected][Hostname=exchange.example.com];ClientSubmitTime:,Intra-Organization SMTP Send Connector,SMTP,SEND,29519319995411,[email protected],0b7099ea-cb95-1234-328e-08dc5f139ac8,[email protected],250 2.1.5Recipient OK,38663,1,,,ein Titel,[email protected],[email protected],2024-01-07T00:00:05.535Z;LSRV=exchange.example.com:TOTAL-HUB=1.921|SMR=0.127(SMRDE=0.002|SMRC=0.125(SMRCL=0.105|X-SMRCR=0.125))|CAT=1.698(CATOS=0.018(CATSM=0.017(CATSM-Malware Agent=0.017))|CATRESL=0.004|CATORES=1.567(CATRS=1.566(CATRS-ScanMail Routing Agent=0.117|CATRS-Transport Rule Agent=0.002(X-ETREX=0.002)|CATRS-Index Routing Agent=1.444))|CATORT=0.108(CATRT=0.107(CATRT-Journal Agent=0.107)))|QDM=0.010|SMSC=0.006(X-SMSDR=0.011)|SMS=0.076(SMSMBXD=0.071),Originating,,,,S:E2ELatency=1.928;S:MsgRecipCount=1;S:IncludeInSla=True;S:Microsoft.Exchange.Transport.MailRecipient.RequiredTlsAuthLevel=Opportunistic;S:IsSmtpResponseFromExternalServer=False;S:DeliveryPriority=Normal;S:AccountForest=example.com,Email,a7ae9ef9-e10c-4111-19bf-08dc0f111bee,15.01.2507.035
+2025-01-14T09:29:05.327Z,216.160.83.56,HELLOWORLD,175.16.199.1,global-mail-onmicrosoft-com.mail.protection.outlook.com,";250 2.6.0 <[email protected]> [InternalId=5450313518243, Hostname=AAAAAAAAAAAAA.bbbbbbbb.prod.outlook.com] 141301 bytes in 0.101, 1361.827 KB/sec Queued mail for delivery;ClientSubmitTime:",Outbound to Office 365 - cb1ce4e7-cbb6-46e8-8a4c-e1f2efbdd202,SMTP,SENDEXTERNAL,8774618205228,<[email protected]>,cb1ce4e7-cbb6-46e8-8a4c-e1f2efbdd202,[email protected],250 2.1.5 Recipient OK,136349,1,,,John Doe,,<>,2025-01-14T09:28:50.814Z;SRV=HELLOWORLD.foo.example.com:TOTAL-FE=0.074|SMR=0.072(SMRPI=0.002(SMRPI-FrontendProxyAgent=0.002))|SMS=0.002;SRV=HELLOWORLD.foo.example.com:TOTAL-HUB=14.439|SMR=0.279(SMRDE=0.174|SMRC=0.105(SMRCL=0.105|X-SMRCR=0.016))|CAT=0.046(CATOS=0.018(CATSM=0.018(CATSM-Malware Agent=0.017))|CATRESL=0.003|CATORES=0.021(CATRS=0.021(CATRS-Prioritization Agent=0.002|CATRS-Index Routing Agent=0.017))|CATORT=0.002(CATRT=0.002(CATRT-CodeTwo Exchange Rules Transport Agent=0.001)))|QDE=13.818|SMS=0.291,Incoming,,,,S:E2ELatency=14.513;S:ExternalSendLatency=0.402;S:ToEntity=Internet;S:FromEntity=Internet;S:MsgRecipCount=1;S:IncludeInSla=True;S:Microsoft.Exchange.Transport.MailRecipient.RequiredTlsAuthLevel=DomainValidation;S:Microsoft.Exchange.Transport.MailRecipient.EffectiveTlsAuthLevel=DomainValidation;S:IsSmtpResponseFromExternalServer=True;S:DeliveryPriority=Normal;S:OriginalFromAddress=<>;S:AccountForest=wgs.wuerth.com,Email,82f8feae-ef9b-41bf-8500-1e10946655ae,15.01.2507.039
+2025-01-14T09:28:39.334Z,2a02:cf40::0000:1234:5678:9abc,HELLOWORLD,,,"MDB:f4adaa08-ff49-4bca-ba70-68e9b80597b0, Mailbox:84dd42d9-8814-4758-a8bb-e8765d62ec90, Event:190620893, MessageClass:IPM.Note, CreationTime:2025-01-14T09:28:39.321Z, ClientType:MOMT, SubmissionAssistant:MailboxTransportSubmissionEmailAssistant",,STOREDRIVER,NOTIFYMAPI,,,,,,,,,,,[email protected],,2025-01-14T09:28:39.321Z;LSRV=HELLOWORLD.foo.example.com:TOTAL-SUB=0.013|SA=0.013|MTSS-PEN=0.000,,,,,S:ItemEntryId=00-00-00-00-39-E7-E4-05-8C-C2-08-4C-AC-96-9A-D0-DE-C4-85-EE-07-00-D8-00-A6-A0-EC-B2-39-4D-BD-2F-31-4C-F7-4F-3F-4C-00-00-00-14-1A-99-00-00-46-7D-7B-CD-1B-EA-FC-40-9C-37-A9-7E-B0-DD-4C-16-00-08-65-53-76-46-00-00,,74364e3b-32b7-4108-9a2c-6dda84007459,15.01.2507.039
+2025-01-14T09:29:05.327Z,216.160.83.56,HELLOWORLD,175.16.199.1,global-mail-onmicrosoft-com.mail.protection.outlook.com,";250 2.6.0 <[email protected]> [InternalId=5450313518243, Hostname=AAAAAAAAAAAAA.bbbbbbbb.prod.outlook.com] 141301 bytes in 0.101, 1361.827 KB/sec Queued mail for delivery;ClientSubmitTime:",Outbound to Office 365 - cb1ce4e7-cbb6-46e8-8a4c-e1f2efbdd202,SMTP,SENDEXTERNAL,8774618205228,<[email protected]>,,[email protected],250 2.1.5 Recipient OK,136349,1,,,John Doe,,<>,2025-01-14T09:28:50.814Z;SRV=HELLOWORLD.foo.example.com:TOTAL-FE=0.074|SMR=0.072(SMRPI=0.002(SMRPI-FrontendProxyAgent=0.002))|SMS=0.002;SRV=HELLOWORLD.foo.example.com:TOTAL-HUB=14.439|SMR=0.279(SMRDE=0.174|SMRC=0.105(SMRCL=0.105|X-SMRCR=0.016))|CAT=0.046(CATOS=0.018(CATSM=0.018(CATSM-Malware Agent=0.017))|CATRESL=0.003|CATORES=0.021(CATRS=0.021(CATRS-Prioritization Agent=0.002|CATRS-Index Routing Agent=0.017))|CATORT=0.002(CATRT=0.002(CATRT-CodeTwo Exchange Rules Transport Agent=0.001)))|QDE=13.818|SMS=0.291,Incoming,,,,S:E2ELatency=14.513;S:ExternalSendLatency=0.402;S:ToEntity=Internet;S:FromEntity=Internet;S:MsgRecipCount=1;S:IncludeInSla=True;S:Microsoft.Exchange.Transport.MailRecipient.RequiredTlsAuthLevel=DomainValidation;S:Microsoft.Exchange.Transport.MailRecipient.EffectiveTlsAuthLevel=DomainValidation;S:IsSmtpResponseFromExternalServer=True;S:DeliveryPriority=Normal;S:OriginalFromAddress=<>;S:AccountForest=wgs.wuerth.com,Email,82f8feae-ef9b-41bf-8500-1e10946655ae,15.01.2507.039