Logging/Alert System #1249
Replies: 19 comments
-
Identify potential logging options that could be deployed in existing infrastructure.Currently we use Azure Log Analytics to query logs in Azure Monitor(?). Feedback is that querying data is not very straightforward. How hard is it to integrate non-Azure things into this system, for example generic Linux logs in Alternatives worthconsidering, Find out if there is a way to automate system alerts at least partly, if not what would manual process look like?A fully automated system seems unlikely, how can we automatically determine what should raise concern? With one of the dashboard type solutions we could configure searches/dashboards to help filter useful information (e.g. authentication logs, antivirus logs). These should also let us create alerts for events which obviously merit investigation (e.g. antivirus positive results). |
Beta Was this translation helpful? Give feedback.
-
|
There is a question here of what needs to be logged. For example, we could (presumably) extract text and journald logs from the containers running internal webapps (GitLab, HackMD). I expect this would take more effort than getting similar data from the host OS on a VM and it feels like this information is less important. Due to the isolation model of the safe haven, if there is malicious activity on one of these webapps then the safe haven has already be compromised. (But maybe we should be more worried about people breaking out?) Similarly, I expect we don't want to keep logs of users command history. |
Beta Was this translation helpful? Give feedback.
-
Graylog
|
Beta Was this translation helpful? Give feedback.
-
|
I think Docker can log to |
Beta Was this translation helpful? Give feedback.
-
PlanTechnical
Policy template
|
Beta Was this translation helpful? Give feedback.
-
Current scope
Future work
|
Beta Was this translation helpful? Give feedback.
-
|
@JimMadge @jemrobinson Can someone spend a decent amount of time working with the Azure logging and log analytics / monitoring / reporting services to see if these can meet our needs? I feel we've not spent much serious effort on this yet. This feels like a good area to collaborate with closely with IT colleagues on, as they may well already be using some of these services to monitor our enterprise systems on Azure. |
Beta Was this translation helpful? Give feedback.
-
|
In the GrayLog context, can we write to Azure storage / database with append only rights and access these for analysis / reporting / monitoring / alerting with read only rights? |
Beta Was this translation helpful? Give feedback.
-
|
Looks like we can configure custom logs via Powershell for the Azure log analytics workspace. |
Beta Was this translation helpful? Give feedback.
-
|
@martintoreilly There are certainly some logs (especially those to do with the Azure fabric/infrastructure) that are difficult to capture except with LogAnalytics. However, the main advantage of GrayLog is that it's not dependent on Azure which makes it easier to port to AWS (which is important for several potential endusers of this setup). It looks like we should be able to forward Azure Monitor Logs to GrayLog using an Azure function. If this works then I'm reasonably comfortable in using GrayLog for log aggregation/alerting and possibly providing a useful dashboard which can ingest data:
This seems like the easiest way to plan for a multi-platform future. |
Beta Was this translation helpful? Give feedback.
-
|
@jemrobinson Ok. I totally get the cross-platform argument, especially if it's also less / no more work that getting the Azure-only option working. My main concern is that ability to delete / edit logs is as restricted as possible, ideally to no more than the group who can deploy / delete infrastructure. Can we configure the underlying datastore for GrayLog so that only infrastructure admins can edit / delete log data and within-safe-haven-admins, local root on the Gray Log box or local root on any box contributing logging data to Gray Log can't? |
Beta Was this translation helpful? Give feedback.
-
Azure Monitor & Azure SentinelAzure Monitor
Sentinel
|
Beta Was this translation helpful? Give feedback.
-
|
Setup_SRE_Logging.ps1 also used to configure the collection of various Windows and Linux performance counters and activation of Log Analytics insight features, but these seem to be missing from master. I can find them in commit 76127bd @jemrobinson @JimMadge Either of you know where we lost this? Edit: It looks like it didn't actually make it into the original PR #743 merge where the changes were made, though I'm struggling to figure out why. The lines are definitely present in commit 198d960 via the "commits" tab of PR #743, but don't appear at all in the history of Setup_SRE_Logging.ps1 (going back from commit 98e1790) via the "files" tab of PR #743. |
Beta Was this translation helpful? Give feedback.
-
|
@martintoreilly I think this is in Setup_SHM_Logging.ps1 |
Beta Was this translation helpful? Give feedback.
-
Yep. There it is. Thanks. I thought I was going mad. My eyes weren't working and I didn't spot that it was the SHM logging setup file rather than the SRE one in commit 198d960. |
Beta Was this translation helpful? Give feedback.
-
|
@JimMadge Here are some Microsoft docs that might be useful: how to bring syslog data into LogAnalytics and overview of Azure Monitor agents. |
Beta Was this translation helpful? Give feedback.
-
Ingesting Azure Monitor logs into GraylogThe scheme set out by the Forward Azure Monitor Logs to Syslog action looks fairly complicated. It involves,
|
Beta Was this translation helpful? Give feedback.
-
Steps required to ingest syslog from Linux VMs
Rough powershell, $ResourceGroup = ""
$WorkspaceName = ""
$WorkspaceId = ""
$WorkspaceKey = ""
$VMName = ""
$Location = ""
$FacilityNames = @()
$FacilityNames += 'auth'
$FacilityNames += 'authpriv'
$FacilityNames += 'cron'
$FacilityNames += 'daemon'
$FacilityNames += 'ftp'
$FacilityNames += 'kern'
$FacilityNames += 'mail'
$FacilityNames += 'syslog'
$FacilityNames += 'user'
$FacilityNames += 'uucp'
$Count = 0
foreach ($FacilityName in $FacilityNames) {
$Count++
$null = New-AzOperationalInsightsLinuxSyslogDataSource `
-ResourceGroupName $ResourceGroupName `
-WorkspaceName $WorkspaceName `
-Name "Linux-syslog-$($Count)" `
-Facility $FacilityName `
-CollectEmergency `
-CollectAlert `
-CollectCritical `
-CollectError `
-CollectWarning `
-CollectNotice `
-CollectDebug `
-CollectInformational
}
$Settings = @{
"workspaceId" = $WorkspaceId
}
$ProtectedSettings = @{
"workspaceKey" = $WorkspaceKey
}
Set-AzVMExtension -Name OMSLinux -ExtensionType OmsAgentForLinux -Publisher Microsoft.EnterpriseCloud.Monitoring -TypeHandlerVersion "1.13" -ResourceGroupName $ResourceGroup -VMName $VMName -Location $Location -Settings $Settings -ProtectedSettings $ProtectedSettings |
Beta Was this translation helpful? Give feedback.
-
Azure monitor private endpointThere's an Azure Monitor Private Endpoint that ensures two-way restriction of DSH / Monitor instance (i.e. our DSH can only connect to our Monitor instance and vice-versa) and that traffic to the various Azure monitor service endpoints runs over the Azure internal network. Credit: Microsoft's Azure TRE design uses this. |
Beta Was this translation helpful? Give feedback.
-
📜 Description
Linked to issue alan-turing-institute/data-safe-haven-production#40 as part of requirements to have a system
requirements don't specify that for either of these it has to be a technical solution, but, was discussed previously that there was a desire for a technical logging solution by the team so think it's worth looking into. could be a combination of technical and manual processes for March (especially for the 'alert' part of requirement).
🍓 Desired behaviour
To have an (ideally) technical solution for logging and alerts.
📄 Tasks
Include specific tasks (if any) in the order in which they need to be done.
Beta Was this translation helpful? Give feedback.
All reactions