An invalid host.json file in combination with a HostInitializationError can cause an endless ScriptHost restart loop

[cjaliaga]

An invalid host.json file in combination with a HostInitializationError can cause an endless ScriptHost restart loop.

We discovered this issue while addressing misconfigurations in applications during a recent release. #10439 reintroduced temporary support for the previous behavior, which reduces the frequency of this issue but doesn't fully resolve it. As this fallback is only temporary and will be removed, it is essential to address the root cause before the fallback behavior is disabled.

Conditions to reproduce

• Invalid host.json, for example a host.json file missing the version property:
  "version": "2.0",
• Error initializing the Host. For example a situation that throws 'Did not find functions with language [dotnet-isolated].'

WebJobsScriptHostService.UnsynchronizedStartHostAsync attempts:

1. First attempt, on this one we try to load host.json as usual

startupMode = JobHostStartupMode.Normal
skipHostJsonConfiguration = false

Since the host.json is invalid, we will throw a HostConfigurationException trying to build the host:

azure-functions-host/src/WebJobs.Script.WebHost/WebJobsScriptHostService.cs

Lines 338 to 347 in 6bccacf

	// If we're in a non-transient error state or offline, skip host initialization
	bool skipJobHostStartup = isOffline || hasNonTransientErrors;
	bool skipHostJsonConfiguration = startupMode == JobHostStartupMode.HandlingConfigurationParsingError;
	string functionsExtensionVersion = _environment.GetFunctionsExtensionVersion();
	_logger.Building(functionsExtensionVersion, skipJobHostStartup, skipHostJsonConfiguration, activeOperation.Id);
	using (_metricsLogger.LatencyEvent(MetricEventNames.ScriptHostManagerBuildScriptHost))
	{
	localHost = BuildHost(skipJobHostStartup, skipHostJsonConfiguration);
	}

2. On this next attempt, we will be handling the Host Configuration Error, skipping the step of parsing the host.json:

startupMode = JobHostStartupMode.HandlingConfigurationParsingError
skipJobHostStartup = false
skipHostJsonConfiguration = true 

We will be able to build the host:

azure-functions-host/src/WebJobs.Script.WebHost/WebJobsScriptHostService.cs

Lines 338 to 347 in 6bccacf

	// If we're in a non-transient error state or offline, skip host initialization
	bool skipJobHostStartup = isOffline || hasNonTransientErrors;
	bool skipHostJsonConfiguration = startupMode == JobHostStartupMode.HandlingConfigurationParsingError;
	string functionsExtensionVersion = _environment.GetFunctionsExtensionVersion();
	_logger.Building(functionsExtensionVersion, skipJobHostStartup, skipHostJsonConfiguration, activeOperation.Id);
	using (_metricsLogger.LatencyEvent(MetricEventNames.ScriptHostManagerBuildScriptHost))
	{
	localHost = BuildHost(skipJobHostStartup, skipHostJsonConfiguration);
	}

However, we won't be able to start the host and we will throw a HostInitializationException here:

azure-functions-host/src/WebJobs.Script.WebHost/WebJobsScriptHostService.cs

Line 382 in 6bccacf

await localHost.StartAsync(currentCancellationToken);

3. On this attempt we're handling an initialization error:

startupMode = JobHostStartupMode.HandlingInitializationError
skipJobHostStartup = true
skipHostJsonConfiguration = false

Now we will skip job host startup, but since the host.json is still invalid, we will throw a HostConfigurationException again trying to build the host:

azure-functions-host/src/WebJobs.Script.WebHost/WebJobsScriptHostService.cs

Lines 338 to 347 in 6bccacf

	// If we're in a non-transient error state or offline, skip host initialization
	bool skipJobHostStartup = isOffline || hasNonTransientErrors;
	bool skipHostJsonConfiguration = startupMode == JobHostStartupMode.HandlingConfigurationParsingError;
	string functionsExtensionVersion = _environment.GetFunctionsExtensionVersion();
	_logger.Building(functionsExtensionVersion, skipJobHostStartup, skipHostJsonConfiguration, activeOperation.Id);
	using (_metricsLogger.LatencyEvent(MetricEventNames.ScriptHostManagerBuildScriptHost))
	{
	localHost = BuildHost(skipJobHostStartup, skipHostJsonConfiguration);
	}

4. This attempt will be exactly as attempt 2
5. This attempt will be exactly as attempt 3