Added retry mechanism for close contentReader in download.

[agrasth]

The changes introduce a refined retry mechanism for simulating and recovering from forced errors encountered during the cleanup of temporary files in the ContentReader's Close method.

Details:
Added Retry Mechanism with Forced Error Simulation:

• Introduced retryOperation function to encapsulate retry logic with a forced error simulation feature.
• The retryOperation function takes the number of attempts, a simulation flag, and the operation to be retried as parameters.
• The retry logic forces errors for the first maxForcedAttempts if the simulateError flag is set to true.
  Controlled Forced Error Simulation:
• Introduced global variables downloadAttempts and maxForcedAttempts to manage the number of simulated download attempts.

---

• All tests passed. If this feature is not already covered by the tests, I added new tests.
• All static analysis checks passed.
• This pull request is on the dev branch.
• I used gofmt for formatting the code before submitting the pull request.

---

[bhanurp]

Suggested change

	// Check if file exists before attempting to remove
	func (cr *ContentReader) Close() error {
	for _, filePath := range cr.filesPaths {
	if filePath == "" {
	continue
	}
	if err := removeFileWithRetry(filePath); err != nil {
	return fmt.Errorf("failed to close reader: %w", err)
	}
	}
	cr.filesPaths = nil
	return nil
	}
	func removeFileWithRetry(filePath string) error {
	// Check if file exists before attempting to remove
	if _, err := os.Stat(filePath); os.IsNotExist(err) {
	log.Debug("File does not exist: %s", filePath)
	return nil
	}
	log.Debug("Attempting to remove file: %s", filePath)
	executor := retryexecutor.RetryExecutor{
	Context: context.Background(),
	MaxRetries: 5,
	RetriesIntervalMilliSecs: 100,
	ErrorMessage: "Failed to remove file",
	LogMsgPrefix: "Attempting removal",
	ExecutionHandler: func() (bool, error) {
	return false, errorutils.CheckError(os.Remove(filePath))
	},
	}
	return executor.Execute()
	}

[fluxxBot]

why do we need this simulate error?
we can simply go ahead and retry

[fluxxBot]

IMO, we should create a general utility function for retry and pass a function along with comparables for the retry operation

[fluxxBot]

why are we referring download here, aren't you just trying to close a opened file?

[fluxxBot]

log.Warn("Retry attempt %d failed: %v", i+1, lastErr)
log.Warn("Retry attempt %d failed: %v", i+1, lastErr.Error())

[fluxxBot]

why is sleep required?

[fluxxBot]

I was able to see the same issue here, please check if this could be of any help.
cc: @agrasth @bhanurp