Add Support for RRULE (iCalendar Recurrence) in Hangfire

[hasansustcse13]

I'm proposing the addition of RRULE support in Hangfire, similar to the recurrence rules used in Google Calendar's ICS files. Currently, Hangfire relies on Cron expressions for scheduling, but they fall short in scenarios where precise recurrence is needed, as defined by RRULE.

Problem:
Consider the following examples:

FREQ=MINUTELY;INTERVAL=7 => Cron: */7 * * * *
FREQ=HOURLY;INTERVAL=7 => Cron: 5 */7 * * *
FREQ=DAILY;INTERVAL=7 => Cron: 5 3 */7 * *
FREQ=WEEKLY;INTERVAL=2;BYDAY=TH,SA;UNTIL=20240831T175959Z
FREQ=MONTHLY;INTERVAL=2;BYMONTHDAY=19;COUNT=3
FREQ=YEARLY;INTERVAL=2;BYYEARDAY=228

For example, the Cron expression for FREQ=MINUTELY;INTERVAL=7 is */7 * * * *, which triggers at the 0th, 7th, 14th, 21st, etc., minute within each hour. However, this does not align with the intended behavior of running a job every 7 minutes from a specific start time.

Let's say the first job runs at 00:07 (7th minute). The next expected run time should be 00:14, followed by 00:21, and so on. The issue arises at the 56th minute. After 00:56, the Cron expression */7 * * * * schedules the next run at 01:00, which is incorrect according to the RRULE. The correct next run time should be 01:03 (7 minutes after 00:56), as the job should run every 7 minutes continuously, not reset at the start of the hour.

This problem exists across all types of frequencies where Cron expressions are used, leading to misaligned scheduling when intervals span across hours, days, or other time units.

Proposed Solutions:

Sol 1 - Recurring Job with Frequent (Every minute, hour, daily, monthly based on the FREQ) Trigger:

Set up a recurring job to trigger every minute (* * * * *) or hour ( 5 * * * *), depending on the frequency.
Use a library like ICal.NET to check whether the current time matches the next occurrence based on the RRULE.

Problem: This solution can lead to a significant increase in server load, as the job triggers frequently and performs additional checks, which is inefficient.

Sol 2 - Dynamically Register Next Job:

Calculate the first occurrence of the job based on the RRULE and register a scheduled job.
When the job executes, calculate the next occurrence and register a new scheduled job accordingly.

Problem: While this reduces server load by only triggering jobs when needed, it introduces complexity. If the server fails to register the next job due to downtime or other issues, the recurrence chain could break. Although Hangfire preserves jobs if the server is temporarily down, there are still edge cases where failures could disrupt the job schedule.

Example Code:
Here's a basic example using ICal.NET to get the next occurrence:

string rrule = "FREQ=HOURLY;INTERVAL=7";
var pattern = new RecurrencePattern(rrule);
string start = "2024-08-23T06:00:00.000Z";
var startDate = DateTime.Parse(start, null, System.Globalization.DateTimeStyles.RoundtripKind);
var calendarEvent = new CalendarEvent
{
    Start = new CalDateTime(startDate),
    RecurrenceRules = new List<RecurrencePattern> { pattern },
};
var occurrences = calendarEvent.GetOccurrences(DateTime.UtcNow, DateTime.UtcNow.AddYears(1))
    .Select(x => x.Period.StartTime.AsUtc).ToList();

Expectation:
It would be highly beneficial for Hangfire to support RRULEs natively, as this is a common requirement for scheduling tasks, especially in applications that handle calendar-like events. Moreover, RRULEs can also include complex exception rules, which are difficult to manage with Cron alone.

Is anyone else encountering similar challenges? How have you addressed them? I'd love to hear your solutions or thoughts on this feature request.

[ShayMusachanov-dev]

Thank you for raising this important issue regarding RRULE support in Hangfire. This feature would be precious, and I'd like to add my support and perspective to this discussion.

The company I work for also desperately needs this feature. We're facing similar challenges with scheduling tasks requiring precise recurrence, which Cron expressions can only handle somewhat. Your examples clearly illustrate Cron's limitations when dealing with intervals that span time boundaries.

This feature is so critical that we would be willing to pay for it, even if it were only available as part of Hangfire Pro. Using RRULE for scheduling would significantly improve our application's functionality and reliability.

A native implementation within Hangfire would be the most robust and efficient solution.

As a side note, I'd like to share some experience regarding the libraries mentioned. While your example uses ICal.NET, I've found that the EWSoftware.PDI NuGet package offers better performance and an API that more closely aligns with the RFC regarding the terminology used.

To illustrate this, I've conducted a naive benchmark comparing EWSoftware.PDI and ICal.NET. While this is not an exhaustive test, it does demonstrate a significant performance difference:

[MemoryDiagnoser]
[Orderer(BenchmarkDotNet.Order.SummaryOrderPolicy.FastestToSlowest)]
[RankColumn]
public class RRuleBenchmarks
{
    private const string RRULE = "FREQ=HOURLY;INTERVAL=7";
    private const string Start = "2024-08-23T06:00:00.000Z";
    private static readonly DateTime StartDate = DateTime.Parse(Start, null, System.Globalization.DateTimeStyles.RoundtripKind);

    private CalendarEvent _ical = null!;
    private Recurrence _pdi = null!;

    [GlobalSetup]
    public void GlobalSetup()
    {
        _ical = new CalendarEvent
        {
            Start = new CalDateTime(StartDate),
            RecurrenceRules = new List<RecurrencePattern> { new(RRULE) },
        };

        _pdi = new Recurrence(RRULE)
        {
            StartDateTime = StartDate
        };
    }

    [Benchmark(Baseline = true)]
    public List<DateTime> Ical_Benchmark() => _ical
        .GetOccurrences(DateTime.UtcNow, DateTime.UtcNow.AddYears(1))
        .Select(x => x.Period.StartTime.AsUtc)
        .ToList();

    [Benchmark]
    public List<DateTime> PDI_Benchmark() => _pdi
        .InstancesBetween(DateTime.UtcNow, DateTime.UtcNow.AddYears(1))
        .ToList();
}

Method	Mean	Error	StdDev	Ratio	RatioSD	Rank	Gen0	Gen1	Allocated	Alloc Ratio
PDI_Benchmark	44.21 us	0.177 us	0.166 us	0.01	0.00	1	2.2583	0.1831	42.46 KB	0.005
Ical_Benchmark	3,207.52 us	63.675 us	65.390 us	1.00	0.03	2	441.4063	328.1250	8136.38 KB	1.000

The results show that the PDI library significantly outperforms ICal.NET in speed and memory usage, which could be crucial for applications dealing with many recurring events.

[hasansustcse13]

Thank you for the benchmark. I want to add one point I just give the example as demonstration purpose. For the the both solution the occurrence array length will be less than 2. So I think benchmark won't be a factor here. You need to handle the search end date based on your frequency. As I have mentioned solution 1 is straightforward only the drawback is hangfire server will be busy for every minute/hour interval which is not needed.

Despite some drawbacks, I have implemented two solutions based on the procedure of 2nd approach. I am middle of something hopefully I will share the code within a couple of days.

Also I want to know are there any other packages which support this kind of RRULE based feature (Even paid) for scheduling.

[ShayMusachanov-dev]

Thank you for your thoughtful response and for sharing your progress on implementing solutions. I appreciate your insights and would like to address a few points:

1. Regarding the benchmark relevance: You're right that the performance difference might be less significant for very short occurrence arrays. However, the EWSoftware.PDI library offers a NextInstance() method, which could be more convenient and efficient for determining the next occurrence, especially in scenarios where we don't need to calculate all occurrences simultaneously. This approach could be particularly beneficial in a scheduling context.

   I ran a benchmark comparing PDI's NextInstance() to ICal's GetOccurrences() (as I couldn't find a direct "next instance" method in ICal), and the performance difference is substantial:

   Method	Mean	Error	StdDev	Median	Ratio	RatioSD	Rank	Gen0	Gen1	Gen2	Allocated	Alloc Ratio
   PDI_Benchmark	783.5 ns	33.88 ns	97.76 ns	761.7 ns	0.000	0.00	1	0.1593	-	-	1000 B	0.000
   ICal_Benchmark	10,670,514.4 ns	631,175.67 ns	1,790,540.45 ns	10,006,406.2 ns	1.025	0.23	2	1328.1250	984.3750	93.7500	8298974 B	1.000

   As you can see from the benchmark results, PDI's NextInstance() method is not only significantly faster but also uses much less memory compared to ICal's 'GetOccurrences () '. This clear advantage of PDI's method underscores its superiority and potential for enhancing our scheduling tasks.

2. About the library's performance: While the benchmark might not be the deciding factor for small sets of occurrences, the overall performance and memory efficiency of EWSoftware.PDI could still be advantageous, especially as the complexity of recurrence rules or the number of scheduled jobs increases. It would be worth considering unless there are specific reasons not to use it.

3. Regarding other RRULE-capable scheduling packages: To my knowledge, no mainstream .NET scheduling libraries natively support RRULE. Quartz has a pull request to add such support, but it hasn't been merged yet.

4. I'm very interested in seeing the code for your solutions based on the second approach. When you share it, could you elaborate on how these solutions handle job grouping? Specifically, I'm curious if previous executions of a recurring job will be grouped or if Hangfire will treat each occurrence as a separate job internally. This could have implications for job management and historical tracking.

Your work on this is greatly appreciated, and I'm looking forward to seeing your implementation. The ability to handle RRULE natively in Hangfire would be a significant improvement for many users, including myself and my company.

[hasansustcse13]

You can use Ical.NET or PDI based on your requirement. But I see Ical.NET supports iCalendar (RFC 5545) where PDI doesn't mention about it. And according to chat-gpt 5545 is the latest version of this kind of event. I don't know details about this RFC.

The executions of the recurring job won't be group. The last job will register next job so hang-fire treat the job separately.

[hasansustcse13]

Solution 1 based Dynamically Register Next Job: (Complete Solution)

Interface

using scheduler.Model;
using System.Net.Http;
using System.Threading.Tasks;

namespace scheduler.Interface
{
    public interface IReminderEventTriggerService
    {
        Task<HttpResponseMessage> TriggerAsync(string reminderName, ReminderEventModel reminderEvent);
    }
}

using scheduler.Model;
using System.Net.Http;
using System.Threading.Tasks;

namespace scheduler.Interface
{
    public interface IReminderEventTriggerService
    {
        Task<HttpResponseMessage> TriggerAsync(string reminderName, ReminderEventModel reminderEvent);
    }
}

Service:

using CloudApper.Common.Models;
using CloudApper.Enumerations;
using Hangfire;
using Hangfire.Storage;
using scheduler.Common;
using scheduler.Extensions;
using scheduler.Interface;
using scheduler.Model;
using Scheduler.Common;
using Serilog;
using System;
using System.Collections.Generic;
using System.Net;
using System.Net.Http;

namespace scheduler.Service
{
    public class ReminderEventService : IReminderEventService
    {
        private readonly IBackgroundJobClient _backgroundJob;
        private readonly JobStorage _storage;

        public ReminderEventService(IBackgroundJobClient backgroundJob)
        {
            _backgroundJob = backgroundJob;
            _storage = JobStorage.Current ?? throw new ArgumentNullException("storage");
        }

        public ResponseMessage<string> AddReminderEvent(ReminderEventModel reminderEvent)
        {
            var nextOccurrenceResponse = GetNextOccurrenceAfterValidation(reminderEvent);
            if (!nextOccurrenceResponse.Success)
            {
                return ResponseMsg<string>.GetResponseMessage(nextOccurrenceResponse.Message, nextOccurrenceResponse.MessageType, nextOccurrenceResponse.ResponseCode, null);
            }

            return RetryHelper.RetryOnException(_ => Create(reminderEvent, nextOccurrenceResponse.Result.Value));
        }

        public ResponseMessage<string> UpdateReminderEvent(string id, ReminderEventModel reminderEvent)
        {
            reminderEvent.Id = id;
            var nextOccurrenceResponse = GetNextOccurrenceAfterValidation(reminderEvent);
            if (!nextOccurrenceResponse.Success)
            {
                return ResponseMsg<string>.GetResponseMessage(nextOccurrenceResponse.Message, nextOccurrenceResponse.MessageType, nextOccurrenceResponse.ResponseCode, null);
            }

            // Delete after the creation successful because if the delete fail but creation success we can delete the old when trigger Make HTTP call via callback API and In that case DB won't rollback
            // And callback API delete the old by providing the id and state in DeleteReminderEventByState API. That's why we are passing the state as query param in callback API to check the state with our main database
            // In this case there are multiple schedule registered but it will deleted eventually
            var oldScheduleResponse = RetryHelper.RetryOnException(_ => GetActiveSchedulesById(id));
            if (!oldScheduleResponse.Success)
                return ResponseMsg<string>.GetResponseMessage(oldScheduleResponse.Message, oldScheduleResponse.MessageType, oldScheduleResponse.ResponseCode, null);

            var createResponse = RetryHelper.RetryOnException(_ => Create(reminderEvent, nextOccurrenceResponse.Result.Value));
            if (createResponse.Success)
            {
                RetryHelper.RetryOnException(_ => Delete(oldScheduleResponse.Result));
            }

            return createResponse;
        }

        public ResponseMessage<bool> DeleteReminderEventById(string id)
        {
            var oldScheduleResponse = RetryHelper.RetryOnException(_ => GetActiveSchedulesById(id));
            if (!oldScheduleResponse.Success)
                return ResponseMsg<bool>.GetResponseMessage(oldScheduleResponse.Message, oldScheduleResponse.MessageType, oldScheduleResponse.ResponseCode, false);
            return RetryHelper.RetryOnException(_ => Delete(oldScheduleResponse.Result));
        }

        public ResponseMessage<bool> DeleteReminderEventByState(string id, long state)
        {
            var oldScheduleResponse = RetryHelper.RetryOnException(_ => GetActiveSchedulesByState(id, state));
            if (!oldScheduleResponse.Success)
                return ResponseMsg<bool>.GetResponseMessage(oldScheduleResponse.Message, oldScheduleResponse.MessageType, oldScheduleResponse.ResponseCode, false);
            return RetryHelper.RetryOnException(_ => Delete(oldScheduleResponse.Result));
        }

        public ResponseMessage<List<DateTime>> GetOccurrencesBasedOnRule(string recurrentRule, DateTime startDateTime, int count = 10)
        {
            List<DateTime> result = new();
            if (!CalDevUtility.IsValidRecurrentRule(recurrentRule))
                return ResponseMsg<List<DateTime>>.GetResponseMessage($"The rule format is invalid.", EnumMessageType.Error, (int)HttpStatusCode.BadRequest, result);
            if (startDateTime == DateTime.MinValue || startDateTime.Kind != DateTimeKind.Utc)
                return ResponseMsg<List<DateTime>>.GetResponseMessage($"The startDateTime must be a valid UTC date.", EnumMessageType.Error, (int)HttpStatusCode.BadRequest, result);
            if (count < 1 || count > 100)
                return ResponseMsg<List<DateTime>>.GetResponseMessage($"The minimum count is 1 and max count is 100.", EnumMessageType.Error, (int)HttpStatusCode.BadRequest, result);

            var nextOccurrence = CalDevUtility.GetNextOccurrence(startDateTime, recurrentRule);
            if (!nextOccurrence.HasValue)
                return ResponseMsg<List<DateTime>>.GetResponseMessage("Unable to determine the next occurrence based on the provided data.", EnumMessageType.Error, (int)HttpStatusCode.BadRequest, result);
            result.Add(nextOccurrence.Value);

            while (count > 1 && nextOccurrence.HasValue)
            {
                nextOccurrence = CalDevUtility.GetNextOccurrence(startDateTime, recurrentRule, nextOccurrence.Value);
                count--;
                if (nextOccurrence.HasValue)
                    result.Add(nextOccurrence.Value);
            }
            return ResponseMsg<List<DateTime>>.GetResponseMessage(Constants.OPERATION_SUCCESSFUL, EnumMessageType.Success, (int)HttpStatusCode.OK, result);
        }

        private ResponseMessage<string> Create(ReminderEventModel reminderEvent, DateTime enqueueAt)
        {
            try
            {
                string createdJobId = _backgroundJob.Schedule<IReminderEventTriggerService>(rEvent => rEvent.TriggerAsync(reminderEvent.Name, reminderEvent), new DateTimeOffset(enqueueAt));
                return ResponseMsg<string>.GetResponseMessage(Constants.OPERATION_SUCCESSFUL, EnumMessageType.Success, (int)HttpStatusCode.OK, createdJobId);
            }
            catch (Exception ex)
            {
                Log.Error(ex, $"Reminder event creation failed: {ex.Message}");
                return ResponseMsg<string>.GetResponseMessage(Constants.OPERATION_FAILED, EnumMessageType.Error, (int)HttpStatusCode.InternalServerError, null);
            }
        }

        private ResponseMessage<bool> Delete(HashSet<string> scheduleIds)
        {
            try
            {
                bool success = true;
                foreach (var scheduleId in scheduleIds)
                {
                    success = success && _backgroundJob.Delete(scheduleId);
                }
                if (success)
                    return ResponseMsg<bool>.GetResponseMessage(Constants.OPERATION_SUCCESSFUL, EnumMessageType.Success, (int)HttpStatusCode.OK, true);
                return ResponseMsg<bool>.GetResponseMessage(Constants.OPERATION_FAILED, EnumMessageType.Error, (int)HttpStatusCode.InternalServerError, false);
            }
            catch (Exception ex)
            {
                Log.Error(ex, $"Reminder event deletion failed: {ex.Message}");
                return ResponseMsg<bool>.GetResponseMessage(Constants.OPERATION_FAILED, EnumMessageType.Error, (int)HttpStatusCode.InternalServerError, false);
            }
        }

        private ResponseMessage<HashSet<string>> GetActiveSchedulesById(string id)
        {
            using IStorageConnection storageConnection = _storage.GetConnection();
            var scheduleIds = storageConnection.GetActiveSchedulesById(id);
            return ResponseMsg<HashSet<string>>.GetResponseMessage(Constants.OPERATION_SUCCESSFUL, EnumMessageType.Success, (int)HttpStatusCode.OK, scheduleIds);
        }

        private ResponseMessage<HashSet<string>> GetActiveSchedulesByState(string id, long state)
        {
            using IStorageConnection storageConnection = _storage.GetConnection();
            var scheduleIds = storageConnection.GetActiveSchedulesByState(id, state);
            return ResponseMsg<HashSet<string>>.GetResponseMessage(Constants.OPERATION_SUCCESSFUL, EnumMessageType.Success, (int)HttpStatusCode.OK, scheduleIds);
        }

        private static ResponseMessage<DateTime?> GetNextOccurrenceAfterValidation(ReminderEventModel reminderEvent)
        {
            if (string.IsNullOrEmpty(reminderEvent.Id))
                return ResponseMsg<DateTime?>.GetResponseMessage(string.Format(Constants.REQUIRED_MESSAGE, nameof(ReminderEventModel.Id)), EnumMessageType.Error, (int)HttpStatusCode.BadRequest, null);
            if (string.IsNullOrEmpty(reminderEvent.Name))
                return ResponseMsg<DateTime?>.GetResponseMessage(string.Format(Constants.REQUIRED_MESSAGE, nameof(ReminderEventModel.Name)), EnumMessageType.Error, (int)HttpStatusCode.BadRequest, null);
            if (reminderEvent.State < 1 || reminderEvent.State.ToString().Length < 10)
                return ResponseMsg<DateTime?>.GetResponseMessage($"'{nameof(ReminderEventModel.State)}' must be a positive number with at least 10 digits.", EnumMessageType.Error, (int)HttpStatusCode.BadRequest, null);
            if (reminderEvent.StartDateTime == DateTime.MinValue || reminderEvent.StartDateTime.Kind != DateTimeKind.Utc)
                return ResponseMsg<DateTime?>.GetResponseMessage($"'{nameof(ReminderEventModel.StartDateTime)}' must be a valid UTC date.", EnumMessageType.Error, (int)HttpStatusCode.BadRequest, null);
            if (!string.IsNullOrEmpty(reminderEvent.RecurrentRule) && !CalDevUtility.IsValidRecurrentRule(reminderEvent.RecurrentRule))
                return ResponseMsg<DateTime?>.GetResponseMessage($"The '{nameof(ReminderEventModel.RecurrentRule)}' format is invalid.", EnumMessageType.Error, (int)HttpStatusCode.BadRequest, null);

            if (reminderEvent.ReminderCallbackInfo == null)
                return ResponseMsg<DateTime?>.GetResponseMessage(string.Format(Constants.REQUIRED_MESSAGE, nameof(ReminderEventModel.ReminderCallbackInfo)), EnumMessageType.Error, (int)HttpStatusCode.BadRequest, null);
            List<HttpMethod> supportedHttpVerbs = new() { HttpMethod.Get, HttpMethod.Post, HttpMethod.Put, HttpMethod.Delete };
            if (string.IsNullOrEmpty(reminderEvent.ReminderCallbackInfo.HttpMethod) || !supportedHttpVerbs.Contains(new HttpMethod(reminderEvent.ReminderCallbackInfo.HttpMethod)))
                return ResponseMsg<DateTime?>.GetResponseMessage($"'{nameof(ReminderCallbackInfoModel.HttpMethod)}' should be one of the following: {string.Join(", ", supportedHttpVerbs)}.", EnumMessageType.Error, (int)HttpStatusCode.BadRequest, null);
            if (string.IsNullOrEmpty(reminderEvent.ReminderCallbackInfo.Path))
                return ResponseMsg<DateTime?>.GetResponseMessage(string.Format(Constants.REQUIRED_MESSAGE, nameof(ReminderCallbackInfoModel.Path)), EnumMessageType.Error, (int)HttpStatusCode.BadRequest, null);
            try
            {
                _ = new Uri(reminderEvent.ReminderCallbackInfo.BaseAddress);
            }
            catch (Exception)
            {
                return ResponseMsg<DateTime?>.GetResponseMessage($"The '{nameof(ReminderCallbackInfoModel.BaseAddress)}' is not a valid URL.", EnumMessageType.Error, (int)HttpStatusCode.BadRequest, null);
            }


            var nextOccurrence = CalDevUtility.GetNextOccurrence(reminderEvent.StartDateTime, reminderEvent.RecurrentRule);
            if (!nextOccurrence.HasValue)
                return ResponseMsg<DateTime?>.GetResponseMessage("Unable to determine the next occurrence based on the provided data.", EnumMessageType.Error, (int)HttpStatusCode.BadRequest, null);

            return ResponseMsg<DateTime?>.GetResponseMessage("Model validation successful. Next occurrence calculated.", EnumMessageType.Success, (int)HttpStatusCode.OK, nextOccurrence);
        }
    }
}

using CloudApper.Common.Models;
using CloudApper.Enumerations;
using Hangfire;
using Newtonsoft.Json;
using scheduler.Common;
using scheduler.Interface;
using scheduler.Model;
using Scheduler.Common;
using Serilog;
using System;
using System.Linq;
using System.Net;
using System.Net.Http;
using System.Text;
using System.Threading.Tasks;
using System.Web;

namespace scheduler.Service
{
    public class ReminderEventTriggerService : IReminderEventTriggerService
    {
        private readonly IHttpClientFactory _httpClientFactory;
        private readonly IBackgroundJobClient _backgroundJob;

        public ReminderEventTriggerService(IHttpClientFactory httpClientFactory, IBackgroundJobClient backgroundJob)
        {
            _httpClientFactory = httpClientFactory;
            _backgroundJob = backgroundJob;
        }

        public async Task<HttpResponseMessage> TriggerAsync(string jobName, ReminderEventModel reminderEvent)
        {
            Console.WriteLine($"--------------{reminderEvent.State} | Trigger------------ {DateTime.UtcNow}");
            DateTime? nextTrigger = CalDevUtility.GetNextOccurrence(reminderEvent.StartDateTime, reminderEvent.RecurrentRule);
            if (nextTrigger.HasValue)
            {
                // First register the next occurrence for recurring event
                // Raise exception if registration failed without make the API call as Hang-fire will try again for the same event
                // By raising exception here we can avoid chain break as much as possible
                var result = RetryHelper.RetryOnException(_ => RegisterNextEvent(reminderEvent, nextTrigger.Value));
                if (!result.Success)
                {
                    Log.Warning($"Next Event Registration Failed - Id: {reminderEvent.Id} State: {reminderEvent.State} Current Time: {DateTime.UtcNow} Next Run Time: {nextTrigger.Value}");
                    throw new InvalidOperationException("Next Event Creation Failed on Current Trigger");
                }
                Log.Information($"Next Event Registration Successful - Id: {reminderEvent.Id} State: {reminderEvent.State} Current Time: {DateTime.UtcNow} Next Run Time: {nextTrigger.Value}");
                Console.WriteLine($"--------------------------------------------------------------------------------------------------");
                Console.WriteLine($"{reminderEvent.State} |  Current: {DateTime.UtcNow} Next:{nextTrigger.Value}");
                Console.WriteLine($"--------------------------------------------------------------------------------------------------");
            }

            Console.WriteLine($"-----------------------API Call-------------- {DateTime.UtcNow}");
            if (!CalDevUtility.HasInRuleExceptionDate(reminderEvent.RecurrentRuleExceptions))
            {
                // Fire and forget and avoid any exception here because if there is an error Hang-fire try again which may register duplicate event
                _ = MakeHttpRequestAsync(reminderEvent);
            }
            Console.WriteLine($"-----------------------Success-------------- {DateTime.UtcNow}");
            Console.WriteLine(string.Concat(Enumerable.Repeat(Environment.NewLine, 3)));
            // Return dummy OK response to make the job succeeded state by Hang-fire
            return new HttpResponseMessage(HttpStatusCode.OK);
        }

        private ResponseMessage<string> RegisterNextEvent(ReminderEventModel reminderEvent, DateTime enqueueAt)
        {
            try
            {
                string createdJobId = _backgroundJob.Schedule<IReminderEventTriggerService>(rEvent => rEvent.TriggerAsync(reminderEvent.Name, reminderEvent), new DateTimeOffset(enqueueAt));
                return ResponseMsg<string>.GetResponseMessage(Constants.OPERATION_SUCCESSFUL, EnumMessageType.Success, (int)HttpStatusCode.OK, createdJobId);
            }
            catch (Exception ex)
            {
                Log.Error(ex, $"Reminder next event creation failed on trigger: {ex.Message}");
                return ResponseMsg<string>.GetResponseMessage(Constants.OPERATION_FAILED, EnumMessageType.Error, (int)HttpStatusCode.InternalServerError, null);
            }
        }

        private async Task<ResponseMessage<bool>> MakeHttpRequestAsync(ReminderEventModel reminderEvent)
        {
            try
            {
                HttpClient client = _httpClientFactory.CreateClient();
                client.Timeout = TimeSpan.FromMinutes(30);

                var uri = new Uri(new Uri(reminderEvent.ReminderCallbackInfo.BaseAddress), reminderEvent.ReminderCallbackInfo.Path);
                var uriBuilder = new UriBuilder(uri);
                var query = HttpUtility.ParseQueryString(uriBuilder.Query);
                // Pass the id & state as query params (not all API need these) to check Hang-fire is align with primary database if not delete by state API
               query["state"] = reminderEvent.State.ToString();
               query["id"] = reminderEvent.Id.ToString();
                uriBuilder.Query = query.ToString();
                Uri requestUri = uriBuilder.Uri;

                HttpRequestMessage request = new()
                {
                    RequestUri = requestUri,
                    Method = new(reminderEvent.ReminderCallbackInfo.HttpMethod),
                    Content = reminderEvent.ReminderCallbackInfo.Payload == null ? null : new StringContent(JsonConvert.SerializeObject(reminderEvent.ReminderCallbackInfo.Payload), Encoding.UTF8, "application/json"),
                };
                HttpResponseMessage response = await client.SendAsync(request);

                if (!response.IsSuccessStatusCode)
                {
                    Log.Error($"Failed to trigger Url:{requestUri} HttpStatusCode:{response.StatusCode} ReasonPhrase: {response.ReasonPhrase} EventName:{reminderEvent.Name} Id:{reminderEvent.Id} State: {reminderEvent.State}");
                    return ResponseMsg<bool>.GetResponseMessage(Constants.OPERATION_FAILED, EnumMessageType.Error, (int)response.StatusCode, false);
                }
                return ResponseMsg<bool>.GetResponseMessage(Constants.OPERATION_SUCCESSFUL, EnumMessageType.Success, (int)HttpStatusCode.OK, true);
            }
            catch (Exception ex)
            {
                Log.Error(ex, $"Failed to trigger Event Name:{reminderEvent.Name} Id:{reminderEvent.Id} State: {reminderEvent.State} Message: {ex.Message}");
                return ResponseMsg<bool>.GetResponseMessage(Constants.OPERATION_FAILED, EnumMessageType.Error, (int)HttpStatusCode.InternalServerError, false);
            }
        }
    }
}

Model:


using System;

namespace scheduler.Model
{
    public class ReminderEventModel
    {
        public string Id { get; set; }
        public string Name { get; set; }
        // This State (DateTime Ticks For version management) will be saved in main database and use for version mismatch if same event register multiple times via update API
        public long State { get; set; }
        public DateTime StartDateTime { get; set; }
        public string RecurrentRule { get; set; }
        public string RecurrentRuleExceptions { get; set; }
        public ReminderCallbackInfoModel ReminderCallbackInfo { get; set; }
    }
}

Here Id will be unique GUID for each recurring event and State will be DateTime.UtcNow.Ticks. This State (aka Version) will be used for any kind of mismatch during delete or update event. Also this will be saved in the main/primary database (not Hang-Fire DB). Main DB means where you save your RRULE and other stuffs. Main DB table should have Id, name, State, RRULE, RecurrentRuleExceptions, StartDateTime columns etc

namespace scheduler.Model
{
    public class ReminderCallbackInfoModel
    {
        public string HttpMethod { get; set; }
        public string BaseAddress { get; set; }
        public string Path { get; set; }
        public dynamic Payload { get; set; }
    }
}

Custom Message Model:

public class ResponseMessage<T>
{
    private EnumMessageType _messageType;

    public T Result { get; set; }

    public EnumMessageType MessageType
    {
        get
        {
            return _messageType;
        }
        set
        {
            switch (value)
            {
                case EnumMessageType.Success:
                    Success = true;
                    break;
                case EnumMessageType.Warning:
                    Success = true;
                    break;
                case EnumMessageType.Error:
                    Success = false;
                    break;
            }

            _messageType = value;
        }
    }

    public bool Success { get; set; }

    public int ResponseCode { get; set; }

    public string Message { get; set; }
}

public static class ResponseMsg<T>
{
    public static ResponseMessage<T> GetResponseMessage(string msg, EnumMessageType messageType, int statusCode, T result)
    {
        return new ResponseMessage<T>
        {
            Message = msg,
            MessageType = messageType,
            ResponseCode = statusCode,
            Result = result
        };
    }
}

Extensions:

using System;

namespace scheduler.Extensions
{
    public static class DateTimeExtensions
    {
        public static DateTime TruncateToMinutes(this DateTime dateTime)
        {
            return new DateTime(dateTime.Year, dateTime.Month, dateTime.Day, dateTime.Hour, dateTime.Minute, 0, 0, dateTime.Kind);
        }
    }
}

using Dapper;
using Hangfire.PostgreSql;
using Hangfire.Storage;
using Npgsql;
using System;
using System.Collections.Generic;

namespace scheduler.Extensions
{
    public static class StorageConnectionExtensions
    {
        public static NpgsqlConnection GetNpgsqlConnection(this IStorageConnection storageConnection)
        {
            if (storageConnection is PostgreSqlConnection pgSqlConnection)
            {
                return pgSqlConnection.Connection;
            }
            throw new InvalidOperationException("The storage connection is not a PostgreSqlConnection.");
        }

        public static HashSet<string> GetActiveSchedulesById(this IStorageConnection storageConnection, string id)
        {
            var npgsqlConnection = storageConnection.GetNpgsqlConnection();
            string sql = @"SELECT id FROM hangfire.job WHERE (statename = 'Scheduled' OR statename = 'Enqueued') AND arguments LIKE @IdPattern;";
            var ids = new HashSet<string>(npgsqlConnection.Query<string>(sql, new { IdPattern = "%" + id + "%" }));
            return ids;
        }

        public static HashSet<string> GetActiveSchedulesByState(this IStorageConnection storageConnection, string id, long state)
        {
            var npgsqlConnection = storageConnection.GetNpgsqlConnection();
            string sql = @"SELECT id FROM hangfire.job WHERE (statename = 'Scheduled' OR statename = 'Enqueued') AND arguments LIKE @IdPattern AND arguments LIKE @StatePattern;";
            var ids = new HashSet<string>(npgsqlConnection.Query<string>(sql, new { IdPattern = "%" + id + "%", StatePattern = "%" + state + "%" }));
            return ids;
        }
    }
}

Note: Here i have used PostgresSQL that's why I have checked the current connection is PostgreSqlConnection . I see many blog where developer want to execute query in the Hangfire DB based on the same connection they can use this. I need to investigate lot's of time to get the connection object. Also you will see, in ReminderEventService I have access JobStorage.Current and then based on this get the IStorageConnection and then get the connection of DB. As per as I know Hangfire auto register it JobStorage as Singleton

Utility:

using Ical.Net;
using Ical.Net.CalendarComponents;
using Ical.Net.DataTypes;
using scheduler.Extensions;
using System;
using System.Collections.Generic;
using System.Globalization;
using System.Linq;

namespace scheduler.Common
{
    public static class CalDevUtility
    {
        public static bool IsValidRecurrentRule(string recurrentRule)
        {
            if (string.IsNullOrWhiteSpace(recurrentRule))
                return false;

            try
            {
                var rPattern = new RecurrencePattern(recurrentRule);
                return rPattern.Frequency != FrequencyType.None;
            }
            catch (Exception)
            {
                return false;
            }
        }


        /// <summary>
        /// Calculates the next occurrence of an event based on the provided recurrence rule and start date.
        /// Handles the recurrence rule's "Count" and "Until" properties automatically.
        /// </summary>
        /// <param name="startDateTime">The start date and time of the event in UTC.</param>
        /// <param name="recurrentRule">The recurrence rule (RRULE) defining the frequency and pattern of the event.</param>
        /// <param name="occurrenceTime">Default current UTC time, Optional param for calculate occurrence recursively</param>
        /// <returns>
        /// The next occurrence date and time in UTC:
        /// - If the current time is greater than or equal to the reminder start time (<param name="startDateTime">), returns the next occurrence after the current time. (e.g. - current time: 12:05 will return 12:06 for every minute interval)
        /// - If the current time is less than the reminder start time, returns the reminder start time.
        /// - Returns null if no valid next occurrence exists based on the recurrence rule.
        /// </returns>
        public static DateTime? GetNextOccurrence(DateTime startDateTime, string recurrentRule, DateTime? occurrenceTime = null)
        {
            var currentTime = (occurrenceTime ?? DateTime.UtcNow).TruncateToMinutes();
            var reminderStartTime = startDateTime.TruncateToMinutes();

            // If no rule provided then it's schedule job (Fire & Forget) and in that case reminderStartTime must be greater than current time
            if (!IsValidRecurrentRule(recurrentRule))
                return reminderStartTime > currentTime ? reminderStartTime : null;

            var rPattern = new RecurrencePattern(recurrentRule);

            // Handle weekly frequency: Adjust start time to match one of the rule's specified days of the week
            if (rPattern.Frequency == FrequencyType.Weekly && currentTime < reminderStartTime)
            {
                var dayofWeeks = rPattern.ByDay.Select(item => item.DayOfWeek);
                while (dayofWeeks.Any() && !dayofWeeks.Contains(reminderStartTime.DayOfWeek))
                {
                    reminderStartTime = reminderStartTime.AddDays(1);
                }
            }

            // Set up the calendar event with the adjusted start time and recurrence pattern
            var reminderEvent = new CalendarEvent
            {
                Start = new CalDateTime(reminderStartTime),
                RecurrenceRules = new List<RecurrencePattern> { rPattern },
            };

            // Take the max as future reminder event may register through API
            var searchStart = reminderStartTime > currentTime ? reminderStartTime : currentTime;
            var searchEnd = GetSearchEndDateTime(rPattern, searchStart);
            // Add -1 for to consider the event start time for future event
            var occurrences = reminderEvent.GetOccurrences(searchStart.AddSeconds(-1), searchEnd)
                   .Select(x => x.Period.StartTime.AsUtc).ToList();
            var nextOccurrence = occurrences.FirstOrDefault(dt => dt > currentTime);
            return nextOccurrence != DateTime.MinValue ? nextOccurrence : null;
        }

        private static DateTime GetSearchEndDateTime(RecurrencePattern rPattern, DateTime searchStart)
        {
            // For monthly and yearly frequency it is possible that next occurrence is in on February 29 (leap year) FREQ=MONTHLY;INTERVAL=12;BYMONTHDAY=29 | FREQ=YEARLY;INTERVAL=1;BYMONTHDAY=29 (2024-02-29T03:00:00.000Z) => 2028 -> 2032 -> 2036.
            // Also there are cases where user set frequency MONTHLY with any INTERVAL and BYMONTHDAY = 29 or 30 or 31 and not all month has this day, in those cases we need to skip the month but not the next month. FREQ=MONTHLY;INTERVAL=4;BYMONTHDAY (2024-02-29T03:00:00.000Z) => ... June -> October -> Feb (skip as no 29) -> June ...
            // That's why multiplication with 5. Instead of 4 (leap year) use 5 to avoid unexpected cases
            int interval = rPattern.Interval * 5;
            return rPattern.Frequency switch
            {
                FrequencyType.Minutely => searchStart.AddMinutes(interval),
                FrequencyType.Hourly => searchStart.AddHours(interval),
                FrequencyType.Daily => searchStart.AddDays(interval),
                FrequencyType.Weekly => searchStart.AddDays(7 * interval),
                FrequencyType.Monthly => searchStart.AddMonths(interval),
                FrequencyType.Yearly => searchStart.AddYears(interval),
                _ => throw new NotSupportedException($"Frequency {rPattern.Frequency} is not supported."),
            };
        }

        public static bool HasInRuleExceptionDate(string exceptionDates)
        {
            try
            {
                if (string.IsNullOrWhiteSpace(exceptionDates))
                    return false;

                var currentTime = DateTime.UtcNow;
                var exceptionList = exceptionDates.Split(',');

                return exceptionList.Any(dateStr =>
                    DateTime.TryParseExact(dateStr, "yyyyMMddTHHmmssZ", CultureInfo.InvariantCulture, DateTimeStyles.RoundtripKind, out DateTime exceptionDate)
                    && exceptionDate.Year == currentTime.Year
                    && exceptionDate.Month == currentTime.Month
                    && exceptionDate.Day == currentTime.Day
                    && exceptionDate.Hour == currentTime.Hour
                    && exceptionDate.Minute == currentTime.Minute
                );
            }
            catch (Exception)
            {
                return false;
            }
        }
    }
}



using CloudApper.Common.Models;
using Serilog;
using System.Threading;
using System;
using CloudApper.Enumerations;
using System.Net;
using Scheduler.Common;

namespace scheduler.Common
{
    public static class RetryHelper
    {
        public static ResponseMessage<T> RetryOnException<T>(
            Func<int, ResponseMessage<T>> action,
            int maxAttempts = 3,
            Func<int, TimeSpan> retryDelayFunc = null
        )
        {
            maxAttempts = maxAttempts < 1 ? 1 : maxAttempts;
            TimeSpan delay = TimeSpan.Zero;
            retryDelayFunc ??= attempt => TimeSpan.FromSeconds(attempt * 2);

            for (int attempt = 0; attempt < maxAttempts; attempt++)
            {
                try
                {
                    if (delay > TimeSpan.Zero)
                    {
                        Thread.Sleep(delay);
                    }

                    var result = action(attempt);

                    if (result.Success)
                    {
                        return result;
                    }
                    delay = retryDelayFunc.Invoke(attempt + 1);
                }
                catch (Exception ex)
                {
                    Log.Error(ex, $"Exception occurred on attempt {attempt + 1}: {ex.Message}");
                    delay = retryDelayFunc.Invoke(attempt + 1);
                }
            }

            return ResponseMsg<T>.GetResponseMessage(Constants.OPERATION_FAILED, EnumMessageType.Error, (int)HttpStatusCode.InternalServerError, default);
        }
    }
}

namespace Scheduler.Common
{
    public static class Constants
    {
        public const string API_VERSION_V3_0 = "3.0";
        public const string REQUIRED_MESSAGE = "The '{0}' field is required.";
        public const string OPERATION_SUCCESSFUL = "Operation Successful.";
        public const string OPERATION_FAILED = "Operation Failed.";
    }
}

Controller:

using Microsoft.AspNetCore.Mvc;
using scheduler.Interface;
using scheduler.Model;
using Scheduler.Common;
using System;

namespace scheduler.Controllers.v3
{
    [ApiController]
    [ApiVersion(Constants.API_VERSION_V3_0)]
    [Route("api/v{version:apiVersion}/[controller]")]
    public class ReminderEventsController : ControllerBase
    {
        private readonly IReminderEventService _reminderEventService;

        public ReminderEventsController(IReminderEventService reminderEventService)
        {
            _reminderEventService = reminderEventService;
        }

        /// <summary>
        /// Create reminder event based on the provided information
        /// </summary>
        /// <param name="reminderEvent"></param>
        /// <returns></returns>
        [HttpPost]
        public IActionResult AddReminderEvent(ReminderEventModel reminderEvent)
        {
            return Ok(_reminderEventService.AddReminderEvent(reminderEvent));
        }

        /// <summary>
        /// Update reminder event by deleting the old and register new one
        /// </summary>
        /// <param name="id"></param>
        /// <param name="reminderEvent"></param>
        /// <returns></returns>
        [HttpPut("{id}")]
        public IActionResult UpdateReminderEvent(string id, ReminderEventModel reminderEvent)
        {
            return Ok(_reminderEventService.UpdateReminderEvent(id, reminderEvent));
        }

        /// <summary>
        /// Delete reminder by reminder id
        /// </summary>
        /// <param name="id"></param>
        /// <returns></returns>
        [HttpDelete("{id}")]
        public IActionResult DeleteReminderEventById(string id)
        {
            return Ok(_reminderEventService.DeleteReminderEventById(id));
        }

        /// <summary>
        /// Delete reminder with id and state when there is a version mismatch with the main DB
        /// </summary>
        /// <param name="id"></param>
        /// <param name="state"></param>
        /// <returns></returns>
        [HttpDelete("{id}/state/{state}")]
        public IActionResult DeleteReminderEventByState(string id, long state)
        {
            return Ok(_reminderEventService.DeleteReminderEventByState(id, state));
        }

        /// <summary>
        /// Get subsequent occurrences based on the provide rule and date to test event
        /// </summary>
        /// <param name="rule"></param>
        /// <param name="startDateTime"></param>
        /// <param name="count"></param>
        /// <returns></returns>
        [HttpGet("occurrences")]
        public IActionResult GetOccurrencesBasedOnRule([FromQuery] string rule, [FromQuery] DateTime startDateTime, [FromQuery] int count = 10)
        {
            return Ok(_reminderEventService.GetOccurrencesBasedOnRule(rule, startDateTime, count));
        }
    }
}

Startup Registration:

services.AddSingleton<IReminderEventService, ReminderEventService>();
services.AddSingleton<IReminderEventTriggerService, ReminderEventTriggerService>();

Observations / Some Key Points:

1. During Update API, first I have stored the current job id of the provided reminder/event id. Then create a new job. If the creation failed I didn't loss the previous chain and you can rollback in the main database (not hangfire db). If the job creation successful but can't delete in that case I also assume it success. But you see in that case there will be two job registered for same reminder/event Id but there State are different. Here State (aka version) comes to play. When the old job trigger I have passed the state as query params in the callback api the API will check the the State and it won't match with current DB state and I will delete it using delete API by providing the Id and State. That's why there are two delete API.
2. According my understanding, There are 5 state of a Job Scheduled, Enqueued, Processing Succeeded and Failed. What happen if the service is down if the job state is Scheduled or Enqueued or Processing. I have checked it and see that hangfire trigger the job when the service is up again so in those state the chain won't break. But you may loss some trigger.
3. In case of Failed state hangfire transfer the job in Scheduled state again. That's why I have throw exception in next registration if next registration failed so that chain the chain won't break. And hangfire try 10 times. So among all of the try hangfire should register next job also I have tried internally using the Retry helper. So I think chain won't break. But you can still face some inconsistent behavior.
4. I have saved the exception rule as comma separate string but you can handle this case in different way based on your requirement.
5. I want this service independent but couldn't make it fully independent because I couldn't manage transaction here. That's why I have to save the State along with RRULE and other stuffs in the main DB and cross check the state when trigger. This is the main drawback of the solution. I make a R&D of another solution which will share later. But that won't be full solution. I will just share my thoughts.
6. You may need the event/reminder id to the callback API to retrieve the row of corresponding RRULE, State and other stuff in that case you have to pass it ReminderCallbackInfoModel.Path when register.

Am I missing any kind of test case? Any feedback will be appreciated.

[hasansustcse13]

Solution 2 based Dynamically Register Next Job: (Partial Solution)

Interface:

public interface ICalendarJobManager
{
    ResponseMessage<string> AddOrUpdate(CalendarJobModel calendarJob, DateTime fireDateTime);
    void AddOrUpdateCalendarJobHash(string calendarJobId, string scheduleId);
    string GetCalendarJobScheduleId(string calendarJobId);
    void RemoveCalendarJobHashIfExists(string calendarJobId);
    bool RemoveCalendarJobIfExists(string calendarJobId, bool removeHash = false);
}

public interface ICalendarJobTriggerService
{
    Task<HttpResponseMessage> TriggerAsync(string jobName, CalendarJobModel calendarJobModel);
}

Service:

public class CalendarJobManager : ICalendarJobManager
{
    private readonly JobStorage _storage;
    private static readonly TimeSpan DefaultTimeout = TimeSpan.FromSeconds(15.0);
    private static readonly string _calendarJobPrefix = "calendar-job:";
    private static readonly string _hashField = "ScheduleJobId";
    private readonly IBackgroundJobClient _backgroundJob;

    public CalendarJobManager(IBackgroundJobClient backgroundJob)
    {
        _backgroundJob = backgroundJob;
        _storage = JobStorage.Current ?? throw new ArgumentNullException("storage");
    }

    public ResponseMessage<string> AddOrUpdate(CalendarJobModel calendarJob, DateTime fireDateTime)
    {
        try
        {
            string createdJobId = _backgroundJob.Schedule<ICalendarJobTriggerService>(job => job.TriggerAsync(calendarJob.JobName, calendarJob), new DateTimeOffset(fireDateTime));
            AddOrUpdateCalendarJobHash(calendarJob.JobId, createdJobId);
            return ResponseMsg<string>.GetResponseMessage(Constants.OPERATION_SUCCESSFUL, EnumMessageType.Success, (int)HttpStatusCode.OK, createdJobId);
        }
        catch (Exception ex)
        {
            Log.Error(ex, $"Calendar job creation failed: {ex.Message}");
            return ResponseMsg<string>.GetResponseMessage(Constants.OPERATION_FAILED, EnumMessageType.Error, (int)HttpStatusCode.InternalServerError, null);
        }
    }

    public void AddOrUpdateCalendarJobHash(string calendarJobId, string scheduleId)
    {
        if (calendarJobId == null)
            throw new ArgumentNullException("calendarJobId");
        if (scheduleId == null)
            throw new ArgumentNullException("scheduleId");

        var changedFields = new Dictionary<string, string>()
        {
            { _hashField, scheduleId }
        };
        using IStorageConnection storageConnection = _storage.GetConnection();
        using IWriteOnlyTransaction writeOnlyTransaction = storageConnection.CreateWriteTransaction();
        writeOnlyTransaction.SetRangeInHash(_calendarJobPrefix + calendarJobId, changedFields);
        writeOnlyTransaction.Commit();
    }

    public string GetCalendarJobScheduleId(string calendarJobId)
    {
        if (calendarJobId == null)
            throw new ArgumentNullException("calendarJobId");
        using IStorageConnection storageConnection = _storage.GetConnection();
        Dictionary<string, string> allEntriesFromHash = storageConnection.GetAllEntriesFromHash(_calendarJobPrefix + calendarJobId);
        if (allEntriesFromHash == null || allEntriesFromHash.Count == 0 || !allEntriesFromHash.ContainsKey(_hashField))
        {
            return null;
        }
        return allEntriesFromHash[_hashField];
    }

    public void RemoveCalendarJobHashIfExists(string calendarJobId)
    {
        if (calendarJobId == null)
            throw new ArgumentNullException("calendarJobId");

        using IStorageConnection storageConnection = _storage.GetConnection();
        using IWriteOnlyTransaction writeOnlyTransaction = storageConnection.CreateWriteTransaction();
        writeOnlyTransaction.RemoveHash(_calendarJobPrefix + calendarJobId);
        writeOnlyTransaction.Commit();
    }

    public bool RemoveCalendarJobIfExists(string calendarJobId, bool removeHash = false)
    {
        if (calendarJobId == null)
            throw new ArgumentNullException("calendarJobId");
        string scheduleId = GetCalendarJobScheduleId(calendarJobId);
        if (!string.IsNullOrEmpty(scheduleId))
        {
            _backgroundJob.Delete(scheduleId);
        }
        if (removeHash)
        {
            RemoveCalendarJobHashIfExists(calendarJobId);
        }
        return true;
    }

    private static IDisposable AcquireDistributedRecurringJobLock(IStorageConnection connection, string calendarJobId, TimeSpan timeout)
    {
        if (connection == null)
            throw new ArgumentNullException("connection");
        if (calendarJobId == null)
            throw new ArgumentNullException("calendarJobId");

        return connection.AcquireDistributedLock($"lock:{_calendarJobPrefix}" + calendarJobId, timeout);
    }
}

public class CalendarJobTriggerService : ICalendarJobTriggerService
{
    private readonly IHttpClientFactory _httpClientFactory;
    private readonly ICalendarJobManager _calendarJobManager;

    public CalendarJobTriggerService(IHttpClientFactory httpClientFactory, ICalendarJobManager calendarJobManager)
    {
        _httpClientFactory = httpClientFactory;
        _calendarJobManager = calendarJobManager;
    }

    public async Task<HttpResponseMessage> TriggerAsync(string jobName, CalendarJobModel calendarJobModel)
    {
        _calendarJobManager.AddOrUpdate(calendarJobModel, DateTime.UtcNow.AddMinutes(1));
        Console.WriteLine($"--------------------------------------------------------------------------------------------------");
        Console.WriteLine($"{jobName} | {GetTZDate(calendarJobModel.StartDateTime)} | {GetTZDate(DateTime.UtcNow)}");
        Console.WriteLine($"--------------------------------------------------------------------------------------------------");
        return new HttpResponseMessage(System.Net.HttpStatusCode.OK);
    }

    private static string GetTZDate(DateTime dateTime)
    {
        const string SOLR_DATE_FORMAT = "yyyy-MM-ddTHH:mm:ss.fffZ";
        return dateTime.ToString(SOLR_DATE_FORMAT);
    }
}

How does the solution work?

We need to track the schedule Job id against the reminder Id to delete or update the event. In previous solution I have solve it to query in Job table and then delete. In this solution I have solved it by storing it in the Hangfire hash table. To every id (Guid of reminder or RRULE) there will be a scheduled id (created by _backgroundJob.Schedule) saved. So when we need to delete or update we can easily find the jobid from the hash. One thing to note that scheduled id changed every time on next register but the Id (Guid or Reminder) won't change. Hash table auto handle it I mean if there are no entry it will create otherwise it will just update the schedule id.
CalendarJobModel is same as ReminderEventModel

How to make this solution more accurate?

1. As described, the solution is store the created job id against the Id (RRULE id, comes with request body) in hash table. So you see when first time register event you have to create the schedule job and save it in hash, and for delete or update you have delete it hash and also from job using backgroundJob.Delete(scheduleId); So you see in all cases there 2 or more db operations. To make the solution work perfectly we need to maintain transaction here (Which I didn't).
2. To make the transaction at the time of create you need to override create behavior of the _backgroundJob.Schedule() method. But it's not possible directly. To do this you need to change the Create method of CoreBackgroundJobFactory. You can't commit in this method as you have to make another write (in Hash table) in DB. BackgroundJobClient -> BackgroundJobFactory -> CoreBackgroundJobFactory. To understand this see the constructor of BackgroundJobClient.
3. As I have mentioned you also have to maintain transaction in delete also. To do this you have to override ChangeState method of BackgroundJobStateChanger (not Sure) and can't commit here . BackgroundJobClient -> BackgroundJobStateChanger. See the BackgroundJobClient constructor to understand more. But I think this will be highly risk as my assumption is hangfire internally use this method to change the state.
4. If you are able to make the transaction without hampering the hangfire internal behavior you need to used lock.

As you see this solution is too complicated and it's possible hangfire will be internally broken if we don't careful enough. That's why I won't go with this solution. But the provided solution will work if we ignore transaction and lock. And also we can introduce State property of previous solution here to any kind of mismatch.
Despite this is a workable solution I won't go with this, because I need to manually enter data in hash table to track the chain (Also there are multiple entries (hash & job) in the table during the next registration) . And I don't know how hangfire will change in future.

[hasansustcse13]

@ShayMusachanov-dev Have you got the time to see the implementation? I have tried PDI NextInstance as it's most straight forward and all kind of cases handle here. But I see that this method return DateTimeKind is Unspecified (though all my input time in UTC) which will cause problem in Hangfire as it converts the date in UTC. My next occurrences function is bit complex and need to handle a lot's of cases (Still may have bug) that's why I wanted to try PDI. If you find any other way or any kind of bug please let me know as you see this is the main core of the feature.