Failed Jobs

Enqueued 0 / 1 Scheduled 0 Processing 4 Succeeded 7,828,633 Failed 13,733 Deleted 5,104 Awaiting 0

Failed jobs do not become expired to allow you to re-queue them without any time pressure. You should re-queue or delete them manually, or apply AutomaticRetry(OnAttemptsExceeded = AttemptsExceededAction.Delete) attribute to delete them automatically.

Id	Job	Failed
#6657895	RetrieveProductStatusTask.ProcessRecurringAsync An exception occurred during performance of the job. Fewer details...	09/04/2025 23:30:24
#6657895	System.InvalidOperationException A second operation was started on this context instance before a previous operation completed. This is usually caused by different threads concurrently using the same instance of DbContext. For more information on how to avoid threading issues with DbContext, see https://go.microsoft.com/fwlink/?linkid=2097913. System.InvalidOperationException: A second operation was started on this context instance before a previous operation completed. This is usually caused by different threads concurrently using the same instance of DbContext. For more information on how to avoid threading issues with DbContext, see https://go.microsoft.com/fwlink/?linkid=2097913. at Microsoft.EntityFrameworkCore.Infrastructure.Internal.ConcurrencyDetector.EnterCriticalSection() at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChangesAsync(IList`1 entriesToSave, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChangesAsync(StateManager stateManager, Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken) at Npgsql.EntityFrameworkCore.PostgreSQL.Storage.Internal.NpgsqlExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.DbContext.SaveChangesAsync(Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.DbContext.SaveChangesAsync(Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken) at EkolPY.Integration.Service.IntegrationTaskService.AddTaskLog(TaskLogLevel logLevel, String[] messages) in /src/Integration/EkolPY.Integration.Service/IntegrationService.cs:line 156 at EkolPY.Integration.Service.IntegrationTaskService.AddTaskError(Exception ex, String[] msgs) in /src/Integration/EkolPY.Integration.Service/IntegrationService.cs:line 148 at EkolPY.Integration.Service.IntegrationTaskService.SetFailedWithException(String message, Exception ex) in /src/Integration/EkolPY.Integration.Service/IntegrationService.cs:line 248 at EkolPY.Integration.Manager.TaskBase`1.ProcessRecurringAsync(PerformContext context, String queueName, TaskType taskType) in /src/Integration/EkolPY.Integration.Manager/TaskBase.cs:line 162 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
#6649301	RetrieveUpdateProductStockBatchResultsTask.ProcessRecurringAsync An exception occurred during performance of the job. More details...	09/04/2025 06:10:39
#6649301	System.InvalidOperationException A second operation was started on this context instance before a previous operation completed. This is usually caused by different threads concurrently using the same instance of DbContext. For more information on how to avoid threading issues with DbContext, see https://go.microsoft.com/fwlink/?linkid=2097913. System.InvalidOperationException: A second operation was started on this context instance before a previous operation completed. This is usually caused by different threads concurrently using the same instance of DbContext. For more information on how to avoid threading issues with DbContext, see https://go.microsoft.com/fwlink/?linkid=2097913. at Microsoft.EntityFrameworkCore.Infrastructure.Internal.ConcurrencyDetector.EnterCriticalSection() at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChangesAsync(IList`1 entriesToSave, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChangesAsync(StateManager stateManager, Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken) at Npgsql.EntityFrameworkCore.PostgreSQL.Storage.Internal.NpgsqlExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.DbContext.SaveChangesAsync(Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.DbContext.SaveChangesAsync(Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken) at EkolPY.Integration.Service.IntegrationTaskService.AddTaskLog(TaskLogLevel logLevel, String[] messages) in /src/Integration/EkolPY.Integration.Service/IntegrationService.cs:line 154 at EkolPY.Integration.Service.IntegrationTaskService.AddTaskError(Exception ex, String[] msgs) in /src/Integration/EkolPY.Integration.Service/IntegrationService.cs:line 146 at EkolPY.Integration.Service.IntegrationTaskService.SetFailedWithException(String message, Exception ex) in /src/Integration/EkolPY.Integration.Service/IntegrationService.cs:line 246 at EkolPY.Integration.Manager.TaskBase`1.ProcessRecurringAsync(PerformContext context, String queueName, TaskType taskType) in /src/Integration/EkolPY.Integration.Manager/TaskBase.cs:line 162 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
#6591309	RetrieveUpdateProductStockBatchResultsTask.ProcessRecurringAsync An exception occurred during performance of the job. More details...	08/30/2025 10:04:00
#6591309	System.InvalidOperationException A second operation was started on this context instance before a previous operation completed. This is usually caused by different threads concurrently using the same instance of DbContext. For more information on how to avoid threading issues with DbContext, see https://go.microsoft.com/fwlink/?linkid=2097913. System.InvalidOperationException: A second operation was started on this context instance before a previous operation completed. This is usually caused by different threads concurrently using the same instance of DbContext. For more information on how to avoid threading issues with DbContext, see https://go.microsoft.com/fwlink/?linkid=2097913. at Microsoft.EntityFrameworkCore.Infrastructure.Internal.ConcurrencyDetector.EnterCriticalSection() at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChangesAsync(IList`1 entriesToSave, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChangesAsync(StateManager stateManager, Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken) at Npgsql.EntityFrameworkCore.PostgreSQL.Storage.Internal.NpgsqlExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.DbContext.SaveChangesAsync(Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.DbContext.SaveChangesAsync(Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken) at EkolPY.Integration.Service.IntegrationTaskService.AddTaskLog(TaskLogLevel logLevel, String[] messages) in /src/Integration/EkolPY.Integration.Service/IntegrationService.cs:line 154 at EkolPY.Integration.Service.IntegrationTaskService.AddTaskError(Exception ex, String[] msgs) in /src/Integration/EkolPY.Integration.Service/IntegrationService.cs:line 146 at EkolPY.Integration.Service.IntegrationTaskService.SetFailedWithException(String message, Exception ex) in /src/Integration/EkolPY.Integration.Service/IntegrationService.cs:line 246 at EkolPY.Integration.Manager.TaskBase`1.ProcessRecurringAsync(PerformContext context, String queueName, TaskType taskType) in /src/Integration/EkolPY.Integration.Manager/TaskBase.cs:line 162 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
#6555832	SendOrderTask.ProcessRecurringAsync An exception occurred during processing of a background job. More details...	08/27/2025 11:00:44
#6555832	System.InvalidOperationException Can't close, connection is in state Connecting System.InvalidOperationException: Can't close, connection is in state Connecting at Npgsql.NpgsqlConnection.Close(Boolean async) at Npgsql.NpgsqlConnection.Close() at Npgsql.NpgsqlConnection.Dispose(Boolean disposing) at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.DisposeDbConnection() at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.ResetState(Boolean disposeDbConnection) at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope.Dispose() at Microsoft.EntityFrameworkCore.DbContext.Dispose() at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope.Dispose() at Hangfire.JobActivatorScope.Dispose() at Hangfire.Server.CoreBackgroundJobPerformer.Perform(PerformContext context) at Hangfire.Server.BackgroundJobPerformer.<>c__DisplayClass9_0.<PerformJobWithFilters>b__0() at Hangfire.Server.BackgroundJobPerformer.InvokePerformFilter(IServerFilter filter, PerformingContext preContext, Func`1 continuation) at Hangfire.Server.BackgroundJobPerformer.PerformJobWithFilters(PerformContext context, IEnumerable`1 filters) at Hangfire.Server.BackgroundJobPerformer.Perform(PerformContext context) at Hangfire.Server.Worker.PerformJob(BackgroundProcessContext context, IStorageConnection connection, String jobId)
#6553567	SendOrderTask.ProcessRecurringAsync An exception occurred during processing of a background job. More details...	08/27/2025 06:28:05
#6553567	System.InvalidOperationException Can't close, connection is in state Connecting System.InvalidOperationException: Can't close, connection is in state Connecting at Npgsql.NpgsqlConnection.Close(Boolean async) at Npgsql.NpgsqlConnection.Close() at Npgsql.NpgsqlConnection.Dispose(Boolean disposing) at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.DisposeDbConnection() at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.ResetState(Boolean disposeDbConnection) at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope.Dispose() at Microsoft.EntityFrameworkCore.DbContext.Dispose() at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope.Dispose() at Hangfire.JobActivatorScope.Dispose() at Hangfire.Server.CoreBackgroundJobPerformer.Perform(PerformContext context) at Hangfire.Server.BackgroundJobPerformer.<>c__DisplayClass9_0.<PerformJobWithFilters>b__0() at Hangfire.Server.BackgroundJobPerformer.InvokePerformFilter(IServerFilter filter, PerformingContext preContext, Func`1 continuation) at Hangfire.Server.BackgroundJobPerformer.PerformJobWithFilters(PerformContext context, IEnumerable`1 filters) at Hangfire.Server.BackgroundJobPerformer.Perform(PerformContext context) at Hangfire.Server.Worker.PerformJob(BackgroundProcessContext context, IStorageConnection connection, String jobId)
#6546486	SendOrderTask.ProcessRecurringAsync An exception occurred during performance of the job. More details...	08/26/2025 16:14:41
#6546486	System.InvalidOperationException A second operation was started on this context instance before a previous operation completed. This is usually caused by different threads concurrently using the same instance of DbContext. For more information on how to avoid threading issues with DbContext, see https://go.microsoft.com/fwlink/?linkid=2097913. System.InvalidOperationException: A second operation was started on this context instance before a previous operation completed. This is usually caused by different threads concurrently using the same instance of DbContext. For more information on how to avoid threading issues with DbContext, see https://go.microsoft.com/fwlink/?linkid=2097913. at Microsoft.EntityFrameworkCore.Infrastructure.Internal.ConcurrencyDetector.EnterCriticalSection() at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChangesAsync(IList`1 entriesToSave, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChangesAsync(StateManager stateManager, Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken) at Npgsql.EntityFrameworkCore.PostgreSQL.Storage.Internal.NpgsqlExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.DbContext.SaveChangesAsync(Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.DbContext.SaveChangesAsync(Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken) at EkolPY.Integration.Service.IntegrationTaskService.AddTaskLog(TaskLogLevel logLevel, String[] messages) in /src/Integration/EkolPY.Integration.Service/IntegrationService.cs:line 154 at EkolPY.Integration.Service.IntegrationTaskService.AddTaskError(Exception ex, String[] msgs) in /src/Integration/EkolPY.Integration.Service/IntegrationService.cs:line 146 at EkolPY.Integration.Service.IntegrationTaskService.SetFailedWithException(String message, Exception ex) in /src/Integration/EkolPY.Integration.Service/IntegrationService.cs:line 246 at EkolPY.Integration.Manager.TaskBase`1.ProcessRecurringAsync(PerformContext context, String queueName, TaskType taskType) in /src/Integration/EkolPY.Integration.Manager/TaskBase.cs:line 162 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
#6307809	RetrieveUpdateProductStockBatchResultsTask.ProcessRecurringAsync An exception occurred during performance of the job. More details...	08/06/2025 19:50:45
#6307809	Grpc.Core.RpcException Status(StatusCode="InvalidArgument", Detail="Document 'projects/ekol-md-prod/databases/ekol-md-prod/documents/TaskLogFireStore/48320067' cannot be written because its size (1,051,983 bytes) exceeds the maximum allowed size of 1,048,576 bytes.") Grpc.Core.RpcException: Status(StatusCode="InvalidArgument", Detail="Document 'projects/ekol-md-prod/databases/ekol-md-prod/documents/TaskLogFireStore/48320067' cannot be written because its size (1,051,983 bytes) exceeds the maximum allowed size of 1,048,576 bytes.") at Google.Api.Gax.Grpc.ApiCallRetryExtensions.<>c__DisplayClass0_0`2.<<WithRetry>b__0>d.MoveNext() --- End of stack trace from previous location --- at Google.Cloud.Firestore.WriteBatch.CommitAsync(ByteString transactionId, CancellationToken cancellationToken) at Google.Cloud.Firestore.DocumentReference.SetAsync(Object documentData, SetOptions options, CancellationToken cancellationToken) at EkolPY.Integration.Data.FireStore.FireStoreRepository`1.UpdateItemAsync(String id, TEntity item) in /src/Integration/EkolPY.Integration.Data.FireStore/Repository/FirestoreRepository.cs:line 78 at EkolPY.Integration.Service.IntegrationTaskService.SaveLog() in /src/Integration/EkolPY.Integration.Service/IntegrationService.cs:line 442 at EkolPY.Integration.Service.IntegrationTaskService.SetFailedAsync() in /src/Integration/EkolPY.Integration.Service/IntegrationService.cs:line 483 at EkolPY.Integration.Service.IntegrationTaskService.SetFailedWithException(String message, Exception ex) in /src/Integration/EkolPY.Integration.Service/IntegrationService.cs:line 497 at EkolPY.Integration.Manager.TaskBase`1.ProcessRecurringAsync(PerformContext context, String queueName, TaskType taskType) in /src/Integration/EkolPY.Integration.Manager/TaskBase.cs:line 162 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
#6307790	RetrieveUpdateProductStockBatchResultsTask.ProcessRecurringAsync An exception occurred during performance of the job. More details...	08/06/2025 19:47:09
#6307790	Grpc.Core.RpcException Status(StatusCode="InvalidArgument", Detail="Document 'projects/ekol-md-prod/databases/ekol-md-prod/documents/TaskLogFireStore/48319788' cannot be written because its size (1,051,983 bytes) exceeds the maximum allowed size of 1,048,576 bytes.") Grpc.Core.RpcException: Status(StatusCode="InvalidArgument", Detail="Document 'projects/ekol-md-prod/databases/ekol-md-prod/documents/TaskLogFireStore/48319788' cannot be written because its size (1,051,983 bytes) exceeds the maximum allowed size of 1,048,576 bytes.") at Google.Api.Gax.Grpc.ApiCallRetryExtensions.<>c__DisplayClass0_0`2.<<WithRetry>b__0>d.MoveNext() --- End of stack trace from previous location --- at Google.Cloud.Firestore.WriteBatch.CommitAsync(ByteString transactionId, CancellationToken cancellationToken) at Google.Cloud.Firestore.DocumentReference.SetAsync(Object documentData, SetOptions options, CancellationToken cancellationToken) at EkolPY.Integration.Data.FireStore.FireStoreRepository`1.UpdateItemAsync(String id, TEntity item) in /src/Integration/EkolPY.Integration.Data.FireStore/Repository/FirestoreRepository.cs:line 78 at EkolPY.Integration.Service.IntegrationTaskService.SaveLog() in /src/Integration/EkolPY.Integration.Service/IntegrationService.cs:line 442 at EkolPY.Integration.Service.IntegrationTaskService.SetFailedAsync() in /src/Integration/EkolPY.Integration.Service/IntegrationService.cs:line 483 at EkolPY.Integration.Service.IntegrationTaskService.SetFailedWithException(String message, Exception ex) in /src/Integration/EkolPY.Integration.Service/IntegrationService.cs:line 497 at EkolPY.Integration.Manager.TaskBase`1.ProcessRecurringAsync(PerformContext context, String queueName, TaskType taskType) in /src/Integration/EkolPY.Integration.Manager/TaskBase.cs:line 162 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
#6024206	RetrieveReturnRequestRejectReasonsTask.ProcessRecurringAsync An exception occurred during performance of the job. More details...	07/13/2025 22:34:18
#6024206	System.NullReferenceException Object reference not set to an instance of an object. System.NullReferenceException: Object reference not set to an instance of an object. at EkolPY.Integration.Service.IntegrationTaskService.SetFailedWithException(String message, Exception ex) in /src/Integration/EkolPY.Integration.Service/IntegrationService.cs:line 496 at EkolPY.Integration.Manager.TaskBase`1.ProcessRecurringAsync(PerformContext context, String queueName, TaskType taskType) in /src/Integration/EkolPY.Integration.Manager/TaskBase.cs:line 162 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
#5988089	RetrieveOrdersTask.ProcessRecurringAsync An exception occurred during performance of the job. More details...	07/10/2025 22:37:31
#5988089	Grpc.Core.RpcException Status(StatusCode="InvalidArgument", Detail="Document 'projects/ekol-md-prod/databases/ekol-md-prod/documents/TaskLogFireStore/44873297' cannot be written because its size (1,477,759 bytes) exceeds the maximum allowed size of 1,048,576 bytes.") Grpc.Core.RpcException: Status(StatusCode="InvalidArgument", Detail="Document 'projects/ekol-md-prod/databases/ekol-md-prod/documents/TaskLogFireStore/44873297' cannot be written because its size (1,477,759 bytes) exceeds the maximum allowed size of 1,048,576 bytes.") at Google.Api.Gax.Grpc.ApiCallRetryExtensions.<>c__DisplayClass0_0`2.<<WithRetry>b__0>d.MoveNext() --- End of stack trace from previous location --- at Google.Cloud.Firestore.WriteBatch.CommitAsync(ByteString transactionId, CancellationToken cancellationToken) at Google.Cloud.Firestore.DocumentReference.SetAsync(Object documentData, SetOptions options, CancellationToken cancellationToken) at EkolPY.Integration.Data.FireStore.FireStoreRepository`1.UpdateItemAsync(String id, TEntity item) in /src/Integration/EkolPY.Integration.Data.FireStore/Repository/FirestoreRepository.cs:line 78 at EkolPY.Integration.Service.IntegrationTaskService.SaveLog() in /src/Integration/EkolPY.Integration.Service/IntegrationService.cs:line 441 at EkolPY.Integration.Service.IntegrationTaskService.SetFailedAsync() in /src/Integration/EkolPY.Integration.Service/IntegrationService.cs:line 482 at EkolPY.Integration.Service.IntegrationTaskService.SetFailedWithException(String message, Exception ex) in /src/Integration/EkolPY.Integration.Service/IntegrationService.cs:line 496 at EkolPY.Integration.Manager.TaskBase`1.ProcessRecurringAsync(PerformContext context, String queueName, TaskType taskType) in /src/Integration/EkolPY.Integration.Manager/TaskBase.cs:line 162 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

System.InvalidOperationException

System.InvalidOperationException

System.InvalidOperationException

System.InvalidOperationException

System.InvalidOperationException

System.InvalidOperationException

Grpc.Core.RpcException

Grpc.Core.RpcException

System.NullReferenceException

Grpc.Core.RpcException