Failed jobs do not become expired to allow you to re-queue them without any
time pressure. You should re-queue or delete them manually, or apply AutomaticRetry(OnAttemptsExceeded = AttemptsExceededAction.Delete)
attribute to delete them automatically.
An exception occurred during processing of a background job. Fewer details...
08/27/2025 06:28:05
System.InvalidOperationException
Can't close, connection is in state Connecting
System.InvalidOperationException: Can't close, connection is in state Connecting
at Npgsql.NpgsqlConnection.Close(Booleanasync)
at Npgsql.NpgsqlConnection.Close()
at Npgsql.NpgsqlConnection.Dispose(Booleandisposing)
at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.DisposeDbConnection()
at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.ResetState(BooleandisposeDbConnection)
at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope.Dispose()
at Microsoft.EntityFrameworkCore.DbContext.Dispose()
at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope.Dispose()
at Hangfire.JobActivatorScope.Dispose()
at Hangfire.Server.CoreBackgroundJobPerformer.Perform(PerformContextcontext)
at Hangfire.Server.BackgroundJobPerformer.<>c__DisplayClass9_0.<PerformJobWithFilters>b__0()
at Hangfire.Server.BackgroundJobPerformer.InvokePerformFilter(IServerFilterfilter, PerformingContextpreContext, Func`1continuation)
at Hangfire.Server.BackgroundJobPerformer.PerformJobWithFilters(PerformContextcontext, IEnumerable`1filters)
at Hangfire.Server.BackgroundJobPerformer.Perform(PerformContextcontext)
at Hangfire.Server.Worker.PerformJob(BackgroundProcessContextcontext, IStorageConnectionconnection, StringjobId)
An exception occurred during performance of the job. More details...
08/26/2025 16:14:41
System.InvalidOperationException
A second operation was started on this context instance before a previous operation completed. This is usually caused by different threads concurrently using the same instance of DbContext. For more information on how to avoid threading issues with DbContext, see https://go.microsoft.com/fwlink/?linkid=2097913.
System.InvalidOperationException: A second operation was started on this context instance before a previous operation completed. This is usually caused by different threads concurrently using the same instance of DbContext. For more information on how to avoid threading issues with DbContext, see https://go.microsoft.com/fwlink/?linkid=2097913.
at Microsoft.EntityFrameworkCore.Infrastructure.Internal.ConcurrencyDetector.EnterCriticalSection()
at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChangesAsync(IList`1entriesToSave, CancellationTokencancellationToken)
at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChangesAsync(StateManagerstateManager, BooleanacceptAllChangesOnSuccess, CancellationTokencancellationToken)
at Npgsql.EntityFrameworkCore.PostgreSQL.Storage.Internal.NpgsqlExecutionStrategy.ExecuteAsync[TState,TResult](TStatestate, Func`4operation, Func`4verifySucceeded, CancellationTokencancellationToken)
at Microsoft.EntityFrameworkCore.DbContext.SaveChangesAsync(BooleanacceptAllChangesOnSuccess, CancellationTokencancellationToken)
at Microsoft.EntityFrameworkCore.DbContext.SaveChangesAsync(BooleanacceptAllChangesOnSuccess, CancellationTokencancellationToken)
at EkolPY.Integration.Service.IntegrationTaskService.AddTaskLog(TaskLogLevellogLevel, String[]messages) in /src/Integration/EkolPY.Integration.Service/IntegrationService.cs:line 154
at EkolPY.Integration.Service.IntegrationTaskService.AddTaskError(Exceptionex, String[]msgs) in /src/Integration/EkolPY.Integration.Service/IntegrationService.cs:line 146
at EkolPY.Integration.Service.IntegrationTaskService.SetFailedWithException(Stringmessage, Exceptionex) in /src/Integration/EkolPY.Integration.Service/IntegrationService.cs:line 246
at EkolPY.Integration.Manager.TaskBase`1.ProcessRecurringAsync(PerformContextcontext, StringqueueName, TaskTypetaskType) in /src/Integration/EkolPY.Integration.Manager/TaskBase.cs:line 162
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Tasktask)
An exception occurred during performance of the job. More details...
08/06/2025 19:50:45
Grpc.Core.RpcException
Status(StatusCode="InvalidArgument", Detail="Document 'projects/ekol-md-prod/databases/ekol-md-prod/documents/TaskLogFireStore/48320067' cannot be written because its size (1,051,983 bytes) exceeds the maximum allowed size of 1,048,576 bytes.")
Grpc.Core.RpcException: Status(StatusCode="InvalidArgument", Detail="Document 'projects/ekol-md-prod/databases/ekol-md-prod/documents/TaskLogFireStore/48320067' cannot be written because its size (1,051,983 bytes) exceeds the maximum allowed size of 1,048,576 bytes.")
at Google.Api.Gax.Grpc.ApiCallRetryExtensions.<>c__DisplayClass0_0`2.<<WithRetry>b__0>d.MoveNext()
--- End of stack trace from previous location ---
at Google.Cloud.Firestore.WriteBatch.CommitAsync(ByteStringtransactionId, CancellationTokencancellationToken)
at Google.Cloud.Firestore.DocumentReference.SetAsync(ObjectdocumentData, SetOptionsoptions, CancellationTokencancellationToken)
at EkolPY.Integration.Data.FireStore.FireStoreRepository`1.UpdateItemAsync(Stringid, TEntityitem) in /src/Integration/EkolPY.Integration.Data.FireStore/Repository/FirestoreRepository.cs:line 78
at EkolPY.Integration.Service.IntegrationTaskService.SaveLog() in /src/Integration/EkolPY.Integration.Service/IntegrationService.cs:line 442
at EkolPY.Integration.Service.IntegrationTaskService.SetFailedAsync() in /src/Integration/EkolPY.Integration.Service/IntegrationService.cs:line 483
at EkolPY.Integration.Service.IntegrationTaskService.SetFailedWithException(Stringmessage, Exceptionex) in /src/Integration/EkolPY.Integration.Service/IntegrationService.cs:line 497
at EkolPY.Integration.Manager.TaskBase`1.ProcessRecurringAsync(PerformContextcontext, StringqueueName, TaskTypetaskType) in /src/Integration/EkolPY.Integration.Manager/TaskBase.cs:line 162
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Tasktask)
An exception occurred during performance of the job. More details...
08/06/2025 19:47:09
Grpc.Core.RpcException
Status(StatusCode="InvalidArgument", Detail="Document 'projects/ekol-md-prod/databases/ekol-md-prod/documents/TaskLogFireStore/48319788' cannot be written because its size (1,051,983 bytes) exceeds the maximum allowed size of 1,048,576 bytes.")
Grpc.Core.RpcException: Status(StatusCode="InvalidArgument", Detail="Document 'projects/ekol-md-prod/databases/ekol-md-prod/documents/TaskLogFireStore/48319788' cannot be written because its size (1,051,983 bytes) exceeds the maximum allowed size of 1,048,576 bytes.")
at Google.Api.Gax.Grpc.ApiCallRetryExtensions.<>c__DisplayClass0_0`2.<<WithRetry>b__0>d.MoveNext()
--- End of stack trace from previous location ---
at Google.Cloud.Firestore.WriteBatch.CommitAsync(ByteStringtransactionId, CancellationTokencancellationToken)
at Google.Cloud.Firestore.DocumentReference.SetAsync(ObjectdocumentData, SetOptionsoptions, CancellationTokencancellationToken)
at EkolPY.Integration.Data.FireStore.FireStoreRepository`1.UpdateItemAsync(Stringid, TEntityitem) in /src/Integration/EkolPY.Integration.Data.FireStore/Repository/FirestoreRepository.cs:line 78
at EkolPY.Integration.Service.IntegrationTaskService.SaveLog() in /src/Integration/EkolPY.Integration.Service/IntegrationService.cs:line 442
at EkolPY.Integration.Service.IntegrationTaskService.SetFailedAsync() in /src/Integration/EkolPY.Integration.Service/IntegrationService.cs:line 483
at EkolPY.Integration.Service.IntegrationTaskService.SetFailedWithException(Stringmessage, Exceptionex) in /src/Integration/EkolPY.Integration.Service/IntegrationService.cs:line 497
at EkolPY.Integration.Manager.TaskBase`1.ProcessRecurringAsync(PerformContextcontext, StringqueueName, TaskTypetaskType) in /src/Integration/EkolPY.Integration.Manager/TaskBase.cs:line 162
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Tasktask)
An exception occurred during performance of the job. More details...
07/13/2025 22:34:18
System.NullReferenceException
Object reference not set to an instance of an object.
System.NullReferenceException: Object reference not set to an instance of an object.
at EkolPY.Integration.Service.IntegrationTaskService.SetFailedWithException(Stringmessage, Exceptionex) in /src/Integration/EkolPY.Integration.Service/IntegrationService.cs:line 496
at EkolPY.Integration.Manager.TaskBase`1.ProcessRecurringAsync(PerformContextcontext, StringqueueName, TaskTypetaskType) in /src/Integration/EkolPY.Integration.Manager/TaskBase.cs:line 162
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Tasktask)
An exception occurred during performance of the job. More details...
07/10/2025 22:37:31
Grpc.Core.RpcException
Status(StatusCode="InvalidArgument", Detail="Document 'projects/ekol-md-prod/databases/ekol-md-prod/documents/TaskLogFireStore/44873297' cannot be written because its size (1,477,759 bytes) exceeds the maximum allowed size of 1,048,576 bytes.")
Grpc.Core.RpcException: Status(StatusCode="InvalidArgument", Detail="Document 'projects/ekol-md-prod/databases/ekol-md-prod/documents/TaskLogFireStore/44873297' cannot be written because its size (1,477,759 bytes) exceeds the maximum allowed size of 1,048,576 bytes.")
at Google.Api.Gax.Grpc.ApiCallRetryExtensions.<>c__DisplayClass0_0`2.<<WithRetry>b__0>d.MoveNext()
--- End of stack trace from previous location ---
at Google.Cloud.Firestore.WriteBatch.CommitAsync(ByteStringtransactionId, CancellationTokencancellationToken)
at Google.Cloud.Firestore.DocumentReference.SetAsync(ObjectdocumentData, SetOptionsoptions, CancellationTokencancellationToken)
at EkolPY.Integration.Data.FireStore.FireStoreRepository`1.UpdateItemAsync(Stringid, TEntityitem) in /src/Integration/EkolPY.Integration.Data.FireStore/Repository/FirestoreRepository.cs:line 78
at EkolPY.Integration.Service.IntegrationTaskService.SaveLog() in /src/Integration/EkolPY.Integration.Service/IntegrationService.cs:line 441
at EkolPY.Integration.Service.IntegrationTaskService.SetFailedAsync() in /src/Integration/EkolPY.Integration.Service/IntegrationService.cs:line 482
at EkolPY.Integration.Service.IntegrationTaskService.SetFailedWithException(Stringmessage, Exceptionex) in /src/Integration/EkolPY.Integration.Service/IntegrationService.cs:line 496
at EkolPY.Integration.Manager.TaskBase`1.ProcessRecurringAsync(PerformContextcontext, StringqueueName, TaskTypetaskType) in /src/Integration/EkolPY.Integration.Manager/TaskBase.cs:line 162
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Tasktask)
An exception occurred during performance of the job. More details...
07/10/2025 22:39:14
Grpc.Core.RpcException
Status(StatusCode="InvalidArgument", Detail="Property array contains an invalid nested entity.")
Grpc.Core.RpcException: Status(StatusCode="InvalidArgument", Detail="Property array contains an invalid nested entity.")
at Google.Api.Gax.Grpc.ApiCallRetryExtensions.<>c__DisplayClass0_0`2.<<WithRetry>b__0>d.MoveNext()
--- End of stack trace from previous location ---
at Google.Cloud.Firestore.WriteBatch.CommitAsync(ByteStringtransactionId, CancellationTokencancellationToken)
at Google.Cloud.Firestore.DocumentReference.SetAsync(ObjectdocumentData, SetOptionsoptions, CancellationTokencancellationToken)
at EkolPY.Integration.Data.FireStore.FireStoreRepository`1.UpdateItemAsync(Stringid, TEntityitem) in /src/Integration/EkolPY.Integration.Data.FireStore/Repository/FirestoreRepository.cs:line 78
at EkolPY.Integration.Service.IntegrationTaskService.SaveLog() in /src/Integration/EkolPY.Integration.Service/IntegrationService.cs:line 441
at EkolPY.Integration.Service.IntegrationTaskService.SetFailedAsync() in /src/Integration/EkolPY.Integration.Service/IntegrationService.cs:line 482
at EkolPY.Integration.Service.IntegrationTaskService.SetFailedWithException(Stringmessage, Exceptionex) in /src/Integration/EkolPY.Integration.Service/IntegrationService.cs:line 496
at EkolPY.Integration.Manager.TaskBase`1.ProcessRecurringAsync(PerformContextcontext, StringqueueName, TaskTypetaskType) in /src/Integration/EkolPY.Integration.Manager/TaskBase.cs:line 162
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Tasktask)
An exception occurred during performance of the job. More details...
07/10/2025 22:38:32
Grpc.Core.RpcException
Status(StatusCode="InvalidArgument", Detail="Property array contains an invalid nested entity.")
Grpc.Core.RpcException: Status(StatusCode="InvalidArgument", Detail="Property array contains an invalid nested entity.")
at Google.Api.Gax.Grpc.ApiCallRetryExtensions.<>c__DisplayClass0_0`2.<<WithRetry>b__0>d.MoveNext()
--- End of stack trace from previous location ---
at Google.Cloud.Firestore.WriteBatch.CommitAsync(ByteStringtransactionId, CancellationTokencancellationToken)
at Google.Cloud.Firestore.DocumentReference.SetAsync(ObjectdocumentData, SetOptionsoptions, CancellationTokencancellationToken)
at EkolPY.Integration.Data.FireStore.FireStoreRepository`1.UpdateItemAsync(Stringid, TEntityitem) in /src/Integration/EkolPY.Integration.Data.FireStore/Repository/FirestoreRepository.cs:line 78
at EkolPY.Integration.Service.IntegrationTaskService.SaveLog() in /src/Integration/EkolPY.Integration.Service/IntegrationService.cs:line 441
at EkolPY.Integration.Service.IntegrationTaskService.SetFailedAsync() in /src/Integration/EkolPY.Integration.Service/IntegrationService.cs:line 482
at EkolPY.Integration.Service.IntegrationTaskService.SetFailedWithException(Stringmessage, Exceptionex) in /src/Integration/EkolPY.Integration.Service/IntegrationService.cs:line 496
at EkolPY.Integration.Manager.TaskBase`1.ProcessRecurringAsync(PerformContextcontext, StringqueueName, TaskTypetaskType) in /src/Integration/EkolPY.Integration.Manager/TaskBase.cs:line 162
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Tasktask)
An exception occurred during performance of the job. More details...
07/10/2025 22:39:51
Grpc.Core.RpcException
Status(StatusCode="InvalidArgument", Detail="Property array contains an invalid nested entity.")
Grpc.Core.RpcException: Status(StatusCode="InvalidArgument", Detail="Property array contains an invalid nested entity.")
at Google.Api.Gax.Grpc.ApiCallRetryExtensions.<>c__DisplayClass0_0`2.<<WithRetry>b__0>d.MoveNext()
--- End of stack trace from previous location ---
at Google.Cloud.Firestore.WriteBatch.CommitAsync(ByteStringtransactionId, CancellationTokencancellationToken)
at Google.Cloud.Firestore.DocumentReference.SetAsync(ObjectdocumentData, SetOptionsoptions, CancellationTokencancellationToken)
at EkolPY.Integration.Data.FireStore.FireStoreRepository`1.UpdateItemAsync(Stringid, TEntityitem) in /src/Integration/EkolPY.Integration.Data.FireStore/Repository/FirestoreRepository.cs:line 78
at EkolPY.Integration.Service.IntegrationTaskService.SaveLog() in /src/Integration/EkolPY.Integration.Service/IntegrationService.cs:line 441
at EkolPY.Integration.Service.IntegrationTaskService.SetFailedAsync() in /src/Integration/EkolPY.Integration.Service/IntegrationService.cs:line 482
at EkolPY.Integration.Service.IntegrationTaskService.SetFailedWithException(Stringmessage, Exceptionex) in /src/Integration/EkolPY.Integration.Service/IntegrationService.cs:line 496
at EkolPY.Integration.Manager.TaskBase`1.ProcessRecurringAsync(PerformContextcontext, StringqueueName, TaskTypetaskType) in /src/Integration/EkolPY.Integration.Manager/TaskBase.cs:line 162
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Tasktask)
An exception occurred during performance of the job. More details...
07/10/2025 22:07:19
Grpc.Core.RpcException
Status(StatusCode="InvalidArgument", Detail="Document 'projects/ekol-md-prod/databases/ekol-md-prod/documents/TaskLogFireStore/44870762' cannot be written because its size (1,458,446 bytes) exceeds the maximum allowed size of 1,048,576 bytes.")
Grpc.Core.RpcException: Status(StatusCode="InvalidArgument", Detail="Document 'projects/ekol-md-prod/databases/ekol-md-prod/documents/TaskLogFireStore/44870762' cannot be written because its size (1,458,446 bytes) exceeds the maximum allowed size of 1,048,576 bytes.")
at Google.Api.Gax.Grpc.ApiCallRetryExtensions.<>c__DisplayClass0_0`2.<<WithRetry>b__0>d.MoveNext()
--- End of stack trace from previous location ---
at Google.Cloud.Firestore.WriteBatch.CommitAsync(ByteStringtransactionId, CancellationTokencancellationToken)
at Google.Cloud.Firestore.DocumentReference.SetAsync(ObjectdocumentData, SetOptionsoptions, CancellationTokencancellationToken)
at EkolPY.Integration.Data.FireStore.FireStoreRepository`1.UpdateItemAsync(Stringid, TEntityitem) in /src/Integration/EkolPY.Integration.Data.FireStore/Repository/FirestoreRepository.cs:line 78
at EkolPY.Integration.Service.IntegrationTaskService.SaveLog() in /src/Integration/EkolPY.Integration.Service/IntegrationService.cs:line 441
at EkolPY.Integration.Service.IntegrationTaskService.SetFailedAsync() in /src/Integration/EkolPY.Integration.Service/IntegrationService.cs:line 482
at EkolPY.Integration.Service.IntegrationTaskService.SetFailedWithException(Stringmessage, Exceptionex) in /src/Integration/EkolPY.Integration.Service/IntegrationService.cs:line 496
at EkolPY.Integration.Manager.TaskBase`1.ProcessRecurringAsync(PerformContextcontext, StringqueueName, TaskTypetaskType) in /src/Integration/EkolPY.Integration.Manager/TaskBase.cs:line 162
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Tasktask)