V2EX = way to explore
V2EX 是一个关于分享和探索的地方
现在注册
已注册用户请  登录
maymay5
V2EX  ›  .NET

.NET 大牛们,这次遇到了一个涉及知识盲区的 BUG,与 Redis 有关

  •  
  •   maymay5 · 83 天前 · 1582 次点击
    这是一个创建于 83 天前的主题,其中的信息可能已经有所发展或是发生改变。
    • 问题场景:我的 webapi 与 Redis 读写操作有关的 API 全部会 500

    • 检查:Redis 服务一切正常

    • 修复方法:清空所有缓存

    • 无法理解的特殊现象:我有两个 webapi ,一个用的 db5 ,一个用的 db1 ,我猜测是 db5 出了问题,但是 db1 的 api 也一起挂掉了,然后我清空了 db5 ,所有服务全部正常了

    • 下面是小弟 debug 到的一段异常,我问了 GPT ,说这是和 缓存副本 相关的异常,可是我的系统并没有做负载均衡类的部署,所以请教各位大佬,这到底是怎么造成的?

    fail: Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware[1]
          An unhandled exception has occurred while executing the request.
          StackExchange.Redis.RedisCommandException: Command cannot be issued to a replica: DEL cache_4f4bc6e0-4278-d758-4312-3be96b11d34e
             at StackExchange.Redis.ConnectionMultiplexer.PrepareToPushMessageToBridge[T](Message message, ResultProcessor`1 processor, IResultBox`1 resultBox, ServerEndPoint& server) in /_/src/StackExchange.Redis/ConnectionMultiplexer.cs:line 1967
             at StackExchange.Redis.ConnectionMultiplexer.TryPushMessageToBridgeAsync[T](Message message, ResultProcessor`1 processor, IResultBox`1 resultBox, ServerEndPoint& server) in /_/src/StackExchange.Redis/ConnectionMultiplexer.cs:line 2013
             at StackExchange.Redis.ConnectionMultiplexer.ExecuteAsyncImpl[T](Message message, ResultProcessor`1 processor, Object state, ServerEndPoint server) in /_/src/StackExchange.Redis/ConnectionMultiplexer.cs:line 2191
             at StackExchange.Redis.RedisBase.ExecuteAsync[T](Message message, ResultProcessor`1 processor, ServerEndPoint server) in /_/src/StackExchange.Redis/RedisBase.cs:line 54
             at StackExchange.Redis.RedisDatabase.KeyDeleteAsync(RedisKey key, CommandFlags flags) in /_/src/StackExchange.Redis/RedisDatabase.cs:line 758
             at aibotPro.Service.RedisService.DeleteAsync(String key) in /Users/mayday/Desktop/GitHub/AIBot-Pro/AIBot-Pro/aibotPro/aibotPro/Service/RedisService.cs:line 75
             at aibotPro.Service.UsersService.GenerateCodeImage(String account, String key) in /Users/mayday/Desktop/GitHub/AIBot-Pro/AIBot-Pro/aibotPro/aibotPro/Service/UsersService.cs:line 151
             at aibotPro.Controllers.UsersController.GenerateCodeImage(String key) in /Users/mayday/Desktop/GitHub/AIBot-Pro/AIBot-Pro/aibotPro/aibotPro/Controllers/UsersController.cs:line 388
             at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.TaskOfIActionResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
             at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeActionMethodAsync>g__Awaited|12_0(ControllerActionInvoker invoker, ValueTask`1 actionResultValueTask)
             at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
             at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
             at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
             at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeInnerFilterAsync()
          --- End of stack trace from previous location ---
             at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResourceFilter>g__Awaited|25_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
             at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)
             at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
             at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeFilterPipelineAsync()
          --- End of stack trace from previous location ---
             at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
             at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
             at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
             at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
             at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.<Invoke>g__Awaited|6_0(ExceptionHandlerMiddleware middleware, HttpContext context, Task task)
    fail: Microsoft.AspNetCore.Mvc.ViewFeatures.ViewResultExecutor[3]
          The view 'Error' was not found. Searched locations: /Views/Home/Error.cshtml, /Views/Shared/Error.cshtml
    fail: Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware[3]
          An exception was thrown attempting to execute the error handler.
          System.InvalidOperationException: The view 'Error' was not found. The following locations were searched:
          /Views/Home/Error.cshtml
          /Views/Shared/Error.cshtml
             at Microsoft.AspNetCore.Mvc.ViewEngines.ViewEngineResult.EnsureSuccessful(IEnumerable`1 originalLocations)
             at Microsoft.AspNetCore.Mvc.ViewFeatures.ViewResultExecutor.ExecuteAsync(ActionContext context, ViewResult result)
             at Microsoft.AspNetCore.Mvc.ViewResult.ExecuteResultAsync(ActionContext context)
             at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResultFilterAsync>g__Awaited|30_0[TFilter,TFilterAsync](ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
             at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResultExecutedContextSealed context)
             at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.ResultNext[TFilter,TFilterAsync](State& next, Scope& scope, Object& state, Boolean& isCompleted)
             at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeResultFilters()
          --- End of stack trace from previous location ---
             at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResourceFilter>g__Awaited|25_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
             at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)
             at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
             at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeFilterPipelineAsync()
          --- End of stack trace from previous location ---
             at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
             at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
             at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
             at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
             at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.HandleException(HttpContext context, ExceptionDispatchInfo edi)
    
    
    5 条回复    2024-09-12 10:37:54 +08:00
    kk2syc
        1
    kk2syc  
       83 天前
    根据报错 < StackExchange.Redis.RedisCommandException: Command cannot be issued to a replica: DEL cache_4f4bc6e0-4278-d758-4312-3be96b11d34e > 来看
    你如果是定义一个实例的时候同时连接 db1 和 db5 的话,组件会默认把 db5 当作 [只读缓存] 副本,通常副本是不允许写入操作的,于是报错了。
    ----

    1. 你应该定义两个 redis 实例并以单例形式添加到服务中:
    ```
    public static IServiceCollection AddStackExchangeRedis(this IServiceCollection services, IConfiguration configuration)
    {
    ConfigurationOptions options1 = new ConfigurationOptions()
    {
    EndPoints = {
    {Address1, Port1 }
    },
    Password = Password1,
    ClientName = Name1,
    DefaultDatabase = DefaultDatabaseNumber1,
    AllowAdmin = true
    };

    ConfigurationOptions options2 = new ConfigurationOptions()
    {
    EndPoints = {
    { Address2, Port2 }
    },
    Password = Password2,
    ClientName = Name2,
    DefaultDatabase = DefaultDatabaseNumber2,
    AllowAdmin = true
    };
    services.AddSingleton<IConnectionMultiplexer>(ConnectionMultiplexer.Connect(options1));
    services.AddSingleton<IConnectionMultiplexer>(ConnectionMultiplexer.Connect(options2));
    return services;
    }
    ```

    2. 使用 IEnumerable 选择对 ClientName ( Name1 或 Name2 )进行查询的任何 redis 实例:
    ```
    public class YourClass
    {
    private readonly IConnectionMultiplexer _connectionMultiplexer;

    public YourClass(IEnumerable<IConnectionMultiplexer> connectionMultiplexer)
    {
    _connectionMultiplexer= connectionMultiplexer.Where(c => c.ClientName == Name1).First();
    }
    }
    ```
    maymay5
        2
    maymay5  
    OP
       83 天前
    @kk2syc 大佬,场景好像不同,我并不是一个实例连接两个 db ,db1 和 db5 ,完全不是同一个项目,他们是分开部署的两个网站,进程都不是同一个,很奇怪
    kk2syc
        3
    kk2syc  
       83 天前
    @maymay5 那把两个 webapi 相关 redis 操作时候的各个数据都打 log ,看看会不会有参数为空、类型错误等等,造成死循环挤满 redis 连接了?主要是不理解你描述的。假如你的两个 webapi 不共用 db1 和 db5 的数据,那么 db1 、5 是数据独立的,不存在 db5 的业务挂了影响 db1 的业务。清除 db5 就恢复肯定指向 redis 因为什么卡了
    wuhuiwenzi
        4
    wuhuiwenzi  
       82 天前
    furion 做的??我遇到过,要定义两个 redis 实例。。类似这种
    _redisConnZK = new FullRedis();
    _redisConnDS = new FullRedis();
    _redisConnZK.Init(cacheOptions.RedisConnectionStringZK);
    _redisConnDS.Init(cacheOptions.RedisConnectionStringDS);
    maymay5
        5
    maymay5  
    OP
       82 天前
    @wuhuiwenzi 谢谢解答,我现在怀疑是 StackExchange.Redis 库的一个 bug ,因为我两个 db 完全不在同一个进程中,但事实就是我一个进程中的 db5 挂了(也不能说挂了,因为我用客户端工具依旧可以增删改查,就是程序里用不了),另一个使用 db1 的进程也挂了
    关于   ·   帮助文档   ·   博客   ·   API   ·   FAQ   ·   实用小工具   ·   1204 人在线   最高记录 6679   ·     Select Language
    创意工作者们的社区
    World is powered by solitude
    VERSION: 3.9.8.5 · 21ms · UTC 18:20 · PVG 02:20 · LAX 10:20 · JFK 13:20
    Developed with CodeLauncher
    ♥ Do have faith in what you're doing.