首页 注册 登录
V2EX = way to explore
V2EX 是一个关于分享和探索的地方
现在注册
已注册用户请  登录
Mrkon
V2EX  ›  Elasticsearch

求助: 没有到主机的路由(no route to host)

  •   
  •   Mrkon · 2021-10-14 16:57:44 +08:00 · 1747 次点击
    这是一个创建于 1154 天前的主题，其中的信息可能已经有所发展或是发生改变。

    系统：ubuntu 16.04 （已经关闭防火墙，sudo ufw disable, telnet 可以联通） ES 版本 : 7.13.1

    目的： 使用两个节点通过外网组成集群

    问题描述： 已经开启一台服务器作为 master 节点，将第二个节点加入时报错: 没有到主机的路由

    关键配置：

    network.host: 0.0.0.0
network.publish_host: 192.168.3.134
discovery.zen.minimum_master_nodes: 2
discovery.seed_hosts: ["61.15.284.58:10091", "1.65.184.8:10081"]
cluster.initial_master_nodes: ["node-1", "node-2"]

xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12

    我的猜想： DNS 原因，我通过路由器将内网端口映射到固定 IP 导致的问题，报错是找不到内网 192.168.3.86:19300

    映射关系： node-1: 61.15.284.58:10091 -> 192.168.3.86:19300

    node-2: 1.65.184.8:10081 -> 192.168.3.134:29300

    错误 log：

    [2021-09-24T09:29:32,786][WARN ][o.e.d.HandshakingTransportAddressConnector] [
node-2] [connectToRemoteMasterNode[61.15.284.58:10091]] completed handshake with [{node-1}{oapoq8khQ9GtiWj63kfbCQ}{VwXYChZEQdKceJYNrCRPmw}{192.168.3.86}{192.168.3.86:19300}{cdfhilmrstw}{ml.machine_memory=33599610880, ml.max_open_jobs=512, xpack.installed=true, ml.max_jvm_size=12884901888, transform.node=true}] but followup connection failed
org.elasticsearch.transport.ConnectTransportException: [node-1][192.168.3.86:19300] connect_exception
        at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:971) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$0(ActionListener.java:277) ~[elasticsearch-7.13.1.jar:7.13.1]
        at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:31) ~[elasticsearch-core-7.13.1.jar:7.13.1]
        at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
        at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
        at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
        at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2158) ~[?:?]
        at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:46) ~[elasticsearch-core-7.13.1.jar:7.13.1]
        at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$addListener$0(Netty4TcpChannel.java:57) ~[?:?]
        at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:577) ~[?:?]
        at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:570) ~[?:?]
        at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:549) ~[?:?]
        at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:490) ~[?:?]
        at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:615) ~[?:?]
        at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:608) ~[?:?]
        at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:117) ~[?:?]
        at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:321) ~[?:?]
        at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:337) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[?:?]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[?:?]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
        at java.lang.Thread.run(Thread.java:831) [?:?]
Caused by: io.netty.channel.AbstractChannel$AnnotatedNoRouteToHostException: 没有到主机的路由: 192.168.3.86/192.168.3.86:19300
Caused by: java.net.NoRouteToHostException: 没有到主机的路由
        at sun.nio.ch.Net.pollConnect(Native Method) ~[?:?]
        at sun.nio.ch.Net.pollConnectNow(Net.java:669) ~[?:?]
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:944) ~[?:?]
        at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330) ~[?:?]
        at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) ~[?:?]
        ... 7 more
  • Java
  • 7.13.1
  • 192.168.3.86
  • ect
    8 条回复  •  2021-11-12 16:00:58 +08:00
    wujiezero
        1
    wujiezero  
       2021-10-14 17:01:09 +08:00
    我上次遇到这个问题改了下 /etc/hosts 好了
    Mrkon
        2
    Mrkon  
    OP
       2021-10-14 17:23:53 +08:00
    @wujiezero 感谢回复，该如何修改呢？
    julyclyde
        3
    julyclyde  
       2021-10-14 17:30:22 +08:00
    iptables 的问题。对方使用了 ICMP host unreachable 来拒绝连接
    Mrkon
        4
    Mrkon  
    OP
       2021-10-14 17:49:55 +08:00
    @julyclyde 感觉不太像我的 iptables 表分别是

    node1

    ```
    Chain INPUT (policy ACCEPT)
    target prot opt source destination
    ACCEPT tcp -- anywhere anywhere tcp dpt:9600

    Chain FORWARD (policy ACCEPT)
    target prot opt source destination
    DOCKER-USER all -- anywhere anywhere
    DOCKER-ISOLATION-STAGE-1 all -- anywhere anywhere
    ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
    DOCKER all -- anywhere anywhere
    ACCEPT all -- anywhere anywhere
    ACCEPT all -- anywhere anywhere
    ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
    DOCKER all -- anywhere anywhere
    ACCEPT all -- anywhere anywhere
    ACCEPT all -- anywhere anywhere
    ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
    DOCKER all -- anywhere anywhere
    ACCEPT all -- anywhere anywhere
    ACCEPT all -- anywhere anywhere
    ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
    DOCKER all -- anywhere anywhere
    ACCEPT all -- anywhere anywhere
    ACCEPT all -- anywhere anywhere
    ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
    DOCKER all -- anywhere anywhere
    ACCEPT all -- anywhere anywhere
    ACCEPT all -- anywhere anywhere
    ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
    DOCKER all -- anywhere anywhere
    ACCEPT all -- anywhere anywhere
    ACCEPT all -- anywhere anywhere
    ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
    DOCKER all -- anywhere anywhere
    ACCEPT all -- anywhere anywhere
    ACCEPT all -- anywhere anywhere
    ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
    DOCKER all -- anywhere anywhere
    ACCEPT all -- anywhere anywhere
    ACCEPT all -- anywhere anywhere
    ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
    DOCKER all -- anywhere anywhere
    ACCEPT all -- anywhere anywhere
    ACCEPT all -- anywhere anywhere
    ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
    DOCKER all -- anywhere anywhere
    ACCEPT all -- anywhere anywhere
    ACCEPT all -- anywhere anywhere
    ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
    DOCKER all -- anywhere anywhere
    ACCEPT all -- anywhere anywhere
    ACCEPT all -- anywhere anywhere

    Chain OUTPUT (policy ACCEPT)
    target prot opt source destination

    Chain DOCKER (11 references)
    target prot opt source destination
    ACCEPT tcp -- anywhere 172.21.0.2 tcp dpt:tproxy
    ACCEPT tcp -- anywhere 172.21.0.2 tcp dpt:http-alt
    ACCEPT tcp -- anywhere 192.168.32.2 tcp dpt:3000
    ACCEPT tcp -- anywhere 172.20.0.2 tcp dpt:6379
    ACCEPT tcp -- anywhere 172.24.0.2 tcp dpt:7687
    ACCEPT tcp -- anywhere 172.24.0.2 tcp dpt:7474
    ACCEPT tcp -- anywhere 172.24.0.3 tcp dpt:27017
    ACCEPT tcp -- anywhere 172.24.0.4 tcp dpt:6379
    ACCEPT tcp -- anywhere 172.24.0.5 tcp dpt:5000
    ACCEPT tcp -- anywhere 172.24.0.5 tcp dpt:3000
    ACCEPT tcp -- anywhere 172.24.0.5 tcp dpt:ssh
    ACCEPT tcp -- anywhere 172.20.0.4 tcp dpt:3000
    ACCEPT tcp -- anywhere 172.20.0.3 tcp dpt:5000

    Chain DOCKER-ISOLATION-STAGE-1 (1 references)
    target prot opt source destination
    DOCKER-ISOLATION-STAGE-2 all -- anywhere anywhere
    DOCKER-ISOLATION-STAGE-2 all -- anywhere anywhere
    DOCKER-ISOLATION-STAGE-2 all -- anywhere anywhere
    DOCKER-ISOLATION-STAGE-2 all -- anywhere anywhere
    DOCKER-ISOLATION-STAGE-2 all -- anywhere anywhere
    DOCKER-ISOLATION-STAGE-2 all -- anywhere anywhere
    DOCKER-ISOLATION-STAGE-2 all -- anywhere anywhere
    DOCKER-ISOLATION-STAGE-2 all -- anywhere anywhere
    DOCKER-ISOLATION-STAGE-2 all -- anywhere anywhere
    DOCKER-ISOLATION-STAGE-2 all -- anywhere anywhere
    DOCKER-ISOLATION-STAGE-2 all -- anywhere anywhere
    RETURN all -- anywhere anywhere

    Chain DOCKER-ISOLATION-STAGE-2 (11 references)
    target prot opt source destination
    DROP all -- anywhere anywhere
    DROP all -- anywhere anywhere
    DROP all -- anywhere anywhere
    DROP all -- anywhere anywhere
    DROP all -- anywhere anywhere
    DROP all -- anywhere anywhere
    DROP all -- anywhere anywhere
    DROP all -- anywhere anywhere
    DROP all -- anywhere anywhere
    DROP all -- anywhere anywhere
    DROP all -- anywhere anywhere
    RETURN all -- anywhere anywhere

    Chain DOCKER-USER (1 references)
    target prot opt source destination
    RETURN all -- anywhere anywhere
    ```

    node2

    ```
    Chain INPUT (policy ACCEPT)
    target prot opt source destination

    Chain FORWARD (policy ACCEPT)
    target prot opt source destination

    Chain OUTPUT (policy ACCEPT)
    target prot opt source destination
    ```
    julyclyde
        5
    julyclyde  
       2021-10-15 11:06:22 +08:00
    看 192.168.3.86 那边的 iptables
    @Mrkon
    Mrkon
        6
    Mrkon  
    OP
       2021-10-17 14:46:55 +08:00
    @julyclyde 192.168.3.86 的 iptables 是：上面 node1 的，就是 ACCEPT 。我实在是找不到问题点了，能帮我看下吗？

    还有如果安装了 docker，当 es network.host 设置为：0.0.0.0 时，好像默认地址就会绑定为 docker 的 IP 段时怎么回事呢？
    julyclyde
        7
    julyclyde  
       2021-10-18 11:24:14 +08:00
    @Mrkon 那你抓包看看吧。看看是不是 ICMP host unreacable 、是谁发过来的
    Mrkon
        8
    Mrkon  
    OP
       2021-11-12 16:00:58 +08:00
    看了下官方文档，问题解决了，需要将所有节点 network.publish_host 设置映射前的 ip 地址 例如：61.15.284.58 ，transport.publish_port 同样需要设置 例如：10091
    关于   ·   帮助文档   ·   博客   ·   API   ·   FAQ   ·   实用小工具   ·   956 人在线   最高记录 6679   ·     Select Language
    创意工作者们的社区
    World is powered by solitude
    VERSION: 3.9.8.5 · 23ms · UTC 23:00 · PVG 07:00 · LAX 15:00 · JFK 18:00
    Developed with CodeLauncher
    ♥ Do have faith in what you're doing.