deepseek-ai/3FS
1
0
Fork 0
mirror of https://github.com/deepseek-ai/3FS.git synced 2026-11-05 00:47:47 +00:00
Code Issues 166 Projects Packages Wiki Activity

[GH-ISSUE #331] Monitor Service 的 Clickhouse Client 初始化失败：Failed to initialize ClickHouse: closed: Operation now in progress #202

New issue
Closed
opened 2026-10-26 09:47:05 +00:00 by bsakaguchi · 2 comments
bsakaguchi commented 2026-10-26 09:47:05 +00:00
Owner
Copy link

Originally created by @wangzhuzhen on GitHub (Aug 27, 2025).
Original GitHub issue: https://github.com/deepseek-ai/3FS/issues/331

Monitor Service 的 Clickhouse Client 初始化失败

  1. monitor_collector_main 运行日志报错
[2025-08-27T02:46:40.162171920+00:00 monitor_collect:12608 OnePhaseApplication.h:91 INFO] [server.monitor_collector.reporter.clickhouse]
[2025-08-27T02:46:40.162171920+00:00 monitor_collect:12608 OnePhaseApplication.h:91 INFO] db = '3fs'
[2025-08-27T02:46:40.162171920+00:00 monitor_collect:12608 OnePhaseApplication.h:91 INFO] host = 'localhost'
[2025-08-27T02:46:40.162171920+00:00 monitor_collect:12608 OnePhaseApplication.h:91 INFO] passwd = 'admin123'
[2025-08-27T02:46:40.162171920+00:00 monitor_collect:12608 OnePhaseApplication.h:91 INFO] port = '8123'
[2025-08-27T02:46:40.162171920+00:00 monitor_collect:12608 OnePhaseApplication.h:91 INFO] user = 'default'
[2025-08-27T02:46:40.166410560+00:00 monitor_collect:12608 OnePhaseApplication.h:93 INFO] Init waiter singleton 0x557299282d50
[2025-08-27T02:46:40.175100797+00:00 monitor_collect:12608 ConstructLog.h:11 INFO] Construct net::Server
[2025-08-27T02:46:40.175114439+00:00 monitor_collect:12608 ConstructLog.h:11 INFO] Construct net::ThreadPoolGroup
[2025-08-27T02:46:40.180332267+00:00 monitor_collect:12608 ConstructLog.h:11 INFO] Construct CPUExecutorGroup
[2025-08-27T02:46:40.189049229+00:00 monitor_collect:12608 ConstructLog.h:11 INFO] Construct CPUExecutorGroup
[2025-08-27T02:46:40.197676322+00:00 monitor_collect:12608 ConstructLog.h:11 INFO] Construct CPUExecutorGroup
[2025-08-27T02:46:40.214699549+00:00 monitor_collect:12608 ConstructLog.h:11 INFO] Construct net::ThreadPoolGroup
[2025-08-27T02:46:40.218742392+00:00 monitor_collect:12608 ConstructLog.h:11 INFO] Construct CPUExecutorGroup
[2025-08-27T02:46:40.226345189+00:00 monitor_collect:12608 ConstructLog.h:11 INFO] Construct CPUExecutorGroup
[2025-08-27T02:46:40.233531795+00:00 monitor_collect:12608 ConstructLog.h:11 INFO] Construct CPUExecutorGroup
[2025-08-27T02:46:40.248060410+00:00 monitor_collect:12608 ConstructLog.h:11 INFO] Construct net::ServiceGroup
[2025-08-27T02:46:40.248070408+00:00 monitor_collect:12608 ConstructLog.h:11 INFO] Construct serde::Services
[2025-08-27T02:46:40.252853666+00:00 monitor_collect:12608 Server.cc:31 INFO] Server::setup ...
[2025-08-27T02:46:40.252862725+00:00 monitor_collect:12608 Server.cc:27 INFO] Setup group (MonitorCollector) ...
[2025-08-27T02:46:40.253148492+00:00 monitor_collect:12608 Listener.cc:110 INFO] Listener setup: TCP://100.70.8.25:10000
[2025-08-27T02:46:40.253159349+00:00 monitor_collect:12608 Server.cc:27 INFO] Setup group (MonitorCollector) finished
[2025-08-27T02:46:40.253163338+00:00 monitor_collect:12608 Server.cc:31 INFO] Server::setup finished
[2025-08-27T02:46:40.253185185+00:00 monitor_collect:12608 OnePhaseApplication.h:118 INFO] Server 0x55729b2ca200 has 1 groups:
[2025-08-27T02:46:40.253185185+00:00 monitor_collect:12608 OnePhaseApplication.h:118 INFO] Group 0
[2025-08-27T02:46:40.253185185+00:00 monitor_collect:12608 OnePhaseApplication.h:118 INFO]   Service: MonitorCollector
[2025-08-27T02:46:40.253185185+00:00 monitor_collect:12608 OnePhaseApplication.h:118 INFO]   Listening: TCP://100.70.8.25:10000
[2025-08-27T02:46:40.253185185+00:00 monitor_collect:12608 OnePhaseApplication.h:118 INFO] 
[2025-08-27T02:46:40.253194386+00:00 monitor_collect:12608 Server.cc:39 INFO] Server::beforeStart ...
[2025-08-27T02:46:40.346604089+00:00 monitor_collect:12608 Server.cc:39 INFO] Server::beforeStart finished
[2025-08-27T02:46:40.346615051+00:00 monitor_collect:12608 Server.cc:42 INFO] Start group (MonitorCollector) ...
[2025-08-27T02:46:40.346729412+00:00 monitor_collect:14243 MonitorCollectorOperator.cc:91 INFO] Sample queue capacity: 0 / 204800.
[2025-08-27T02:46:40.349094685+00:00 monitor_collect:12608 Server.cc:42 INFO] Start group (MonitorCollector) finished
[2025-08-27T02:46:40.349100937+00:00 monitor_collect:12608 Server.cc:44 INFO] Server::afterStart ...
[2025-08-27T02:46:40.349103762+00:00 monitor_collect:12608 Server.cc:44 INFO] Server::afterStart finished
[2025-08-27T02:46:40.349199313+00:00 SvrEL0:14244 EventLoop.cc:116 INFO] EventLoop::loop() started.
[2025-08-27T02:46:45.347066832+00:00 monitor_collect:14243 MonitorCollectorOperator.cc:91 INFO] Sample queue capacity: 0 / 204800.
[2025-08-27T02:46:50.347197430+00:00 monitor_collect:14243 MonitorCollectorOperator.cc:91 INFO] Sample queue capacity: 0 / 204800.
[2025-08-27T02:46:55.347444036+00:00 monitor_collect:14243 MonitorCollectorOperator.cc:91 INFO] Sample queue capacity: 0 / 204800.
[2025-08-27T02:47:00.347644985+00:00 monitor_collect:14243 MonitorCollectorOperator.cc:91 INFO] Sample queue capacity: 0 / 204800.
[2025-08-27T02:47:05.347871604+00:00 monitor_collect:14243 MonitorCollectorOperator.cc:91 INFO] Sample queue capacity: 0 / 204800.
[2025-08-27T02:47:10.348074822+00:00 monitor_collect:14243 MonitorCollectorOperator.cc:91 INFO] Sample queue capacity: 0 / 204800.
[2025-08-27T02:47:15.348231494+00:00 monitor_collect:14243 MonitorCollectorOperator.cc:91 INFO] Sample queue capacity: 0 / 204800.
[2025-08-27T02:47:20.348364582+00:00 monitor_collect:14243 MonitorCollectorOperator.cc:91 INFO] Sample queue capacity: 0 / 204800.
[2025-08-27T02:47:25.348512764+00:00 monitor_collect:14243 MonitorCollectorOperator.cc:91 INFO] Sample queue capacity: 0 / 204800.
[2025-08-27T02:47:30.348645005+00:00 monitor_collect:14243 MonitorCollectorOperator.cc:91 INFO] Sample queue capacity: 0 / 204800.
[2025-08-27T02:47:35.348871089+00:00 monitor_collect:14243 MonitorCollectorOperator.cc:91 INFO] Sample queue capacity: 0 / 204800.
[2025-08-27T02:47:40.349213960+00:00 monitor_collect:14243 MonitorCollectorOperator.cc:91 INFO] Sample queue capacity: 0 / 204800.
[2025-08-27T02:47:45.282377857+00:00 monitor_collect:14204 ClickHouseClient.cc:25 ERROR] Failed to initialize ClickHouse: closed: Operation now in progress
[2025-08-27T02:47:45.282513602+00:00 monitor_collect:14204 MonitorCollectorOperator.cc:49 FATAL] Initializing reporter failed. MonitorInitFailed(12)
  1. clickhouse-server 错误日志：
2025.08.27 02:51:33.556638 [ 12683 ] {} <Error> ServerErrorHandler: Code: 209. DB::NetException: Timeout exceeded while reading from socket (peer: 127.0.0.1:57222, local: 127.0.0.1:8123, 30000 ms). (SOCKET_TIMEOUT), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000fbf72fb
1. DB::NetException::NetException<String, String, long>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<long>::type>, String&&, String&&, long&&) @ 0x000000000fdeb4a6
2. DB::ReadBufferFromPocoSocketBase::socketReceiveBytesImpl(char*, unsigned long) @ 0x000000000fdeb07e
3. DB::ReadBufferFromPocoSocketBase::nextImpl() @ 0x000000000fdeb715
4. DB::HTTPServerRequest::HTTPServerRequest(std::shared_ptr<DB::IHTTPContext>, DB::HTTPServerResponse&, Poco::Net::HTTPServerSession&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x00000000155c8ccc
5. DB::HTTPServerConnection::run() @ 0x00000000155c656e
6. Poco::Net::TCPServerConnection::start() @ 0x0000000018d04d87
7. Poco::Net::TCPServerDispatcher::run() @ 0x0000000018d051d9
8. Poco::PooledThread::run() @ 0x0000000018cd01db
9. Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000018cce6bd
10. ? @ 0x0000000000094ac3
11. ? @ 0x0000000000126850
 (version 25.6.4.12 (official build))
  1. Clickhouse 正常运行（采用默认配置），客户端访问正常
    Clickhouse API 调用正常
# curl "http://default:admin123@localhost:8123?query=SELECT+name+FROM+system.databases"
3fs
INFORMATION_SCHEMA
default
information_schema
system

Clickhouse-client 访问正常

# clickhouse-client -h 127.0.0.1 --password admin123
ClickHouse client version 25.6.4.12 (official build).
Connecting to 127.0.0.1:9000 as user default.
Connected to ClickHouse server version 25.6.4.

Warnings:
 * Delay accounting is not enabled, OSIOWaitMicroseconds will not be gathered. You can enable it using `echo 1 > /proc/sys/kernel/task_delayacct` or by using sysctl.

GPU25-N204-C01-6U :) show databases;

SHOW DATABASES

Query id: e1c8a7b5-0462-4f42-a3a6-cdd20d8c324e

   ┌─name───────────────┐
1. │ 3fs                │
2. │ INFORMATION_SCHEMA │
3. │ default            │
4. │ information_schema │
5. │ system             │
   └────────────────────┘

5 rows in set. Elapsed: 0.005 sec. 

GPU25-N204-C01-6U :) use 3fs;

USE `3fs`

Query id: 9da90389-29c0-45cd-9e77-8970aedfd9e5

Ok.

0 rows in set. Elapsed: 0.003 sec. 

GPU25-N204-C01-6U :) show tables;

SHOW TABLES

Query id: 870a2f39-83ab-4977-8423-493a1972d7ed

   ┌─name──────────┐
1. │ counters      │
2. │ distributions │
   └───────────────┘

2 rows in set. Elapsed: 0.005 sec. 

GPU25-N204-C01-6U :) 
```


4.  monitor service 配置文件内容如下：
```
[common]
cluster_id = 'stage'

[common.ib_devices]
allow_unknown_zone = true
default_network_zone = 'UNKNOWN'
device_filter = ['enp218s0np0']
subnets = []

[[common.log.categories]]
categories = [ '.' ]
handlers = [ 'normal', 'err', 'fatal' ]
inherit = true
level = 'INFO'
propagate = 'NONE'

[[common.log.handlers]]
async = true
file_path = '/var/log/3fs/monitor_collector_main.log'
max_file_size = '100MB'
max_files = 10
name = 'normal'
rotate = true
rotate_on_open = false
start_level = 'NONE'
stream_type = 'STDERR'
writer_type = 'FILE'

[[common.log.handlers]]
async = false
file_path = '/var/log/3fs/monitor_collector_main-err.log'
max_file_size = '100MB'
max_files = 10
name = 'err'
rotate = true
rotate_on_open = false
start_level = 'ERR'
stream_type = 'STDERR'
writer_type = 'FILE'

[[common.log.handlers]]
async = false
file_path = '/var/log/3fs/monitor_collector_main-fatal.log'
max_file_size = '100MB'
max_files = 10
name = 'fatal'
rotate = true
rotate_on_open = false
start_level = 'FATAL'
stream_type = 'STDERR'
writer_type = 'STREAM'

[server.base.independent_thread_pool]
bg_thread_pool_stratetry = 'SHARED_QUEUE'
collect_stats = false
enable_work_stealing = false
io_thread_pool_stratetry = 'SHARED_QUEUE'
num_bg_threads = 2
num_connect_threads = 2
num_io_threads = 2
num_proc_threads = 2
proc_thread_pool_stratetry = 'SHARED_QUEUE'

[server.base.thread_pool]
bg_thread_pool_stratetry = 'SHARED_QUEUE'
collect_stats = false
enable_work_stealing = false
io_thread_pool_stratetry = 'SHARED_QUEUE'
num_bg_threads = 2
num_connect_threads = 2
num_io_threads = 2
num_proc_threads = 2
proc_thread_pool_stratetry = 'SHARED_QUEUE'

[[server.base.groups]]
#default_timeout = '1s'
#drop_connections_interval = '1h'
network_type = 'TCP'
services = [ 'MonitorCollector' ]
use_independent_thread_pool = false

[server.base.groups.io_worker]
num_event_loop = 1
rdma_connect_timeout = '5s'
read_write_rdma_in_event_thread = false
read_write_tcp_in_event_thread = false
tcp_connect_timeout = '1s'
wait_to_retry_send = '100ms'

[server.base.groups.io_worker.ibsocket]
buf_ack_batch = 8
buf_signal_batch = 8
buf_size = 16384
drop_connections = 0
event_ack_batch = 128
#gid_index = 0
max_rd_atomic = 16
max_rdma_wr = 128
max_rdma_wr_per_post = 32
max_sge = 16
min_rnr_timer = 1
pkey_index = 0
record_bytes_per_peer = false
record_latency_per_peer = false
retry_cnt = 7
rnr_retry = 0
send_buf_cnt = 32
sl = 0
start_psn = 0
timeout = 14
traffic_class = 0

[server.base.groups.io_worker.transport_pool]
max_connections = 1

[server.base.groups.listener]
filter_list = ['enp218s0np0']
listen_port = 10000
listen_queue_depth = 4096
rdma_listen_ethernet = true
reuse_port = false

[server.base.groups.processor]
enable_coroutines_pool = true
max_coroutines_num = 256
max_processing_requests_num = 4096

[server.monitor_collector]
batch_commit_size = 4096
conn_threads = 32
queue_capacity = 204800

[server.monitor_collector.reporter]
type = 'clickhouse'

[server.monitor_collector.reporter.clickhouse]
db = '3fs'
host = 'localhost'
passwd = 'admin123'
port = '8123'
user = 'default'
```
Originally created by @wangzhuzhen on GitHub (Aug 27, 2025). Original GitHub issue: https://github.com/deepseek-ai/3FS/issues/331 Monitor Service 的 Clickhouse Client 初始化失败 1. monitor_collector_main 运行日志报错 ``` [2025-08-27T02:46:40.162171920+00:00 monitor_collect:12608 OnePhaseApplication.h:91 INFO] [server.monitor_collector.reporter.clickhouse] [2025-08-27T02:46:40.162171920+00:00 monitor_collect:12608 OnePhaseApplication.h:91 INFO] db = '3fs' [2025-08-27T02:46:40.162171920+00:00 monitor_collect:12608 OnePhaseApplication.h:91 INFO] host = 'localhost' [2025-08-27T02:46:40.162171920+00:00 monitor_collect:12608 OnePhaseApplication.h:91 INFO] passwd = 'admin123' [2025-08-27T02:46:40.162171920+00:00 monitor_collect:12608 OnePhaseApplication.h:91 INFO] port = '8123' [2025-08-27T02:46:40.162171920+00:00 monitor_collect:12608 OnePhaseApplication.h:91 INFO] user = 'default' [2025-08-27T02:46:40.166410560+00:00 monitor_collect:12608 OnePhaseApplication.h:93 INFO] Init waiter singleton 0x557299282d50 [2025-08-27T02:46:40.175100797+00:00 monitor_collect:12608 ConstructLog.h:11 INFO] Construct net::Server [2025-08-27T02:46:40.175114439+00:00 monitor_collect:12608 ConstructLog.h:11 INFO] Construct net::ThreadPoolGroup [2025-08-27T02:46:40.180332267+00:00 monitor_collect:12608 ConstructLog.h:11 INFO] Construct CPUExecutorGroup [2025-08-27T02:46:40.189049229+00:00 monitor_collect:12608 ConstructLog.h:11 INFO] Construct CPUExecutorGroup [2025-08-27T02:46:40.197676322+00:00 monitor_collect:12608 ConstructLog.h:11 INFO] Construct CPUExecutorGroup [2025-08-27T02:46:40.214699549+00:00 monitor_collect:12608 ConstructLog.h:11 INFO] Construct net::ThreadPoolGroup [2025-08-27T02:46:40.218742392+00:00 monitor_collect:12608 ConstructLog.h:11 INFO] Construct CPUExecutorGroup [2025-08-27T02:46:40.226345189+00:00 monitor_collect:12608 ConstructLog.h:11 INFO] Construct CPUExecutorGroup [2025-08-27T02:46:40.233531795+00:00 monitor_collect:12608 ConstructLog.h:11 INFO] Construct CPUExecutorGroup [2025-08-27T02:46:40.248060410+00:00 monitor_collect:12608 ConstructLog.h:11 INFO] Construct net::ServiceGroup [2025-08-27T02:46:40.248070408+00:00 monitor_collect:12608 ConstructLog.h:11 INFO] Construct serde::Services [2025-08-27T02:46:40.252853666+00:00 monitor_collect:12608 Server.cc:31 INFO] Server::setup ... [2025-08-27T02:46:40.252862725+00:00 monitor_collect:12608 Server.cc:27 INFO] Setup group (MonitorCollector) ... [2025-08-27T02:46:40.253148492+00:00 monitor_collect:12608 Listener.cc:110 INFO] Listener setup: TCP://100.70.8.25:10000 [2025-08-27T02:46:40.253159349+00:00 monitor_collect:12608 Server.cc:27 INFO] Setup group (MonitorCollector) finished [2025-08-27T02:46:40.253163338+00:00 monitor_collect:12608 Server.cc:31 INFO] Server::setup finished [2025-08-27T02:46:40.253185185+00:00 monitor_collect:12608 OnePhaseApplication.h:118 INFO] Server 0x55729b2ca200 has 1 groups: [2025-08-27T02:46:40.253185185+00:00 monitor_collect:12608 OnePhaseApplication.h:118 INFO] Group 0 [2025-08-27T02:46:40.253185185+00:00 monitor_collect:12608 OnePhaseApplication.h:118 INFO] Service: MonitorCollector [2025-08-27T02:46:40.253185185+00:00 monitor_collect:12608 OnePhaseApplication.h:118 INFO] Listening: TCP://100.70.8.25:10000 [2025-08-27T02:46:40.253185185+00:00 monitor_collect:12608 OnePhaseApplication.h:118 INFO] [2025-08-27T02:46:40.253194386+00:00 monitor_collect:12608 Server.cc:39 INFO] Server::beforeStart ... [2025-08-27T02:46:40.346604089+00:00 monitor_collect:12608 Server.cc:39 INFO] Server::beforeStart finished [2025-08-27T02:46:40.346615051+00:00 monitor_collect:12608 Server.cc:42 INFO] Start group (MonitorCollector) ... [2025-08-27T02:46:40.346729412+00:00 monitor_collect:14243 MonitorCollectorOperator.cc:91 INFO] Sample queue capacity: 0 / 204800. [2025-08-27T02:46:40.349094685+00:00 monitor_collect:12608 Server.cc:42 INFO] Start group (MonitorCollector) finished [2025-08-27T02:46:40.349100937+00:00 monitor_collect:12608 Server.cc:44 INFO] Server::afterStart ... [2025-08-27T02:46:40.349103762+00:00 monitor_collect:12608 Server.cc:44 INFO] Server::afterStart finished [2025-08-27T02:46:40.349199313+00:00 SvrEL0:14244 EventLoop.cc:116 INFO] EventLoop::loop() started. [2025-08-27T02:46:45.347066832+00:00 monitor_collect:14243 MonitorCollectorOperator.cc:91 INFO] Sample queue capacity: 0 / 204800. [2025-08-27T02:46:50.347197430+00:00 monitor_collect:14243 MonitorCollectorOperator.cc:91 INFO] Sample queue capacity: 0 / 204800. [2025-08-27T02:46:55.347444036+00:00 monitor_collect:14243 MonitorCollectorOperator.cc:91 INFO] Sample queue capacity: 0 / 204800. [2025-08-27T02:47:00.347644985+00:00 monitor_collect:14243 MonitorCollectorOperator.cc:91 INFO] Sample queue capacity: 0 / 204800. [2025-08-27T02:47:05.347871604+00:00 monitor_collect:14243 MonitorCollectorOperator.cc:91 INFO] Sample queue capacity: 0 / 204800. [2025-08-27T02:47:10.348074822+00:00 monitor_collect:14243 MonitorCollectorOperator.cc:91 INFO] Sample queue capacity: 0 / 204800. [2025-08-27T02:47:15.348231494+00:00 monitor_collect:14243 MonitorCollectorOperator.cc:91 INFO] Sample queue capacity: 0 / 204800. [2025-08-27T02:47:20.348364582+00:00 monitor_collect:14243 MonitorCollectorOperator.cc:91 INFO] Sample queue capacity: 0 / 204800. [2025-08-27T02:47:25.348512764+00:00 monitor_collect:14243 MonitorCollectorOperator.cc:91 INFO] Sample queue capacity: 0 / 204800. [2025-08-27T02:47:30.348645005+00:00 monitor_collect:14243 MonitorCollectorOperator.cc:91 INFO] Sample queue capacity: 0 / 204800. [2025-08-27T02:47:35.348871089+00:00 monitor_collect:14243 MonitorCollectorOperator.cc:91 INFO] Sample queue capacity: 0 / 204800. [2025-08-27T02:47:40.349213960+00:00 monitor_collect:14243 MonitorCollectorOperator.cc:91 INFO] Sample queue capacity: 0 / 204800. [2025-08-27T02:47:45.282377857+00:00 monitor_collect:14204 ClickHouseClient.cc:25 ERROR] Failed to initialize ClickHouse: closed: Operation now in progress [2025-08-27T02:47:45.282513602+00:00 monitor_collect:14204 MonitorCollectorOperator.cc:49 FATAL] Initializing reporter failed. MonitorInitFailed(12) ``` 2. clickhouse-server 错误日志： ``` 2025.08.27 02:51:33.556638 [ 12683 ] {} <Error> ServerErrorHandler: Code: 209. DB::NetException: Timeout exceeded while reading from socket (peer: 127.0.0.1:57222, local: 127.0.0.1:8123, 30000 ms). (SOCKET_TIMEOUT), Stack trace (when copying this message, always include the lines below): 0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000fbf72fb 1. DB::NetException::NetException<String, String, long>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<long>::type>, String&&, String&&, long&&) @ 0x000000000fdeb4a6 2. DB::ReadBufferFromPocoSocketBase::socketReceiveBytesImpl(char*, unsigned long) @ 0x000000000fdeb07e 3. DB::ReadBufferFromPocoSocketBase::nextImpl() @ 0x000000000fdeb715 4. DB::HTTPServerRequest::HTTPServerRequest(std::shared_ptr<DB::IHTTPContext>, DB::HTTPServerResponse&, Poco::Net::HTTPServerSession&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x00000000155c8ccc 5. DB::HTTPServerConnection::run() @ 0x00000000155c656e 6. Poco::Net::TCPServerConnection::start() @ 0x0000000018d04d87 7. Poco::Net::TCPServerDispatcher::run() @ 0x0000000018d051d9 8. Poco::PooledThread::run() @ 0x0000000018cd01db 9. Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000018cce6bd 10. ? @ 0x0000000000094ac3 11. ? @ 0x0000000000126850 (version 25.6.4.12 (official build)) ``` 3. Clickhouse 正常运行（采用默认配置），客户端访问正常 Clickhouse API 调用正常 ``` # curl "http://default:admin123@localhost:8123?query=SELECT+name+FROM+system.databases" 3fs INFORMATION_SCHEMA default information_schema system ``` Clickhouse-client 访问正常 ```` # clickhouse-client -h 127.0.0.1 --password admin123 ClickHouse client version 25.6.4.12 (official build). Connecting to 127.0.0.1:9000 as user default. Connected to ClickHouse server version 25.6.4. Warnings: * Delay accounting is not enabled, OSIOWaitMicroseconds will not be gathered. You can enable it using `echo 1 > /proc/sys/kernel/task_delayacct` or by using sysctl. GPU25-N204-C01-6U :) show databases; SHOW DATABASES Query id: e1c8a7b5-0462-4f42-a3a6-cdd20d8c324e ┌─name───────────────┐ 1. │ 3fs │ 2. │ INFORMATION_SCHEMA │ 3. │ default │ 4. │ information_schema │ 5. │ system │ └────────────────────┘ 5 rows in set. Elapsed: 0.005 sec. GPU25-N204-C01-6U :) use 3fs; USE `3fs` Query id: 9da90389-29c0-45cd-9e77-8970aedfd9e5 Ok. 0 rows in set. Elapsed: 0.003 sec. GPU25-N204-C01-6U :) show tables; SHOW TABLES Query id: 870a2f39-83ab-4977-8423-493a1972d7ed ┌─name──────────┐ 1. │ counters │ 2. │ distributions │ └───────────────┘ 2 rows in set. Elapsed: 0.005 sec. GPU25-N204-C01-6U :) ``` 4. monitor service 配置文件内容如下： ``` [common] cluster_id = 'stage' [common.ib_devices] allow_unknown_zone = true default_network_zone = 'UNKNOWN' device_filter = ['enp218s0np0'] subnets = [] [[common.log.categories]] categories = [ '.' ] handlers = [ 'normal', 'err', 'fatal' ] inherit = true level = 'INFO' propagate = 'NONE' [[common.log.handlers]] async = true file_path = '/var/log/3fs/monitor_collector_main.log' max_file_size = '100MB' max_files = 10 name = 'normal' rotate = true rotate_on_open = false start_level = 'NONE' stream_type = 'STDERR' writer_type = 'FILE' [[common.log.handlers]] async = false file_path = '/var/log/3fs/monitor_collector_main-err.log' max_file_size = '100MB' max_files = 10 name = 'err' rotate = true rotate_on_open = false start_level = 'ERR' stream_type = 'STDERR' writer_type = 'FILE' [[common.log.handlers]] async = false file_path = '/var/log/3fs/monitor_collector_main-fatal.log' max_file_size = '100MB' max_files = 10 name = 'fatal' rotate = true rotate_on_open = false start_level = 'FATAL' stream_type = 'STDERR' writer_type = 'STREAM' [server.base.independent_thread_pool] bg_thread_pool_stratetry = 'SHARED_QUEUE' collect_stats = false enable_work_stealing = false io_thread_pool_stratetry = 'SHARED_QUEUE' num_bg_threads = 2 num_connect_threads = 2 num_io_threads = 2 num_proc_threads = 2 proc_thread_pool_stratetry = 'SHARED_QUEUE' [server.base.thread_pool] bg_thread_pool_stratetry = 'SHARED_QUEUE' collect_stats = false enable_work_stealing = false io_thread_pool_stratetry = 'SHARED_QUEUE' num_bg_threads = 2 num_connect_threads = 2 num_io_threads = 2 num_proc_threads = 2 proc_thread_pool_stratetry = 'SHARED_QUEUE' [[server.base.groups]] #default_timeout = '1s' #drop_connections_interval = '1h' network_type = 'TCP' services = [ 'MonitorCollector' ] use_independent_thread_pool = false [server.base.groups.io_worker] num_event_loop = 1 rdma_connect_timeout = '5s' read_write_rdma_in_event_thread = false read_write_tcp_in_event_thread = false tcp_connect_timeout = '1s' wait_to_retry_send = '100ms' [server.base.groups.io_worker.ibsocket] buf_ack_batch = 8 buf_signal_batch = 8 buf_size = 16384 drop_connections = 0 event_ack_batch = 128 #gid_index = 0 max_rd_atomic = 16 max_rdma_wr = 128 max_rdma_wr_per_post = 32 max_sge = 16 min_rnr_timer = 1 pkey_index = 0 record_bytes_per_peer = false record_latency_per_peer = false retry_cnt = 7 rnr_retry = 0 send_buf_cnt = 32 sl = 0 start_psn = 0 timeout = 14 traffic_class = 0 [server.base.groups.io_worker.transport_pool] max_connections = 1 [server.base.groups.listener] filter_list = ['enp218s0np0'] listen_port = 10000 listen_queue_depth = 4096 rdma_listen_ethernet = true reuse_port = false [server.base.groups.processor] enable_coroutines_pool = true max_coroutines_num = 256 max_processing_requests_num = 4096 [server.monitor_collector] batch_commit_size = 4096 conn_threads = 32 queue_capacity = 204800 [server.monitor_collector.reporter] type = 'clickhouse' [server.monitor_collector.reporter.clickhouse] db = '3fs' host = 'localhost' passwd = 'admin123' port = '8123' user = 'default' ```
bsakaguchi commented 2026-10-26 09:47:06 +00:00
Author
Owner
Copy link

@wangzhuzhen commented on GitHub (Aug 27, 2025):

环境是 k8s 环境，也尝试过其他方式部署 clickhouse，包括 clickhouse-operator 的方式，docker run 命令行改成 statefulset 方式部署的 clickhouse，包括上面列出的 systemctl service 方式部署的 clickhouse，几种方式下，现象完全相同。

clickhouse 也增加了如下配置。仍然没有解决这个问题，目前不知道该如何解决。

# cat  /etc/clickhouse-server/users.d/http_send_timeout.xml 
<?xml version="1.0" ?>
<clickhouse>
    <profiles>
        <default>
            <http_send_timeout>3600</http_send_timeout>
        </default>
    </profiles>
</clickhouse>
<!-- gh-comment-id:3226585290 --> @wangzhuzhen commented on GitHub (Aug 27, 2025): 环境是 k8s 环境，也尝试过其他方式部署 clickhouse，包括 clickhouse-operator 的方式，docker run 命令行改成 statefulset 方式部署的 clickhouse，包括上面列出的 systemctl service 方式部署的 clickhouse，几种方式下，现象完全相同。 clickhouse 也增加了如下配置。仍然没有解决这个问题，目前不知道该如何解决。 ``` # cat /etc/clickhouse-server/users.d/http_send_timeout.xml <?xml version="1.0" ?> <clickhouse> <profiles> <default> <http_send_timeout>3600</http_send_timeout> </default> </profiles> </clickhouse> ```
bsakaguchi commented 2026-10-26 09:47:06 +00:00
Author
Owner
Copy link

@wangzhuzhen commented on GitHub (Aug 27, 2025):

解决了，Monitor Service 配置文件中 clickhouse 的端口用错了，server.monitor_collector.reporter.clickhouse 中的 port 应该用 Clickouse 的 tcp_port (默认 9000)，而不是用 http_port(默认 8123)

<!-- gh-comment-id:3226640200 --> @wangzhuzhen commented on GitHub (Aug 27, 2025): 解决了，Monitor Service 配置文件中 clickhouse 的端口用错了，server.monitor_collector.reporter.clickhouse 中的 port 应该用 Clickouse 的 tcp_port (默认 9000)，而不是用 http_port(默认 8123)
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
SF-Zhou-patch-1
No results found.
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
deepseek-ai/3FS#202
No description provided.