Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dynomite log meaning #798

Open
abhinav0101 opened this issue Jun 25, 2021 · 3 comments
Open

Dynomite log meaning #798

abhinav0101 opened this issue Jun 25, 2021 · 3 comments

Comments

@abhinav0101
Copy link

abhinav0101 commented Jun 25, 2021

Can someone explain what do these logs mean? Specially the "on event 00FF eof 1 done 1 rb 3536 sb 1221" part ?

proxy_accept:220 accepted CLIENT 14 on PROXY 8 from '10.150.114.162:54142'
[2021-06-24 13:22:14.032] core_close_log:343 close CLIENT 166 '10.150.114.144:41368' on event 00FF eof 1 done 1 rb 3536 sb 1221

@abhinav0101
Copy link
Author

@ipapapa @diegopacheco

@shailesh33
Copy link
Contributor

proxy_accept : function name
CLIENT: type of connection
PROXY: (if my memory serves me well) type of listener in dynomite (there is PROXY for listening on 8102 and peer proxy for listerning on 8101 for talking to other dynomite nodes)
166: every connection is given an internal identifier for debugging purpose in the logs
00FF: I think this is the events received by select call.
eof 1 : received EOF
done 1: I think it means that its done sending data.
rb: read buffer size
sb: send buffer size

I hope I got it all correct. @smukil can confirm.

@abhinav0101
Copy link
Author

abhinav0101 commented Jun 25, 2021

Hey @shailesh33 Thanks a lot for the quick reply. Support from you guys would ensure that this community keeps growing.

I would really like some help with the below questions.

1.) What do the codes "220" & "343" in the above logs mean?
2.) Recently we experienced an issue with our dynomite setup. We are running 3 racks 1 dc dynomite cluster with RF = 3. One of the 3 dynomite.yml files looks like:-

dyn_o_mite:
  datacenter: dc1
  rack: dc1-rack1
  dyn_listen: 0.0.0.0:8101
  dyn_seeds:
  - 10.150.114.157:8101:dc1-rack2:dc1:0
  - 10.150.114.158:8101:dc1-rack3:dc1:0
  listen: 0.0.0.0:8102
  client_connections: 1500
  servers:
  - 127.0.0.1:6379:1
  tokens: '0'
  pem_key_file: /havail/dynomite/conf/dynomite.pem
  data_store: 0
  read_consistency : DC_QUORUM
  write_consistency : DC_QUORUM

We had some issue with our system recently due to which a client started bombarding dynomite. As you can see above, max client connections we had set is 1500 and the client requests exceeded this threshold. Dynomite was no longer accepting any redis Commands after this happened. For example on running redis-cli -p 8102 on dynomite box we were getting, "Could not connect to redis at 127.0.0.1:8102. Connection refused". I am attaching a log which helped us debug this. What does the code proxy_accept: 173 mean ?

dynomite logs

To resolve for this we are going ahead and increasing client_connections to 10000.

Question :- Is it possible to configure dynomite such that on exceeding max client requests, it stops accepting new connections but keeps working, instead of completely stopping to work?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants