Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Failing connection to graphite leads to error loop blocking all sockets on the machine #635

Open
FRosner opened this issue Nov 6, 2017 · 0 comments
Open

Failing connection to graphite leads to error loop blocking all sockets on the machine #635

FRosner opened this issue Nov 6, 2017 · 0 comments
Labels
bug A bug within statsd or it's bundled proxies

Comments

@FRosner
Copy link

FRosner commented Nov 6, 2017

Problem

  1. Start statsd specifying graphiteHost=my-graphite.internal.address:1704
  2. Let's assume my-graphite.internal.address resolves to 192.168.0.1
  3. Machine 192.168.0.1 goes down and graphite gets restarted on 192.168.0.2
  4. Statsd fails to connect to 192.168.0.1, logging the error but attempting to reconnect again to 192.168.0.1. However it does not terminate the TCP connection properly, leaving it orphaned in FIN_WAIT2. This keeps happening until all sockets are blocked and no other service on this machine can use any socket.

Expected Behaviour

  1. Statsd should resolve the DNS name on every connection attempt and not save the IP address. This way, when the IP address changes, resolution will happen on the next reconnect attempt.
  2. If reconnecting is not possible, statsd should terminate the connection properly, not leaving it orphaned.

Relevant Logs

statsd    30923     root 1719u  IPv4 62567113      0t0  TCP 192.168.0.10:50826->192.168.0.1:1704 (FIN_WAIT2)
statsd    30923     root 1720u  IPv4 62567351      0t0  TCP 192.168.0.10:50857->192.168.0.1:1704 (FIN_WAIT2)
statsd    30923     root 1721u  IPv4 62568759      0t0  TCP 192.168.0.10:50910->192.168.0.1:1704 (FIN_WAIT2)
statsd    30923     root 1722u  IPv4 62568818      0t0  TCP 192.168.0.10:50916->192.168.0.1:1704 (FIN_WAIT2)
statsd    30923     root 1723u  IPv4 62568856      0t0  TCP 192.168.0.10:50922->192.168.0.1:1704 (FIN_WAIT2)
statsd    30923     root 1724u  IPv4 62568913      0t0  TCP 192.168.0.10:50928->192.168.0.1:1704 (FIN_WAIT2)
statsd    30923     root 1725u  IPv4 62568969      0t0  TCP 192.168.0.10:50934->192.168.0.1:1704 (FIN_WAIT2)
statsd    30923     root 1726u  IPv4 62569016      0t0  TCP 192.168.0.10:50942->192.168.0.1:1704 (FIN_WAIT2)
statsd    30923     root 1727u  IPv4 62569063      0t0  TCP 192.168.0.10:50948->192.168.0.1:1704 (FIN_WAIT2)
statsd    30923     root 1728u  IPv4 62569111      0t0  TCP 192.168.0.10:50954->192.168.0.1:1704 (FIN_WAIT2)
statsd    30923     root 1729u  IPv4 62569154      0t0  TCP 192.168.0.10:50960->192.168.0.1:1704 (FIN_WAIT2)
statsd    30923     root 1730u  IPv4 62569194      0t0  TCP 192.168.0.10:50966->192.168.0.1:1704 (FIN_WAIT2)
statsd    30923     root 1731u  IPv4 62569257      0t0  TCP 192.168.0.10:50972->192.168.0.1:1704 (FIN_WAIT2)
statsd    30923     root 1732u  IPv4 62569288      0t0  TCP 192.168.0.10:50980->192.168.0.1:1704 (FIN_WAIT2)
statsd    30923     root 1733u  IPv4 62569340      0t0  TCP 192.168.0.10:50986->192.168.0.1:1704 (FIN_WAIT2)
statsd    30923     root 1734u  IPv4 62572739      0t0  TCP 192.168.0.10:51164->192.168.0.1:1704 (FIN_WAIT2)
statsd    30923     root 1735u  IPv4 62569431      0t0  TCP 192.168.0.10:51000->192.168.0.1:1704 (FIN_WAIT2)
statsd    30923     root 1736u  IPv4 62569471      0t0  TCP 192.168.0.10:51006->192.168.0.1:1704 (FIN_WAIT2)
statsd    30923     root 1737u  IPv4 62571582      0t0  TCP 192.168.0.10:51012->192.168.0.1:1704 (FIN_WAIT2)
statsd    30923     root 1738u  IPv4 62572240      0t0  TCP 192.168.0.10:51103->192.168.0.1:1704 (FIN_WAIT2)
statsd    30923     root 1739u  IPv4 62571685      0t0  TCP 192.168.0.10:51024->192.168.0.1:1704 (FIN_WAIT2)
statsd    30923     root 1740u  IPv4 62572793      0t0  TCP 192.168.0.10:51172->192.168.0.1:1704 (FIN_WAIT2)
statsd    30923     root 1741u  IPv4 62571770      0t0  TCP 192.168.0.10:51038->192.168.0.1:1704 (FIN_WAIT2)
statsd    30923     root 1742u  IPv4 62570811      0t0  TCP 192.168.0.10:51044->192.168.0.1:1704 (FIN_WAIT2)
statsd    30923     root 1743u  IPv4 62571849      0t0  TCP 192.168.0.10:51050->192.168.0.1:1704 (FIN_WAIT2)
statsd    30923     root 1744u  IPv4 62571903      0t0  TCP 192.168.0.10:51059->192.168.0.1:1704 (FIN_WAIT2)
statsd    30923     root 1745u  IPv4 62571941      0t0  TCP 192.168.0.10:51064->192.168.0.1:1704 (FIN_WAIT2)
statsd    30923     root 1746u  IPv4 62571015      0t0  TCP 192.168.0.10:51070->192.168.0.1:1704 (FIN_WAIT2)
statsd    30923     root 1747u  IPv4 62573158      0t0  TCP 192.168.0.10:51220->192.168.0.1:1704 (FIN_WAIT2)
statsd    30923     root 1748u  IPv4 62573201      0t0  TCP 192.168.0.10:51228->192.168.0.1:1704 (FIN_WAIT2)
statsd    30923     root 1749u  IPv4 62572182      0t0  TCP 192.168.0.10:51096->192.168.0.1:1704 (FIN_WAIT2)
statsd    30923     root 1750u  IPv4 62573240      0t0  TCP 192.168.0.10:51235->192.168.0.1:1704 (FIN_WAIT2)
statsd    30923     root 1751u  IPv4 62573293      0t0  TCP 192.168.0.10:51240->192.168.0.1:1704 (FIN_WAIT2)
statsd    30923     root 1752u  IPv4 62573329      0t0  TCP 192.168.0.10:51246->192.168.0.1:1704 (FIN_WAIT2)
statsd    30923     root 1753u  IPv4 62573368      0t0  TCP 192.168.0.10:51252->192.168.0.1:1704 (FIN_WAIT2)
statsd    30923     root 1754u  IPv4 62573421      0t0  TCP 192.168.0.10:51258->192.168.0.1:1704 (FIN_WAIT2)
statsd    30923     root 1755u  IPv4 62573468      0t0  TCP 192.168.0.10:51264->192.168.0.1:1704 (FIN_WAIT2)
statsd    30923     root 1756u  IPv4 62573520      0t0  TCP 192.168.0.10:51270->192.168.0.1:1704 (FIN_WAIT2)
statsd    30923     root 1757u  IPv4 62575631      0t0  TCP 192.168.0.10:51276->192.168.0.1:1704 (FIN_WAIT2)
statsd    30923     root 1758u  IPv4 62575688      0t0  TCP 192.168.0.10:51284->192.168.0.1:1704 (FIN_WAIT2)
statsd    30923     root 1759u  IPv4 62575748      0t0  TCP 192.168.0.10:51290->192.168.0.1:1704 (FIN_WAIT2)
statsd    30923     root 1760u  IPv4 62575796      0t0  TCP 192.168.0.10:51297->192.168.0.1:1704 (FIN_WAIT2)
statsd    30923     root 1761u  IPv4 62575836      0t0  TCP 192.168.0.10:51302->192.168.0.1:1704 (FIN_WAIT2)
statsd    30923     root 1762u  IPv4 62575874      0t0  TCP 192.168.0.10:51308->192.168.0.1:1704 (FIN_WAIT2)
statsd    30923     root 1763u  IPv4 62575941      0t0  TCP 192.168.0.10:51315->192.168.0.1:1704 (FIN_WAIT2)
statsd    30923     root 1764u  IPv4 62575978      0t0  TCP 192.168.0.10:51320->192.168.0.1:1704 (FIN_WAIT2)
statsd    30923     root 1765u  IPv4 62576030      0t0  TCP 192.168.0.10:51326->192.168.0.1:1704 (FIN_WAIT2)
statsd    30923     root 1766u  IPv4 62576091      0t0  TCP 192.168.0.10:51332->192.168.0.1:1704 (FIN_WAIT2)
statsd    30923     root 1767u  IPv4 62576150      0t0  TCP 192.168.0.10:51340->192.168.0.1:1704 (FIN_WAIT2)
statsd    30923     root 1768u  IPv4 62576197      0t0  TCP 192.168.0.10:51346->192.168.0.1:1704 (FIN_WAIT2)
statsd    30923     root 1769u  IPv4 62576266      0t0  TCP 192.168.0.10:51352->192.168.0.1:1704 (FIN_WAIT2)
statsd    30923     root 1770u  IPv4 62576324      0t0  TCP 192.168.0.10:51358->192.168.0.1:1704 (FIN_WAIT2)
statsd    30923     root 1771u  IPv4 62576384      0t0  TCP 192.168.0.10:51366->192.168.0.1:1704 (FIN_WAIT2)
statsd    30923     root 1772u  IPv4 62576422      0t0  TCP 192.168.0.10:51372->192.168.0.1:1704 (FIN_WAIT2)
statsd    30923     root 1773u  IPv4 62576477      0t0  TCP 192.168.0.10:51378->192.168.0.1:1704 (FIN_WAIT2)
statsd    30923     root 1774u  IPv4 62576540      0t0  TCP 192.168.0.10:51384->192.168.0.1:1704 (FIN_WAIT2)
statsd    30923     root 1775u  IPv4 62576578      0t0  TCP 192.168.0.10:51390->192.168.0.1:1704 (FIN_WAIT2)
statsd    30923     root 1776u  IPv4 62576616      0t0  TCP 192.168.0.10:51398->192.168.0.1:1704 (FIN_WAIT2)
statsd    30923     root 1777u  IPv4 62576667      0t0  TCP 192.168.0.10:51404->192.168.0.1:1704 (FIN_WAIT2)
statsd    30923     root 1778u  IPv4 62576719      0t0  TCP 192.168.0.10:51412->192.168.0.1:1704 (FIN_WAIT2)
statsd    30923     root 1779u  IPv4 62576768      0t0  TCP 192.168.0.10:51418->192.168.0.1:1704 (FIN_WAIT2)
statsd    30923     root 1780u  IPv4 62576803      0t0  TCP 192.168.0.10:51424->192.168.0.1:1704 (FIN_WAIT2)
statsd    30923     root 1781u  IPv4 62576869      0t0  TCP 192.168.0.10:51430->192.168.0.1:1704 (FIN_WAIT2)
statsd    30923     root 1782u  IPv4 62576907      0t0  TCP 192.168.0.10:51436->192.168.0.1:1704 (FIN_WAIT2)
statsd    30923     root 1783u  IPv4 62576950      0t0  TCP 192.168.0.10:51442->192.168.0.1:1704 (FIN_WAIT2)
statsd    30923     root 1784u  IPv4 62577005      0t0  TCP 192.168.0.10:51448->192.168.0.1:1704 (FIN_WAIT2)
statsd    30923     root 1785u  IPv4 62577059      0t0  TCP 192.168.0.10:51456->192.168.0.1:1704 (FIN_WAIT2)
statsd    30923     root 1786u  IPv4 62577100      0t0  TCP 192.168.0.10:51462->192.168.0.1:1704 (FIN_WAIT2)
statsd    30923     root 1787u  IPv4 62577154      0t0  TCP 192.168.0.10:51468->192.168.0.1:1704 (FIN_WAIT2)
statsd    30923     root 1788u  IPv4 62577203      0t0  TCP 192.168.0.10:51474->192.168.0.1:1704 (FIN_WAIT2)
statsd    30923     root 1789u  IPv4 62578093      0t0  TCP 192.168.0.10:51480->192.168.0.1:1704 (FIN_WAIT2)
statsd    30923     root 1790u  IPv4 62577322      0t0  TCP 192.168.0.10:51486->192.168.0.1:1704 (FIN_WAIT2)
statsd    30923     root 1791u  IPv4 62577372      0t0  TCP 192.168.0.10:51493->192.168.0.1:1704 (FIN_WAIT2)
statsd    30923     root 1792u  IPv4 62577416      0t0  TCP 192.168.0.10:51498->192.168.0.1:1704 (FIN_WAIT2)
statsd    30923     root 1793u  IPv4 62578250      0t0  TCP 192.168.0.10:51508->192.168.0.1:1704 (FIN_WAIT2)
statsd    30923     root 1794u  IPv4 62577525      0t0  TCP 192.168.0.10:51516->192.168.0.1:1704 (FIN_WAIT2)
statsd    30923     root 1795u  IPv4 62577574      0t0  TCP 192.168.0.10:51522->192.168.0.1:1704 (FIN_WAIT2)
statsd    30923     root 1796u  IPv4 62577628      0t0  TCP 192.168.0.10:51528->192.168.0.1:1704 (FIN_WAIT2)
statsd    30923     root 1797u  IPv4 62578451      0t0  TCP 192.168.0.10:51534->192.168.0.1:1704 (FIN_WAIT2)
statsd    30923     root 1798u  IPv4 62578731      0t0  TCP 192.168.0.10:51540->192.168.0.1:1704 (FIN_WAIT2)
statsd    30923     root 1799u  IPv4 62578783      0t0  TCP 192.168.0.10:51546->192.168.0.1:1704 (FIN_WAIT2)
statsd    30923     root 1800u  IPv4 62578834      0t0  TCP 192.168.0.10:51553->192.168.0.1:1704 (FIN_WAIT2)
statsd    30923     root 1801u  IPv4 62578886      0t0  TCP 192.168.0.10:51558->192.168.0.1:1704 (FIN_WAIT2)
statsd    30923     root 1802u  IPv4 62578924      0t0  TCP 192.168.0.10:51564->192.168.0.1:1704 (FIN_WAIT2)
statsd    30923     root 1803u  IPv4 62578975      0t0  TCP 192.168.0.10:51572->192.168.0.1:1704 (FIN_WAIT2)
statsd    30923     root 1804u  IPv4 62579023      0t0  TCP 192.168.0.10:51578->192.168.0.1:1704 (FIN_WAIT2)
@BlueHatbRit BlueHatbRit added the bug A bug within statsd or it's bundled proxies label Jul 11, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug A bug within statsd or it's bundled proxies
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@BlueHatbRit @FRosner