Memory optimization when high concurrency

[JobberRT]

I'm running a service that has 100-200/s requests at idle and 700-1000/s requests at peak(usually the peak will last for 10~30 min). Now I'm running into memory issues.Here are what I could found:


Echo will use *http.Server as under lying server:

echo/echo.go

Lines 642 to 651 in 6b09f3f

	func (e *Echo) Start(address string) error {
	e.startupMutex.Lock()
	e.Server.Addr = address
	if err := e.configureServer(e.Server); err != nil {
	e.startupMutex.Unlock()
	return err
	}
	e.startupMutex.Unlock()
	return e.Server.Serve(e.Listener)
	}

Then *http.Server will create a new goroutines that running *conn.serve. Eventually, At these three lines, it will allocate buffer and read the request for each connection(I don't know how to reference other repo's code preview like the last one, so I use a link): server.go

The question is, how to optimize the memory between these behaivor? More specifily:

1. Does every connection allocates both a new read buffer and a write buffer with the size of 4KB?
2. What if I set the request to keep-alive and reuse the connection? Will that avoid creating new buffer(just simply reset the buffer and then reuse it)? I think it will still create new buffer...
3. Does the buffer keep held when the connection are still alived? If I disable echo's keep-alive like Tcp connection keepalive period time shouble be configable #1942 , will the created buffer released right after the connection is complete and being available to GC and finally reduce the average memory usage? (I know this will causing much heavier cpu load or resource waste, but memory is my first priority)
4. If none of my thoughts are good or will work, could someone please give me advice to reduce the memory usage without reducing the concurrency?

Here's pprof and pyroscope information:

(pprof) top -cum
Showing nodes accounting for 77.83MB, 73.12% of 106.45MB total
Dropped 331 nodes (cum <= 0.53MB)
Showing top 10 nodes out of 72
      flat  flat%   sum%        cum   cum%
    0.38MB  0.36%  0.36%    79.14MB 74.35%  net/http.(*conn).serve
   33.50MB 31.48% 31.83%    33.50MB 31.48%  bufio.NewWriterSize (inline)
         0     0% 31.83%    33.47MB 31.45%  net/http.newBufioWriterSize
   32.77MB 30.79% 62.62%    32.77MB 30.79%  bufio.NewReaderSize (inline)
         0     0% 62.62%    32.70MB 30.72%  bufio.NewReader (inline)
         0     0% 62.62%    32.70MB 30.72%  net/http.newBufioReader
         0     0% 62.62%    11.44MB 10.74%  runtime.systemstack
         0     0% 62.62%    11.43MB 10.74%  runtime.newproc.func1
         0     0% 62.62%    11.43MB 10.74%  runtime.newproc1
   11.18MB 10.50% 73.12%    11.18MB 10.50%  runtime.malg

[aldas]

This is how standard library http.Server works - this is nothing inherently Echo specific. You definitely find better answers when you google for http.Server concurrency. Echo/Gin and similar std based "framework" role here is only to do request routing and run middlewares and eventually execute handler function.

This is post from another thread that details how it works #2205 (comment) but I assume that you know that already, having checked the profiler/debugger

And about #1942 If I recall correctly this is about how sockets behave (linger and are reused etc) and not how http.Server handles the connections. Keep-alive header is both server and client-side decision. If client wants to have new connections - it will have that.

If you run Echo with your own http.Server instance you will not use tcpKeepAliveListener mentioned in #1942

func main() {
	e := echo.New()

	s := http.Server{
		Addr:              ":8080",
		Handler:           e,
	}
	
	if err := s.ListenAndServe(); errors.Is(err, http.ErrServerClosed) {
		log.Fatal(err)
	}
}

[JobberRT]

Thanks for answering, I will give Fiber a try

[aldas]

You probably should write small demo only using http.Server and standard library http.Handler and see if that has same buffer problems. If it so maybe next step would be to ask in https://groups.google.com/g/golang-nuts as Authorities of Go dwell there and get really in-depth answers.

[aldas]

In case of OOM killer kicking in you might want to try GOMEMLIMIT

Just saw this post on reddit https://weaviate.io/blog/gomemlimit-a-game-changer-for-high-memory-applications

[JobberRT]

Thanks, Im testing Fiber, and if it doesn't help, I will do a minmum demo in standrad library for debuging the memory issue