Manager (agent) should use Unix socket (instead of creds) to connect to local CQL

[mykaul]

See scylladb/scylladb#16172 , scylladb/scylla-cqlsh#67

[mykaul]

Hmmm, that support is not available in Gocql ? - @sylwiaszunejko - is that a Python + cqlsh only?

[kbr-scylla]

6.0 and later only

[sylwiaszunejko]

Hmmm, that support is not available in Gocql ? - @sylwiaszunejko - is that a Python + cqlsh only?

@mykaul AFAIK there is no support for unix socket in gocql

[karol-kokoszka]

It would be beneficial for us to have it in gocql. This would allow us to avoid using CQL credentials for operations such as checking if something is a view (and not backing it up) and for performing restores. However, CQLPing would still require credentials unless we alter its behavior so that it's executed through an agent. In that case, we would need to accept the fact that we're testing it in a different manner than the typical CQL session created and used by end users.

[Michal-Leszczynski]

I tried to experiment with a custom Dialer (which connects to unix socket) set in ClusterConfig for both regular Session and SingleHostQueryExecutor, but I couldn't make it work because the code always panics (interface conversion: net.Addr is *net.UnixAddr, not *net.TCPAddr) when creating control connection (which cannot be disabled):

func (c *controlConn) setupConn(conn *Conn) error {
	// we need up-to-date host info for the filterHost call below
	iter := conn.querySystemLocal(context.TODO())
	host, err := c.session.hostInfoFromIter(iter, conn.host.connectAddress, conn.conn.RemoteAddr().(*net.TCPAddr).Port)

Used dialer

// unixSocketDialer is a special dialer which connects only to the maintenance_socket.
type unixSocketDialer struct {
	dialer     net.Dialer
	socketPath string
}

func (d unixSocketDialer) DialContext(_ context.Context, _, _ string) (net.Conn, error) {
	return d.dialer.Dial("unix", d.socketPath)
}

Used config

        c := gocql.NewCluster("192.168.100.11")
	c.NumConns = 1
	c.DisableInitialHostLookup = true
	c.ProtoVersion = 3
	c.ReconnectInterval = 0
        c.WriteCoalesceWaitTime = 0

	c.Events.DisableNodeStatusEvents = true
	c.Events.DisableTopologyEvents = true
	c.Events.DisableSchemaEvents = true

	d := net.Dialer{
		Timeout: c.Timeout,
	}
	if c.SocketKeepalive > 0 {
		d.KeepAlive = c.SocketKeepalive
	}

	c.Dialer = unixSocketDialer{
		dialer:     d,
		socketPath: socketPath,
	}

So it looks like there is no way (or at least not a very hacky way) for SM to configure gocql so that it connects to unix socket without dedicated changes in the driver itself.

Note: when I exported hidden ClusterConfig option disableControlConn bool, I was able to setup Session and query DESCRIBE SCHEMA WITH INTERNALS.

cc: @mykaul @karol-kokoszka @piodul

[avikivity]

Could we tunnel a unix socket via ssh? I don't think ssh supports crossing the boundary this way.

[Michal-Leszczynski]

I tried to make it work by adding regular agent API endpoint which would simply create session (to a single unix socket), get DESCRIBE SCHEMA, close session and return the output.

[Michal-Leszczynski]

Also, even if SM uses the maintenance_socket for obtaining schema, it won't work every time as the maintenance_socket can be disabled in Scylla config (AFAIR it is even disabled by default). So SM would still need either credentials or maintenance_socket in order to back up schema.

[piodul]

Another idea, for a similar workaround to Avi's - Go has lots of stuff that I think should make it easy to write a proxy between a TCP (TLS even) socket and a Unix socket. It could open only for the duration of the schema fetch.

I didn't touch the gocql driver for a while so it's hard for me to tell whether fixing it there is harder than implementing that workaround, but the gocql driver fix would should be the proper solution that we implement eventually.

[karol-kokoszka]

grooming notest

We can implement the "work around" in Scylla Manager directly.
The idea is:
(in the context of Scylla Manager's agent)

• to have separate background goroutine acting as a proxy between the CQL session and the maintenance sockets
• it's expected to be simple forward proxy
• it would require to consider the security as the listener would be expected to listen at certain localhost port

At the first glance, it looks as a dirty work-around that is not needed if the functionality is implemented in gocql driver (python driver has it already.
What are the chances of having it included into the gocql ? scylladb/gocql#175
cc: @piodul @avelanarius