Missing animations when minimizing/maximizing/restoring undecorated window

[mahozad]

Describe the bug
I want to have custom title bar for my Desktop application (especially important to implement dark theme).
So, I set undecorated = true.

With undecorated in Windows OS, when maximizing or minimizing a window or restoring a window already minimized to taskbar, the window does not animate. Decorated windows, in contrast, have proper animations:

Decorated window	Undecorated window
decorated-window.mp4	undecorated.mp4

I tried the workaround in #3166 but it does not seem to work for undecorated windows (window minimizes but with no animation):

fun minimize() {
    User32.INSTANCE.ShowWindow(
        WinDef.HWND(Pointer((window as ComposeWindow).windowHandle)),
        User32.SW_MINIMIZE // == 6
    )
    // OR
    User32.INSTANCE.CloseWindow(
        WinDef.HWND(Pointer((window as ComposeWindow).windowHandle))
    )
}

Electron seems to have had this problem too in the past:

• [Bug]: Frameless windows have no animation on Windows electron/electron#32993
• No animation when minimizing and restoring frameless window electron/electron#751

This is the code they use to set proper windows decoration and preserve animations:

https://github.com/electron/electron/blob/3a5e2dd90c099b10679364642a0f7fa398dce875/shell/browser/native_window_views.cc#L367-L390

So, then tried setting just the flags that Electron sets like this for the window:

window(// ...) {
    val frameStyle = User32.WS_OVERLAPPED or User32.WS_MINIMIZEBOX
    User32.INSTANCE.SetWindowLong(
        WinDef.HWND(Pointer(window.windowHandle)),
        User32.GWL_STYLE,
        frameStyle
    )
    App()
}

OR modifying the existing window flags:

window(// ...) {
    val defaultStyle = User32.INSTANCE.GetWindowLong(WinDef.HWND(Pointer(window.windowHandle)), GWL_STYLE)
    val newStyle = defaultStyle and User32.WS_THICKFRAME.inv() and User32.WS_DLGFRAME.inv()
    User32.INSTANCE.SetWindowLong(
       WinDef.HWND(Pointer(window.windowHandle)),
       User32.GWL_STYLE,
       newStyle
    )
    App()
}

The default close, maximize, minimize buttons are removed but cannot get rid of the title bar.

The IntelliJ IDEA 2023.1 is undecorated and has proper animations.
Maybe you could consider getting a little help from them?

Fixing this problem could probably resolve or help resolve all the following issues:

• How do I add shadows to a window when its undecorated attribute was set to true?  #3295
• [Titlebar] Support Transparent Titlebar API. #3164
• Is it possible to activate Aero Snap when using the undecorated option? #2062
• How to make "Title bar window shake" work, when Window undecorated is enabled. #1472

Affected platforms

• Desktop

Versions

• Kotlin version*: 1.8.20
• Compose Multiplatform version*: 1.4.1
• OS version(s)*: Windows 10, Windows 11
• OS architecture (x86 or arm64): x64
• JDK (for desktop issues): Oracle JDK 17.0.5

To Reproduce
Compare app minimize animation in these two:

decorated (default title bar):

fun main() {
    application {
        Window(
            state = rememberWindowState(size = DpSize(400.dp, 300.dp)),
            undecorated = false,
            transparent = false,
            onCloseRequest = ::exitApplication,
        ) {
            Text(text = "This is an example text")
        }
    }
}

undecorated (custom title bar):

fun main() {
    application {
        Window(
            state = rememberWindowState(size = DpSize(400.dp, 300.dp)),
            undecorated = true,
            transparent = true, // for rounded corners
            onCloseRequest = ::exitApplication,
        ) {
            Surface(
                modifier = Modifier
                    .fillMaxSize()
                    .border(Dp.Hairline, Color.Gray, RoundedCornerShape(8.dp))
                    .clip(RoundedCornerShape(8.dp))
            ) {
                Column {
                    WindowDraggableArea(modifier = Modifier.fillMaxWidth().height(30.dp)) {
                        Row(
                            horizontalArrangement = Arrangement.SpaceBetween,
                            modifier = Modifier.fillMaxWidth()
                        ) {
                            Text(text = "Title")
                            Row {
                                Box(
                                    contentAlignment = Alignment.Center,
                                    modifier = Modifier
                                        .width(48.dp)
                                        .fillMaxHeight()
                                        .clickable { window.isMinimized = true }
                                ) {
                                    Icon(
                                        imageVector = Icons.Custom.Minimize,
                                        contentDescription = null,
                                        modifier = Modifier.size(12.dp)
                                    )
                                }
                                Box(
                                    contentAlignment = Alignment.Center,
                                    modifier = Modifier
                                        .width(48.dp)
                                        .fillMaxHeight()
                                ) {
                                    Icon(
                                        imageVector = Icons.Custom.Close,
                                        contentDescription = null,
                                        modifier = Modifier.size(14.dp)
                                    )
                                }
                            }
                        }
                    }
                    Text(text = "This is an example text")
                }
            }
        }
    }
}

Expected behavior
Undecorated windows should have animations when minimizing/maximizing/restoring them.

[pjBooms]

Relates to #3166

[mahozad]

A good example:

• https://github.com/rossy/borderless-window
• https://discourse.glfw.org/t/making-a-custom-titlebar/2392/3

[mahozad]

demo-windows-10.mp4

Fixed the problem in Windows (don't know if the problem applies to Linux or macOS).
Tried it successfully in Windows 7, Windows 10, Windows 11.

The animation solution was adopted from https://github.com/kalibetre/CustomDecoratedJFrame which was for Swing (Jframe) windows.

With this, native animations for window appearance (open), window disappearance (close), window minimize, window restore, and (possibly) window maximize work correctly. The repo above also implemented Aero snap feature (and possibly, shaking window title to hide other windows) but I omitted it.

It also provides native window shadows and border (and rounded corners in Windows 11).

The only downside is that the window cannot be made transparent (otherwise, it loses animations and shadows).

To be honest, I don't know how the CustomWindowProcedure below works. But made it to work.

Update: 2024-05-07

implementation("net.java.dev.jna:jna-jpms:5.14.0")
implementation("net.java.dev.jna:jna-platform-jpms:5.14.0")
// ...

import com.sun.jna.Native
import com.sun.jna.Pointer
import com.sun.jna.platform.win32.BaseTSD.LONG_PTR
import com.sun.jna.platform.win32.User32
import com.sun.jna.platform.win32.WinDef.*
import com.sun.jna.platform.win32.WinUser.WM_DESTROY
import com.sun.jna.platform.win32.WinUser.WindowProc
import com.sun.jna.win32.W32APIOptions
// ...

fun main() = application {
    Window(
        undecorated = true, // This should be true
        transparent = false,
        // ...
    ) {
        val windowHandle = remember(this.window) {
            val windowPointer = (this.window as? ComposeWindow)
                ?.windowHandle
                ?.let(::Pointer)
                ?: Native.getWindowPointer(this.window)
            HWND(windowPointer)
        }
        remember(windowHandle) { CustomWindowProcedure(windowHandle) }
        MyAppContent()
    }
}

@Suppress("FunctionName")
private interface User32Ex : User32 {
    fun SetWindowLong(hWnd: HWND, nIndex: Int, wndProc: WindowProc): LONG_PTR
    fun SetWindowLongPtr(hWnd: HWND, nIndex: Int, wndProc: WindowProc): LONG_PTR
    fun CallWindowProc(proc: LONG_PTR, hWnd: HWND, uMsg: Int, uParam: WPARAM, lParam: LPARAM): LRESULT
}

// See https://stackoverflow.com/q/62240901
@Structure.FieldOrder(
    "leftBorderWidth",
    "rightBorderWidth",
    "topBorderHeight",
    "bottomBorderHeight"
)
data class WindowMargins(
    @JvmField var leftBorderWidth: Int,
    @JvmField var rightBorderWidth: Int,
    @JvmField var topBorderHeight: Int,
    @JvmField var bottomBorderHeight: Int
) : Structure(), Structure.ByReference

private class CustomWindowProcedure(private val windowHandle: HWND) : WindowProc {
    // See https://learn.microsoft.com/en-us/windows/win32/winmsg/about-messages-and-message-queues#system-defined-messages
    private val WM_NCCALCSIZE = 0x0083
    private val WM_NCHITTEST = 0x0084
    private val GWLP_WNDPROC = -4
    private val margins = WindowMargins(
        leftBorderWidth = 0,
        topBorderHeight = 0,
        rightBorderWidth = -1,
        bottomBorderHeight = -1
    )
    private val USER32EX =
        runCatching { Native.load("user32", User32Ex::class.java, W32APIOptions.DEFAULT_OPTIONS) }
        .onFailure { println("Could not load user32 library") }
        .getOrNull()
    // The default window procedure to call its methods when the default method behaviour is desired/sufficient
    private var defaultWndProc = if (is64Bit()) {
        USER32EX?.SetWindowLongPtr(windowHandle, GWLP_WNDPROC, this) ?: LONG_PTR(-1)
    } else {
        USER32EX?.SetWindowLong(windowHandle, GWLP_WNDPROC, this) ?: LONG_PTR(-1)
    }

    init {
        enableResizability()
        enableBorderAndShadow()
        // enableTransparency(alpha = 255.toByte())
    }

    override fun callback(hWnd: HWND, uMsg: Int, wParam: WPARAM, lParam: LPARAM): LRESULT {
        return when (uMsg) {
            // Returns 0 to make the window not draw the non-client area (title bar and border)
            // thus effectively making all the window our client area
            WM_NCCALCSIZE -> { LRESULT(0) }
            // The CallWindowProc function is used to pass messages that
            // are not handled by our custom callback function to the default windows procedure
            WM_NCHITTEST -> { USER32EX?.CallWindowProc(defaultWndProc, hWnd, uMsg, wParam, lParam) ?: LRESULT(0) }
            WM_DESTROY -> { USER32EX?.CallWindowProc(defaultWndProc, hWnd, uMsg, wParam, lParam) ?: LRESULT(0) }
            else -> { USER32EX?.CallWindowProc(defaultWndProc, hWnd, uMsg, wParam, lParam) ?: LRESULT(0) }
        }
    }

    /**
     * For this to take effect, also set `resizable` argument of Compose Window to `true`.
     */
    private fun enableResizability() {
        val style = USER32EX?.GetWindowLong(windowHandle, GWL_STYLE) ?: return
        val newStyle = style or WS_CAPTION
        USER32EX.SetWindowLong(windowHandle, GWL_STYLE, newStyle)
    }

    /**
     * To disable window border and shadow, pass (0, 0, 0, 0) as window margins
     * (or, simply, don't call this function).
     */
    private fun enableBorderAndShadow() {
        val dwmApi = "dwmapi"
            .runCatching(NativeLibrary::getInstance)
            .onFailure { println("Could not load dwmapi library") }
            .getOrNull()
        dwmApi
            ?.runCatching { getFunction("DwmExtendFrameIntoClientArea") }
            ?.onFailure { println("Could not enable window native decorations (border/shadow/rounded corners)") }
            ?.getOrNull()
            ?.invoke(arrayOf(windowHandle, margins))
    }

    /**
     * See https://learn.microsoft.com/en-us/windows/win32/api/winuser/nf-winuser-setlayeredwindowattributes
     * @param alpha 0 for transparent, 255 for opaque
     */
    private fun enableTransparency(alpha: Byte) {
        val defaultStyle = User32.INSTANCE.GetWindowLong(windowHandle, GWL_EXSTYLE)
        val newStyle = defaultStyle or User32.WS_EX_LAYERED
        USER32EX?.SetWindowLong(windowHandle, User32.GWL_EXSTYLE, newStyle)
        USER32EX?.SetLayeredWindowAttributes(windowHandle, 0, alpha, LWA_ALPHA)
    }

    private fun is64Bit(): Boolean {
        val bitMode = System.getProperty("com.ibm.vm.bitmode")
        val model = System.getProperty("sun.arch.data.model", bitMode)
        return model == "64"
    }
}

To minimize the window, instead of using
(window as? ComposeWindow)?.isMinimized = true use User32.INSTANCE.CloseWindow(windowHandle):

@Composable
fun WindowScope.MyCustomMinimizeButton() {
    val windowHandle = remember(this.window) {
        val windowPointer = (this.window as? ComposeWindow)
            ?.windowHandle
            ?.let(::Pointer)
            ?: Native.getWindowPointer(this.window)
        HWND(windowPointer)
    }
    Button(onclick = { User32.INSTANCE.CloseWindow(windowHandle) }) {
        Text("Minimize")
    }
}

I also added the following rule to my Proguard rules.pro file for the app release version to work correctly:

-keep class com.sun.jna.** { *; }

[mahozad]

@pjBooms Fixed the problem with native animations and shadows as described in the above comment.

Can they be incorporated into Compose Multiplatform code?

[pjBooms]

@mahozad currently we are busy with other tasks

[mahozad]

OK, no problem.

[sheng-ri]

@mahozad
I tried this,it works, but I can't resize the window.

[mahozad]

@sheng-ri
My app did not need to be resized (I set resizable = false).
Sorry.

[sheng-ri]

@sheng-ri My app did not need to be resized (I set resizable = false). Sorry.

I found a solution for resizable window base your code. You need make window style has WS_CAPTION.
.

[sleinexxx]

@sheng-riРазмер моего приложения не нужно было изменять (я установил resizable = false). Извини.

Я нашел решение для изменения размера окна в вашем коде. Вам нужно, чтобы стиль окна имел WS_CAPTION. .

how to set this style?

[sheng-ri]

@sleinexxx

See win32 api
You can convert this to use JNA.
Here a example(Using FFM):

final var style = (long)GetWindowLongA.invoke(hWnd, GWL_STYLE);
SetWindowLongA.invoke(hWnd, GWL_STYLE, style |  WS_CAPTION);

[sleinexxx]

@sleinexxx

See win32 api You can convert this to use JNA. Here a example(Using FFI):

final var style = (long)GetWindowLongA.invoke(hWnd, GWL_STYLE);
SetWindowLongA.invoke(hWnd, GWL_STYLE, style |  WS_CAPTION);

I tried to do this, but it didn't work. WS_SIZEBOX too

[sheng-ri]

I tried to do this, but it didn't work. WS_SIZEBOX too

@sleinexxx
I forget one thing,This solution need set undecorated=true
I still can't undertstand that, but seems this works.

[sleinexxx]

I tried to do this, but it didn't work. WS_SIZEBOX too

@sleinexxx I forget one thing,This solution need set undecorated=true I still can't undertstand that, but seems this works.

it works! thanks a lot