research_notes.md

Research Notes

Inference

• LM Studio
• microsoft/monitors4codegen: Code and Data artifact for NeurIPS 2023 paper - "Monitor-Guided Decoding of Code LMs with Static Analysis of Repository Context". `multispy` is a lsp client library in Python intended to be used to build applications around language servers. (github.com)

Fine Tuning

• allenai/OLMo: Modeling, training, eval, and inference code for OLMo (github.com)
• hiyouga/LLaMA-Factory: Unify Efficient Fine-tuning of 100+ LLMs (github.com)
• OpenAccess-AI-Collective/axolotl: Go ahead and axolotl questions (github.com)
• LLaVA/docs/Finetune_Custom_Data.md at main - haotian-liu/LLaVA (github.com)

Vision

• PTA-Text: A Text Only Click Model - Prompt image, it tells you where it would click (demo)
• Set-of-Mark Visual Prompting for GPT-4V
• LLaVA
• YOLOv9
• Ty on X: "Open Interpreter 0.2.1 is out! -  5X launch speed -  Local OSS model for GUI control -  Native control over Apple Mail, Calendar, Contacts, SMS -  New LLM-first web browser (powered by @perplexity_ai) -  Profiles, Docker, Jupyter export... ↓ https://t.co/XU4gibTXWk" / X (twitter.com)
• (13) Victor Sanh on X: "💬🔥Releasing idefics2-8b-chatty, the chat-optimized version of Idefics2! It is a very efficient (8B parameters) state-of-the-art VLM, has been red-teamed, and comes with a few surprises: - 📖Paper dissecting a lot of the experimental insights we learned building Idefics2: - https://t.co/DDQkg5ddza" / X
• SeeMoE: Implementing a MoE Vision Language Model from Scratch (huggingface.co)

Large Context

• gradientai/Llama-3-8B-Instruct-Gradient-4194k - Hugging Face

Image Processing

• Template Matching by Correlation | Image Processing I - YouTube
• Seam Carving Algorithm in Python | Liquid Resizing - YouTube
• Using Generative AI to Organize Video Game Screenshots (raymondcamden.com)
• Page dewarping (mzucker.github.io)

RPA

• OpenAdaptAI/OpenAdapt: AI-First Process Automation with Large Multimodal Models (LMMs)
• askui/askui: "What can be said can be solved-"
• ddupont808/GPT-4V-Act: AI agent using GPT-4V(ision) capable of using a mouse/keyboard to interact with web UI
• TobiasNorlund/UI-Act: An AI agent for interacting with a computer using the graphical user interface
• KillianLucas/open-interpreter: A natural language interface for computers

Windows UI Automation

• Accessibility tools - AccEvent (Accessible Event Watcher) - Win32 apps | Microsoft Learn
• Accessibility tools - Inspect - Win32 apps | Microsoft Learn
• Accessibility Insights
• Navigation events for WebView2 apps - Microsoft Edge Developer documentation | Microsoft Learn
• c# - UIAutomation won't retrieve children of an element - Stack Overflow
• comEle = _AutomationClient.instance().dll.GetNextSiblingElement(self.Element) Hangs - Issue #30 - yinkaisheng/Python-UIAutomation-for-Windows (github.com)
• microsoft/WinAppDriver: Windows Application Driver (github.com)
• How to force UI automation tree refresh
  • mostly interested in the screen reader flag mentioned

    SystemParametersInfo( SPI_SETSCREENREADER, TRUE, NULL, SPIF_UPDATEINIFILE | SPIF_SENDCHANGE);
    PostMessage( HWND_BROADCAST, WM_WININICHANGE, SPI_SETSCREENREADER, 0);
    

• UI Automation Fundamentals - .NET Framework | Microsoft Learn
• Understanding Threading Issues - Win32 apps | Microsoft Learn
• UI Automation Threading Issues - .NET Framework | Microsoft Learn
• c# - System.Windows.Automation is extremely slow - Stack Overflow
• msdn-code-gallery-microsoft/Official Windows Platform Sample/UI Automation document content client sample at master - microsoftarchive/msdn-code-gallery-microsoft (github.com)
  • Browse code samples | Microsoft Learn
• caching
  • UIAutomation not catching all elements - Microsoft Q&A
  • Caching UI Automation Properties and Control Patterns - Win32 apps | Microsoft Learn
  • IUIAutomationElement::FindAllBuildCache (uiautomationclient.h) - Win32 apps | Microsoft Learn
• Writing a UI Automation Provider for a Win32-based Custom Control (codemag.com)
• Incorrect BoundingRectangle value - Issue #129 - yinkaisheng/Python-UIAutomation-for-Windows (github.com)
  • the UI Automation tree has bounding boxes for windows that include a padding and IDK why, found this issue which is probably unrelated lol
• microsoft/accessibility-insights-windows: Accessibility Insights for Windows (github.com)

PowerAutomate

• 🤖 How to use Microsoft Power Automate Desktop - Full tutorial - YouTube

Win32

• How does Microsoft's "inspect.exe" application keep its window on top? - Microsoft Q&A === WS_EX_TOPMOST, UAC Bypass UI
  • "Automatic dismissal of the start menu" and other crazyness - Windows-classic-samples/Samples/DesktopAutomationDismiss at 27ffb0811ca761741502feaefdb591aebf592193 - microsoft/Windows-classic-samples (github.com)
• [Convert]::ToInt32("0x80070012", 16) then ctrl+F win32::Foundation to find the error code
• Process Hacker / Code / [r6350] /2.x/trunk (sourceforge.net)
• Process Monitor - Sysinternals | Microsoft Learn
• (1) Building 25+ years of SysInternals: Exploring ZoomIt | BRK200H - YouTube
• c# - Getting icon of "modern" Windows app from a desktop application? - Stack Overflow
• DLL Export Viewer - view exported functions list in Windows DLL (nirsoft.net)

OneDrive

• How can I check status of files/folders in OneDrive folder whether it is synced or not using c# or PowerShell?
• rodneyviana/ODSyncService: OneDrive service/DLL for Sync State (github.com)

RL

• stillonearth/bevy_rl
• Saving RenderTarget image data to a file #5603
• paulkre/bevy_image_export: Bevy plugin for rendering image sequences

Prompting

• guidance-ai/guidance: A guidance language for controlling large language models.
• Eladlev/AutoPrompt: A framework for prompt tuning using Intent-based Prompt Calibration (github.com)

RAG, Tools and Actions

• ACT-1: Transformer for Actions
• LlamaIndex 🦙 v0.10.6
• Supercharge Your RAG with Contextualized Late Interactions (youtube.com)
  • bclavie/RAGatouille
• PromtEngineer/YoutTube-Tutorial: This repo contains codes covered in the youtube tutorials. (github.com)
  • How Good is Phi-3-Mini for RAG, Routing, Agents (youtube.com)
  • Advanced RAG with ColBERT in LangChain and LlamaIndex (youtube.com)
  • Supercharge Your RAG with Contextualized Late Interactions (youtube.com)
  • Feature Test for Phi-3-mini-4k-instruct - LlamaIndex

Database

• Quickstart | Get Started | EdgeDB Docs
• bclavie/RAGatouille: Easily use and train state of the art late-interaction retrieval methods (ColBERT) in any RAG pipeline. Designed for modularity and ease-of-use, backed by research. (github.com)
• duckdb/duckdb-rs: Ergonomic bindings to duckdb for Rust (github.com)
  • Rust API -- DuckDB

Sandboxing

• copy/v86: x86 PC emulator and x86-to-wasm JIT, running in the browser (demo)

Speech to Text

• openai/whisper-large-v2: Hugging Face
• m-bain/whisperX: Automatic Speech Recognition with Word-level Timestamps (& Diarization)
• SYSTRAN/faster-whisper: Faster Whisper transcription with CTranslate2
• collabora/WhisperLive: A nearly-live implementation of OpenAI's Whisper
• gaborvecsei/whisper-live-transcription: Live-Transcription (STT) with Whisper PoC (github.com)
• FL33TW00D/whisper-turbo: Cross-Platform, GPU Accelerated Whisper 🏎️ (github.com)

Multiplayer

• Veilid: open-source, peer-to-peer, mobile-first, networked application framework.
• stillonearth/bevy_veilid: Build two-player turn-based p2p games with Bevy and Veilid (github.com)

Impl

• beartype
• facebookresearch/torchdim: Named tensors with first-class dimensions for PyTorch
• Are we learning yet? A work-in-progress to catalog the state of machine learning in Rust
• PyO3/pyo3: Rust bindings for the Python interpreter

Assets

• Euphoric Heavy Regular
• Kenney Fonts
• Fira Mono - Google Fonts
• Xelu's FREE Controller Prompts (thoseawesomeguys.com)
• [DIVIDE BY ZERO] Fonts : 1998-infinity (tom7.com)
• [DIVIDE BY ZERO] Fonts : 1993-1997 (tom7.com)

Bevy stuff

• janhohenheim/foxtrot: The all-in-one Bevy 3D game template for desktop. (github.com)
• StarArawn/bevy_ecs_tilemap: A tilemap rendering crate for bevy which is more ECS friendly. (github.com)
• djeedai/bevy_hanabi: 🎆 Hanabi --- a GPU particle system plugin for the Bevy game engine. (github.com)
• QueryLens in bevy::ecs::system - Rust (docs.rs)
  • keywords for when I am trying to find this later: telescope, queryparams, systemparams
• Entity-entity relations 🌈 - Issue #3742 - bevyengine/bevy (github.com)
  • Flecs: Relationships
  • iiYese/aery: A plugin that enables a subset of entity relationship features for bevy (github.com)

Game inspiration

• The Last Clockwinder

Issues 👀

• Transparent example not working - Issue #2502 - rust-windowing/winit (github.com)

SDF

• SDF arbitary 2D polygon (shadertoy.com)
• Arbitrary Polygons, Blur/Border (shadertoy.com)

Video making

• Quickstart | Motion Canvas
  • examples/examples at master - motion-canvas/examples (github.com)
  • reviewing your motion canvas animations

Art

• LibreSprite/LibreSprite: Animated sprite editor & pixel art tool -- Fork of the last GPLv2 commit of Aseprite (github.com)

UI

• linebender/vello: An experimental GPU compute-centric 2D renderer. (github.com)
  • loopystudios/bevy_vello: An integration to render with Vello in Bevy game engine. (github.com)

Evolutionary Algorithms

• bones-ai/rust-snake-ai: Training a Neural Network to play snake, uses genetic algorithm for neuro-evolution (github.com)

Hot Reloading

• Hot Reloading Rust --- for Fun and Faster Feedback Cycles | Robert Krahn
• draivin/rust-hotswap: Easily hotswap functions in running executables (github.com)
• Hot Reloading Rust: Windows and Linux --- John Austin

Docs

• tbillington/bevy_game_docs: Generate documentation for Bevy games (github.com)
  • bevy_game_docs/src/main.rs at main - tbillington/bevy_game_docs (github.com)
  • Motion Blur, Visualizations, and Beautiful Renders - This Week in Bevy Engine

Permissions

• Open Policy Agent | Policy Language

Audio

• Added Audio Input Support by bushrat011899 - Pull Request #10072 - bevyengine/bevy (github.com)
• VB-Audio VoiceMeeter Banana
• Virtual Audio Device/Card/Adapter - help - The Rust Programming Language Forum (rust-lang.org)
• c++ - How to create a virtual audio input device to simulate a microphone on windows? - Stack Overflow
• Windows-driver-samples/audio/simpleaudiosample at main - microsoft/Windows-driver-samples (github.com)
• wdmaudiodev Mailing List Archive (freelists.org)

Animations

• rive-app/rive-bevy (github.com)
  • How Duolingo Animates Its World Characters

Models

• mxbai-embed-large (ollama.com)
• starcoder2 (ollama.com)
• dolphincoder (ollama.com)
• Function Calling in Ollama vs OpenAI (youtube.com)

Transparent windows not working

• Bevy Transparent Window not working - Issue #10929 - bevyengine/bevy (github.com)
• Window transparency broken on Windows - Issue #7544 - bevyengine/bevy (github.com)
• Transparent example not working - Issue #2502 - rust-windowing/winit (github.com)
• Issues - rust-windowing/winit (github.com)
• transparent example doesn't work because softbuffer doesn't support transparency - Issue #2960 - rust-windowing/winit (github.com)
• Transparent example not working - Issue #2502 - rust-windowing/winit (github.com)
• On Windows, improve support for undecorated windows by msiglreith - Pull Request #2419 - rust-windowing/winit (github.com)
• fixed window transparency by mgalos999 - Pull Request #2503 - rust-windowing/winit (github.com)
• rust-windowing/glutin: A low-level library for OpenGL context creation, written in pure Rust. (github.com)
• Issues - rust-windowing/glutin (github.com)
• X11 Window is always transparent with 0.30 - Issue #1559 - rust-windowing/glutin (github.com)
• Transparency broken on X11 - Issue #1191 - rust-windowing/glutin (github.com)
• Triangle example fails - cargo.toml problem? - Issue #32 - coderedart/egui_overlay (github.com)
• winit::changelog::v0_30 - Rust (docs.rs)

Reference - AI Agents

• SysCV/sam-hq: Segment Anything in High Quality [NeurIPS 2023] (github.com)
• OpenAdaptAI/OpenAdapt: AI-First Process Automation with Large ([Language (LLMs) / Action (LAMs) / Multimodal (LMMs)] / Visual Language (VLMs)) Models (github.com)
  • OpenAdapt.AI
  • (12) Richard Abrich on X: "Check out the latest from @OpenAdaptAI. Now you can quickly and easily automate repetitive tasks in desktop apps for free! Compatible with @OpenAI, @Anthropic, @GoogleAI, and more. Go to https://t.co/7WzHcDRrRH to get started. https://t.co/KkC24CxJFK" / X (twitter.com)
• OpenInterpreter/open-interpreter: A natural language interface for computers (github.com)
  • Open Interpreter - YouTube
  • George Kedenburg III on X: "ai pin 🤝 open interpreter https://t.co/cUU2zGoCfd" / X (twitter.com)
• Techfrens AI Agents Rubrik - Google Drive
• jordan singer on X: "✨ talk to your computer remotely from your phone i call it Teleport https://t.co/EvktAB1Lrz" / X (twitter.com)
• princeton-nlp/SWE-agent: SWE-agent takes a GitHub issue and tries to automatically fix it, using GPT-4, or your LM of choice. It solves 12.29% of bugs in the SWE-bench evaluation set and takes just 1.5 minutes to run.
  • John Yang @ ICLR 🇦🇹 on X: "SWE-agent is our new system for autonomously solving issues in GitHub repos. It gets similar accuracy to Devin on SWE-bench, takes 93 seconds on avg + it's open source! We designed a new agent-computer interface to make it easy for GPT-4 to edit+run code https://t.co/CTzMxDiouH https://t.co/VW9FuZGIUf" / X (twitter.com)

Articles

• Why it is time to start thinking of games as databases | by Sander Mertens | Medium

Screen Capturing / Recording

• Show HN: I made an open-source Loom alternative | Hacker News (ycombinator.com)
  • Capture/Desktop -- FFmpeg
  • Cap --- Effortless, instant screen sharing. Open source and cross-platform.

Controlling Controllers

• TODO: add links about steam controller chords
  • meta+rstick move mouse
  • meta+r1 click
  • meta+y turn off (doesn't work? haven't restarted since installing extended xbox support)

Music Archival

• librespot-org/librespot: Open Source Spotify client library (github.com)
• zotify-dev/zotify: A fast and customizable music and podcast downloader. (github.com)
• GuillemCastro/spotify-dl: A command-line utility to download songs and playlists directly from Spotify's servers (github.com)

Text editor

• 
  • tsoding/ded: Dramatic EDitor (github.com)
  • Rope (data structure) - Wikipedia
  • tsoding/noed: Not ed(1) text editor (github.com)