Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Deploy to KeyBank VM (most likely) #8

Closed
25 tasks done
andygello555 opened this issue Jan 13, 2023 · 0 comments · Fixed by #12
Closed
25 tasks done

Deploy to KeyBank VM (most likely) #8

andygello555 opened this issue Jan 13, 2023 · 0 comments · Fixed by #12
Assignees
@andygello555
Labels
deployment Deployment creation/modification/fixes

Comments

@andygello555
Copy link
Owner

andygello555 commented Jan 13, 2023 •
edited

Deploy the Robo-scout to the KeyBank VM

  • Write a supervisor config for it
  • Create a directory to hold the log files
  • Install go on the VM
  • Build an executable for game-scout
  • Install mono on the VM
  • Install the steamcmd executable
    • I followed the Debian 11 and above note on the SteamCMD install page
    • Had to add non-free packages to the Debian source lists
    • Have to update the way that steamcmd processes are killed as for some reason it's not working on Linux.
      • It was actually a problem with the way that the steamcmd process is started on Linux
      • I had to create a copy of the /home/$USER/.local/share/Steam/steamcmd/steamcmd.sh in directory that is earlier on in the PATH such as /home/$USER/.local/bin/steamcmd
      • I then had to modify the STEAMROOT variable to point to the /home/$USER/.local/share/Steam/steamcmd/ directory
      • Finally, I added an extra branch to the debugger check to check if the DEBUGGER variable is not set. If so, then it will exec $STEAMEXE $@. The old version of the script just creates a new process for the steamcmd executable, leaving the steamcmd.sh process behind when killed.
      • I'll add the new version of this file to the repo
  • Build ScoutWebPipes
  • Install dependencies for playwright-go package
    • To do this successfully on Debian 11 I had to aid the apt repositories for buster to the sources list
    • This is the answer that I used as reference
  • Copy over the local config.json and update it for production uses
    • Debug flags
    • Email addresses
    • ScoutWebPipes binary location
    • Headless flags
  • Create a new user in the postgres server
  • Create a new DB in the postgres server using the new user
  • Create a backup of the local DB and restore it in the new DB on production
  • Iron out any https://github.com/RichardKnop/machinery quirks and kinks
    • Scheduling issues with periodic tasks
    • Logging output probably needs to be redirected for Scout task
    • Need to make it so that Scout task retries if failed
    • Need to add some simple email templates for text only templates so that I can notify admins when a scrape has
      • Started
      • Ended
      • When an error has occurred

Monday.com Item ID: #3808494168
@andygello555 andygello555 added the deployment Deployment creation/modification/fixes label Jan 13, 2023
@andygello555 andygello555 self-assigned this Jan 13, 2023
@andygello555 andygello555 added deployment Deployment creation/modification/fixes and removed deployment Deployment creation/modification/fixes labels Jan 13, 2023
andygello555 added a commit that referenced this issue Jan 25, 2023
@andygello555
ScoutResult model and hooks #3
d9266f3 
- Added a function to tasks to enable registering periodic tasks #8 (25/01/2023 - 11:39:58)
- Added the ScoutResult model that tracks the statistics of a single run of the Scout procedure (25/01/2023 - 12:58:33)
- Added the Result field to ScoutState. This can be set using the StateType.SetOrAdd method that uses reflection to descend arbitrarily down inside the ScoutResult and set the field represented by the path embedded into the previous arguments to the final argument provided (25/01/2023 - 13:01:00)
- Added the Email.Size method (25/01/2023 - 14:18:11)
- Added more methods to email.Profiling (25/01/2023 - 14:27:38)
- Not actually calling the deferred Profiling.Start methods in email.go. I am now tho (25/01/2023 - 14:31:24)
- Added hooks for ScoutResult.DiscoveryStats to the DiscoveryPhase (25/01/2023 - 15:57:52)
- Added hooks for ScoutResult.UpdateStats to the UpdatePhase (this just changed the same DiscoveryBatch to handle Update Phase) (25/01/2023 - 16:41:24)
- Added hooks for ScoutResult.SnapshotStats to the SnapshotPhase (25/01/2023 - 16:41:47)
- Added hooks for ScoutResult.DisableStats to the DisablePhase (25/01/2023 - 16:42:19)
- Added hooks for ScoutResult.EnableStats to the EnablePhase (25/01/2023 - 16:46:13)
- Added hooks for ScoutResult.DeleteStats to the DeletePhase (25/01/2023 - 16:50:19)
- Added hooks for ScoutResult.MeasureStats to the MeasurePhase (25/01/2023 - 16:53:31)
- Added a RegisterModel line for the new ScoutResult model (25/01/2023 - 19:07:05)
- Moved the creation of the ScoutResult instance to the end of the Scout procedure (25/01/2023 - 23:04:36)
andygello555 added a commit that referenced this issue Jan 31, 2023
@andygello555
server dir + create_backup + create_sample_csv for #8
2bc9800 
- Added the server directory to keep track of config files and scripts (31/01/2023 - 12:10:35)
- Added the script that I use to start steamcmd to the server/local/ dir (31/01/2023 - 12:10:55)
- Moved the create_backup.sh and create_sample_csv.sh scripts to the server/tools dir (31/01/2023 - 12:11:34)
- Updated the create_backup and create_sample_csv scripts to use the root of the git repository as a reference for where the config.json exists (31/01/2023 - 12:32:49)
andygello555 added a commit that referenced this issue Jan 31, 2023
@andygello555
supervisor config + build.sh
b11a609 
- Added the supervisor config for game-scout. This only works if a built game-scout executable exists in the repo #8 (31/01/2023 - 13:14:48)
- Added the game-scout binary to the gitignore (31/01/2023 - 13:15:35)
- Added a script to build the entire project including the ScoutWebPipes executable (31/01/2023 - 13:22:10)
andygello555 added a commit that referenced this issue Feb 1, 2023
@andygello555
StartedContext + changes to PathsToBytes and Template sending
b8ca9e1 
- Added the error and started templates which are both new txt based templates (01/02/2023 - 10:35:17)
- Renamed Context.HTML to Context.Execute as it now can call either Template.HTML or Template.Text (01/02/2023 - 10:35:51)
- Added Text as a new TemplateBufferContentType (01/02/2023 - 10:36:17)
- Moved all implementations of email.Context to contexts.go (01/02/2023 - 10:42:28)
- Template.Template is now the new parsedType which can store a reference to both a parsed HTML template and a parsed text template (01/02/2023 - 12:00:28)
- parsedTemplate.Execute will call the execute method of the set parsed template (01/02/2023 - 12:01:05)
- Renamed Template.HTML to Execute and it now can also execute text templates along with HTML templates (01/02/2023 - 12:15:17)
- Added the Template.SendAsyncAndConsume method for use cases where we want to send an email asynchronously and continue doing other stuff (01/02/2023 - 13:33:07)
- The StartedContext and ErrorContext will probably need to exist within the main package so that we can pass a ScoutState instance to them (01/02/2023 - 14:51:22)
- Added new interfaces for reading and writing to PathsToBytes. These are PathsToBytesInterface, PathsToBytesReader, PathsToBytesWriter, and PathsToBytesReadWriter. This is so I could create another type similar to the existing PathsToBytes that writes to a zip archive for email sending in StartedContext and ErrorContext (01/02/2023 - 15:55:12)
- Added the ScoutState.LoadFrom and ScoutState.SaveTo methods which can be given a PathsToBytesReader and a PathsToBytesWriter respectively to load/save a ScoutState to (01/02/2023 - 15:56:37)
- StartedContext should now be finished and ready for implementation in the Scout procedure #8 (01/02/2023 - 16:34:37)
- Added a check to Part.Encoder which stops zip attachment from being re-archived (01/02/2023 - 16:35:28)
andygello555 added a commit that referenced this issue Feb 2, 2023
@andygello555
Finished template + refac state.go + refac periodic tasks #8
6b91d1f 
- Email.AddPart now checks whether an encoded attachment exceeds the max number of bytes for an attachment. If this is the case and Part.DropIfBig is not set then an error will be returned, otherwise the Part will not be added to the Email silently (02/02/2023 - 10:48:39)
- Added the Finished TemplatePath and the FinishedContext (02/02/2023 - 10:49:04)
- Moved all the PathsToBytes stuff to a new file in main along with some further documentation on the new PathsToBytes interface introduced previously (02/02/2023 - 10:59:08)
- Moved all cached field types and symbols to the new cached_fields.go file in main (02/02/2023 - 11:32:59)
- Implemented sends for all the new templates into the Scout procedure (02/02/2023 - 12:31:31)
- Fully implemented the Finished template (02/02/2023 - 13:43:17)
- Periodic tasks are now registered from the config using a new field within TaskConfig called PeriodicTaskSignatures (02/02/2023 - 14:24:02)
- Updates to tasks.StartServer to display registered tasks + updates to errors (02/02/2023 - 14:51:04)
@andygello555 andygello555 linked a pull request Feb 2, 2023 that will close this issue
Deployment changes and a few QoL changes #12
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@andygello555 andygello555

Labels
deployment Deployment creation/modification/fixes
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Deployment changes and a few QoL changes andygello555/game-scout
1 participant
@andygello555