Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implementation of @quicklisp runner #473

Closed
Kreyren opened this issue Oct 8, 2020 · 20 comments
Closed

Implementation of @quicklisp runner #473

Kreyren opened this issue Oct 8, 2020 · 20 comments
Assignees
Milestone

Comments

@Kreyren
Copy link
Contributor

Kreyren commented Oct 8, 2020

DISCLAIMER: I tried to make this as short as possible, but there is lots of info to be processed x.x

DISCLAIMER: I am not a lisp backend developer (i just use the language on preferred implementation atm) so the information is provided to the best of my ability and may be inaccurate, peer-reviews were made and addressed.Any relevant information/criticism is welcomed.

github_quicklisp

This is a feature request to implement quicklisp https://www.quicklisp.org/beta/ which is a deployed through loadable lisp library expected to allow for implementation-independent code in cargo-make https://github.com/sagiegurari/cargo-make which is arguably a better alternative alternative to make command reading Makefile.

Expectation

The ability to use lisp and/or common-lisp (programming language) called from cargo-make on all supported devices by rustlang and/or *lisp (https://doc.rust-lang.org/nightly/rustc/platform-support.html) with readable and fault tolerant implementation-independent implementation while not preventing implementation of other touring complete systems.

Issue

Currently cargo-make version =0.32.6 requires the following entry to run common lisp through Embedded Common Lisp (ecl) to use implementation independent code style:

[env]
MESSAGE = "something"

# NOTICE(Krey): You will need quicklisp installed

[tasks.kreyren]
script_runner = "ecl"
script_runner_args = [ "--norc", "--quiet", "--shell" ]
script_extension = "cl"
script = [
'''
(setf *load-verbose* nil)
(load "/home/kreyren/quicklisp/setup.lisp" :verbose nil)
(ql:quickload :uiop :silent t)

(write-line (uiop:getenv "MESSAGE"))
'''
]

to return something which is hard to read and maintain as it requires duplicate code (namely script_runner_args and lisp lines above write-line)

Where the expected is:

[env]
MESSAGE = "something"

[tasks.kreyren]
script_runner = "@quicklisp"
script = [
'''
(write-line (uiop:getenv "MESSAGE"))
'''
]

Implementation compatibility

Common lisp has many implementations alike:

  • CLISP (clisp) - Implementation of Common lisp written in C
  • CCL (Clozure CL)
  • Clasp - Effort to write C++/LLVM common lisp implementation
  • ABCL (Armed Bear CL)
  • ACL (Allegro CL)
  • LW (LispWorks)
  • CMUCL (Carnegie Mellon University)
  • MKCL (ManKai)
  • Scieneer CL
  • SICL - Work in progress implementation
  • Embedded Common Lisp (ECL)
  • Steel Bank Common Lisp (SBCL)
  • possibly more..

Where following are dialects of lisp that are not supported by quicklisp as quicklisp is depending on ASDF that is using CLOS that is close to impossible to port to these (See statement below):

Where hard-coded logic is mentioned in specification: http://www.lispworks.com/documentation/HyperSpec/Body/03_ababa.htm which are implemented in quicklisp that allows to write implementation-independent common lisp (meaning that the common lisp code written will work on all other implementations).

Example of implemendation dependent code printing value of environment variable MESSAGE:

(write-line (ext:getenv "MESSAGE"))

as ext is specific to ecl and sbcl (possibly others..).

Whereas this implementation works on all implementation provided:

(load #p"~/quicklisp/setup.lisp")
(write-line (uiop:getenv "MESSAGE"))

Thus the script_runner = "@quicklisp" should be looking for executables capable of processing the runtime instead of depending only on the hard-coded.

Silencing unwanted output

By default quicklisp is outputting lots of unwanted informations:

kreyren@leonid:~$ export MESSAGE=kreyren
kreyren@leonid:~$ ecl
ECL (Embeddable Common-Lisp) 20.4.24 (git:UNKNOWN)
Copyright (C) 1984 Taiichi Yuasa and Masami Hagiya
Copyright (C) 1993 Giuseppe Attardi
Copyright (C) 2013 Juan J. Garcia-Ripoll
Copyright (C) 2018 Daniel Kochmanski
Copyright (C) 2020 Daniel Kochmanski and Marius Gerbershagen
ECL is free software, and you are welcome to redistribute it
under certain conditions; see file 'Copyright' for details.
Type :h for Help.  
Top level in: #<process TOP-LEVEL 0x7fb1d054af80>.
> (load "/home/kreyren/quicklisp/setup.lisp")

;;; Loading "/home/kreyren/quicklisp/setup.lisp"
;;; Loading #P"/usr/lib/x86_64-linux-gnu/ecl-20.4.24/asdf.fas"
#P"/home/kreyren/quicklisp/setup.lisp"
> (ql:quickload :uiop)
To load "uiop":
  Load 1 ASDF system:
    uiop
; Loading "uiop"

(:UIOP)
> (write-line (uiop:getenv "MESSAGE"))
kreyren
"kreyren"
> 

To silence these on ecl it's expected to use:

  • --shell to silence the header including copyright
  • (setf *load-verbose* nil) to silence the ;; Loading .. messages from ecl
  • (load "/home/kreyren/quicklisp/setup.lisp" :verbose nil) to silence the ;;; Loading #P"/usr/lib/x86_64-linux-gnu/ecl-20.4.24/asdf.fas" #P"/home/kreyren/quicklisp/setup.lisp" from quicklisp itself
  • (ql:quickload :uiop :silent t) to silence the loading of quicklisp in the said implementation

Additionally we need argument --norc to avoid sourcing of ~/.eclrc which could interfiere with the logic in cargo-make's script.

on clisp this is getting:

kreyren@leonid:~$ MESSAGE=kreyren clisp test.lisp 
WARNING: DEFGENERIC: redefining function DIST in
         /home/kreyren/.cache/common-lisp/clisp-2.49.92-unix-x64/home/kreyren/quicklisp/quicklisp/dist.fas,
         was defined in top-level
WARNING: DEFGENERIC: redefining function SYSTEM-INDEX-URL in
         /home/kreyren/.cache/common-lisp/clisp-2.49.92-unix-x64/home/kreyren/quicklisp/quicklisp/dist.fas,
         was defined in top-level
WARNING: DEFGENERIC: redefining function RELEASE-INDEX-URL in
         /home/kreyren/.cache/common-lisp/clisp-2.49.92-unix-x64/home/kreyren/quicklisp/quicklisp/dist.fas,
         was defined in top-level
WARNING: DEFGENERIC: redefining function AVAILABLE-VERSIONS-URL in
         /home/kreyren/.cache/common-lisp/clisp-2.49.92-unix-x64/home/kreyren/quicklisp/quicklisp/dist.fas,
         was defined in top-level
WARNING: DEFGENERIC: redefining function RELEASE in
         /home/kreyren/.cache/common-lisp/clisp-2.49.92-unix-x64/home/kreyren/quicklisp/quicklisp/dist.fas,
         was defined in top-level
WARNING: DEFGENERIC: redefining function NAME in
         /home/kreyren/.cache/common-lisp/clisp-2.49.92-unix-x64/home/kreyren/quicklisp/quicklisp/dist.fas,
         was defined in top-level
WARNING: DEFGENERIC: redefining function BASE-DIRECTORY in
         /home/kreyren/.cache/common-lisp/clisp-2.49.92-unix-x64/home/kreyren/quicklisp/quicklisp/dist.fas,
         was defined in top-level
WARNING: DEFGENERIC: redefining function METADATA-NAME in
         /home/kreyren/.cache/common-lisp/clisp-2.49.92-unix-x64/home/kreyren/quicklisp/quicklisp/dist.fas,
         was defined in top-level
WARNING: DEFGENERIC: redefining function PREFERENCE-PARENT in
         /home/kreyren/.cache/common-lisp/clisp-2.49.92-unix-x64/home/kreyren/quicklisp/quicklisp/dist.fas,
         was defined in top-level
WARNING: DEFGENERIC: redefining function SHORT-DESCRIPTION in
         /home/kreyren/.cache/common-lisp/clisp-2.49.92-unix-x64/home/kreyren/quicklisp/quicklisp/dist.fas,
         was defined in top-level
WARNING: DEFGENERIC: redefining function PROVIDED-RELEASES in
         /home/kreyren/.cache/common-lisp/clisp-2.49.92-unix-x64/home/kreyren/quicklisp/quicklisp/dist.fas,
         was defined in top-level
WARNING: DEFGENERIC: redefining function PROVIDED-SYSTEMS in
         /home/kreyren/.cache/common-lisp/clisp-2.49.92-unix-x64/home/kreyren/quicklisp/quicklisp/dist.fas,
         was defined in top-level
WARNING: DEFGENERIC: redefining function ARCHIVE-URL in
         /home/kreyren/.cache/common-lisp/clisp-2.49.92-unix-x64/home/kreyren/quicklisp/quicklisp/dist.fas,
         was defined in top-level
kreyren

To silence these we need --quiet

Handling of arguments per implementation

Recommend implementing script_runner_args at the background depending on found executable namely this should be doing if the end-user did not overwrite script_runner_args already:

# ECL
script_runner_args = [ "--norc", "--quiet", "--shell" ]
script_extension = "cl"
# clisp
script_runner_args = [ "--norc", "--quiet" ]
script_extension = "cl"
# elisp - Doesn't work atm
script_runner_args = [ "--quick" ,"--script" ]
script_extension = "cl"
# sbcl
script_runner_args = [ "--no-userinit", "--script" ]
script_extension = "cl"

rlisp

rlisp was concluded to be not usable https://github.com/Kreyren/rust-lisp/actions/runs/295136199

Created swgillespie/rust-lisp#7 to track the code usability

Filed swgillespie/rust-lisp#6 to get more info

elisp

elisp is able to process the file using script_runner_args = [ "--quick" ,"--script" ], but does not work:

Loading /home/kreyren/quicklisp/setup.lisp...
Symbol’s function definition is void: defpackage

Filed quicklisp/quicklisp-bootstrap#21 for the elisp compatibility of quicklisp

EDIT: Is not supported

Prepending lisp code to the created scripts

For the implementation to be able to implement quicklisp we need to source quicklisp library which can be done by prepending

(load #p"~/quicklisp/setup.lisp")

assuming that it has been installed on the system.

Deployment of quicklisp

To be able to run quicklisp we need to run https://beta.quicklisp.org/quicklisp.lisp to get the backend to be used in the implementation.

FIXME: How to implement this in cargo-make?

Quicklisp goal

Allegedly the goal is to make it easy to distribute and update Lisp code over the Internet, which may interfere in presented usecase.

Caches

Worth mentioning that common-lisp is caching it's functions in ~/.cache/common-lisp which might influence the runtime as the changes might not be present in the real-time

Quicklisp compatibility with non-common lisp

This is a quote of Zach Beane (@xach) from irc.freenode.net/#lisp (was allowed to quote):

<Xach> quicklisp is a common lisp program. other lisps are not common lisp and are not compatible.
<Xach> other lisps could be made compatible. it's a lot of work. nobody has done it. common lisp has a lot of features and quicklisp uses a lot of 
them.
<Xach> also, quicklisp uses extra-standard functionality that would also need implementation - networking and filesystem work mostly.
<kreyren> Is quicklisp implemented by design to allow possible implementation of non-CL interpretations or would that require lots of rewritting?
<Xach> kreyren: it is implemented by design to take full advantage of Common Lisp and I think trying to make it work elsewhere would be difficult and not very rewarding - since it is meant to allow you to run other common lisp programs, which also are not portable to other lisps.
<Xach> "Lisp" isn't generic - there are only specifics
<Xach> I think the ideas of quicklisp are pretty portable, even if the code itself is not especially

My usecase

I want to use (C)lisp on my repositories that are aiming to be cross-platform compatible with ideology to support as many devices as possible where it's not limiting me in terms of technology for things where it's not practical to implement them in rustlang (scripts).

I prefer rustlang since it allow me to outsource my code in libraries (crates) allowing for passive maintenance of my codebase which is much less efficient on C + i am fed up with keep hotfixing of C standard issues and wasting time looking for memory leaks where it seems that rustlang is not less efficient then C assuming optimization made i.e. comparing fibonacci in rustlang https://github.com/Kreyren/rustlang-fibonacci/tree/kreyren/case-study-performance-2 to C and Lisp.

Rustlang currently works on less devices compared to lisp which i want to in worst case scenario implement though lisp wrapper to read the Makefile.toml.

Clisp maintenance

Based on activity of https://gitlab.com/gnu-clisp/clisp it was advised to not rely on clisp as it seems somewhat unmaintained which might be subjective assuming that there doesn't seem to be any actionable issues.

EDIT: Merging the https://gitlab.com/gnu-clisp/clisp/-/merge_requests/3 seems actionable enough assuming unmaintained.

EDIT2: requested an official statement from GNU about maintenance.

References

  1. https://courses.cs.washington.edu/courses/cse341/04wi/lectures/14-scheme-quote.html
  2. emacs scripting https://www.emacswiki.org/emacs/EmacsScripts
  3. Quicklisp on github https://github.com/quicklisp
  4. http://clhs.lisp.se/
  5. https://web.archive.org/web/20200426054415/http://home.pipeline.com/~hbaker1/TInference.html
  6. Rust your own lisp https://dev.to/deciduously/rust-your-own-lisp-50an
  7. Risp https://stopa.io/post/222
  8. rust_lisp https://crates.io/crates/rust_lisp
  9. Ketos https://crates.io/crates/ketos
@sagiegurari
Copy link
Owner

thanks for all the details.
are you familiar with the extend task attribute?

you can do something like the following

  • define a base task with all the stuff you need to run ecl the way you want it, but without the script
  • have other tasks provide the actual script and just extend that base task
[tasks.ecl]
script_runner = "ecl"
script_runner_args = [ "--norc", "--quiet", "--shell" ]
script_extension = "cl"

[tasks.script1]
extend = "ecl"
script = [
'''
(setf *load-verbose* nil)
(load "/home/kreyren/quicklisp/setup.lisp" :verbose nil)
(ql:quickload :uiop :silent t)

(write-line (uiop:getenv "MESSAGE"))
'''
]

[tasks.script2]
extend = "ecl"
script = [
'''
(setf *load-verbose* nil)
(load "/home/kreyren/quicklisp/setup.lisp" :verbose nil)
(ql:quickload :uiop :silent t)

(write-line (uiop:getenv "MESSAGE"))
'''
]

@sagiegurari
Copy link
Owner

by the way, in the base task you can also setup installation instructions on how to setup ecl if missing in your machine.
you can do that via install_script attribute:
https://github.com/sagiegurari/cargo-make#usage-installing-dependencies

@sagiegurari
Copy link
Owner

@Kreyren did you have a chance to look at my solution?

@Kreyren
Copy link
Contributor Author

Kreyren commented Oct 15, 2020

@sagiegurari yep, sorry for the delay had/have issues on infra (DNS Server, ISP and sDNS provider are ruining my life) x.x


are you familiar with the extend task attribute?

That would solve just a part of the issue as this needs quicklisp installed and configured to run per implementation to not have:

(setf *load-verbose* nil)
(load "/home/kreyren/quicklisp/setup.lisp" :verbose nil)
(ql:quickload :uiop :silent t)

At the header of the file which is different per implementation.

Installing dependencies noted, do you propose to make a logic there for the invidual implementations? I would ideally want to avoid installing ecl on systems that already have an implementation installed.

@sagiegurari
Copy link
Owner

  • header - sorry i'm not a lisp person. i only did few things during my university time 20 years ago :)
    So you are saying you need to write those 3 lisp lines all of the time? because (and i'm not a lisp person) those seem very specific to how you do things (like loading setup.lisp from specific path) and not a must for lisp execution in general.
    so if i did have a @ecl runner, it would still not trigger those.

  • installation - checkout the section i talked about.
    for rust you just define meta data and i do the rest, including ensuring not to install if already installed or actually upgrade if too old and so on... so if you are using a lisp implementation written in rust, you are set.
    if not, still, the installation section does talk about installing shell script to do whatever you want, so its up to you. including first checking the lisp executable is available check.

@Kreyren
Copy link
Contributor Author

Kreyren commented Oct 15, 2020

header - sorry i'm not a lisp person. i only did few things during my university time 20 years ago :)

Noted

So you are saying you need to write those 3 lisp lines all of the time? because (and i'm not a lisp person) those seem very specific to how you do things (like loading setup.lisp from specific path) and not a must for lisp execution in general.

Yes these three lines and parsed arguments per implementation (i.e. ecl) are requires to run the code alike:

Expecting to be able to use:

[env]
MESSAGE = "something"

[tasks.kreyren]
script_runner = "@quicklisp"
script = [
'''
(write-line (uiop:getenv "MESSAGE"))
'''
]

Without them the implementation either doesn't parse the created file correctly or prints unwanted messages (see Silencing unwanted output in OP.)

so if i did have a @ECL runner, it would still not trigger those.

FWIW this is a quicklisp which is a dialect of common lisp so using @ecl would be confusing.

;; Printing value from env var in quicklisp
(write-line (uiop:getenv "MESSAGE"))

;; Printing value from env var in ecl
(write-line (ext:getenv "MESSAGE"))

Where ext only works on ecl and sbcl implementations (possibly more) which makes it implementation dependent (unwanted).


Mentioned installation seems to allow implementing the flags, but doesn't seem to allow the header and would be probably pita to implement at which case it would in my mind make more sense as part of cargo (with hopefully common lisp interpreter at some point)

@sagiegurari
Copy link
Owner

So, i'm re-reading the whole thing and still bottom line the requirement is that you would have a new runner @quicklist for example which adds 3 lines automatically for all such scripts:

(setf *load-verbose* nil)
(load "/home/kreyren/quicklisp/setup.lisp" :verbose nil)
(ql:quickload :uiop :silent t)

but

  • 1 line is specific to your installation path
  • silent is something you want, but its not default for quicklist so why should it be default here?

so, at the end, it boils down to something really really specific for quicklisp + your preferences.

I think the base task

[tasks.ecl]
script_runner = "ecl"
script_runner_args = [ "--norc", "--quiet", "--shell" ]
script_extension = "cl"

that you add in your makefile and than reuse it

[tasks.script1]
extend = "ecl"
script = [
'''
(setf *load-verbose* nil)
(load "/home/kreyren/quicklisp/setup.lisp" :verbose nil)
(ql:quickload :uiop :silent t)

(write-line (uiop:getenv "MESSAGE"))
'''
]

gets you 95% covered and all you need is to add those lines to your script code.
it feels to me its the more correct approach here.
adding a runner to do these things feels like a bit out of scope for cargo make.

the fact is i have 3 unique runners (apart of the generic runner you can put)

  • @shell - converts sh to windows bat
  • @rust - Runs rust code - cargo make is mainly for rust projects so this gets special handling
  • @duckscript - I had to write this one because @shell is super limited and i needed a way to give out of the box cross platform shell like experience.

i wasn't planning on adding more. thats what the generic runner setup is for (as i gave an example with tasks.ecl).

@Kreyren
Copy link
Contributor Author

Kreyren commented Oct 23, 2020

Sorry for taking long time to respond again x.x


These three lines are not my preferences, but configuration for the implementation to allow running from cargo-make so that it can process the provided scripts.

Without these the implementation either spams in the cargo-make output or fails to process the script.

Currently this requires the quicklisp backend installed on the target userland which ideally i would like vendored through cargo-make on demand within a userland independent directory that doesn't make changes to the target system.

gets you 95% covered and all you need is to add those lines to your script code.
it feels to me its the more correct approach here.

That's not a practical implementation as it needs these three lines each time and is depending on the implementation that the code was written for instead of utilizing the implementation independency introduced through quicklisp.

@sagiegurari
Copy link
Owner

let me think of how to enable you to configure this....

@roblabla
Copy link
Contributor

roblabla commented Oct 31, 2020

Couldn't extend "merge" scripts sections? e.g. have:

[tasks.ecl]
script_runner = "ecl"
script_runner_args = [ "--norc", "--quiet", "--shell" ]
script_extension = "cl"
script = [
'''
(setf *load-verbose* nil)
(load "/home/kreyren/quicklisp/setup.lisp" :verbose nil)
(ql:quickload :uiop :silent t)
''']

[tasks.script1]
extend = "ecl"
script = [
'''
(write-line (uiop:getenv "MESSAGE"))
'''
]

I think this would be useful beyond quicklisp. You could imagine a runner setting up some variables (not env vars, actual variables specific to that runner) before execution. Maybe even add a pre_script and post_script meant to be used by task-runner?

@sagiegurari
Copy link
Owner

pre/post acript is an interesting idea. I'll take a look.
thanks for the suggestion.

sagiegurari added a commit that referenced this issue Nov 1, 2020
…) to enable sharing common script content between tasks #473
@sagiegurari
Copy link
Owner

@roblabla @Kreyren i implemented pre/main/post script sections as suggested so we can now share scripts between tasks partially. so you should be able to do something like this:

[tasks.ecl]
script_runner = "ecl"
script_runner_args = [ "--norc", "--quiet", "--shell" ]
script_extension = "cl"
script.pre = '''
(setf *load-verbose* nil)
(load "/home/kreyren/quicklisp/setup.lisp" :verbose nil)
(ql:quickload :uiop :silent t)
'''

[tasks.script1]
extend = "ecl"
script.main = '''
(write-line (uiop:getenv "MESSAGE"))
'''

[tasks.script2]
extend = "ecl"
script.main = '''
(write-line (uiop:getenv "MESSAGE"))
'''

@Kreyren
Copy link
Contributor Author

Kreyren commented Nov 3, 2020

Thanks for working on this! ^-^

This resolves the issue for hard-codded executables, but is there a way to apply the script.pre depending on the executable used?

@sagiegurari
Copy link
Owner

just create different base tasks to extend from
instead of selecting executable in you actual task, you select base task to extend.
the base task can also extend other tasks as needed.

@sagiegurari
Copy link
Owner

maybe a more complete example will explain better:

[tasks.ecl]
script_runner_args = [ "--norc", "--quiet", "--shell" ]
script_extension = "cl"

[tasks.ecl_1]
extend = "ecl"
script_runner = "ecl_1"
script.pre = '''
(setf *load-verbose* nil)
(load "/home/kreyren/quicklisp/setup.lisp" :verbose nil)
(ql:quickload :uiop :silent t)
'''

[tasks.ecl_2]
extend = "ecl"
script_runner = "ecl_2"
script.pre = '''
(setf *load-verbose* nil)
(load "/home/kreyren/quicklisp/setup.lisp" :verbose nil)
(ql:quickload :uiop :silent t)
'''

[tasks.script1]
extend = "ecl_1"
script.main = '''
(write-line (uiop:getenv "MESSAGE"))
'''

[tasks.script2]
extend = "ecl_2"
script.main = '''
(write-line (uiop:getenv "MESSAGE"))
'''

@Kreyren
Copy link
Contributor Author

Kreyren commented Nov 5, 2020

That example is kinda bloated, decided to use ecl only atm.

@sagiegurari
Copy link
Owner

its bloated as it is an example of having 2 lisp langs with minor differences sharing logic.
it is meant to answer your last question

@sagiegurari sagiegurari added this to the 0.32.8 milestone Nov 5, 2020
@sagiegurari
Copy link
Owner

this feature is now published.

@Kreyren
Copy link
Contributor Author

Kreyren commented Jan 21, 2021

That example is kinda bloated, decided to use ecl only atm.

I take it back i like it!

@sagiegurari
Copy link
Owner

happy to hear :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants