Broken methods

[Kirtai]

I was looking through the Smalltalk-78 image on the Zoo and noticed that some of the methods could not be found when selected in the browser. I also noticed a lot of uncategorised methods and missing Class comments.

Would you be willing to accept fixes from the ST-76 sources extracted from the xmsmall.dsk disk image? Example is attached.

It's Point with the broken #fromPress:value:, #hideData: and #pressCode replaced and recategorized, a class comment added, and #copy recategorized.

This was with a fully updated base image. I checked a fully updated full image but it seemed to be missing even more in the way of temps etc.
Is there a preferred image to work on?

Point.zip

[codefrau]

The original image (or rather, the two files found on the disk pack, object table and object space) had no categories or method sources at all. To keep the system small (to make it fit into the Notetaker's memory) everything was decompiled from bytecode. It is in the Lively version at the Smalltalk Zoo (the code to read it is in vm.js, see class ObjectTableReader).

Come to think of it, we should add those two files to the GitHub repo so we can replicate updating from the original to the current version (see updates/makeBaseImage-bf.st). We switched the update stream URL a few times, but we can map them all to the current URL in the VM primitive). This would give us a reproducible way to build an updated image.

In that original version, Point>>fromPress:value is there (see my screenshot). The sources must have been broken by something in the update stream. Ted imported a lot of sources from St76 (e.g. 5621fdb), my guess is the broken sources come from that. See section 4.3 and Appendix B in our “Reviving Smalltalk-78” paper.

And yes, I'd happily accept fixes. Preferably as updates in the update stream. The way we used to work is by pushing updates to the update stream, and later me committing them to git and pushing to github. I could explain how to do that – or we could come up with a process to commit to git first and then pull that into the update stream? Ideally, the image would load updates from and push to GitHub directly, but I'm not entirely sure how much effort that would be.

[codefrau]

Come to think of it, we should add those two files to the GitHub repo

I just did that, and you can run the original now. After pasting the contents of updates/makeBaseImage-bf.st and executing it, I was able to update all the way from the original to the current version (there's one notifier where you need to proceed manually).

[Kirtai]

In that original version, Point>>fromPress:value is there (see my screenshot). The sources must have been broken by something in the update stream.

I see. I was just looking at an image with the latest updates and I see the original images have the decompiled source so yes, it does seem to have been an update that did it.

For now I'll read up on how to make updates/changesets and either add them here in an issue or as a pull request, whichever you prefer. (I'm afraid of adding directly to the update stream because I'm just a beginner at ST-76/78)

Checking further I also noticed that the following look in different places for the update stream.

At https://smalltalkzoo.thechm.org/HOPL-St78.html all of the following look in https://lively-web.org/users/bert/St78/updates/

• Sim Kit
• ThingLab
• base image

In this repo all of the following look in https://lively-web.org/users/bert/St78/updates/

• updates/0001InitUpdates-bf.st
• updates/makeBaseImage-bf.st
• updates/updateStreamInit-bf.st

The following at https://smalltalkzoo.thechm.org/HOPL-St78.html looks in https://smalltalkzoo.thechm.org/users/codefrau/St78/updates/ instead.

• The default image
• fetch updated st78 from server image (these two might be the same?)

I believe these should all point to codefrau.

[codefrau]

For now I'll read up on how to make updates/changesets and either add them here in an issue or as a pull request, whichever you prefer. (I'm afraid of adding directly to the update stream because I'm just a beginner at ST-76/78)

Great. Let’s start by attaching them to an issue until we figure out a good process.

Checking further I also noticed that the following look in different places for the update stream. https://lively-web.org/users/bert/St78/updates/

<https://lively-web.org/users/bert/St78/updates/>

https://smalltalkzoo.thechm.org/users/codefrau/St78/updates/ instead. I believe these should all point to codefrau.

Agreed – but I think I’d only update the .st sources for now. The images, even if they still point to Lively Web, will automatically switch to the Smalltalk Zoo update stream, because I manually added an additional changeset to the Lively stream that does that. So if someone tried to update those images it should work and they will point to the zoo after.

…

[Kirtai]

With the missing methods it appears that they were deleted by updates from the image but not removed from the ClassOrganization. Would simply removing those selectors from their categories in the ClassOrganization be sufficient?

Also I just noticed that there are two "As yet unclassified" method categories. One starting with uppercase and one with lowercase A.
Some updates use one, some the other.

When making updates, would you prefer that I use the ST76 categories?

[codefrau]

With the missing methods it appears that they were deleted by updates from the image but not removed from the ClassOrganization. Would simply removing those selectors from their categories in the ClassOrganization be sufficient?

Should be sufficient, yes. Instead of doing this manually, we could put an expression into an update that goes through all classes.

Also I just noticed that there are two "As yet unclassified" method categories. One starting with uppercase and one with lowercase A. Some updates use one, some the other.

I believe the capitalized one is the right one.

When making updates, would you prefer that I use the ST76 categories?

Yes, this would make sense.

Thank you!

[Kirtai]

Some interesting breakage here :)

It seems some method sources reference missing classes so the class names were replaced by 'unknown', see Object>>notify:at:in:for: which contains

t5 ← unknown new

where unknown should be SyntaxWindow according to the ST-76 sources.
Other missing ones I've found so far are ParsedObjectReference and ParsedFieldReference in Decompiler>>remoteReference: and loadByte:code:

Attached is a fix containing all three classes and the methods fixed to reference them properly.
(All comes from Smalltalk-76 source)

fix-unknowns-ff.st.zip

Also recategorizations of Object and Parser. (Yes, I did them in an odd order but I was tracking a bug) :)

object-recategorization-ff.zip
parser-recategorization-ff.zip

[codefrau]

It's not so much that they were "replaced" by unknown, but rather that classes like SyntaxWindow never existed in Smalltalk-78. The St78 image was created by exporting only a subset of the St76 classes and methods, and my guess is that those methods were simply overlooked, and should have been removed. Checking the senders of notify:at:in: I'm not sure it's even being used? But it's good to have them back. I'm just not sure how to test them.

I tried your object-recategorization-ff.st but it did not seem to actually change the categories of methods? And parser-recategorization-ff.st has all the method sources even if they are the same it seems? It does not change only the categories.

Nonetheless, this is exciting! Thank you!

[Kirtai]

Hmm, very odd. I must have done something wrong with Parser, it should only have the recategorizations.

I'll have a look at them and see if they need an extra doit to make the changes. (Currently I'm having trouble with the lively file upload not working properly in both chromium and firefox).

Speaking of tests, is there anything like a port of SUnit to ST78? It looks simple enough except for figuring out how exceptions work.

Edit: yes, I messed up Parser somehow. The reorg of object should work, I needed to refresh the pane to see the changes though.

[Kirtai]

Ok, I may have found some weirdness in the updates.

Form>#figure is in the browsers method list but gives a "not found" error in an updated image.
This method was removed in 0332ReshapeForm.st but in 0343Source-ok-07A-tk.st: I find

Form understands: 'figure: figure 
        ["change the figure color (color associated with black) for this Image"]' classified: false

which should add it. But at the top of the update are the words

'Methods with original comments and temp names.
Only those that do not affect actual byteCode will be installed.'.

Could this have left Form>#figure with completely blank source?

I'm seeing similar things with other methods that give this error.
I'm puzzled here :)

[codefrau]

(Currently I'm having trouble with the lively file upload not working properly in both chromium and firefox).

You might like the devserver I just committed (377c5ce) ... it allows you to work with files on your local disk via http:-prefixed file names in the image, just like it works on the lively site.

Speaking of tests, is there anything like a port of SUnit to ST78?

Not that I know of – unit testing was invented 20 years later, and St76/78 were Xerox-internal prototypes, only St80 was released to the public back then.

It looks simple enough except for figuring out how exceptions work.

There isn't really an exception mechanism but it should not be too hard to add. Even the debugger is not working entirely correctly (although Dan got it working to some extent). The reason for that is that the debugger basically comes straight from St76, but St78 uses a linearized stack rather than actual context objects.

Edit: yes, I messed up Parser somehow. The reorg of object should work, I needed to refresh the pane to see the changes though.

Ah I messed up – the file browser menu items were broken when browsing a different directory than the default one (I put your files on my local disk under test/ and then switched a file browser to http:test/). I pushed fc2308a to fix that.

And I pushed your reorg changeset now as 6c0c82a. I should have designated you as the GIT_AUTHOR but I don't think I have your email ... would you want attribution? If you prefer to not reveal your actual email, you could use a private GitHub email.

[codefrau]

Re Form>>figure and Form>>figure::

• 0240Removals-di.st they were added to SymbolTable>>unforgetableMessages (which we should fix now I guess)
• 0294Source-ok-07A-tk.st both got source code
• 0295Source-ok-07B-tk.st a typo was fixed in their comment
• 0332ReshapeForm.st: removed both
• 0344Source-ok-07B-tk.st has source for them but should not have affected the system because MergingSources was true

The idea of MergingSources is to be able to file in ST76 sources to get their comments, without affecting the actual ST78 behavior, in case they were actually changed between 76 and 78. The way that's done is by compiling the St76 source, and only accepting it if the method produced is exactly the same as the existing method, by comparing their decompiled code. This logic was defined in 0283Merge-srcs-fix-tk.st and then employed in Ted's various Source-ok change sets.

In Class>>install:method:literals:code:backpointers: there is a check messagedict has: sel so this would not have brought back these methods (and indeed, we don't have them)

And Class>>derstands: (I love that cheeky name for un-understands) has had 4 versions but each one has the line self organization delete: sel

So it all seems like it should be okay, but it's not, obviously. Somehow both figure and figure: snuck back into Form's organization, or were never removed. We could figure out what happened using user loadUpdatesThrough: N but it seems to be easier to just fix it:

| nm cls sel .
for⦂ nm from: Smalltalk allClassNames do⦂ [
	cls ← Smalltalk ◦ nm.
	for⦂ sel from: cls organization asStream do⦂ [
		(cls canunderstand: sel) ⇒ []
		cls organization delete: sel.
			user show: 'del '; show: nm; space; show: sel; cr ]]

seems to do the trick. What do you think?

[Kirtai]

Dev-server

That's amazing. It took me a bit to figure out how to set the buttons the same as on the Lively site but I got it in the end. Great stuff :)

Unit tests

I read the original paper and the core of SUnit seems very very small. Unfortunately it relies on exceptions to handle failures but correct tests should run.

Exceptions

I imported the ExceptionHandler and related classes to the image but it falls foul of the same problem that hits the debugger. I expect that if the debugger was fixed, this would follow.

Attributions

I got a github email of "3085457+Kirtai@users.noreply.github.com" which would be fine. :)

Missing methods

Yep, your script works :)
It's particularly handy since I've been going through the ST-76 code, ripping out everything but the class names, method names and categories and converting them into simple scripts to reorganize a class. This will help find and remove any selectors in ST76 that aren't in ST78 that try sneaking in :)

(they're along the lines of

Class organization classify: ↪ed: asVector , ↪edit: , ↪edit:para:formerly: , ↪execute: under: 'Editing'.

followed by going in and manually fixing up anything left.)

End

BTW, I was wondering, is restoring removed classes fine? It would be nice if ST78 was ST76-Plus rather than ST76-Minus but I don't know if the system is up to it. Things like the Events, Picture-Editor and Exception-Handling categories.

Also, here's the reorganisation of Class and VariableLengthClass :)

class-recategorization-ff.st.zip
variablelengthclass-recategorization-ff.st.zip

[codefrau]

Exceptions

I imported the ExceptionHandler and related classes to the image but it falls foul of the same problem that hits the debugger. I expect that if the debugger was fixed, this would follow.

I added some details about that in #7.

Missing methods

Yep, your script works :) It's particularly handy since I've been going through the ST-76 code, ripping out everything but the class names, method names and categories and converting them into simple scripts to reorganize a class. This will help find and remove any selectors in ST76 that aren't in ST78 that try sneaking in :)

The question is if I should run this script as an update, or if we want to fix the broken class orgs manually ... I'm tending to just publish it.

End

BTW, I was wondering, is restoring removed classes fine? It would be nice if ST78 was ST76-Plus rather than ST76-Minus but I don't know if the system is up to it.

The system should be able to handle that fine – the only limitation would be total number of objects (30720) and classes (2048), where we currently use about 43% of object oops and 6% of classOops. I'd like to keep the base system 16 bit with enough room to do meaningful work (even though my 32 bit largeOops work lifts that limit).

Things like the Events, Picture-Editor and Exception-Handling categories.

Is Picture-Editor the same as our BitRectEditor? And what are the Events about? I don't have a working St76 here.

Also, here's the reorganisation of Class and VariableLengthClass :)

class-recategorization-ff.st.zip variablelengthclass-recategorization-ff.st.zip

Thank you! Published as cdeb490

[Kirtai]

For access to the source, if you go to this repo the readme contains details on how to extract and convert the source of Smalltalk-76 from the xmsmall.dsk using Contralto.

It also contains pre-extracted source but I did the extraction myself to make sure it was correct and have been using that. Note that while it also has the bootstrap files as text, these files have an error where ↪ has been substituted for ".

While the source is also on a linked website, the web pages uses the wrong kind of single quote so methods with them don't copy/paste cleanly.

It also has the source for quicksilver which seems to be a publishing tool?

The Events category has the classes 'EventQueue', 'MessageTally', 'PriorityInterrupt', 'PriorityScheduler', 'Timer' & 'UserEvent'. So scheduling and system events it seems. These are not in the image.

Yes, Picture-Editor is a category for the BitRectEditor & related tools (along with RadioButtons for some reason). These are in the image. They could be left alone or popped into the original category. (Sorry for not digging deeper in that before mentioning it, I never thought of classes being moved to different categories too)

Would it be okay to change the class categories to match ST76? Most of them are fine but a few seem to be merged. e.g. 'Picture-Editor' was merged into 'Windows'. It would make comparing easier and match the ST76 documentation better.

The system should be able to handle that fine – the only limitation would be total number of objects (30720) and classes (2048), where we currently use about 43% of object oops and 6% of classOops. I'd like to keep the base system 16 bit with enough room to do meaningful work (even though my 32 bit largeOops work lifts that limit).

That shouldn't be a problem. Many of the missing classes relate to hardware access (e.g. ethernet) or no longer extant filesystems (Juniper, Alto, IFS) so won't be needed. Some missing classes that maybe should be added are Path, Image and maybe things like Timer.

Regarding the script, I'll defer to you but the missing methods should be cleaned up with the reorganization I'm doing but publishing it wouldn't hurt afaict.

Edit: I just saw the mention of porting ST76 forward to ST78 on the zoo page.
That's what I kinda have in mind :)

[codefrau]

I talked to @Daningalls:

Regarding the restoration of missing methods, such things as writing pressFile format was consciously omitted (by me) because it’s not relevant/useful for anything. the inconsistency is just a manifestation of an incomplete job of elimination. To restore it is not the right thing IMO. But more to say about making a real St-76 (not 78).

So I think we are in agreement – we don't want to change this into "a St76" by restoring more features, just making what's there actually work (like we started to do with the text handling fixes and debugger), and more accessible (like with method and class comments). We should instead make a St76 VM, which would mostly differ in the stack/context handling, and need more primitives (one of the big innovations in St78 over -76 was switching fully to BitBlt for all graphics ops rather than bespoke primitives).

So recategorization would be fine if it makes sense – OTOH having BitRectEditor be grouped with the other windows would reinforce the notion that it's not a separate "app" but just an object like everything else (I'm sure you saw Alan's demo, about 10:00 minutes in). We're mainly doing this for historical education, it just happens to be fun in itself.

Dan also wrote:

Fixing the debugger would be great. My goal is to get it to where all the simulation stuff like tracing works again (and of course the debugger be robust).

I've started #7 to track this.

[Kirtai]

Alright, I'll stick to what's there and just reorganise the methods and replace missing class comments. Keeping this one as a historical artifact sounds good.

BTW, the only features I actually wanted added to ST78 was the exception handling to support a test framework for testing. Not really an issue for a system that's for preservation though.

I would really like an ST76 based system to work with though, so a VM for that sounds great. Turning it into a "modern" ST76 with bitblt would be fun :)

I suppose a preservation of ST76 wouldn't need much if anything done on the image side.

[codefrau]

BTW, the only features I actually wanted added to ST78 was the exception handling to support a test framework for testing. Not really an issue for a system that's for preservation though.

Maybe we can find a way to easily share optional changesets / images. Adding SUnit in the main update stream would feel too anachronistic. We added some stuff that was not in the original system, like the gesture recognizer. But Alan argued that gesture recognition was an established UI tech back then, so we felt it would still represent the spirit, and he did great demos with it. Same with the window dragging / refreshing, where they would have done it if they'd had the compute power (or the sheer audacity of Bill Atkinson who misremembered seeing it so he invented quickdraw regions to do it).

I would really like an ST76 based system to work with though, so a VM for that sounds great. Turning it into a "modern" ST76 with bitblt would be fun :)

That's what the XEROX PARC engineers thought too, which became Smalltalk 80.

Then again, St80 added a whole lot of other stuff, like the dual class hierarchy (I'm still amazed that the VM has no idea about that – the power of Turtles All The Way Down).

I suppose a preservation of ST76 wouldn't need much if anything done on the image side.

All my VM projects aim to not require any image-side modifications. The St78 VM can run the original image fine, the image just happened to be rather buggy (and there are still bugs, e.g. when mixing different styles in the same text, or the debugger).

[Kirtai]

Well yes, I meant that the St76 image wouldn't have the problems the St78 image has due to the incomplete conversion. No disrespect meant to your VMs :)

re tests, all they actually need is the stack handling bugs fixed well enough that ExceptionHandler et al could be loaded and work. Where the tests are stored doesn't bother me. (Either a changeset or separate Testing image is good.)

(I was just about to post this when realized that most of the tests would be the same for St76.)

While I do like ST80 a lot I am very intrigued with using an St76 system (or even earlier ones) for day to day programming and seeing what different paths it could evolve along. I wish there was an easy way to get files in and out of ContrAlto, slow as it is.

(While the metaclass structure is incredible, I do admit I find it complex and wonder how things would have turned out if Dan Ingalls had made St80 prototype based as he mentioned considering in his HOPL paper)

I will keep on with the recategorization of methods and looking for bugs.

[Kirtai]

UserView seems to have a lot of changes so I've only moved the methods from St76 and left the rest alone.
SystemDictionary isn't in St76 so I skipped it.
This is all the "Kernel Classes" I believe :)

UserView.st.zip

And more
Number.st.zip
Date.st.zip
Float.st.zip

Integer.st.zip