Smalltalk in small steps: Cuis

Showing posts with label Cuis. Show all posts

Monday, 7 December 2015

UTF-8 for Cuis, Pharo and Squeak

Unicode UTF-8 for Smalltalks

The aim - Sortable, equivalency-testable Unicode strings for Smalltalks.

Phase 1,

Sortable, equivalency-testable UTF-8 strings which

encode ONLY ASCII and ISO 8859-1 ("ISO Latin-1")

Step 1 - Thinking

0) a) ASCII and ISO-8859-1 consist of characters encoded in 1 byte.
b) UTF-8 can encode all of those characters in 1 byte, but can prefer some of them to be encoded as sequences of multiple bytes. And can encode additional characters as sequences of multiple bytes.

1) Smalltalk has long had multiple String classes.

2) Any UTF16 Unicode codepoint which has a codepoint of 00nn hex
is encoded as a UTF-8 codepoint of nn hex.

3) All valid ISO-8859-1 characters have a character code between 20 hex and 7E hex, or between A0 hex and FF hex.
https://en.wikipedia.org/wiki/ISO/IEC_8859-1

4) All valid ASCII characters have a character code between 00 hex and 7E hex.
https://en.wikipedia.org/wiki/ASCII

5) a) All character codes which are defined within ISO-8859-1 and also defined within ASCII. (i.e. character codes 20 hex to 7E hex) are defined identically in both.

b) All printable ASCII characters are defined identically in both ASCII and ISO-8859-1

6) All character codes defined in ASCII (00 hex to 7F hex) are defined identically in Unicode UTF-8.

7) All character codes defined in ISO-8859-1 (20 hex - 7E hex ; A0 hex - FF hex ) are defined identically in UTF-8.

8) => some Unicode codepoints map to both ASCII and ISO-8859-1.
all ASCII maps 1:1 to Unicode UTF-8
all ISO-8859-1 maps 1:1 to Unicode UTF-8

9) All ByteStrings elements which are either a valid ISO-8859-1 character or a valid ASCII character are *also* a valid UTF-8 character.

10) ISO-8859-1 characters representing a character with a diacritic, or a two-character ligature, have no ASCII equivalent. In Unicode UTF-8, those character codes which are representing compound glyphs, are called "compatibility codepoints".

11) The preferred Unicode representation of the characters which have compatibility codepoints is as a a short set of codepoints representing the characters which are combined together to form the glyph of the convenience codepoint, as a sequence of bytes representing the component characters.

12) Some concrete examples:

A - aka Upper Case A
In ASCII, in ISO 8859-1
ASCII A - 41 hex
ISO-8859-1 A - 41 hex
UTF-8 A - 41 hex

BEL (a bell sound, often invoked by a Ctrl-g keyboard chord)
In ASCII, not in ISO 8859-1
ASCII : BEL - 07 hex
ISO-8859-1 : 07 hex is not a valid character code
UTF-8 : BEL - 07 hex

£ (GBP currency symbol)
In ISO-8859-1, not in ASCII
ASCII : A3 hex is not a valid ASCII code
UTF-8: £ - A3 hex
ISO-8859-1: £ - A3 hex

Upper Case C cedilla
In ISO-8859-1, not in ASCII, in UTF-8 as a compatibility codepoint *and* a composed set of codepoints
ASCII : C7 hex is not a valid ASCII character code
ISO-8859-1 : Upper Case C cedilla - C7 hex
UTF-8 : Upper Case C cedilla (compatibility codepoint) - C7 hex
Unicode preferred Upper Case C cedilla (composed set of codepoints)
Upper case C 0043 hex (Upper case C)
followed by
cedilla 00B8 hex (cedilla)

13) For any valid ASCII string *and* for any valid ISO-8859-1 string, aByteString is completely adequate for editing and display.

14) When sorting any valid ASCII string *or* any valid ISO-8859-1 string, upper and lower case versions of the same character will be treated differently.

15) When sorting any valid ISO-8859-1 string containing letter+diacritic combination glyphs or ligature combination glyphs, the glyphs in combination will treated differently to a "plain" glyph of the character
i.e. "C" and "C cedilla" will be treated very differently. "ß" and "fs" will be treated very differently.

16) Different nations have different rules about where diacritic-ed characted and ligature pairs should be placed when in alphabetical order.

17) Some nations even have multiple standards - e.g. surnames beginning either "M superscript-c" or "M superscript-a superscript-c" are treated as beginning equivalently in UK phone directories, but not in other situations.

Some practical upshots
==================

1) Cuis and its ISO-8859-1 encoding is *exactly* the same as UTF-8, for any single character it considers valid, or any ByteString it has made up of characters it considers valid.

2) Any ByteString is valid UTF-8 in any of Squeak, Pharo, Cuis or any other Smalltalk with a single byte ByteString following ASCII or ISO-8859-1.

3) Any Smalltalk (or derivative language) using ByteString can immediately consider it's ByteString as valid UTF-8, as long as it also considers the ByteSring as valid ASCII and/or ISO-8859-1.

4) All of those can be successfully exported to any system using UTF-8 (e.g. HTML).

5) To successfully *accept* all UTF-8 we much be able to do either:
a) accept UTF-8 strings with composed characters
b) convert UTF-8 strings with composed characters into UTF-8 strings that use *only* compatibility codepoints.

Having written all this, I have realised a picture is worth a thousand words, and I've now drawn this as a Venn diagram.
This note
It can be found at both:
http://smalltalk.uk.to/unicode-utf8.html

and at my Smalltalk in Small Steps blog at:

I therefore as a first step, propose these classes

a Utf8CompatibilityString class.

asByteString - ensure only compatibility codepoints are used. Ensure it doews not encode characters above 00FF hex.

asIso8859String - ensures only compatibility codepoints are used, and that the characters are each valid ISO 8859-1

asAsciiString - ensures only characters 00hex - 7F hex are used.

asUtf8ComposedIso8859String - ensures all compatibility codepoints are expanded into small OrderedCollections of codepoints

a Utf8ComposedIso8859String class - will provide sortable and comparable UTF8 strings of all ASCII and ISO 8859-1 strings.

Then a Utf8SortableCollection class - a collection of Utf8ComposedIso8859Strings words and phrases.

Custom sortBlocks will define the applicable sort order.

We can create a Dictionary of named, prefabricated sortBlocks.

This will work for all UTF8 composed strings of ISO-8859-1 and ASCII strings.

If anyone has better names for the classes, please let me know.

If anyone else wants to help

- build these,

- create SUnit tests for these

- write documentation for these

Please let me know. euan mee at a well known email system run by Google.com - gmail

Monday, 23 November 2015

Timing Issues in Squeak and Pharo and Cuis Smalltalks

I was profiling yesterday, and discovered these issues:

Issue:
=====
Time and DateAndTime both show human-readable time information to
micro-second precision. Time also holds information to nano-second
precision.

Both objects are only providing information to the nearest millisecond.

The digits beyond milliseconds are always (on my PC) filled in as '902'.

In the nanoseconds instvar of the Time objects, the nanoseconds are
always filled in as '000'.

Root cause:
=========
When Times (and DateAndTimes) are created, they use the
Time>>primUTCMicrosecondsClock method to generate the time

primUTCMicrosecondsClock
"Answer the number of micro-seconds ellapsed since Squeak epoch.
That is since 00:00 on the morning of January 1, 1901 UTC.
At least a 60-bit unsigned integer is used internally which is enough
for dates up to year 38435.
Essential. See Object documentation whatIsAPrimitive. "

<primitive: 240>
self primitiveFailed

Potential for solution:
================
[The following is taken from Stack Overflow,
http://stackoverflow.com/questions/2607263/how-precise-is-the-internal-clock-of-a-modern-pc]
Modern PCs have a hardware High Precision Event Timer (HPET) which
specifies a clock speed of 10MHz
http://www.intel.com/hardwaredesign/hpetspec_1.pdf
Even newer PC architectures take this and put it on the Northbridge
controller and the HPET can run at 100MHz or even greater. At 10Mhz we
should be able to get a resolution of 100 nano-seconds and at 100MHZ
we should be able to get 10 nano-second resolution.

The following operating systems are known not to be able to use HPET:
Windows XP, Windows Server 2003, and earlier Windows versions, older
Linux versions
The following operating systems are known to be able to use HPET:
Windows Vista, Windows 2008, Windows 7, x86 based versions of Mac OS
X, Linux operating systems using the 2.6 kernel and FreeBSD.

Tuesday, 17 November 2015

Enumerating using keysAndValuesDo:

**Part 1 - Displaying to the screen**
The most straight forward way to print to the screen in a GUI-based Smalltalk is to use the message:
`Transcript show: 'some text'`
(The Transcript is a system object that displays into a scrollable window).

To make sure there is a newline before each line of display text, we send the message `cr` to the Transcript

Transcript cr.
Transcript show: 'some text'.

A shorthand method, that saves us re-typing `Transcript` over and over, is to send Transcript a series of messages one after another. This is called a *message cascade*. Each time we end a message in `;` it means *send to the same receiver as the last message was sent to*.

We can then shorten this again, as Smalltalk pays no attention to newlines in expressions.

The final display message cascade becomes:
` Transcript cr; show: 'some text'.`

**Part 2: Enumerating aSortedCollection using keysAndValuesDo:**
This keyword message is SequencableCollectionand its method header is:
`keysAndValuesDo: aBlock
"Enumerate the receiver with all the keys (aka indices) and values."`

(It works the same way in Dolphin, and in Squeak and its derivatives, Pharo and Cuis).

The keyword message `keysAndValuesDo:` takes a block argument.
A block is an anonymous object, with one method. Its method is defined between a matched pair of square brackets - a `[` ... `]` pair.

In this case, we need a local variable in the block for the key of each element of the collection, and another local variable for the value of each element.

We can call them anything we like, and in this case, it is the order that they appear in that is important. `keysAndValuesDo:` will put the element's key into the first local variable in the block, and will put the element's value into the second local variable in the block.

Local variables in a block are declared at the start of the block, and each variable name is identified by prefixing it with `:`. The local variable declarations are ended with a `|`.

The block then looks like
`[:local1 :local2 |
"do something for each element, with the key in local1 and the value in local2"
]`

I prefer meaningful local variable names, so I'll use `eachKey` and `eachValue`.

**Part 3: Putting it all together**
To enumerate through mySortedCollection

"Declare the mySortedCollection variable"
|mySortedCollection|

"Initialise the collection"
mySortedCollection := SortedCollection new.

"add in some data to the collection"
mySortedCollection add: ('First') ;
add: ('Second') ;
add: ('Third').

"Enumerate through the collection, displaying to the Transcript window"
mySortedCollection keysAndValuesDo:
[:eachKey :eachValue |
Transcript cr; show: eachKey; show: ' '; show: eachValue
] .

Paste the code into a Workspace (known as a Playground in Pharo, from version 4.0 onwards). Select the text. Once selected, right-click (on a two or three button mouse) and select "Do it" from the menu. Or use Ctrl-d as a keyboard shortcut. (The exact chording key may vary on your platform)

**Final notes**
In a SortedCollection or an OrderedCollection, the key is the index. The value is what is stored at element[index].

In a Dictionary, the key of the element is the key, and the value of the element is the value.

SortedCollections are sorted in order of the element values, according to the definition of the collections *sort block*. In the absence of a custom sort block, they will be added in ascending order. `'First'`, `'Second'` and `'Third'` are, coincidentally, in alphabetical order. It happens to work out nicely in this example.

Monday, 16 November 2015

Beginning to Smalltalk has begun!

I've started a new blog - "Beginning to Smalltalk".

This is where I'll be putting my more polished posts, and I'll hopefully manage to put them in a sensible order.

I'm confident Beginning to Smalltalk will become a useful step-by-step tutorial guide.

I'll continue to log my notes to myself, and my 'aha' moments, here.

Thursday, 12 November 2015

Get a Smalltalk up and running

Scope: This how-to covers Squeak, Pharo, Cuis and Dolphin.

Many of the concepts also apply other Smalltalks.

Smalltalk installations come in two principal parts.

The first part is known as the "Virtual Machine"

More usually called the "VM". This is the underlying engine that Smalltalk runs on. It interfaces with the operating system of the computer that you are running the Smalltalk on, and also provides low-level services e.g. garbage-collection.

Most Virtual Machines are cross-platform. When you use a cross-platform VM, all the Smalltalks that run on top of the VM are cross-platform too.

Cuis, Pharo and Squeak can all run on the same Virtual Machines, which run on MacOS, Windows and Linux.

Dolphin Smalltalk is designed to run on its own Virtual Machine, which runs on Windows.

The second part is called the "image" .

The image holds a bunch of Smalltalk code. And it holds more, too. It holds a bunch of data. And it holds all the information about the Smalltalk GUI as it was when the image was last saved. Which windows the user had open, and where they were positioned.

It holds everything that Smalltalk knew about, and all information relevant to running the Smalltalk installation.

The image file can be considered to be a collection of all information about a particular installation of Smalltalk from just before it went into hibernation.

Choosing a Smalltalk

A little bit about each Smalltalk, and platform-specific notes

Squeak has given rise to both Pharo and Cuis, and also both the Seaside and AidaWeb web applications frameworks, along with much else.

The Scratch graphical programming language is written in Squeak, for example.

A lot of work goes in to getting Squeak running fast on the Raspberry Pi , because Scratch is written in Squeak. Scratch is a key package for teaching children to program, and the Rasberry Pi is a platform designed for teaching children to program. This work is done so that Scratch will run fast on the Pi, but it has a very useful side-effect for people who want to use Smalltalk on the Pi.

Cuis is a fork of Squeak, intending to radically simplify the numbers of libraries in the distribution, and radically simplify the way GUIs are drawn and coded.

It contains only 500 classes, compared to the 2,300 in Squeak and the 5,000 in Pharo.

This makes it a good candidate for understanding a Smalltalk from top to bottom. Potentially it might lead to speedy performance.

Pharo is a fork of Squeak where the focus is on extending the functionality and ability of Squeak, without much regard to backward compatibility. Considerable effort goes in to making sure that Pharo always remains a capable platform for the Seaside web server.

Pharo evolves rapidly, with vast majority of the additional functionality added by academic computer science research groups. Nonetheless, the Pharo management team intend it also to be a useful platform for commercial application developments - as can be seen by the effort that goes into keeping the GUI and desktop themes looking clean and professional.

Cuis, Pharo and Squeak can all run on the same Virtual Machines, which run on MacOS, Windows and Linux.

Squeak 5.0 includes extensive optimisations for when it runs on the Raspberry Pi.

Dolphin Smalltalk is designed to run on its own Virtual Machine, which runs on Windows.

The key advantages are that Dolphin is designed to provide a Windows native look-and-feel; and a working system can be packaged up as a .exe install file. This means there is no learning curve for Windows end-users when they are installing an end-user app written in Dolphin Smalltalk.

Dolphin Community Edition cannot store code and classes to repositories in the cloud, or automatically retrieve packages from the internet. Downloading packages that other people have already written for you to use from the internet is really useful as you start out. So you'd be better to download Squeak or Pharo, to start out with - unless you really want native look-and-feel from day 1.

Downloading packages from the internet is available in the paid-licence version of Dolphin. I have not yet used this, so cannot vouch for how well it works.

So unless you want to learn only Windows-GUIs, I personally advise you to start on a different Smalltalk, and then later come back and learn the Windows-GUI facilities in Dolphin, once you are used to Smalltalk.

If you want to start writing web applications as you learn, then you can use a package called Seaside, which has a once-click installer for Pharo and Seaside, all in one step.

Seaside will also run on Squeak, but I don't know of any current All-in-one installer for it.

Licencing Issues

Cuis, Pharo and Squeak all use the MIT licence. This allows commercial use, and is also GPL-compatible.

Dolphin's licence allows you to distribute the DLLs and the Dolphin VM as an integral part of executable applications created by you using Dolphin Smalltalk.

Download a Smalltalk

With all Smalltalks

If you have a specific pre-existing application, package or library you want to use, you may need to use an older version. If you just want to get started with the latest version, choose the latest version.

On Windows

All of the cross-platform Smalltalks can be installed as a 'portable' installation. You can copy the installed folder to anywhere on your PC and the Smalltalk will still run. The folder contains all the necessary code and .dlls to run successfully.

For specific Smalltalks

Cuis:

Cuis currently has you download the Smalltalk VM and the Cuis image separately. Probably best to start with one of the others, then come back to Cuis once you have got the hang of VMs and Images.

The Cuis Image is available from
https://github.com/Cuis-Smalltalk

Any Squeak VM should work with Cuis.

Dolphin:

http://www.object-arts.com/downloads/downloads.html

At time of writing (2015-11-11), there is one download link
Dolphin X6.0 (Community and Pro Editions)

This downloads a .exe file. (And the first time it runs, it will prompt you to email for a (zero-cost) Community edition licence number).

If later you wish to prepare .exe files for your application's end-users, you can upgrade to the Pro Edition for a fee of ~£160(GBP) or approx. $225(USD)/€225(EUR)

Pharo:

http://pharo.org/download

At time of writing (2015-11-11), the site offers Pharo 4.0 via three download links, one for each of MacOS, Windows and Linux.

MacOS : http://files.pharo.org/platform/Pharo4.0-mac.zip

Windows : http://files.pharo.org/platform/Pharo4.0-win.zip
Linux : http://pharo.org/gnu-linux-installation

Once you have a little experience with VMs and images, you can download the latest Pharo 5.0 development image from http://files.pharo.org/get-files/50/pharo.zip

Seaside

http://www.seaside.st/download/pharo

Seaside One-Click Experience 3.1 (36 MB) For Windows, MacOS and Linux

Squeak:

http://squeak.org/downloads/

At time of writing (2015-11-11), there are two versions listed under "Quick Download"
All-in-One (Spur) - Version 5.0
All-in-One (Non-Spur) - Version 4.6

"Spur" / "Non-Spur" refers to the Virtual Machine (VM) included in the download. Cog is currently the most prevalent VM in the Squeak/Pharo/Cuis worlds.

A recent improvement to Cog was labelled "Spur". So '(Spur)' means that the install includes the latest Cog VM with added Spur. '(Non-Spur)' means that it includes a slightly older version of the Cog VM.

To start A Smalltalk

Once you have the files unzipped and installed on your computer:

For Cuis

On Windows
Open the folder with the VM in it.
Drag the Cuis .image file you want to use and drop it on the VM

On Linux