What's in Your Vocabulary Deck?

I am something of a student of New Testament Greek and made my beginnings a few years ago with Mounce's Basics of Biblical Greek. Along with the text book was a flash card program called FlashWorks, and it is a solid way to learn your vocab. The vocab was indexed with frequency and chapter and user level difficulty (based on how often you answered correctly) which was almost all that I wanted to help me keep track of which words I should be working on. 

There was one thing that I felt was missing, though I wasn't sure how to define it or to decide what could be done about it. Something like, "how do I get a balanced randomness to my vocab selection if I only want to do a few right now?"

I don't want to randomly select 10 words from out of 500 words. I won't get much repetition if I do that every day. On the other hand, if I only focus on the most frequent, hard words, then I repeat them every day, then the risk might be that I "cram" them and don't really get them into long term memory. So, maybe the thing to do is to start with a selection of 30 words that are the most difficult/frequent words and randomly select 10 from that list. I call it a "pool factor". The pool factor in that case, would be 3. Make a selection of the most difficult, frequent words up to 30 long (and include a random sorting factor in case there are contenders for the last spots in the 30) and then randomly select 10 out of the 30.

The 30 words are a kind of "working set" that I am working toward mastering, although, I don't need to specifically decide on them, they get selected based on criteria. I do 10 words today out of those 30. As I do the daily work, some of those 30 words drop out because I got them right a bunch of time and new words enter the selection of 30.

The next issue is to decide what is meant by hardest, frequent words. If I use a score that starts high for words that have never been marked correct, then I can do a descending sort on score, then by frequency. The whole deck will start at the highest score and initially I am getting the most frequent words. In order to prevent words from disappearing from this set too soon, keep track of not just a score, but the number of times marked correct. Only decrease the score after after getting the word right correct, say, 3 consecutive times. (Since the score of a word doesn't change until you have reached a level of mastery with it, you don't run into the scenario of interchanging sets of words that you then never master.)

Note that you can probably apply this strategy to other types of vocabulary that don't reference a fixed body of literature, but you have to come up with some kind of importance rating on each word in your database that serves the same purpose as the frequency field here.

The code below belongs to a C# project of mine that is using SQLiteAsyncConnection with some fields that are still missing data (hence, ifnull):

        public Task<List<Vocab>> GetNRandomHardFrequent(int numberOfWords, double poolFactor)

        {

            int PoolSize = Convert.ToInt32(numberOfWords * poolFactor);

            Random rand = new Random();

            string query = @"

                    select 

                        sub.*

                    from

                        (select 

                            v.*

                        from

                            Vocab v

                        order by

                            ifnull(v.score, 0) desc, ifnull(v.frequency, 0) desc, RANDOM()

                        limit ?) sub

                    order by

                        RANDOM()

                    limit ?;";

                        

            var words = database.QueryAsync<Vocab>(query, PoolSize, numberOfWords); 

            return words;

        }

EMACS with Slime on Chromebook

Because of a felt-insufficiency of nerdness in my life, I decided it was high time I get set up to use slime in EMACS on a Chromebook. I have been an EMACS user on Windows, using it for my first experience of Common Lisp development. I now use LispWorks on my Windows machine but if there was a way to do Common Lisp on my Chromebook I thought I would hazard some of my time to learn how to go about it.

First, get Linux on your Chromebook. For newer systems, this is as simple as turning on Linux Beta in the user settings of your Chromebook. I don't know how well this works on older Chromebooks, but I understand there are ways.

With Linux enabled or installed as required, go to the terminal (settings key, type linux or terminal and you should see it) and type sudo apt-get install emacs25. I had to run an update to apt-get for this to work properly (sudo apt-get update). Then I redid the emacs25 install (same command line) and it caught up the parts it missed the first time. 

Similarly, you can install a version of Common Lisp. I chose Steel Bank Common Lisp (sbcl). The command line for that is (perhaps predictably) sudo apt-get install sbcl.

Quicklisp is next. I downloaded it from here. I moved it into my Linux files because it seemed like a good idea off hand, though I'm not sure that it is strictly necessary. Before trying to load this, I checked out the (rather simple) file structure on my Linux system from the terminal perspective. Use the ls command to see the files in the current directory and the cd command to change directory (I'm not a real Linux user, maybe you're not either). Next, I started sbcl by typing it at the command line. This results in bringing you to a command line version of sbcl where you can type Common Lisp commands. We use this to load Quicklisp. I issue the following, pressing enter after each line:

    (load "quicklisp.lisp")

    (quicklisp-quickstart:install)

    (ql:add-to-init-file)

    (ql:quickload "quicklisp-slime-helper")

    (quit)

Now, you need some code added to the the init file, but I didn't see such a file and wasn't sure where to put it. Fortunately, there's a way for EMACS to help you with that. Go into EMACS and type CTRL+x CTRL+f, it will assume you want the home directory and lead you with "~/", and so you add to that emacs.d/init.el and press enter. Or, in EMACSese, that's C-x C-f ~/emacs.d/init.el RET. HT to the Wiki. This will create and open the file for you. 

The code you will want to put in this file is suggested at the end of installing the quicklisp-slime-helper and it is this:

    (load (expand-file-name "~/quicklisp/slime-helper.el"))

      (setq inferior-lisp-program "sbcl")

Paste the code into the file you just opened. Now, be prepared for a "wha?" moment. The shortcut sequence for pasting into EMACS is CTRL+y. Or, C-y as the EMACS users like to say. This will not be your last such moment, but carry on.

You are now prepared to start writing and executing Common Lisp code in EMACS. In the below, C- means press and hold CTRL and M- means press and hold ALT. I do the following sequence as a standard way to get started:

    C-x 3 (gets you set with vertical division--left and right panes. one for code file and one for REPL)

    M-x slime RET (starts the REPL)

Here is the classic video tutorial (HT Baggers) of most of these steps done for Windows that also gives a bit of a flavor of what it is like to use Slime in EMACS for Common Lisp development: Installing Common Lisp, Emacs, Slime & Quicklisp.

Recursion and Accumulators

Working through Paradigms in Artificial Intelligence Programming, I have worked to wrap my head around accumulators and I have implemented one that worked surprisingly quickly. My example is a factorial design experiment. I have 4 reference implementations of a factorial design on an experiment with 7 factors, with each factor having 10 treatments in it. This is a bit of a ridiculous size, but it gave me clear separation of performances. I have split the code up into sections so you can focus on the parts of the program that matter you.

Supporting Code

(defmacro make-enumerator (head tail) `(cons ,head #'(lambda () ,tail))) (defun get-current (enumerator) (if (consp enumerator) (first enumerator) nil)) (defun get-next (enumerator) (when (functionp (rest enumerator)) (funcall (rest enumerator)))) (defvar empty-indices '(nil)) ;; this is a non-destructive implmentation (defun multiple-radix-counter (radices &key indices terminate) (when (null indices) (setf indices (make-array (length radices) :initial-element 0))) (if (and terminate (equal indices empty-indices)) nil (make-enumerator indices (multiple-radix-counter radices :indices (loop with dest = (make-array (length indices) :initial-contents indices) for i from (1- (length indices)) downto 0 if (< (1+ (aref dest i)) (aref radices i)) do (incf (aref dest i)) (return dest) else do (setf (aref dest i) 0) finally (return (if (and terminate (< i 0)) empty-indices dest))) :terminate terminate)))) (defun extract-by-multiple-radix (source indices) (loop with res = (make-array (length indices)) for i from 0 to (1- (length indices)) do (setf (aref res i) (aref (aref source i) (aref indices i))) finally (return res))) (defmacro discard (&body body) `(let () ,@body nil))

Main Functions

(defun factorialize (factors &optional (accumulator '(nil))) "Given a list of lists, where each list is of treatments, give a complete factorial." (cond ((null factors) accumulator) ((atom factors) (mapcar #'(lambda (u) (cons factors u)) accumulator)) ((atom (first factors)) (loop with res = '(nil) with ending = res for f in factors do (loop for acc in accumulator do (setf (cdr ending) (cons (cons f acc) nil)) (setf ending (cdr ending))) finally (pop res) (return res))) (t (factorialize (first factors) (factorialize (rest factors) accumulator)))) (defun factorialize-2 (factors) "This is the slow version using append. sample usage: (factorialize-2 '((1 2 3) (4 5 6) (a b c))) produces: ((1 4 A) (1 4 B) (1 4 C) (1 5 A) (1 5 B) (1 5 C) (1 6 A) (1 6 B) (1 6 C) (2 4 A) (2 4 B) (2 4 C) (2 5 A) (2 5 B) (2 5 C) (2 6 A) (2 6 B) (2 6 C) (3 4 A) (3 4 B) (3 4 C) (3 5 A) (3 5 B) (3 5 C) (3 6 A) (3 6 B) (3 6 C))" (cond ((null factors) nil) ((null (rest factors)) (mapcar #'(lambda (u) (list u)) (first factors))) (t (loop with remaining = (factorialize-2 (rest factors)) for f in (first factors) append (loop for r in remaining collect (cons f r)))))) (defun factorialize-3 () (loop with res = '(nil) with ending = res for i from 0 to 9 do (loop for j from 0 to 9 do (loop for k from 0 to 9 do (loop for l from 0 to 9 do (loop for m from 0 to 9 do (loop for n from 0 to 9 do (loop for p from 0 to 9 do (setf (cdr ending) (cons (list i j k l m n p) nil)) (setf ending (cdr ending)))))))) finally (pop res) (return res))) (defun factorialize-4 (factors) (let* ((radices (map 'vector #'length factors)) (combinations (reduce #'* radices)) (res (make-array (list combinations)))) (loop for enum = (multiple-radix-counter radices :terminate t) then (get-next enum) for index = 0 then (1+ index) for indices = (get-current enum) while enum do (setf (aref res index) (extract-by-multiple-radix factors indices)) finally (return res))))

Speed Testing

To make the testing as fair as possible, I started emacs fresh for each test run. Under these conditions, factorialize took 2.5 seconds, factorialize-2 took 3.7 seconds, factorialize-3 took 22 seconds, and factorialize-4 took 12 seconds.

A large amount of the time spent in these examples is spend in garbage collection. The factorialize-4 example could probably be improved by allocating a large chunk of memory in a 2-dimensional array and then the rest of the work involves a lot less memory allocation time. The land slide difference that marks factorialize off from the other methods appears to be the amount of shared memory it involves. Some reassignments of various locations of the resulting list, produce changes in other lists since there is shared cons cells among the lists. For my particular application this was proving to be a problem and so I have changed over to the factorialize-4 version of the function. Since my actual likely data is much smaller than that used in the speed tests of this post, I'm not overly concerned about the difference in speed.

An advantage of the method of factorialize-4, which uses an enumerator is that this method could be used to output data to a database or file in a way that didn't need to use very much run-time memory. Very likely in these cases, the time spent sending and saving data to persistent storage is much greater than the computational effort. The factorialize and factorialize-2 methods do not lend themselves to this as they do not generate complete tuples until all of the tuples are created. factorialize-4 creates complete tuples one at at time and these can be acted on immediately without waiting for all of the output to be generated.

Common Lisp Pipes→Enumerators

I've been plodding along through Peter Norvig's Paradigm's in Artificial Intelligence Programming and finding some helpful tips along the way. I especially enjoyed the discussion of pipes in chapter 9. I was intrigued by the pipes implementation of the sieve or Eratosthenes and wondered what else I could do with pipes.

Something that I didn't like about pipes is that they keep around a list of the results, when maybe you are actually done with those results. I was taking combinations of factors to produce a factorial experimental design and I decided to do so using something I have called multiple-radix-counting. Multiple-radix, meaning the radix (or base) for each digit is different. Imagine an odometer where each spinner has a different number of marks on it. Instead of always going from 0 to 9, some of the spinners might be 0 to 1 or 0 to 5. The sequencing is the same. When you get the end of the right most spinner, you wrap around on the first and advance the next spinner. Here's some sample output the shows the pattern:

(loop for enum = (multiple-radix-counter #(3 2 3) :terminate t) then (get-next enum) while enum do (print (get-current enum))) #(0 0 0) #(0 0 1) #(0 0 2) #(0 1 0) #(0 1 1) #(0 1 2) #(1 0 0) #(1 0 1) #(1 0 2) #(1 1 0) #(1 1 1) #(1 1 2) #(2 0 0) #(2 0 1) #(2 0 2) #(2 1 0) #(2 1 1) #(2 1 2) NIL

I have taken the definitions from PIAP and modified them to avoid hanging on to the part of the pipe that has gone past. I have renamed certain functions/macros and made minor changes to them to produce a result that behaves differently. Instead of make-pipe, I have make-enumerator, although the macro is identical. It's important that I have a new name for this macro that suits the different usage in case I decide to implement enumerators differently in the future. The function get-current and get-next are the replacements for head and tail, respectively.  The representation is very similar to that used for pipes. I use a cons cell where the car is the current value and the cdr is a function call.

(defmacro make-enumerator (head tail) `(cons ,head #'(lambda () ,tail))) (defun get-current (enumerator) (if (consp enumerator) (first enumerator) nil)) (defun get-next (enumerator) (when (functionp (rest enumerator)) (funcall (rest enumerator)))) (defun integer-enum (&optional (start 0) end) (if (or (null end) (<= start end)) (make-enumerator start (integer-enum (1+ start) end)) nil)) (defvar empty-indices '(nil)) ;; this is a non-destructive implmentation (defun multiple-radix-counter (radices &key indices terminate) (when (null indices) (setf indices (make-array (length radices) :initial-element 0))) (if (and terminate (equal indices empty-indices)) nil (make-enumerator indices (multiple-radix-counter radices :indices (loop with dest = (make-array (length indices) :initial-contents indices) for i from (1- (length indices)) downto 0 if (< (1+ (aref dest i)) (aref radices i)) do (incf (aref dest i)) (return dest) else do (setf (aref dest i) 0) finally (return (if (and terminate (< i 0)) empty-indices dest))) :terminate terminate)))) (defun extract-by-multiple-radix (source indices) (loop with res = (make-array (length indices)) for i from 0 to (1- (length indices)) do (setf (aref res i) (aref (aref source i) (aref indices i))) finally (return res)))

The purpose of the extract-by-multiple-radix is to take a selection out of source (an array of arrays) based on the indices that were specified. If you were doing multiplication of multiple term factors, you would take the terms by index from each factor and multiply them. To get the factors based on index, you would use extract-by-multiple-radix.

Trimming the Split Ends of Waveforms

In a recent post about making sounds in F# (using Pulse Code Modulation, PCM) I noted that my sound production method was creating popping sounds at the ends of each section of sound. I suggested this was due to sudden changes in phase, which I am persuaded is at least involved. Whether or not it is the fundamental cause, it may be a relevant observation about instrumental sound production.

One way to make the consecutive wave forms not have pops between them might well be to carry the phase from one note to the next as it would lessen the sudden change in the sound. Another way, probably simpler, and the method I pursue in this post is to "back off" from the tail—the "split end" of the wave form—and only output full cycles of waves with silence filling in the left over part of a requested duration. My experimentation with it suggests that this approach still results in a percussive sound at the end of notes when played on the speakers. 

The wave forms I was previously producing can be illustrated by Fig. 1. We can reduce the popping by avoiding partial oscillations (Fig. 2.). However, even on the final wave form of a sequence wave forms, there is an apparent percussive sound suggesting that this percussive effect is not (fully) attributable to the sudden start of the next sound. Eliminating this percussive effect would probably involve dampening.[1] Either the dampening would need to be a tail added to the sound beyond the requested duration or a dampening period would need to be built in to the duration requested.

Fig. 1. A partial oscillation is dropped and a new wave form starts at a phase of 0.0. The "jog" is audible as a "pop".

Fig. 2. "Silent partials" means we don't output any pulse signal for a partial oscillation. The feature as played on my speakers left the appearance of a slight percussive sound, though less annoying than the pop.

It's worth thinking a bit about how typical "physical" sound production has mechanisms in it which "naturally" prevent the popping that we have to carefully avoid in "artificial" sound production.

• In a wind instrument, you can't have a sudden enough change in air pressure or sudden enough change in the vibration of the reed or instrument body to create pops.
• In a stringed instrument, alteration of the frequency of the vibration of the string will maintain the phase of the vibration from one frequency to the next. 
• The wave form is not "interrupted" by anything you can do to the string. There are no truly "sudden" changes you can make to the vibration of the string. 
• Any dampening you can do is gradual in hearing terms. 
• A "hammer-on" a guitar string does not suddenly move the position of the string with respect to its vibration period—phase is maintained.
• In a piano, dampening does not create sufficiently sudden changes in the wave form to create a pop.

In short, (non-digital) instrumental sound production avoids "pops" by physical constraints that produce the effects of dampening and/or phase continuity. Digital sound production is not inherently constrained by mechanisms that enforce these effects.

I have altered the sinewave function to fill the last section of the sound with silence, following the pattern suggested by Fig. 2. This does leave an apparent percussive effect, but is a slight improvement in sound.

// duration is in seconds let sinewave volume freq_hertz sample_rate duration = let TWOPI = 2.0 * System.Math.PI // how long is a single step? let timestep = 1.0 / (float sample_rate) // how many total steps will fit? let steps = int(floor(duration / timestep)) // number of full cycles in duration let fullcycles = int(floor(duration * freq_hertz)) // time for one cycle let cycletime = 1.0 / freq_hertz // time for all of the full cycles let fullcyclestime = cycletime * (float fullcycles) let fullCycleSteps = int(floor(fullcyclestime / timestep)) let silentSteps = steps - fullCycleSteps // amplitude ranges between negative and positive (shifted down by half volume) let amp = volume / 2.0 let theta = freq_hertz * TWOPI * timestep seq { // full oscillations for i = 1 to fullCycleSteps do yield amp * sin(theta * float(i)) // fill remainder with silence for i = 1 to silentSteps do yield 0.0 }

Something this experiment does not deal with is whether we are hearing an effect of the wave form, per se, or whether we are hearing the behavior of the speakers that are outputting the sound. A sudden stop of something physical within the speaker might generate an actual percussive sound. Also still outstanding is whether the human ear can perceive phase discontinuity in the absence of an amplitude discontinuity.

I decided to pick this up again from the other direction and look at fast Fourier transforms done in Maxima here.

¹ Dampening is a (progressive) reduction in amplitude over several consecutive oscillations of a wave.

Sounds Like F#

Last time, I thought that I was going to have to do some special research to account for multiple sources of sound. Reading this short physics snippet I can see that multiple sources is really as simplistic as my initial thinking was inclined to. You can ignore all of the decibel stuff and focus on the statements about amplitude, because, of course, I am already working in terms of amplitude. What can I say, I let the mass of voices on the internet trouble my certainty unduly? Sort of.

There's two important issues to consider:

1. Offsetting of the signals.
2. Psychological issues regarding the perception of sound.

Regarding number 2, here's a quick hit I found: http://hyperphysics.phy-astr.gsu.edu/hbase/Sound/loud.html. It seems credible, but I'm not sufficiently interested to pursue it further. And so, my dear Perception, I salute you in passing, Good day, Sir!

Offsetting of the signals is something that is implicitly accounted for by naively adding the amplitudes, but there is a question of what the sineChord() function is trying to achieve. Do I want:

1. The result of the signals beginning at exactly the same time (the piano player hits all his notes exactly[?] at the same time—lucky dog, guitar player not as much[?]. Hmm...)?
2. The average case of sounds which may or may not be in phase?

Regarding case 2, Loring Chien suggests in a comment on Quora that the average case would be a 41% increase in volume for identical sounds not necessarily in phase. At first glance, I'm inclined to think it is a plausible statement. Seeming to confirm this suggestion is the summary about Adding amplitudes (and levels) found here, which also explains where the the 41% increase comes from, namely, from \(\sqrt{2}\approx 1.414\dots\). Good enough for me.

All such being said, I will pass it by as I am approaching this from a mechanical perspective. I'm neither interested in the psychological side of things, nor am I interested in getting average case volumes. I prefer the waves to interact freely. If would be neat, for example, to be able to manipulate the offset on purpose and get different volumes. The naive approach is little more subject to experimentation.

I would like to go at least one step further though and produce a sequence of notes and chords and play them without concern of fuzz or pops caused by cutting the tops of the amplitudes off. We need two pieces: a) a limiter and b) a means of producing sequences of sounds in a musically helpful way.

A limiter is pretty straight-forward and is the stuff of basic queries. I have the advantage of being able to determine the greatest amplitude prior to playing any of the sound. The function limiter() finds the largest absolute amplitude and scales the sound to fit.

let limiter volume wave = let amp = volume / 2.0 let max = wave |> Seq.map (fun u -> abs(u)) |> Seq.max if max <= amp then wave else let compression = amp / (float max) Seq.map (fun u -> u*compression) wave
Here is an example usage:

let ChordAOpen = seq[69; 76; 81; 85; 88] notes.sineChord 65535.0 ChordAOpen sample_rate 1.0 |> notes.limiter 65535.0 |> Seq.map (fun u -> int(u)) |> notes.PlayWaveAsync sample_rate

One concern here is that we may make some sounds that we care about too quiet by this approach. To address such concerns we would need a compressor that affects sounds from the bottom and raises them as well. The general idea of a sound compressor is to take sounds within a certain amplitude range and compress them into a smaller range. So, sounds outside your range (above or below) get eliminated and sounds within your range get squeezed into a tighter dynamic (volume) range. This might be worth exploring later, but I do not intend to go there in this post.

Last post I had a function called sineChord(), which I realize could be generalized. Any instrument (or group of identical instruments) could be combined using a single chord function that would take as a parameter a function that converts a frequency into a specific instrument sound. This would apply to any instruments where we are considering the notes of the chord as starting at exactly the same time (guitar would need to be handled differently). So, instead of sineChord(), let's define instrumentChord():

// expects: // volume (2 x amplitude) // frequency (Hertz) // sample_rate (Hertz) // duration (seconds) let instrumentChord (instrument: (float->float->int->float->seq<float>)) (volume:float) (sample_rate:int) (duration:float) notes = notes |> Seq.map frequency |> Seq.map (fun f -> instrument volume f sample_rate duration) |> Seq.reduce (fun a b -> Seq.map2 (fun sa sb -> sa + sb) a b)

Next we will create a simple class to put a sequence of notes/chords and durations into which we can then play our sounds from.

Reference Chords and Values

Here are a few reference chord shapes and duration values. I define the chord shape and the caller supplies the root sound to get an actual chord. The durations are relative to the whole note and the chord shapes are semi-tone counts for several guitar chord shapes. The root of the shapes is set to 0 so that by adding the midi-note value to all items you end up with a chord using that midi note as the root. This is the job of chord().

let m1st = 1.0 // whole note let m3_4 = 0.75 // dotted-half let m2nd = 0.5 // half let m4th = 0.25 // quarter let m8th = 0.125 // eighth let m16th = 0.0625 // sixteenth let m32th = 0.03125 // thirty-secondth let EShape = seq[0; 7; 12; 16; 19; 24] let AShape = seq[0; 7; 12; 16; 19] let DShape = seq[0; 7; 12; 16] let DAShape = seq[-5; 0; 7; 12; 16] let CShape = seq[0; 4; 7; 12; 16] let CGShape = seq[-5; 0; 4; 7; 12; 16] let GShape = seq[0; 4; 7; 12; 16; 24] let Gno3Shape = seq[0; 7; 12; 19; 24] let EmShape = seq[0; 7; 12; 15; 19; 24] let AmShape = seq[0; 7; 12; 15; 19] let DmShape = seq[0; 7; 12; 15] let DmAShape = seq[-5; 0; 7; 12; 15] let chord root shape = shape |> Seq.map (fun u -> root + u)

Next, InstrumentSequence is an accumulator object that knows how to output a wave form and play it. For simplicity, it is limited to the classic equal temperment western scale. Note that it takes a function has the task of turning a frequency into a wave form, which means that by creating a new function to generate more interesting wave forms, we can have them played by the same accumulator. If we only want it to produce the wave form and aggregate the wave forms into some other composition, we can access the wave form through the routine WaveForm.

type InstrumentSequence () = let chords = new ResizeArray<seq<int>>() let metrics = new ResizeArray<float>() member this.AddNote note duration = chords.Add(Seq.singleton note) metrics.Add(duration) member this.AddChord notes duration = chords.Add(notes) metrics.Add(duration) member this.Chords = chords member this.WaveForm instrument volume tempo sample_rate = let whole_note_duration = 240.0 / (float tempo) let runs = Seq.map2 (fun chord metric -> let duration = metric * whole_note_duration instrumentChord instrument volume sample_rate duration chord) chords metrics seq { for r in runs do yield! r} member this.PlayInstrument instrument volume tempo sample_rate = this.WaveForm instrument volume tempo sample_rate |> limiter volume |> Seq.map int |> PlayWave sample_rate member this.PlayInstrumentAsync instrument volume tempo sample_rate = this.WaveForm instrument volume tempo sample_rate |> limiter volume |> Seq.map int |> PlayWaveAsync sample_rate

A sample use of the class which follows a very basic E, A, B chord progression follows:

let song = notes.InstrumentSequence() // E major song.AddChord (notes.chord 40 notes.EShape) notes.m2nd // A major song.AddChord (notes.chord 45 notes.AShape) notes.m4th // B major bar chord song.AddChord (notes.chord 47 notes.DShape) notes.m4th song.AddChord (notes.chord 40 notes.EShape) notes.m2nd song.AddChord (notes.chord 45 notes.AShape) notes.m4th song.AddChord (notes.chord 47 notes.DShape) notes.m4th song.AddChord (notes.chord 40 notes.EShape) notes.m2nd song.PlayInstrumentAsync notes.sinewave 65535.0 92 44100

To get the same progression with minors, use the EmShape, etc., values. But, at the end of the day, we've still got sine waves, lots of them. And, it's mostly true that,

                           boring + boring = boring.

I will not attempt the proof. You can make some more examples to see if you can generate a counter-example.

There is also some unpleasant popping that I would like to understand a bit better—for now I'm going to guess it relates to sudden changes in the phases of the waves at the terminations of each section of sound. Not sure what the solution is but I'll guess that some kind of dampening of the amplitude at the end of a wave would help reduce this.

The Sound of F#

I'm doing a bit or reading and dabbling the in the world of sound with the hope that it will help me to tackle an interesting future project. One of the first steps to a successful project is to know more about what the project will actually entail. Right now I'm in exploratory mode. It's a bit like doing drills in sports or practicing simple songs from a book that are not really super interesting to you except for helping you learn the instrument you want to learn. I'm thinking about learning to "play" the signal processor, but it seemed like a good start to learn to produce signals first. And sound signals are what interest me at the moment.

Sound is pretty complicated, but we'll start with one of the most basic of sounds and what it takes to make them in Windows, writing in F#. Our goal in this post is to play sounds asynchronously in F# Interactive.

The first iteration of this comes from Shawn Kovac's response to his own question found here at Stack Overflow. Below is my port to F#. I have used the use keyword to let F# handle the Disposable interfaces on the stream and writer to take care of all the closing issues. It's important to note that the memory stream must remain open while the sound is playing, which means this routine is stuck as synchronous.

// ported from Shawn Kovac's response to his own question at: // http://stackoverflow.com/questions/19672593/generate-morse-code-or-any-audio-in-net-c-or-vb-net-without-3rd-party-depe let PlaySound frequency msDuration volume = let TAU = 2.0 * System.Math.PI let formatChunkSize = 16 let headerSize = 8 let formatType = int16(1) let tracks = int16(1) let samplesPerSecond = 44100 let bitsPerSample = int16(16) let frameSize = int16(tracks * ((bitsPerSample + int16(7)) / int16(8))) let bytesPerSecond = samplesPerSecond * int(frameSize) let waveSize = 4 let samples = (samplesPerSecond * msDuration) / 1000 let dataChunkSize = samples * int(frameSize) let fileSize = waveSize + headerSize + formatChunkSize + headerSize + dataChunkSize // var encoding = new System.Text.UTF8Encoding(); use mStrm = new MemoryStream() use writer = new BinaryWriter(mStrm) writer.Write(0x46464952) // = encoding.GetBytes("RIFF") writer.Write(fileSize) writer.Write(0x45564157) // = encoding.GetBytes("WAVE") writer.Write(0x20746D66) // = encoding.GetBytes("fmt ") writer.Write(formatChunkSize) writer.Write(formatType) writer.Write(tracks) writer.Write(samplesPerSecond) writer.Write(bytesPerSecond) writer.Write(frameSize) writer.Write(bitsPerSample) writer.Write(0x61746164) // = encoding.GetBytes("data") writer.Write(dataChunkSize) let theta = frequency * TAU / (float samplesPerSecond) // 'volume' is UInt16 with range 0 thru Uint16.MaxValue ( = 65 535) // we need 'amp' to have the range of 0 thru Int16.MaxValue ( = 32 767) let amp = volume >>> 2 // so we simply set amp = volume / 2 for step = 0 to samples-1 do let s = int16(float(amp) * sin(theta * (float step))) writer.Write(s) // set ourselves up at the beginning of the file mStrm.Seek(int64(0), SeekOrigin.Begin) |> ignore // we use a player object which requires its memory stream to exist in order to play use player = new System.Media.SoundPlayer(mStrm) player.PlaySync()

To get an asynchronous sound we can shove the PlaySound function into a background thread and let it go. This looks like the following:

let PlaySoundAsync frequency msDuration volume = let soundThread = Thread((fun () -> PlaySound frequency msDuration volume)) soundThread.IsBackground = true |> ignore soundThread.Start()

A limitation of the PlaySound() approach given above is that it limits your method of sound production. The details of the sound produced are buried inside the function that plays the sound. I don't want to have a bunch of separate routines: one that plays sine waves, one that plays saw tooth, one that plays chords with two notes, one that plays chords with three notes, one that include an experimental distortion guitar—whatever. The sound player shouldn't care about the details, it should just play the sound, whatever it is. (Not a criticism against Kovac, he had a narrower purpose than I do; his choice was valid for his purpose.)

I want to decouple the details of the sound to be produced from the formatting and playing of the sound. Here we have the basic formatting and playing of the sound packaged up that will take any wave form sequence we give it (sequence of integer values):

// suggested sample_rate = 44100 let PlayWave sample_rate wave = let TWOPI = 2.0 * System.Math.PI let formatChunkSize = 16 let formatType = int16(1) let tracks = int16(1) let bitsPerSample = int16(16) let frameSize = int16(tracks * ((bitsPerSample + int16(7)) / int16(8))) let bytesPerSecond = sample_rate * int(frameSize) use mStrm = new MemoryStream() use writer = new BinaryWriter(mStrm) // we write the initial byte of each item of data in the comment to the right // this is so we can come back later and write to two of these locations writer.Write(0x46464952) // = encoding.GetBytes("RIFF"): 0 writer.Write(0) // filesize needs to be calculated and filled in at byte 4 writer.Write(0x45564157) // = encoding.GetBytes("WAVE"): 8 writer.Write(0x20746D66) // = encoding.GetBytes("fmt "): 12 writer.Write(formatChunkSize) //: 16 writer.Write(formatType) //: 20 writer.Write(tracks) //: 22 writer.Write(sample_rate) //: 24 writer.Write(bytesPerSecond) //: 28 writer.Write(frameSize) //: 32 writer.Write(bitsPerSample) //: 34 writer.Write(0x61746164) // = encoding.GetBytes("data"): 36 writer.Write(0) // dataChunkSize needs to be calculated and filled in: 40 let mutable samples = 0 wave |> Seq.iter (fun (s:int) -> writer.Write((int16 s)) samples <- samples + 1) // fill in the datachunk mStrm.Seek(int64(40), SeekOrigin.Begin) |> ignore let dataChunkSize = samples * int(frameSize) writer.Write(dataChunkSize) // fill in the filesize let filesize = 4 + (8 + formatChunkSize) + (8 + dataChunkSize) mStrm.Seek(int64(4), SeekOrigin.Begin) |> ignore writer.Write(filesize) // set ourselves up at the beginning of the file to play it mStrm.Seek(int64(0), SeekOrigin.Begin) |> ignore // we use a player object which requires its memory stream to exist in order to play use player = new System.Media.SoundPlayer(mStrm) player.PlaySync() let PlayWaveAsync sample_rate wave = let soundThread = Thread((fun () -> PlayWave sample_rate wave)) soundThread.IsBackground <- true soundThread.Start()

I haven't decoupled this as far as we could. If we wanted to handle multiple tracks, for example, you would need to take this abstraction a little further.

Now, we also need some functions for generating wave forms. Sequences are probably a good choice for this because you will not need to put them into memory until you are ready to write them to the memory stream in the PlayWave function. What exactly happens to sequence elements after they get iterated over for the purpose of writing to the MemoryStream, I'm not quite clear on. I know memory is allocated by MemoryStream, but whether the sequence elements continue to have an independent life for some time after they are iterated over, I'm not sure about. The ideal thing, of course, would be for them to come into existence at the moment they are to be copied into the memory stream (which they will, due to lazy loading of sequences) and then fade immediately into the bit bucket—forgotten the moment after they came into being.

Here are a few routines that relate to making chords that are not without an important caveat: the volume control is a bit tricky.

let ChordEOpen = seq[64; 71; 76; 80; 83; 88] let ChordAOpen = seq[69; 76; 81; 85; 88] // duration is in seconds let sinewave volume freq_hertz sample_rate duration = let TWOPI = 2.0 * System.Math.PI let timestep = 1.0 / (float sample_rate) let steps = int(floor(duration / timestep)) let theta = freq_hertz * TWOPI * timestep seq { for i = 1 to steps do yield volume * sin(theta * float(i)) } let frequency note = 440.0 * exp(float(note - 69)/12.0 * log(2.0)) // produce sine wave form for several different notes // volume is the volume of each individual note; // it is up to the caller to deal with the possible need for compression on the resultant signal // or to keep the volume under control from the start let sineChord volume notes sample_rate duration = notes |> Seq.map frequency |> Seq.map (fun f -> sinewave volume f sample_rate duration) |> Seq.reduce (fun a b -> Seq.map2 (fun sa sb -> sa + sb) a b)

Here is a sample script to run in F# Interactive:

let sample_rate = 44100 notes.sineChord (65535.0/6.0) (seq [69; 76; 81; 85; 88; 93]) sample_rate 1.0 |> Seq.map (fun u -> int(u)) |> notes.PlayWaveAsync sample_rate

The above plays a version of an A chord (5th fret E-form bar chord on the guitar).

Note that I have divided the volume by the number of notes in the chord. If we don't watch the volume like this and we let it get away on us, it will change the sound fairly dramatically. At worst something like the "snow" from the old analog TVs when you weren't getting a signal, but if you just go a little over, you will get a few pops like slightly degraded vinyl recordings.

We could include volume control in our wave form production routines. This wouldn't necessarily violate our decoupling ideals. The main concern I have is that I want the volume indicated to represent the volume of one note played and the other notes played should cause some increase in volume. I want the dynamic fidelity of the wave forms to be preserved with different sounds joining together. Even after that, you may want to run a limiter over the data to make sure the net volume of several sounds put together doesn't take you over your limit.

However, I need to do more research to achieve these ideals. For now, I will break it off there.

>> Imitation Forward Pipe for Common Lisp

Since learning a little F# about a year ago, I took a liking to its forward pipe operator (|>). The semantics of the forward pipe work great with F#'s sequence data type (with so-called, "lazy" computation) and functional style programming. It allows you to express a progression of consecutive steps in the order they will be done, while allowing you to directly feed the results of the last function into the input of the next function, avoiding redundant variable declarations. I suppose you could call these types of consecutive actions "transitive" combinations of operations. Imperative style programming would preserve the apparent sequence of events (meaning, the order you type the operations in, is the order the operations happen in), but it also involves declaring intermediate variables, which perhaps you will only use once.

I'm not an expert on functional style programming and don't intend, right now, to explain or justify its merits, except to say that sometimes it produces cleaner, clearer code. But while I have a current preference for Common Lisp above most other (programming) languages at the present, I really like that forward pipe operator in F#. This morning, I thought it was time to cross that bridge—try to make an imitation forward pipe in Common Lisp.

In F#, if you wanted to turn a into b into c, you might do something like

let a2c x =
    a2b x
    |> b2c

where b2c also takes one parameter. In Lisp, you end up with something more like

(defun a->c (x)

  (b->c (a->b x)))

This is fine for being concise but it doesn't preserve the (apparent) logical ordering of \(a\to b\to c\). We could do that with a change to something more imperative, such as

(defun a->c (x)

  (let ((b (a->b x)))

    (b->c b)))

and this works okay. However, I would like to write

(defun a->c (x)
  (>> 
    (a->b x)
    (b->c)))

Or,

(defun a->e (x)
  (>> :pipe-in
    (a->b x)
    (b->c)
    (c->d :pipe-in 0.0625d0)
    (d->e)))

where we want this last one to expand into

(D->E (C->D (B->C (A->B X)) 0.0625))

I've introduced a keyword so that we can dictate which parameter of the next function the previous result goes into. You can specify any keyword to use as a pipe-in parameter, but make sure you don't use a keyword of one of the functions being called in the sequence—it'll really mess with your mind. Here's the macro definition along with an error condition:

(define-condition received-bad-pipe-keyword (error) ((text :initarg :text :reader text))) (defmacro >> (opt-symbol &rest li) (destructuring-bind (pipe-sym lis) (if (keywordp opt-symbol) (list opt-symbol li) (if (listp opt-symbol) (list :pipe-in (cons opt-symbol li)) (error 'received-bad-pipe-keyword :text "Received bad pipe keyword."))) (loop for i on lis unless (cdr i) do (return (car i)) do (if (find pipe-sym (cadr i)) (nsubstitute (car i) pipe-sym (cadr i) :count 1) (nconc (cadr i) (list (car i)))))))

To test that the code works, we can try this at the REPL,

CL-USER> (macroexpand-1 '(>> :pipe-in
 (a->b x)
 (b->c) 
 (c->d :pipe-in 0.0625) 
 (d->e)))
(D->E (C->D (B->C (A->B X)) 0.0625))

This macro does not attempt to handle multiple return values—though such a development might lead to an interesting result. It might be as easy as introducing more keywords.

Merging Uniquishly in Common Lisp

I've been working on some code with dxf files and I needed to have some generalized merging. In particular, for my first function, I wanted to merge headers without creating duplicate entries. My reader sorts the header values, so when I go to merge I just need to have a simple merge that looks at each of two lists and collects the lowest value and only advances the list that had an element "consumed" (as in, I collected it). If the values are equal, it should collect one instance, and consume from both lists (i.e., advance—take the cdr of—both lists).

The approach I've taken does not guarantee a list of unique items, only that if the original lists are sorted and contain no duplicates, then the resulting list will be sorted and contain no duplicates.  It is also non-destructive, returning a new list

(Confession: Actually, I did it the hard way the first time. Custom code that can only do one thing. Now I'm going back and amending my ways as an ardent Lisper.)

I went with a loop macro for one reason: the collect key word. This is one of the things that keeps me coming back to the loop macro. I like it overall, but when I'm struggling with the loop macro, I sometimes want to use the do* macro.  But I dislike the way do* lends itself to consing and a final call to reverse.  Hence, I come back whimpering to the loop macro.

The code:

(defgeneric non-duplicating-merge (a b triple-test) (:documentation "Non-destructive merge.")) (defmethod non-duplicating-merge ((a list) (b list) tri-test) (loop for (alist blist) = (list a b) then (list nexta nextb) for ait = (car alist) for bit = (car blist) for nexta = alist for nextb = blist for tri-test-res = (funcall tri-test ait bit) if (< tri-test-res 0) collect ait into clist and do (setf nexta (cdr alist)) else if (> tri-test-res 0) collect bit into clist and do (setf nextb (cdr blist)) else collect ait into clist and do (setf nexta (cdr alist) nextb (cdr blist)) end when (not nexta) append nextb into clist and return clist when (not nextb) append nexta into clist and return clist))

Here's some sample output from an EMACS session:

CL-USER> (non-duplicating-merge (list 1 2 5) (list 1 2 4) #'-) (1 2 4 5)

The custom code solution was 23 lines and not so pleasant looking. The tri-test follows the custom of using a negative value to indicate a < b, positive for a > b, and 0 for a = b.  For numeric data, subtraction is the natural choice for a tri-test.

I tried using some for keywords inside of if statements and my horse (compiler) wouldn't make the jump.  But that's okay, I used a for statement at the beginning to initialize some variables (nexta and nextb) which I could then setf within do clauses as necessary.

The more I use the loop macro, the more I like it. Eventually, maybe I will discover all of the things that cause it to choke, stop doing those things, and then I will like it even more.

Round 'em up!

Over the past several months I have been dabbling in Common Lisp (LISt Processing language).  Lisp, I think, is the ugly duckling of the programming languages. The facetious "Lost In Stupid Parentheses" seems more apt as an acronym.

My first encounter with this ungainly language (Lisp) was in AutoLisp (and Visual Lisp). I was on a hiatus from programming regularly and discovered this immediately discouraging language that was the "easiest" way to interact with AutoCAD programmatically. (I don't consider AutoCAD scripts to be programming, though I would still include them under the more general category automation.) I did not make much headway at that time and it frankly amazes me that any non-programmer will step up to the challenge and learn it.

But then there was Maxima. I was happy with Maxima as a free computer algebra system. But in the documentation I found the spectre of Lisp. Specifically, Common Lisp. The computations in Maxima are all in Lisp (though there is probably some non-Lisp they use to glue together a Windows interface for Maxima). Maxima itself supports a fairly robust set of programming constructs as far as doing calculations are concerned.

• Input/output from files
• Controlled loops
• if-then-else
• functions
• plotting (via gnuplot—included in standard installs)
• formatted output
• nice math notation display
• export to latex or image

Maxima is built on top of Lisp and you really have all the abilities of Lisp in Maxima (if you know how to get at them). But, being as I am, always in search of a better way of doing something, I am also concerned with whether a technology built on top of another is "hiding" some of the features of the underlying technology. Perhaps hiding isn't exactly the right way to put it, but learning Lisp is a paradigm shift from most other (Algol-like) programming languages I was familiar with, while on the other hand, learning Maxima didn't take me through that paradigm shift.

The two most interesting features of Lisp to me are the emphasis on function references and macros. But they both help you to do the same thing: reduce the repetition in your code. Mind you, they serve this end in very different ways.

When I wanted to implement Jarvis's march [1, p. 955] to encircle a list of points in a "convex hull" it was particularly the function referencing that made the implementation of the code fairly simple.

Fig 1. The red lines join together the red + signs which indicate the convex hull for these points.

The general idea of Jarvis's march is you find an outside point to start from and then keep picking the next point so that you never have to switch directions as you walk around. I started at the left most point. In the event of a tie, I choose the lower of these. (That would place us at about (0.1, 15.0).) I decided to walk around the points in a counter-clockwise direction. To decide which point comes next, I need to find the point which requires the right-most turn. So I begin by choosing the first of the remaining points (in whatever order I received them). I go through the remaining points and test whether I would need to make a right turn or left turn to point toward them (from my vantage point on the starting point of the hull chosen at my first step). If it is a right turn, then I have a new winner which replaces the first and I continue moving forward to the points not yet checked. The algorithm is really a repeated application of a maximum function where maximum is defined according to a different function than normal. What I need are predicate functions which define the rules for choosing a winner (a "maximum") and a routine to separate the winner from the remaining points that can accept a predicate function as a parameter.  Here's the code for, remove-winner:

(defun remove-winner (test li) (let ((mx (car li)) (temp nil) (therest nil)) (setf therest (loop for i in (cdr li) collect (if (funcall test i mx) (progn (setf temp mx mx i) temp) i))) (list mx therest)))
The idea of remove-winner is to use the test function (supplied by the caller) to determine whether the next item is "better" than the current winner. If you passed in the function < (using the syntax #'<) you would get the smallest value in the list (provided it was a list of numbers). Notice the thought process in this function is not just to find the max and return it, but rather to produce a list which includes the "winner" and the rest of the values (that didn't win) separately. We don't just need to know the winner, we need to separate the winner from the rest.

When we get to the convex-hull routine, we can use the same routine (remove-winner) using two different tests: one to find the initial element and another to find the successive elements.

(defun convex-hull (points) (cond ((< (length points) 3) (list nil points)) ((= (length points) 3) (list points nil)) (t (do* ((init (remove-winner (lambda (a b) (if (= (aref a 0) (aref b 0)) (< (aref a 1) (aref b 1)) ;; pick the lower (< (aref a 0) (aref b 0)))) points)) ;; pick the lefter (start (car init)) (rest (cadr init)) (next nil) (hull (list start)) (result nil)) ((or (eq start next) (not rest)) (list hull rest)) (setf result (remove-winner (lambda (a b) (right-turn-from (vect- a (car hull)) (vect- b (car hull)))) rest)) (setf next (car result)) (if (right-turn-from (vect- start (car hull)) (vect- next (car hull))) (setf next start) ;; set terminating condition (progn ;; else (setf hull (cons next hull) rest (cadr result))))))))

The first occasion we use remove-winner, the function is of the same sort we would use to perform a two key sort. In the main body of the routine, we have a more interesting function which accepts two points, calculates their direction vectors (using the current head node of the hull as the origin) and decides whether the second is a right turn away from the first, in which case it declares the second to be the (current) winner.

Can you do this [abstract the predicate] in other languages? Yes. In many other languages you could implement the routine substantially like this. But the key is this: would you ever think to do it this way if you were using VB.NET or C# or C++? Maybe. But not because of your experience with those languages, but because of your experiences with Lisp or F# or other languages which feature a functional style of programming.

Fig. 1 was produced from running a small bit of Maxima code which loads some routines from a separate package written in Lisp (including the above routines).  You can download both files to try them out:

• points.lisp
• circlethempoints.wxm

You will need to have Maxima installed to try this example out.

Sources:

1. Cormen, T.H., Leiserson, C.E., Rivest, R.L., & Stein, C., Introduction to Algorithms, Second Ed., McGraw-Hill Book Company / The MIT Press, 2001.

Introspecting the Lisp Representation of a Maxima Variable

I recently wanted to know how Maxima (a computer algebra system) implemented something.  I searched and searched and couldn't solve my problem.  Specifically, I wanted to access a Maxima structure from within Lisp code.  However, I didn't know how the structure was implemented in Lisp and therefore didn't know how to access it.

All you need is a single line of inline Lisp:

:lisp (print $structures)

structures is a global variable which stores the structures that have been defined in the session by using defstruct().  I wasted a few hours looking for this information to come up with 25 characters that would answer my question.  Applying the same general idea to an instance of PanelStruct() tells me how it is implemented.  By the way, the answer to my questions looks like this:

((MLIST) ((|$PanelStruct|) $SIZE $TOP $CENTER $BOTTOM $ZENITH $AZIMUTH |$rdv|) ((|$RayStruct|) $START $DIRECTION) ((|$PlaneStruct|) $NORMAL $CONSTANT) ((|$PanelStruct|) $SIZE $TOP $CENTER $BOTTOM $ZENITH $AZIMUTH |$rdv|) ((|$RayStruct|) $START $DIRECTION) ((|$PlaneStruct|) $NORMAL $CONSTANT)) ((MLIST) (($PanelStruct) $SIZE $TOP $CENTER $BOTTOM $ZENITH $AZIMUTH $rdv) (($RayStruct) $START $DIRECTION) (($PlaneStruct) $NORMAL $CONSTANT) (($PanelStruct) $SIZE $TOP $CENTER $BOTTOM $ZENITH $AZIMUTH $rdv) (($RayStruct) $START $DIRECTION) (($PlaneStruct) $NORMAL $CONSTANT))

The near repeat is caused by the print function returning what it has printed and Maxima outputting it.  The $ indicates a references to a Maxima variable and the | | symbols indicate a case sensitive reference.  Oddly, the case is reversed for RDV (which is documented) but not for PanelStruct which surprises me.  (These structures are not built-in but were user-defined in my Maxima session.)

I will still need to figure out how I'm going to use the information, but I have a working concept.

Making a Double Out of Parts in VB.NET

Recently I picked up on a piece of code I had started several months ago and didn't have time to finish.  I was a bit stumped actually, though I've got it pretty well beat into shape now (maybe more on that later).  I was making a VB.NET type Fraction using BigIntegers as the base type.  As such, the class would theoretically be able to represent any rational number.  But what if I wanted to convert that result to a double?  So, I started into the code necessary to do the job.  I began with the easy part, the cases where the numerator and denominator can both be converted to doubles.  In that case, .NET already knows how to do the needed conversions.  But if the numerator or denominator is too large to be converted to a double, the result of division (mathematically) may still be within the range of a double.  For example,

4.445 × 10⁴⁰⁰ / 4.222 × 10³⁹⁹ = 10.528185693983894

An important finishing piece to solving this programming problem is the following function:

Public Shared Function DoubleByParts(positive As Boolean, exponent As Integer, mantissa As Int64) As Double Dim res As Int64 = 0 If Not positive Then res = &H8000000000000000 Dim biasedExp As Int64 = (exponent + 1023) If biasedExp > &H7FF Then Throw New OverflowException("Exponent too large for conversion to type Double.") If biasedExp < 0 Then Throw New OverflowException("Exponent too small for conversion to type Double.") If (mantissa And &HFFF0000000000000) <> 0 Then Throw New OverflowException("Significand too large for conversion to type Double.") biasedExp <<= 52 res = res Or biasedExp res = res Or mantissa Return BitConverter.Int64BitsToDouble(res) End Function

First, for a good summary of the double format, see Double-precision floating-point format (Wikipedia).

Code Walkthrough

The double type has 64 bits as does the Int64 type.  We begin with initializing our result variable to 0.  Depending on whether the number is indicated as positive or not, we set the sign bit.  The sign bit (bit 63, the most significant bit) needs to be set (1) to indicate a negative number.  For simplicity of use, I wanted to let the caller work in terms of the exponent (base 2) rather than with a biased result.  The 1023 "excess" or bias is one of those values you don't want to have to remember.  This function obviates the need to be overly conscious of that value.

After making our biased exponent (biasedExp), we check that it is in range.  We have exactly 11 bits to work with.  If our result is greater than 11 bits (0x7FF in hex, 11111111111 in binary), then we complain to the caller with an OverflowException.  If the result is negative, then it still does not fit into the 11 bits and we complain with an OverflowException in this case as well.  The mantissa is required to be no more than 52 bits.  That's 13 "nibbles" (4 bits, a single hex digit) or 6 1/2 bytes.  If the mantissa has any bits set in the upper 12 bits, we complain.  By calling the variable mantissa I have already declared my draconian intent with this function.  I will not coddle the caller by shifting things that are in the wrong place to the right place.  If the caller sends a mantissa which is too large, they may have done the shifting wrong or they may have incorrectly included the implicit 1 at the beginning of the "binary scientific notation."  Either way, the caller has made an error and we cannot know which one it is.  If we could know which error they made, it might make sense to correct it silently and move on.  However, a wrong assumption here could be somebody's program's undoing.  Therefore, we just complain and let the caller figure out what to do about it.

The biased exponent is now shifted into place—this, like the 1023 excess, falls into the category of "painful detail we want to encapsulate so we can forget about it."  (This is not ambiguous like the too-large-mantissa.)  In our final steps we "Or" the biased exponent and mantissa into place and use a call to BitConverter.Int64BitsToDouble() to ensure we have the proper format.

Room For Improvement

To make this function more robust, an exception class could be defined, either several different ones (one for each error) or a class with properties and enumerations to allow the caller to efficiently test for the particular condition that caused the "overflow".  I note also that I have used OverflowException where an underflow has occurred.  In program debugging, an accurate error message ("over" versus "under") can provide a clue to the person debugging the program.

How to make a .NET (Excel) Add-In Using Visual Basic Express 2010

Automation add-ins are easier than what follows and are the way to go if you just want a few custom functions.  Actually, for efficiency's sake, you might want to just use VBA functions added to Module1  (must be Module1—no renaming to a more suitable name) and then save the workbook as an Excel Add-In (*.xlam).  But, if it is user interaction or in some way making Excel "do stuff" (rather than just return a single result or an array of results), an add-in which implements the IDTExtensibility2 and IRibbonExtensibility interfaces may be a good strategy.  (The main alternative is to use VBA.)

Making an Excel Add-in in VB.NET involves the following (the ordering of the steps is not absolute):

1. Ensure you have Office installed on the development machine and have the Primary Interop Assemblies (PIAs) installed in the Global Assembly Cache (GAC).  You can download them from Microsoft.  (The Microsoft installer will put the PIAs in the GAC.)
2. Start a Class Library project in VB.NET.
3. It's a good idea to generate your own strong name using the sn.exe tool (part of the .NET framework).  You only need one for multiple projects.  In a nutshell, you use the sn.exe tool to create a *.snk file which you will tell VB.NET about.
• let VB.NET know where your *.snk file is by going to Project/Properties..., under Signing check "Sign the assembly" and choose a strong name key file.  You will be able to browse to where your *.snk file is
• if your add-in is intended to be widely distributed, and especially if you want to sell your add-in, you would be well-advised to learn more about this topic to ensure your end-users don't get warning messages that cause them to be (perhaps unduly) alarmed.
4. Apparently, step 4 is unnecessary, but I did step 4, so here it is:  In the project properties, go to Application and click the button Assembly Information....  In this screen, check the Make assembly COM-Visible box.
5. Create a GUID.  There are several ways but a program called guidgen.exe may already be on your computer which will do the job.  Just search your computer for it.  It is located under C:\Program Files\Microsoft SDKs\Windows\... on my computer.
• Copy and Paste the GUID into the Assembly Information... dialog box referred to in step 4.
6. Encapsulate your project in a namespace either by typing it explicitly in the files or by specifying a root namespace in the project properties.
7. Add references to needed PIAs, and extensibility (.NET tab)
• It is recommended that you add these by referencing the Interop Assembly on the .NET tab, not the COM tab. You may need to browse for this: Look under C:\Windows\assembly\gac_msil\Microsoft.Office.Interop.Excel\...
8. Create a class named Connect (seems that the name has to be Connect) which implements the IDTExtensibility2 and IRibbonExtensibility interfaces.
• Implements Extensibility.IDTExtensibility2, Microsoft.Office.Core.IRibbonExtensibility
9. Implement the methods of the interfaces.
10. Decorate the class with COM visibility specifications and include functions for creating and deleting custom registry settings.
11. Register your assembly with regasm.exe.

Steps 8, 9, and 10 are implemented in the following code snippet, which has been parameterized somewhat.

Imports System.Runtime.InteropServices Imports Extensibility Imports XL = Microsoft.Office.Interop.Excel <ClassInterface(ClassInterfaceType.AutoDual), ComVisible(True)> Public Class Connect Implements Extensibility.IDTExtensibility2, Microsoft.Office.Core.IRibbonExtensibility Dim oExcel As XL.Application Public Function GetCustomUI(RibbonID As String) As String Implements Microsoft.Office.Core.IRibbonExtensibility.GetCustomUI Dim res As String res = "<customui xmlns=""http://schemas.microsoft.com/office/2009/07/customui"">" res += "<ribbon>" res += "<qat/>" res += "<tabs>" res += "<tab id=""ArithmosTab"" label=""Arithmos"" insertBeforeQ=""TabFormat"">" res += "<group id=""exportGroup"" label=""Export"" autoScale=""true"">" res += "<button id=""secret111"" label=""Write Secret Number"" screentip=""See what the secret number is..."" imageMso=""DiagramTargetInsertClassic"" onAction=""WriteSecretNumber""/>" res += "</group>" res += "</tab>" res += "</tabs>" res += "</ribbon>" res += "</customUI>" Return res End Function Public Sub WriteSecretNumber(control As Microsoft.Office.Core.IRibbonControl) MsgBox("I don't know what you're talking about.") End Sub Public Sub OnStartupComplete(ByRef custom As System.Array) Implements IDTExtensibility2.OnStartupComplete End Sub Public Sub OnBeginShutdown(ByRef custom As System.Array) Implements IDTExtensibility2.OnBeginShutdown End Sub Public Sub OnDisconnection(RemoveMode As ext_DisconnectMode, ByRef custom As System.Array) Implements IDTExtensibility2.OnDisconnection End Sub Public Sub OnConnection(Application As Object, ConnectMode As ext_ConnectMode, AddInInst As Object, ByRef custom As System.Array) Implements IDTExtensibility2.OnConnection oExcel = Application End Sub Public Sub OnAddInsUpdate(ByRef custom As System.Array) Implements IDTExtensibility2.OnAddInsUpdate End Sub <ComRegisterFunctionAttribute()> Public Shared Sub RegisterFunction(ByVal pType As Type) Microsoft.Win32.Registry.ClassesRoot.CreateSubKey("CLSID\{" + pType.GUID.ToString.ToUpper() + "}\Programmable") Dim key As Microsoft.Win32.RegistryKey = Microsoft.Win32.Registry.ClassesRoot.OpenSubKey("CLSID\{" + pType.GUID.ToString.ToUpper() + "}\InprocServer32", True) key.SetValue("", System.Environment.SystemDirectory + "\mscoree.dll", Microsoft.Win32.RegistryValueKind.String) ' Add-in registry settings so Excel knows that this is an add-in and what it should do with it / register for all users key = Microsoft.Win32.Registry.LocalMachine.CreateSubKey("Software\Microsoft\Office\Excel\AddIns\<yournamespace>.Connect") key.SetValue("CommandLineSafe", 0, Microsoft.Win32.RegistryValueKind.DWord) key.SetValue("Description", "<description of your Add-in.>", Microsoft.Win32.RegistryValueKind.String) key.SetValue("LoadBehavior", 3, Microsoft.Win32.RegistryValueKind.DWord) ' if something goes wrong, tell the user what it is key.SetValue("FriendlyName", "<your friendly addin name>", Microsoft.Win32.RegistryValueKind.String) End Sub <ComUnregisterFunctionAttribute()> Public Shared Sub UnregisterFunction(ByVal pType As Type) Microsoft.Win32.Registry.ClassesRoot.DeleteSubKey("CLSID\{" + pType.GUID.ToString.ToUpper() + "}\Programmable", False) Microsoft.Win32.Registry.LocalMachine.DeleteSubKey("Software\Microsoft\Office\Excel\Addins\<yournamespace>.Connect", False) End Sub End Class

The functions RegisterFunction and UnregisterFunction are invoked by utilities like regasm.exe which comes with the .NET Framework.  I have created a batch file which I run in order to register the assembly.  You will need to modify it to suit your assembly.

cd C:\Users\<userid>\Documents\Visual Studio 2010\Projects\<projectname or names>\bin\Release C:\Windows\Microsoft.NET\Framework\v4.0.30319\regasm /codebase yourassembly.dll pause

Note that once the Add-in is registered and added, you will not need to reregister it after every Build (while developing).  

Local Variables, Function Parameters, and Blocks In Maxima

One of the things that was a real hang-up for me when I started programming in Maxima was its seeming lack of variable scoping rules.  It made some irritating and non-intuitive things happen.  But, because I was always in “get-it-done” mode, I didn’t stop to figure out if there was a better way.  I would resort to adding letters to the name that I wanted to use.  If my functions were going to use a, I would instead name those aa, so that my variables in the “true” global realm wouldn’t be overwritten after the next function call.  My convention would then be, “nice names in the global realm, ugly names in the local realm.”  This is far from ideal.  The fact is, Maxima defaults to global variables, but makes it easy to declare local variables.

Maxima’s primary scoping mechanism is the “block.”  A block starts with block( and ends with ).  Each statement within a block is separated by a comma (,).  The first statement in your block should normally be [v_1, v_2,…,v_n], where v_1, v_2, etc., are variables that you wish to be local (don’t literally type v_1, etc.).  If you don’t want any local variables, then omit the local statement.

What Maxima actually does is not quite the same as in most programming languages, although the apparent runtime behaviour will be the same, if the [v_1, v_2,…,v_n] statement is used as above.  Maxima saves the current values of the variables in the [v_1, v_2,…,v_n] statement and essentially “zeros out” (removes their properties) those variables.  They become blank slates.  When Maxima exits the block() in which the variable was declared as local, its current properties are removed and the saved values/properties are restored.  As far as I know, Maxima does not support multi-threading.  This is probably a good thing, as temporarily preventing a variable from “existing” (as itself, that is; a variable with the same name is not the same variable itself, philosophically) is problematic if that variable could be accessed at any time by a separate thread.

This behaviour is different from most other programming languages which use “dynamic scoping” in that there are no save and restore steps of the kind we are talking about here (in most other languages).  In Maxima, variables have names at run-time and are referenced according to their name.  In most languages, the name of a variable is solely a convenience for programmers.  At run-time, values will move to various parts of the program memory space and the memory addresses will be determined on the fly (e.g., "placed on the stack").  Just because the same name is used in the source code for two variable references, doesn’t mean the compiled code will have references to the same memory location.  For example, t declared in function1() has nothing to do with t declared in function2().  These memory locations will be on the stack and exist only while each function is being executed.  This, again, is what is true in most programming languages, not in Maxima.

When it comes to function parameters, however, Maxima`s behaviour is not unlike most other languages.  That is, the parameters are formal.  They are not considered to constitute a variable declaration.  When I write

f(x) := x^2;

x is not declared as a variable.
Even if I write

RotationalEquivalenceSet(n, bits) := block(
    local(res, size),
    res: {n},
    size: 2^bits,
    if n >= size then return({}),
    n: 2*n,
    if n >= size then n: n - size + 1,
    do (
        if elementp(n, res) then return(res),
        res: union(res, {n}),
        n: 2 * n,
        if  n >= size then n: n - size + 1
    )   
);

n is not declared as a variable.  This is a very helpful thing.  If n was assigned some value before the function call, nothing that I do inside this function call (which uses n as the name of a parameter) affects that value.  The value of n will remain the same after this function has been invoked.  Here is an input/output example of the behaviour:

/* [wxMaxima: input   start ] */
n: x^2;
RotationalEquivalenceSet(3,5);
n;
RotationalEquivalenceSet(5,5);
n;
/* [wxMaxima: input   end   ] */
(%o21) x^2
(%o22) {3,6,12,17,24}
(%o23) x^2
(%o24) {5,9,10,18,20}
(%o25) x^2

Note that I am using := to declare these functions.  You may end up with different (and undesired) behaviour if you don’t use the “delayed evaluation” assignment statement.  If you want the usual programming paradigm with your function declaration, use := as per my sample code above.