"Ray, when someone asks you if you're a god, you say 'YES'!"Winston Zeddemore, Ghostbusters (1984)
I’ve been part of a few discussions recently that have been centered around how awesome maths is. Just how much of it can be used to improve your own programming capabilities, and improve the lot of programming for our entire industry. Both as a tool for designing and building, but also for analysis of new concepts and ideas.
Sadly these discussions usually also turn to lament the current state of mathematics instruction. There is often a feeling that the systems are being drilled into us so that we can provide “the right answer” when it is requested of us. There is no sense of discovery, if we are presented with a problem we cannot solve, it must be because we weren’t taught the techniques we need. What else could it be?
Maybe we were taught the right techniques but we were passing notes that day? Maybe we were taught the techniques we needed but we didn’t know that those techniques could be applied in this situation. We look at those people that are “maths people” or “good with maths” and are often left in wonder as they apply these arcane techniques. We throw up our hands and declare ourselves “bad at maths” or just “not a maths person”.
But as some point out during our discussions on these topics, someone points out that this is a crap excuse as there is no such thing as a “maths person”. We simply haven’t found our feet yet. As part of this post I will present my thoughts on a different way of teaching maths and some of the processes around it. I should probably point out that I am one of the “not a maths person” people. I’ve only recently rediscovered mathematics and I’ve still so much to learn. But I’m having a great time doing so, and it would be great if I could share this with more people.
So without further ado, lets get on with the crazy and see where we end up.
As I alluded to earlier in this post, one the problems presented by the current maths teaching is that we are expected to know ‘the formula’ that is to be used in a given mathematical context. This memorised truth will provide the correct answer provided our rote skills were up to the task. If you don’t know the forumla then you’re either bad at maths or you didn’t pay enough attention that day in class, or some combination there of.
This occurs at even the most basic level, with the droning repetition
of the ‘multiplication tables’.
3 * 3 = 9.. 3 * 4 = 12.. 3 * 5 = 15.. and so
on it went. Even addition is taught in the same manner, drumming
various simple formulae into our heads.
But how else can we expect children to understand? Not treating them like idiots would be a good start, but that’s not possible with the mechanised, standard test driven, approach that is currently used. Why do we wait so long to introduce things like algebra and calculus? These topics are hard right? Well, yes and no..
They are difficult concept if the only thing you’ve ever seen is a number based equation that contains occurrences of problems you’ve been forced to memorise through threats of failure. But just like addition, multiplication and friends, these are built on a grounding of core concepts that are themselves quite simple.
So how can we shake things up? Well for a start we need to focus on the how and why of maths and mathematical reasoning. This analogous to the adage of “Give someone a fish, they are fed for a day. Teach them to fish and they will eat for the rest of their life”.
Teaching someone their ‘multiplication tables’ is not required if they understand the core concepts around HOW multiplication works. Sure they might be faster with some calculations cached ahead of time. But with the core concepts learned and practiced then there is no calculation that is beyond them because they understand the mechanism that makes the magic happen.
After long you don’t even need to include numbers in the discussions around the concepts that are being discussed. You’ve cemented the core concepts and suddenly you’re in a position to be able to move to more advanced concepts and build on what exists.
Let’s start with a simple example…
The problem: Define a new operator that demonstrates what happens to the count of apples in a box when a bucket of apples is emptied into it.
Lets call our new operator,
bucket. So a simple expression may be
0 bucket 5 = ??
For the sake of argument, the apples fall out of the buckets one at a time into the box. So our expression above becomes
0 bucket 1 bucket 1 bucket 1 bucket 1 bucket 1 = ??
Now as some very clever people have already made plain, there are many aspects of maths that are just Fancy Counting. You count from the value on the left to the end if you will, of the value on the right. Similar to standing there and counting each apple as it fell from the bucket into the box.
We know what addition is DOING now so we can write the first solution using our bucket expression:
0 bucket 1 = 1 1 bucket 1 = 2 2 bucket 1 = 3 3 bucket 1 = 4 4 bucket 1 = 5
Phew, that was annoying but now we know exactly what our addition is doing. It’s counting along from our starting value on the left, it stops when we’ve counted the number of steps provided on the right. However what we have written above now highlights two patterns that are crucial to understanding addition..
The first is what we’ve already mentioned, we’re just counting. Our
addition follows a distinct pattern that is easy to follow and
replicate. But also it allows us to simplify our
bucket operation to
something more useful, more on that in a moment.
The second thing to notice is that we have established the
Commutativity as it applies to addition. Based on what we know about
bucket operator works, we know that it counts, starting from
the left value, and stopping after we’ve counted the number of steps
on the right.
So from our previous example…
2 bucket 1 = 3 1 bucket 2 = ?? ## Count out the steps 1 bucket 1 = 2 2 bucket 1 = 3 1 bucket 2 = 3 tada!
We’ve done nothing but count from zero to five and we know two vital aspects of how addition operates. Most importantly we know what addition does and how it actually goes about doing it.
Now we can afford to make some assumptions to speed things along and make our lives a little easier. So..
2 bucket 2 = 4 0 bucket 5 = 5 5 bucket 0 = 5
This is awesome, you’ve just defined your own addition operator and you understand the underlying mechanism of how it actually functions. But this isn’t too dissimilar from the sort of introduction that probably occurs already.
Demonstrating these things on the board in a class room won’t really change things since you’re still working through the same steps you might normally work through. But it’s 2014 and we have access to some amazing tools that can provide a very interesting perspective on these otherwise mundane exercises!
How can we advance these sorts of teachings so that we’re able to demonstrate that so much of mathematics is a combination of very simple steps. How can we empower students understanding such that the idea of new operators or techniques doesn’t immediately mean pages of droning exercises? What if they were able to prove they had an understanding of how the concept operated?
Thankfully we have the capability to do all of that! Lets use
multiplication to demonstrate these ideas, building on our shiny new
bucket operator which we’ll define first, then showing how it can be
used to make multiplication work.
But first there is something we need to discuss.
The tools I will use for this demonstration are the Coq Language and its interactive companion CoqIDE. Although they may seem heavy handed for the task at hand, the capabilities and design of these tools provides a unique method for teaching even the most basic fundamentals of mathematics.
A common mistake would be to start quibbling over syntax or fit for purpose, excessive complexity, or suitability for a class room that is not a university level mathematics class. I don’t particularly care for any opinions on the aforementioned topics because it means you’ve missed the point about what I’m trying to convey.
There may be plenty of other options for environments other than Coq that could be suitable. I will not touch on the specifics of why I chose it in this article, it will most likely come up in part #2, when or if there is one.
We will continue to use our examples from before to demonstrate the next steps of the process. But we also need to introduce some additional rules that will help to keep things sane and relatively easy to follow.
Firstly we will discard the use of numbers as denoted in our original examples. Since we’re after an understanding of the core meaning here, the numbers themselves do not aid us and can be safely ignored. We will however maintain the use of whole numbers equal to or greater than, zero. Or natural numbers.
As per many a canonical example or implementation for Coq, we will use
the following definition for our numbers. They can be either
which is zero. Or
S n, which represents the successor of another
natural number. To represent
1 that is simply the successor of zero
S Z = 1.
Inductive nat : Type := | Z : nat | S : nat -> nat.
Depending on your audience you may simply hand-wave this away or explain its relevance in more depth. That is entirely up to you. Its purpose is to provide a mechanism for understanding how whole numbers that are equal to or greater than zero work. So that we’re able to keep our definitions simple. Lets keep moving!
Let’s re-define our
bucket operator based on what we know..
Fixpoint bucket (n : nat) (m : nat) : nat :=
This first line could be described as mere ceremony, but it gives us a
more concrete perspective of our bucket operator. But also clues to
other important facts. For our purposes, the
Fixpoint at the
beginning of the line, for those interested, denotes that we’re about
to define a recursive function.
We then name our operator and indicate that it will accept two inputs,
both of which must be natural numbers:
(n : nat) (m : nat). This
may seem trivial, but knowing a particular mechanism will only be
given particular types of inputs ensures that your mind is only
focused on what is important to its operation.
Finally we declare the result of our operator will be another natural
: nat. This, like the previous point, may seem like a trivial
thing to mention. But like most of mathematics, we need to construct a
solid foundation of concrete principles that can be absolutely relied
upon. In this case, we declare that our operator will only ever
provide us with natural numbers. This ensures that we always know
what we’re dealing with when we use it. We don’t have to ponder or
consider it, we know.
Now we come to the meaty bit about defining, precisely, how our
bucket operator works. Recall that it simply counts, starting from
the left value, a number of steps indicated by the value on the
right. Only now we must define something that works for any natural
number that is provided. Let’s layout our definition and walk through
Fixpoint bucket (n : nat) (m : nat) : nat := match m with | Z => n | S m' => S (bucket n m') end.
Woah nelly there is a lot going on there!! Or is there? If you look
closely there are only two lines with which we need to concern
ourselves with. The rest fall away as mere syntax or mechanics. The
first point to examine is:
match m with. This means the following
expressions are concerned with the current value of
m. This is
great for us because it means we can further narrow our focus.
Lets examine our first line:
| Z => n
We know that we’re only looking at the value of
m, and we have to
decide what we’re going to do if it’s currently a value of
zero. Remember that we’re building addition, and if you take a journey
of zero steps, where will you end up? Right where you started! So if
someone attempts to
n bucket Z then, we know that zero steps from
Now lets have a look at the next line:
| S m' => S (bucket n m')
Now we’re doing something a little more interesting. Remember that
we’re only looking at values of
m. We’ve already handled what
m is zero, so now we’re dealing with whenever
greater than zero, or any
S of any other natural number.
When we match like this we’re in effect reducing the value of
mto the next lowest number, because we know that
S (S Z)is two, and we’ve matched on
S m'. So using two as example the value of
S Zbecause we’ve reduced it by one.
This line is performing one step of our
bucket definition. The
extra step inside the brackets calls our
bucket operation again
using the reduced value of
m. This ensures that our operator will
take as many steps as is required, all the way to zero (
ensure our starting place
n is incremented by the correct number of
steps. Because when we reach
Z for a value of
m, our operator will
n wrapped up in the correct number of steps. Producing our
This may seem all funky and weird and why on earth would you try
to explain addition that way?! Because we haven’t taken the route of
simply droning on with
2 + 2 = 4, 2 + 3 = 5, 2 + 4 = 6 ad infinitum
in order to get our point across. We’ve explained the core concept of
addition in a way that is reusable and free from the dangers of
learning by repetition. Since we’ve conveyed the underlying framework,
there is now the foundations for understanding the next thing we
explain at a deeper level.
Because we’ve approached this instruction by breaking things down, the
same approach can be applied to teaching multiplication. With the
added bonus that we can reuse a student created operator,
help define how multiplication works. This helps demonstrate how
mathematics is a sequence of very small steps applied to a much larger
The depth that which the smaller steps is understood, allows for much greater confidence when presented with far more complex problems. This means we’ve begun, and continue with, teaching mathematics by teaching the techniques of problem solving instead of teaching them how to use the formulas they are given.
If a student is already used to decomposing a problem into smaller pieces, identifying what is required at the smaller stages, and working through them. They will already be practiced at establishing their own solutions, or at the very least identifying what is missing. If they can do this, then once limits have been reached, or after problems have already been solved. Introducing the students to formulae that exist for given problems, or well established mathematical techniques provides immense value.
Because you’re no longer saying “use these formulae for these problems”. You’re supporting and empowering their discoveries by presenting work that suddenly has so much more meaning because it is directly connected to their efforts of trying to solve the problem. For some it will validate their efforts, others it may provide the direction they need, or the final piece of the puzzle for others.
The point is to teach techniques for discovery and investigation. To teach skills for creating solutions to the subproblems and techniques for confirming that their solutions will work. Something that most maths teachers have been trying to do for quite a long time no doubt! But the point of this is not to say that maths teachers are doing it wrong, but that the constant repetition is potentially harmful. That it may in fact prevent a deeper understanding of the concepts they are trying to teach.
This may lead to increased difficulty in understanding newer concepts because the same repetition has left gaps in the understanding of core concepts. They’ve just been accepted as correct “because the teacher or textbook said so”.
To wrap up this particular diatribe, we will continue our little
lesson and explain multiplication using the same techniques we
established earlier. As well as reusing our
bucket operator to
demonstrate the reusability of various mathematical concepts as you
work towards a larger solution.
Before we begin with defining our new
spintastic operator, as before
we need to clue in to how it will actually function. What does it
Based on our previous investigations we know that addition is a form of Fancy Counting, and if we examine multiplication it’s highly likely we’ll discover that it is simply more Fancy Counting.
Lets examine the term, ‘multiplication’, similar to addition the term
gives us clues as to what it is expected to do. Addition is a process
of changing a value by moving a given number of steps from a starting
2 bucket 3 produces
5 because we start at
three more steps to end up at
Multiplication on the other hand is dealing with ‘multiples’. If
is the result of adding
2 steps to
2 spintastic 3 will
produce a result of
3 multiples of
3 occurrences of
2. This seems like a form of Fancy Counting to me. Lets try it.
2 spintastic 3 = ?? 2 spintastic 0 = nothing ! We have no 2s 2 spintastic 1 = one 2 2 spintastic 2 = two 2s 2 spintastic 3 = three 2s, or [2,2,2]
Well this is interesting, now we have 3 occurrences of 2, or a list of
2s. That doesn’t tell us much though because that’s simply another
way of expressing what our
spintastic operator is telling us. How
can we reduce it to a more simple value? Maybe a final result even!
If you have three piles of two apples in our box, what is one way we
can easily determine how many we have? We add them together of
course. You have three
2s, so we can take our list of occurrences
and simply add them all together to find out our final result!
2 spintastic 3 = ?? 2 spintastic 0 = No 2s 2 spintastic 1 = 2 (we only have one 2, nothing to bucket!) 2 spintastic 2 = 2 bucket 2 (getting interesting) 2 spintastic 3 = 2 bucket 2 bucket 2 (oooh!)
So multiplication is just Fancy Counting after all. Multiplication
is just the result of counting all the ‘multiples’ of some other value
or occurrence. Do you think we know enough now to be able define our
spintastic operator like we did with
bucket? Let’s try.
Fixpoint spintastic (n : nat) (m : nat) : nat :=
Does that seem familiar? Of course it does, it’s nearly identical to
bucket definition from earlier! We’re being very particular and
specifying that our
spintastic operator will only work with numbers
that are equal to, or greater than zero. We also only want a single
natural number as a result of this operation. In this case it’s no use
to have our operator give us back a list of natural numbers, because
we already know that’s what
Lets fill out our definition and discuss…
Fixpoint spintastic (n : nat) (m : nat) : nat := match m with | Z => Z | S m' => bucket n (spintastic n m') end.
Yee gads! Remember fear is the mind killer, the little death that brings total oblivion. There is nothing that we here that we have not already covered or explained. We’re just trying to express our flimsy wordy definition in a concrete form. As before we’ll work through it line by line.
match m with
Nothing new here, as with our
bucket operator we’re using the second
m to ensure we create the correct number of multiples of the first
n. We’re just counting.
| Z => Z
This is different! But remember that when you’re counting, if you have
nothing of something, how much do you have? You have nothing! This is
different to counting with
bucket because when we’re counting with
bucket we start with one value, and then start stepping. However
spintastic we’re not starting with anything, so if we have zero
multiples of something, then we have nothing.
Conveniently this expresses one of the laws of multiplication, in that anything multiplied by zero, is zero! Again, by simply working through how something works at its core, you’ve stumbled on and defined one of its laws.
| S m' => bucket n (spintastic n m')
This is a little more hairy isn’t it? Well, not really. Since from
before we know that the
S m' is simply us counting down our
value. Because we’re _counting the number of multiples.
But that next part is interesting…
bucket n (spintastic n m')
Recall from before that we’ve established
spintastic is Fancy
Counting, and that we’re simply counting the occurrences of one of
n. Also in order to achieve our final result of one
number we need to
bucket everything together. But we’re not counting
single steps this time, we need
bucket together every occurrence of
one of our numbers.
2 spintastic 3…
spintastic 2 3 = bucket 2 (spintastic 2 2) bucket 2 bucket 2 (spintastic 2 1) bucket 2 bucket 2 bucket 2 (spintastic 2 Z) bucket 2 bucket 2 bucket 2 Z
Remember we know that anything
bucket Z is the value of simply the
value of anything, because we’ve not taken any steps. So we can
discard that and our result becomes.
2 spintastic 3 = 2 bucket 2 bucket 2 = (2 bucket 2) bucket 2 = 4 bucket 2 = 6
Show your working. ;)
In this example you can see how we used our existing knowledge of the
bucket or addition, and we were able to apply this
knowledge to determining how multiplication works! Except instead of
forcing the memorisation of multiplication tables for treats, I mean
grades. We broke the operations down to its fundamentals and exposed
the core mechanism that made it work.
In doing so we demonstrated that one mathematical concept can be explained by using the mechanism of another. We demonstrated that multiplication is another form of addition, both of which are instances of Fancy Counting. One can in fact define one in terms of the other!
We not only exposed how both operators work underneath, but we also
revealed some of the laws that apply to them. As an exercise to the
reader you can go back to the
spintastic operator and see if you can
work out what happens if you swap the left and right values around.
Hopefully you can now see how changing how some aspects of mathematics are taught can completely change the approach to various problems. We remove the aspect of repetition and replace it with investigation and discovery about underlying mechanisms. The aspects that are repeated are techniques for problem solving and breaking down big problems to little ones, building on previously acquired knowledge to enhance our understanding.
I sincerely hope you do not focus on the syntax of my explanations or get hang up on my suggestion of Coq as a tool that could be employed for this sort of instruction. As truly that is far from the point I am trying to make. I am not trying to turn every maths lesson into a programming session and if I’ve conveyed my point sufficiently you will see that it is not what I suggest.
The point is shifting the focus from repetition and memorisation of formulae that are applied in arbitrary situations, to a concentration on pure problem solving techniques such as; Analysis, investigation, experimentation, and discovery. I dare not assume that this approach will work for everyone, there is no such thing! But by focusing on maths as a process of dissection, analysis, discovery, and invention. We might just open more minds to what is possible.
If you aren’t aware of the existence of Lisp Flavoured Erlang then you really need to get on that. It is a thing of beauty, combining the true deliciousness of two amazing languages - Lisp & Erlang. This post assumes you’re aware of LFE in some capacity and I will continue without regard for your level of interest.
Primarily because this is a “My Crazy Idea Post(TM)”. You’ve been warned.
Currently in LFE there is no method (that I know of) for defining type specs for the Dialyzer system to interrogate. So whilst in Erlang you can incrementally provide greater information about your functions like so:
-spec(add_one(number()) -> number()). add_one(N) -> N+1.
This is something that, because of a slight obsession with Haskell and Idris, is something I’m quite interested in making happen. Additionally taking inspiration from the brilliant “Typed Clojure”. It seems like this is something that LFE could draw some real power from.
I am partial to static types and I am the sort of person that considers them to be a real boon to nearly any application. That said I am able to understand the benefit that comes from ‘incrementally typing’ an application. Providing a base level of surety around certain functionality and expanding it or reducing it as required by emergent circumstances.
LFE is of course a Lisp, and so many of you might be wondering why I don’t just write some macros and call it a day. First, I don’t think macros in LFE can do this because there is no way to generate the spec information in the first place. There may be scan/parse steps involved to generate the attribute information. I don’t think that it’s a huge job but it’s not done yet so I would have to take that into account.
The more I read about Dialyzer the more it seems that simply including support for it within LFE would provide some seriously amazing benefits. If you’re the sort of loon that likes these shenanigans. Pimping it out with a dash of extra functionality could prove useful too!
Such as the
so operator from Idris in this highly contrived example:
;; A required persistent truth. (defun targetInZone ['float 'float -> 'boolean] ((lat long) (and (and (> lat 0) (<= lat (doge-arena-limit 'latitude))) (and (> long 0) (<= long (doge-arena-limit 'longtitude))))) ((0 0) 'false)) (defun deployDoge [(float x y) -> (so (targetInZone x y))] ((lat long) (let ((doge (untasked-doge)) (retask-doge doge lat long 'such-investigate)))))
Could be kinda cool.
Interestingly the more straight forward application of types to an LFE application
could support the
match-lambda functionality in weird and wonderful algebraic ways.
(defun foo [[a] 'int -> [a]] ;; Provide a generic function signature here ((a b) ['list-float 'int -> 'list-float] ;; More specific constraints (: lists map (lambda (x) (wow x b)) a)) ((a b) ['list-int 'int -> 'list-int] ;; are applied to each branch. (: lists map (lambda (x) (such x b)) a)))
I’ve not spent enough time with these systems to gain a sufficient understanding of how things work together. From just my initial reading the type system available courtesy of Dialyzer is really really powerful. I am quite literally just thinking out loud about things I think could be fun/cool/useful and not necessarily in that order.
Because this sort of thing in Haskell is sweet.
foo :: [a] -> (a -> b -> a) -> [a]
Dialyzer supports something similar, although you notice the lack of support for the type of the second argument being different to the first.
-spec foo([any()], fun((any(), any()) -> any()) -> [any()]
As some of you may have noticed though, the type specification above includes the
deliciousness of polymorphic types. Which in ultra-simplistic terms means types
that can take other types as arguments. In the example above,
[any()] is an
example of a polymorphic type,
list(integer()) would be another. This is a
tremendously sexy feature and can be used to great effect when documenting, err,
I mean typing, your application.
Similar to Haskell (I said similar, go rage somewhere else) you are also able to specify your own types in a module and construct your own type classes, in a manner of speaking! Jumping the gun here as I’ve not finished my initial readings on the intracacies of type classes but here is an example from Learn You Some Erlang on Dialyzer.
-type red_panda() :: bamboo | birds | eggs | berries -type squid() :: sperm_whale %% The type for Food is determined in the specification later on -type food(A) :: fun(() -> A)
In our LFE system we could have something like
(deftypes red_panda_food '('bamboo 'birds 'eggs 'berries) squid_food '('sperm_whale) (food a) '((fun '() -> a)))
In Erlang the above types are used in specifications like so:
-spec feeder(red_panda) -> food(red_panda()); (squid) -> food(squid()). -spec feed_red_panda(food(red_panda())) -> red_panda(). -spec feed_squid(food(squid())) -> squid().
For our delicious LFE version it’d be neat to leverage this awesomeness:
(defun feeder #| Match lambdas could support more generic type signatures to enable a quick understanding at a high level, with more precise return types being available |# ['atom -> a] ((red_panda) [(food red_panda_food)] ...) ((squid) [(food squid_food)] ...)) (defun feed_red_panda [(food red_panda_food) -> red_panda_food] (...)) (defun feed_squid [(food squid_food) -> squid_food] (...))
There are obviously nicer ways to handle it and I am going to play around with a few different options to see if anything sticks. I’d like to be able to replicate a Haskell type signature style, predominantly because I believe it to be clear and flexible without adding too much noise.
But that’s not even a concern at the moment since (in case you hadn’t noticed) I still have so much to learn regarding the functionality that is offered by Dialyzer and type systems in general. The growth rate of my reading list shows no signs of abating and like everyone else I must content with ‘real world’ things, I’ve no idea of how much I will be able to commit to this. :(
This post is mostly me just airing my thoughts on this, so don’t expect a link to a repo with a PR to try out or papers in progress under my name (on this topic). I think LFE is one of the most awesome languages around at the moment, even from being neck deep in Haskell and Idris experiments. Type systems have so much to offer when they are built to function as a system for stability, surety, and documentation.
So I think LFE could, COULD, really benefit from having access to this sort of functionality and I sincerely hope I will be able to provide a meaningful contribution. All the same, I think it’d be hell cool to be able to whip out super powerful BODOL style types within your already magnificent distributed Lisp app!
Hopefully more to come ! :)
Now the reports and code I had orignally written aren’t as directly related as I remember. But the conclusions I came to have stuck with me and I’ve been mulling them over ever since.
The first report is related to using the GPU via WebGL for computations and visualisation of Sammon Map data. For investigating the relationships between genomes and genes for various organisms etc.
It can be found here. Don’t laugh too hard! This was my first real attempt at GPU programming for visualisation and computation, but it was immediately appealing.
The second report was related to a project that was targetting a larger web application that was being designed to collect various visualisations in a modular structure. One of the problems we faced was that because we sometimes had larger datasets, the front end would freeze whilst the computations were performed and the visualisation created, and finally rendered.
My solution was to take the data and chunk-a-fi it, perform the computation within a Web Worker and return that package to the front end, where the main thread would handle the visualisation. This meant we could handle much larger data sets and still have a responsive front end, with a visualisation that was handling the constant stream of input data from the Worker.
I am getting to a point, just bear with me.
The part that I ended up being most interested in, but not in a position (at the time) to follow through with. Was the Bonus Level from the presentation…
Obviously the people running those bitcoin ads already worked this out and to a lot of people this wouldn’t be new information…
But to me this seemed really really awesome.. :D
Given a truly massive dataset and an embarrassingly parallel computation that needs to be performed. I think it should be possible to create an application that collects a data packet from a central source, perform the computation and return the result. Asking for another packet if it’s appropriate.
More importantly, this should be possible to achieve using a web based application that can co-exist on a webpage, or be the sole purpose of the page itself. Thus representing a ‘distributed by design’ application for mass computation. You don’t even have to design the delivery mechanism or create operating system specific binaries that then require constant maintenance. …Okay mostly. Cross browser functionality and close-to-universal web apps aren’t simple beasts.
This sort of thing has been done before, the finest example I know of is the SETI @ Home program that operates in an almost identical fashion.
In a given office you may have ~100 employees, of those, maybe 80 have a smartphone. Each smartphone may have anywhere from 1-4 cores, with 1-4 or even more cores on the GPU. Additionally there may be many tablet owners amongst them. So maybe another 40 tablets, with 1-4 cores and even MORE GPU cores available.
That’s not counting the desktop machines around the place, but for this examples lets assume everybody needs those machines dedicated to their current task.
Using napkin pseudo-nothing-like-maths, that’s…
(80 * avg(1-4 cpu_cores) & avg(1-n GPU cores)) + (40 * avg(1-4 CPU cores) & avg(1-n GPU cores))
That makes for many many cores that for a large proportion of the day are sitting, relatively, idle.
A web page that was established on the internal network could contain a dedicated page that would contain the processing application. Happily and quietly exchanging packages and results with a supervisor and utilising all the devices in an internal distributed computing cluster that is:
Basically the same example above except with maybe ~10k students and staff, although the sysadmins may hate you when they see what you’re doing to their network throughput.
For a large number of organisations this would allow them to have a in-house computing cluster with minimal expenditure on high end servers. Schools and other why-the-fudge-aren’t-they-better-funded organisations would be able to use them for all sorts of weird and wonderful things!
Another aspect is that the devices in question are often considered to be ‘low power’ devices. However if we’re suddenly utilising their full computing potential in this way, then we will quickly drain their batteries and annoy a lot of people.
Although there is a simple solution - “Just plug all the devices into the nearest USB port!”. This will add the cost of powering all of these devices to the cost of running the ‘cluster’.
An alternative to this and one that has been proven out by the truly brilliant RaspberryPI community, is to utilise solar power stations for charging all of the devices. Providing not only a free and effective method for keeping these devices charged under normal use. But in essence provides a solar powered, air cooled, silent, computing cluster!!
It should be obvious by now as well that there is MASSIVE potential for all of the devices that would normally be trashed. Simply plug them in to your ever growing cluster for more computing capability! You might need to upgrade your solar systems a little bit over time. But the savings on power and infrastructure investments alone should nicely cover that.
These ideas haven’t actually resulted in me creating anything yet. Which feels silly as the ideas, at least on the face of it, seem relatively straight forward.
“How hard can it be?”
Given my current obsession with learning programming languages I am sure I have sufficient tools in the box to achieve at least a prototype. I should probably get on that…