Elite Computing

Posted: December 14, 2011 in Complete Stupidity, Out In The Fringe

Lower That Bar!

So recently I saw a H I L A R I O U S post on Quora: What do all of the Mediocre People Do In Silicon Valley?  I was so pleased by this that I decided to write yet another blog on what I am now calling Elite Computing.  As a matter of fact this is a moniker that I cannot claim.  Someone who I had a long discussion with said and I quote, ” I like the fact that you talk about Elite Computing and acknowledge people when they are smart.”  They went on to discuss the differences between The Valley and almost everywhere else was that in The Valley you better know and show your smart or else.

A while a go I wrote a blog on Social Darwinism and Software Development.  Lets take an excerpt from this Quora article that appears to be a little more palatible:

“They work in less important roles at larger companies. Certain parts of Yahoo, AOL, Oracle, Microsoft have lots of these people. The companies actually make reasonably productive use of them. They generally have structures in place that prevent them from doing too much damage and make sure they are always making forward progress, even if it’s slow”

Then if you go down in the comments there is the usual bitching about “Why do you have to have over confidence?  So what IF just WHAT IF?  The same pathetic atitude existed in 1) The Military 2) The Police Force 3) Sports.

Stronger Better Faster More.  You want a 1x return on your investment or you want a 10x?  Think about  that the next time some of the top performers seem overly intense, passionate or crazy.

It put the whole 99% and 1% argument in a different light.

To the victor go the spoils.  May the best company win and live large.

Until Then,

Go Big or Go Home!



The USS Sand Lance Nuclear Submarine

I have been doing some work lets say ‘out of the ordinary’ as of late.  I have also been extremely honored to be working around some former people who were directly involved “when peace broke out” during the Cold War.   I have also had the opportunity to work with people who have backgrounds in literature who now work in the technical industry.  Also, dear readers, as most of you know I am fond of the entrepreneurial spirit.    This  led me to thinking about how they all interrelate and then like magic!  Submarines, Shakespeare and Startups.


So lets pick the first one: Submarines.  Some of submariners that I know were directly involved with designing, building and operating submarines.  Specifically Nuclear Powered submarines.   In listening to the stories I noticed how much in common Submarines have with Startups.   Let me be specific herewith:  I am referring to fast attack nuclear submarines.  Here is a wikipedia page for reference: Fast Attack Nuclear Submarines – Los Angeles Class.  I am also referring to problems areas posed as to why Russian Submarines were quieter than USA submarines (if your interested do the google grunt work).

So first let me explain why I believe in this day and age of Lean Startup mentality why we can learn a great deal from those that travel below the ocean surface.

Submariners are consumate planners.

Submariners are by definition – independent operations.  From the perspective of information and physical support the design, building and operation places a premium on planning.   The Commanding Officer (CO) must “go with what they know and have got”.  Efficiency is key.  Space is a premium.  Intelligence at the highest level are mandatory across all aspects of the design, operation and mission.

Submarines are the epitome of stealth

Submarines have been recognized as the preeminence of contributions to stealth and survival.  For instance the greater degree of enemy’s knowledge of the submarine – in both space and time – the greater the diminsihment of the submarines effectiveness.  Stealth relates to safety and the mission of the ship relates to efficiency.  Signature Management is the term given to the ability to lower the footprint of detection.  It ia also known as Stealth Technology.  Stealth technology also termed LO technology (low observable technology) is a sub-discipline of military tactics and passive electronic countermeasures,which cover a range of techniques used with personnel, aircraftshipssubmarines, and missiles, to make them less visible (ideally invisible) to radarinfraredsonar and other detection methods.  Stealth Startup anyone?

Submariners are Concise and Focused

Submariners need to be extremely consice in all matters of communication.  In most cases receiving communications may always be accomplished but it may need to be managed around mission critical requirements.  These mission critical requirements are in most cases time sensitive  In many cases communication transmission may not always be available without mission impact.  Thus one needs to be concise and make decision in-situ at the moment and go with what they know.  Communications are minimized while maximizing the ability to war-fight.


Nowadays, literature folks abound in the technical sector.  I also include english majors and people who dig 17th century poems like Milton’s Paradise Lost in the group.  Why?  They know how to communicate and weave disparate information.  They also know how to tell a story.  They also know how to use commas accordingly.   Shakespeare being the preeminent example of author and playwright wrote stories that naturally mirrored life’s twists and turns.  What does this have to do with startups?  Let us take an example from Hamlet (NOTE: this a cliff notes exceprt from SparkNotes)

” On a dark winter night, a ghost walks the ramparts of Elsinore Castle in Denmark. Discovered first by a pair of watchmen, then by the scholar Horatio, the ghost resembles the recently deceased King Hamlet, whose brother Claudius has inherited the throne and married the king’s widow, Queen Gertrude. When Horatio and the watchmen bring Prince Hamlet, the son of Gertrude and the dead king, to see the ghost, it speaks to him, declaring ominously that it is indeed his father’s spirit, and that he was murdered by none other than Claudius. Ordering Hamlet to seek revenge on the man who usurped his throne and married his wife, the ghost disappears with the dawn.” ~ SparkNotes on Hamlet

Ok so how does this relate to startups?  Or submarines for that matter?  Inquiring minds want to know!  We have Stealth, Revenge and Strategy all rolled into one simple paragraph! Notwithstanding the whole sword fighting thingy (technical term) between Hamlet and Laertes! This!  This is the stuff of true competition!   Moreover Shakespeare mirrors life, life is about humans, humans are bat-house crazy, ergo startups are crazy!

So lets look at three aspects of many of the characters in Shakespeare’s plays and stories:

Shakespeare characters are consumate planners

Revenge usually takes planning.  Lets take the character Fortinbras, who has led an army to Denmark and attacked Poland earlier in the play, enters with ambassadors from England, who report that Rosencrantz and Guildenstern are dead (ok think all that took some planning by Fortinbras or he just waltzed in?) . Fortinbras is stunned by the gruesome sight of the entire royal family lying sprawled on the floor dead (Ok Fortinbras time to make a quick decision). He moves to take power of the kingdom. Horatio, fulfilling Hamlet’s last request, tells him Hamlet’s tragic story (how nice of him).

Shakespeare characters are the epitome of stealth

Ghost and Witches.  Need I say more?  Don’t we create apparitions when we sell our pitch deck?  How about the reality distortion field?  Yes?

Shakespeare characters are consise and focused

Well they are  when it comes to salient aspects such as revenge, de-throning and well murder!   Things that carry high price tags if you really that can go off the rails – people get really motivated.  Kind of like startups.

So let me be specific here on startups.  I really like the terminology Lean Startups.  The book is great if you havent read it.  That said its really nothing new.  Incremental feedback based on whether a customer could really use something is a basic business premise.  We all have gotten caught up in making something “kewl” because we could but so what?  A little known fact is that process doesnt make great companies or great software.  Getting stuff done is about making a great startup.    Also having the ability to NOT PANIC is a good attribute for your founders and initial team.   Think a submariner stuck under the ice can afford to panic?  What good will panicking do?

Which in turn leads me to another issue on a recent development in the startup arena and that is the issue of Aegism and Founder Prejudice. Yep I said it.  It also cuts both ways.  Older founders and employers are seen as dinasours or cannot go the distance and younger persons do not have the pedigree to get through the tough times.  You need both.  Its a syncopated system.  I do prefer to work with those that have been stuck under the ice and do not panic.  If your in a startup you will get stuck under the ice.  Then what?  I dont care how many lines of ruby,perl,python etc you can write.  So can the next person.  Be calm.  Always remember to breathe and use the restroom, head or latrine.  Take your pick.

Remember:  To do a dangerous thing with style is the mark of a professional.  – #TCTRules.

For those that are interested here are some references:

1) Submarine Plans  from Alexander the Great to 1994

2) Submarine Technology for the 21st Century by Zimmerman

until then go big or go home!


Today I spoke at BarCampCHS. After many requests I decided to upload my deck.  Here is my pitch deck pitching How To Do A Quick and Dirty StartUp – BarCampCHS: DirtyStartups.   I have been pretty busy lately but have been taking a ton of notes for several blogs in the future.  Also note please follow me now @tctjr.

Until then,

Go Big Or Go Home!


Visualize Your Visualizations

Visualization has come into vogue in the past years.  Historically this was the world of straight reporting and key performance indicators.  This is not the case now.  FlowingData has brought interactive data analysis to the forefront.   This is due to the ability for us to process massive amounts of data and the uptake in what I like to call fluid  data systems.  If you have seen the books Beautiful Visualization, Data Analysis or Visualize This then you know there are several choices of frameworks and libraries to choose from in this area.  As of late I have been asked a ton of questions about what/why/when of these libraries so I decided to put them all in one place.  While no means comprehensive (e.g. I didn’t include Haskell, MATLAB,’R” et al) Here is the list.  Hit me up if you find something else as I will keep a running list.



Although Axiis claims open source Flex is not open source. Axiis is an open source data visualization framework designed for beginner and expert developers alike.  Whether you are building elegant charts for executive briefings or exploring the boundaries of advanced data visualization research, Axiis has something for you. Axiis provides both pre-built visualization components as well as abstract layout patterns and rendering classes that allow you to create your own unique visualizations.

Axiis is built upon the Degrafa graphics framework and Adobe Flex 3.


We have used prefuse in the past and it is very flexible.  JAVA based and can be modified to run extremely efficient.  That said one must have a programming background.

Prefuse is a set of software tools for creating rich interactive data visualizations. The original prefuse toolkit provides a visualization framework for the Java programming language. The prefuse flare toolkit provides visualization and animation tools for ActionScript and the Adobe Flash Player.

Prefuse supports a rich set of features for data modeling, visualization, and interaction. It provides optimized data structures for tables, graphs, and trees, a host of layout and visual encoding techniques, and support for animation, dynamic queries, integrated search, and database connectivity. Prefuse is written in Java, using the Java 2D graphics library, and is easily integrated into Java Swing applications or web applets. Prefuse is licensed under the terms of a BSD license, and can be freely used for both commercial and non-commercial purposes.

The visualization gallery and demonstration video provide numerous examples of the types of applications that can be built with the prefuse toolkit.



Protovis composes custom views of data with simple marks such as bars and dots. Unlike low-level graphics libraries that quickly become tedious for visualization, Protovis defines marks through dynamic properties that encode data, allowing inheritancescales and layouts to simplify construction.

Protovis is free and open-source, provided under the BSD License. It uses JavaScript and SVG for web-native visualizations; no plugin required (though you will need a modern web browser)! Although programming experience is helpful, Protovis is mostly declarative and designed to be learned by example.


JavaScript InfoVis Toolkit:

The JavaScript InfoVis Toolkit provides tools for creating Interactive Data Visualizations for the Web.

The toolkit implements advanced features of information visualization like TreeMaps, an adapted visualization of trees based on theSpaceTree, a focus+context technique to plot Hyperbolic Trees, a radial layout of trees with advanced animations -called RGraph and other visualizations.

In November 2010 the toolkit was acquired by the Sencha Labs Foundation.



Processing.js is the sister project of the popular Processing visual programming language, designed for the web. Processing.js makes your data visualizations, digital art, interactive animations, educational graphs, video games, etc. work using web standards and without any plug-ins. You write code using the Processing language, include it in your web page, and Processing.js does the rest. It’s not magic, but almost.



While more of a diagraming solution than a visualization mxGraph/JGraph is used in conjunction with the Swing library.  JGraph has been providing leading diagramming software components since 2001, first with ever popular JGraph Swing library, then in 2006 with the leading edge development of mxGraph.  The libraries are designed to drop in and offer you the complete range of functionality required to add complex diagramming to your application or web site.



We have had good experiences with Flot. It is fast and flexible with low overhead.  Flot is a pure Javascript plotting library for jQuery. It produces graphical plots of arbitrary datasets on-the-fly client-side.  The focus is on simple usage (all settings are optional), attractive looks and interactive features like zooming and mouse tracking.  The plugin works with Internet Explorer 6+, Firefox 2.x+, Safari 3.0+, Opera 9.5+ and Konqueror 4.x+ with the HTML canvas tag (the excanvasJavascript emulation helper is used for IE < 9).



If your into snake charming and use Python then MatPlotLib is for you.  While the installation is not straightforward the flexibility is there from a user perspective.  I see a huge uptake in this library and personally use it.



This library works extremely well for development on the iPhone.  If you need a complete graphing functionality that is cost effective download this lib.

Core Plot is a plotting framework for Mac OS X and iOS. It provides 2D visualization of data, and is tightly integrated with Apple technologies like Core Animation, Core Data, and Cocoa Bindings.



This is a derivative of the InfoViz libs from sencha .  I do not know much about them but it looks very useful.



Written in C++ the framework enables the development of algorithms, visual encodings, interaction techniques, data models, and domain-specific visualizations. One of the goal of Tulip is to facilitates the reuse of components and allows the developers to focus on programming their application. This development pipeline makes the framework efficient for research prototyping as well as the development of end-user applications.  They have libraries for several implementations (Python,Open GL etc).



We tried out ggobi a long time ago.  While it does appear noteworthy we opted not to utilize it on the respective project.  GGobi is an open source visualization program for exploring high-dimensional data. It provides highly dynamic and interactive graphics such as tours, as well as familiar graphics such as the scatterplot, barchart and parallel coordinates plots. Plots are interactive and linked with brushing and identification.



Mondrian is a general purpose statistical data-visualization system. It features outstanding interactive visualization techniques for data of almost any kind, and has particular strengths, compared to other tools, for working with Categorical Data, Geographical Data and LARGE Data.

All plots in Mondrian are fully linked, and offer many interactions and queries. Any case selected in a plot in Mondrian is highlighted in all other plots.

Currently implemented plots comprise Histograms, Boxplots y by x, Scatterplots, Barcharts, Mosaicplots, Missing Value Plots, Parallel Coordinates/Boxplots, SPLOMs and Maps.

Mondrian works with data in standard tab-delimited or comma-separated ASCII files and can load data from R workspaces. There is basic support for working directly on data in Databases (please email for further info).

Mondrian is written in JAVA and is distributed as a native application (wrapper) for MacOS X and Windows. Linux users need to start the jar-file.  It is heavily used and leveraged against ‘R’.


Once again while this is not a comprehensive list it is more than adequate to get your feet wet in the world of visualization and datascience.  One of the most salient things you can do for your self is a little scripting or programming.  Also a little math doesnt hurt either.  So do not be scared of a summation sign.  Dust off those old stat books and get to it then you can start delving into more advanced topics in BigData.

Until Then,

Go Big or Go Home,


Sven To You

Most will consider 9/11  as remembrance of an atrocity that occurred within our nation.  Some cried.  Some died. Some cheered.  I think 9/11 as it is commonly known within the vernacular and as “The Event”, was a pathetic display of religion and government gone awry.  I personally know people who made it out of the towers – they are near and dear to me.  Their stories are gut wrenching.  Running till they puked with bodies flying like rain, splattering on pavement, with an additional hailstorm of aviation gas, fumes and concrete.   Yet in elite fashion the people that I know rebuilt their lives both in terms of monetary success, physical and mental performance.

That said I know September 11 as another day. Over the years since September 11, 2005  I continually think about my comrade at arms: Steven Swenson (aka Sven).   For those that do not know he died in a free diving accident in Costa Rica on this day in 2005.  Many will say that one should “move on” and forget.  Yet I say Time can be crystallized much like a wound that is cauterized with high heat.   Time is a construct –  we can stop time at a moment of our affinity – surrounding yourself with what you enjoy.  Many of you who “follow” my blog, tweets or facebook pages will have a succinct understanding of my views on life.  The elite perform.  There are no free lunches and the aspects of Lex Talionis are alive and well at certain echelons of life for those that love the game within the game.

I am continually asked about proverbial secrets of success and tales of certain times in my life  – especially when a group of us banded together during the hay day of silicon valley (then not now).

Loose lips sink ships and dead pirates tell no tales.  There are many tales that will never be told as half of them went to the watery abyss with Sven.   Given his favorite hobby was free diving his ‘relaxation’ was focused around pushing limits and boundaries.  Most of our copious free hobby time was usually spent in some discussion of philosophy, math,music or making money.  He was the most intense nihilist that loved life and living “the great indulgence” more than anyone I knew.  He continually said Schrodinger cat is eviscerated not just dead.  His computer science skills are now tales of valley folklore.  He read hexadecimal like a Dr. Suess book.  He was also a very well trained martial artist and an obvious waterman.

Yet I will tell a tale that I remember as it were last week.

Late one night we were involved in a marathon session of billiards  on the “purple monster” as we affectionately called my 9 foot purple tiger maple table.

We had a song list in rotation using some software that we created or were testing and laughing how stupid it was to be making money –  making something so stupid.  Yet I digress.  On the playlist came Jimmy Buffet’s “That’s What Living Is To Me” which is based on Following The Equator by Mark Twain,  of which both Sven and I would incessantly  discuss  how Mr. Samuel Clemmons was so beautifully scathing with pen and word.  We turned the song up extremely loud and he was screaming the following stanza:

“Be good and you will be lonesome

Be lonesome and you will be free

Live a lie and you will live to regret it

That’s what living is to me”

So while this isn’t some prurient, lurid or debauched tale, it reflects many salient aspects of why I miss my comrade.  He despised liars and loved life, as well as being extremely responsible.

One should never tell enemies anything, tell your friends everything and assume enemy until proven otherwise.  Sven proved himself a true friend of like mind and for me there are few  – as I prefer quality over quantity.

I recently had supper with his beautiful wife and two handsome sons.  Amazing the older one looked just like him and the younger one was asking me technical programming questions (still a youngster) in the same cadence as SVEN.  They were arguing with each other in the same point-counter point as their dad.  I answered patiently and as best I could intently looking and listening to both of them.

I was attempting to catch that cauterized timeline, that time crystallization and for a brief instance – a flash: my comrade was there.

With that I will leave with you a quote, a Sven assertion and a song dedicated to my comrade:

“With heart and hand I pledge you while I load my gun again, you will never be forgotten or the enemy forgiven, my good comrade.”

~ Dr. A. Lavey

 “Mediocrity is NOT an option!” ~ Steven Swenson

Comrade, by Blood Axis.





I was having a moment of self reflection and truly thought about the essence of humanity.   Without straying into a value or ethics system or some meta-physical looney tunes land,  I have come to realize that the creation of software (besides being the quickest way to make a boat-load of money) is also one of the ultimate expressions of “being human”.

One interesting aspect of software is that it can realize an ephemeral idea into a novel useful process and at the same instance can easily be destroyed with a simple command such as this : rm -f -r {file-name} POOF! – all the ideas are gone!

It is in fact the creation and destruction duality that make this so human.

Humanity basically accomplishes two things: Creation and Destruction.

Software also exhibits a level of self determination and elitism that I must say these days does not exist in many aspects of our society.  The intelligent, fast and strong usually make the mark.  Software is stratifying – that alone is an outlier.  For reference here is a link to a blog that I wrote sometime ago on (taboo subject) Social Darwinism and Software development.

The balance and duality of software creation (and destruction) are not really addressed across the industry.  I recently read Coders at Work.  This is a great tome from some of the best-of-the-best in  the software industry.  The great ones interviewed in this tome continually discussed the creation and destruction aspects – as well as adapting of software.  Also they eloquently discussed getting rid of the tripe or things that did not appear worthy of creation.  Most are bothered by this duality.  Most are bothered by the extreme individualistic nature of software creation/destruction.

No one needs to tell you that you “can” create a piece of software.  Grab Eclipse or IDLE etc and go to it with your idea.  Some of the best software ever written has never been experienced outside the compiler,interpreter,browser, desktop or mobile.

Why?  Brutal Honestly –  because some deemed that most cannot deal with the results of a piece of software.  Just dont eat your own tail.

Until Then,

Go Big Or Go Home!


Enterprise Version X

Posted: May 5, 2011 in Direct Thoughts
Your Enterprise Is Changing

What No EDI?

Yes Germonio this is no longer Kansas.  I am amazed and maybe havent been keeping abreast of what is actually happening in the so called Enterprise world (even though I have been working in it for a while).  Here is my confusion.  If you take the definition of Electronic data interchange (EDI) one will find that it is “It is used to transfer electronic documents or business data from one computer system to another computer system without human intervention.”  So given that (and here is where I may be getting confused) many companies for several disparate reasons are requesting that they want to scale – writ large – these batch processing EDI systems – without changing anything – just by adding a SOA layer on top of the existing RDBMS architecture.  Really?  Now recently I wrote a blog about map reduction of machine learning algorithms and mentioned NoSQL architectures however  Nathan Hurst wrote a great blog and took a page out of Bower’s CAP theorem for a visualization of NoSQL and RDBMS:

Pick Two NOT Three!

Where most of these companies and entities get into problems is not wanting to change the fundamental data model.  Many of these “conversions” or the “wanting to convert” start with understanding the data that presides in these decades old RDBMS or the basic process in general of what is happening within these enterprise systems:

This is Your Enterprise

Basically we are running into a problem where the fundamental issues are:  (1) the data model (2) creation of a set of canonical rules (3) choosing an architecture based on the triangle mentioned above.  Most companies do not want to take stock in understanding from a programmatic perspective what is in the data base.  Most companies keep adding data, stored procedures, key value pairs and columns and call that scale.  Then they realize that 7M, 10M, 50M, 200M people are going to hit this system in a stochastic manner.   We have to hire more analysts, more coders, more people.  That is not scale.  Scale is creating a canonical set of rules, reducing the data model to a core set of attributes and creating machine learning for data hygiene and integrity.  Then naturally most people believe or inquire as to the data accuracy.  Well it is a give and take, humans are not 100% accurate 100% percent of the time.  I would rather have an 80/20 rule and get the data results faster then operate on the focus of 100% correct data 100% of the time.  It is more natural to operate on the the 20% outliers and re-train the system based on those anomalies. Remember the first step is acknowledgement of the actual problem.   As I said in an much earlier blog on adapatibility that homeostasis is the death knell of most companies.  You must build an evolution or revolutionary strategy into your technology roadmap or at least plan with some contingencies.  Thus this brings into question what exactly is enterprise software?

Until Then,

Go Big Or Go Home!


The Machine Is Learning

The past couple of weeks have been rather tumultus for me and several others.  I wont go into details and “kiss and tell” but suffice to say it is has led me to the land of “Free Agency”.

As such over the past couple of weeks I have met several people and have been discussing items such as Big Data, Semantics, Hadoop, NoSQL, Data Science <<insert favorite bingo buzzword here>>. One issue that I see over and over concerns the usage of different distributed compute frameworks. Saying “Hadoop” is not a panacea for your Big Data problems.  One must understand the problem your trying to solve.  Many people fall prey to the “new shiny thing” paradigm.  On the issue of BigData concerns if you want to scale horizontal go with a NoSQL solution – if it is necessary.  The NoSQL database is implemented as a data grid for processing (mapReduce, queries, CRUD, etc.).  Most big websites and players that have moved towards non-relational datastores include LinkedIn, Amazon, Digg and Twitter. One of the main reasons for using NoSQL solutions is relational databases place computation on reads, which is considered wrong for large-scale web applications such as Digg etc.

Aspects of this behavior are:
• The serial nature of applications1 often waiting for I/O from the data store which does no good to scalability and low response times.
• Huge amounts of data and a high growth factor lead Twitter towards facilitating Cassandra, which is designed to operate with large scale data.
• Furthermore, operational costs of running and maintaining systems like Twitter et al escalate. Web applications of this size therefore “need a system that can grow in a more automated fashion and be highly available.”.

Amazon’s James Hamilton agrees:

“Scale-first applications are those that absolutely must scale without bound and being able to do this without restriction is much more important than more features. These applications are exemplified by very high scale web sites such as Facebook, MySpace, Gmail, Yahoo, and Amazon.com. Some of these sites actually do make use of relational databases but many do not. The common theme across all of these services is that scale is more important than features and none of them could possibly run on a single RDBMS.”

Ok so lets assume you have your high availability horizontal scale framework (HAHSF)  in place and you are harvesting, ingressing and egressing data.  Now what?  You must figure out the DataScience strategy and what to do with the data.  Otherwise its like hoarding – your just harvesting and storing it, which by definition means that you need to do some type of statistics, machine learning or data mining.  Here is where I believe most people get slightly sideways.  Map Reduction is not Hadoop.  Map Reducing of Machine Learning algorithms are at the forefront of what is occurring within scale applications.

Machine Learning Review

For a refresher I would suggest breaking out that Grey Book called Machine Learning by Professor Mitchell or Haykin’s, Neural Network tome.  So lets start with something we all probably learned in one of your AI or machine learning classes.  Remember the Back-Propagation Network?  If not here is the diagram for a refresher:

Essentially the system is a linear combination of weights with a trigonometric function that provides a threshold then the error is fed back the the weights are changed.  There are three types of neurons in a neural network that is created BP ANN algorithm:
  • Input neurons

For discrete input attributes, an input neuron typically represents a single state from the input attribute. This includes missing values, if the training data contains nulls for that attribute. A discrete input attribute that has more than two states generates one input neuron for each state, and one input neuron for a missing state, if there are any nulls in the training data. A continuous input attribute generates two input neurons: one neuron for a missing state, and one neuron for the value of the continuous attribute itself. Input neurons provide inputs to one or more hidden neurons.

  • Hidden neurons

Hidden neurons receive inputs from input neurons and provide outputs to output neurons.

  • Output neurons

Output neurons represent predictable attribute values for the data mining model. For discrete input attributes, an output neuron typically represents a single predicted state for a predictable attribute, including missing values. For example, a binary predictable attribute produces one output node that describes a missing or existing state, to indicate whether a value exists for that attribute. A Boolean column that is used as a predictable attribute generates three output neurons: one neuron for a true value, one neuron for a false value, and one neuron for a missing or existing state.

A neuron receives input from other neurons, or from other data, depending on which layer of the network it is in. An input neuron receives inputs from the original data. Hidden neurons and output neurons receive inputs from the output of other neurons in the neural network. Inputs establish relationships between neurons, and the relationships serve as a path of analysis for a specific set of cases.

Each input has a value assigned to it, called the weight, which describes the relevance or importance of that particular input to the hidden neuron or the output neuron. The greater the weight that is assigned to an input, the more relevant or important the value of that input. Weights can be negative, which implies that the input can inhibit, rather than activate, a specific neuron. The value of each input is multiplied by the weight to emphasize the importance of an input for a specific neuron. For negative weights, the effect of multiplying the value by the weight is to deemphasize the importance.

Each neuron has a simple non-linear function assigned to it, called the activation function, which describes the relevance or importance of a particular neuron to that layer of a neural network. Hidden neurons use a hyperbolic tangent function (tanh) for their activation function, whereas output neurons use a sigmoid function for activation. Both functions are nonlinear, continuous functions that allow the neural network to model nonlinear relationships between input and output neurons.

Map Reducing The Algorithm

Now here is where the world of real time systems and distributed systems collide.  Suppose we have a epoch and we have an error propagated back and we need to update the weights?  We need to map reduce this functionality to fully gain benefit from the map reduction – hadoop-it-izing magic.  As a specific note the performance that results depends intimately on the design choices underlying the MapReduce implementation, and how well those choices support the data processing pattern of the respective Machine Learning algorithm.

However, not only does Hadoop not support static reference to shared content across map tasks, as far as I know the implementation also prevented us from adding this feature. In Hadoop, each map task runs in its own Java Virtual Machine (JVM), leaving no access to a shared memory space across multiple map tasks running on the same node. Hadoop does provide some minimal support for sharing parameters in that its streaming mode allows the distribution of a file to all compute nodes (once per machine) that can be referenced by a map task. In this way, data can be shared across map tasks without incurring redundant transfer costs. However, this work-around requires that the map task be written as a separate applications, furthermore, it still requires that each map task load a distinct copy of its parameters into the heap of its JVM.  That said almost all ML learning algorithms and particularly ANN are commonly done with stochastic or direct gradient descent/ascent (depending on sign), which poses a challenge to parallelization.

The problem is that in every step of gradient ascent, the algorithm updates a common set of parameters (e.g. the update matrix for the weights in the BPANN case). When one gradient ascent step (involving one training sample) is updating W , it has to lock down this matrix, read it, compute the gradient, update W , and finally release the lock. This “lock-release” block creates a bottleneck for parallelization; thus, instead of stochastic gradient ascent many methods use a batch method which greatly slows the process down and takes away from the real time aspect.  Work arounds that I have personally seen used and help create are memory mapped shared functions (think signal processing and CPU shared memory modules) which allows a pseudo persistence to occur and the weights to be updated at and paralleled at “epoch time”.   As we reduce the latencies of the networked systems we are starting to get back to some of the very problems that plagued real time parallel signal processing systems – the disc i/o and disc read-write heads are starting to become the bottlneck.  So you move to solid state devices or massive in memory systems.  That does not solve the map reduction problem of machine learning algorithms.  That said over the past years the Apache Foundation and more specifically the Mahout project has been busy adding distributed algorithms to its arsenal.  Of particular interest is the Singular value decomposition which is the basis kernel for processes such as Latent Semantic Indexing and the like.  Even given the breadth of additions fundamental computational blocks such as Kohonen networks and the algorithm mentioned here – Backpropagation are not in the arsenal.  Some good stuff such as an experimental Hidden Markov Model are there.

That said I really do believe we will start to see more of a signal processing architecture on the web.  Just maybe “cycle-stealing” via true grids will make comeback?

Go Big Or Go Home!


The Other PDF To You

The Probability Something Will Happen


Digital Remote for Your Life

Well folks we are going to shift gears here a little and get back to some hardcore Ideas2Bank discussions concerning technologies.  As of late I have been interested once again in Finding-Not-Searching types of behaviors.  Affinity based systems are once again on the rise.  I will go so far as to say the Age of Affinity is here.  TechCrunch did a writeup recently concerning relevance.  At the end it turned into a pitch for Quora.  However it did have some good ideas concerning continuum of Personalization functionality from complete serendipity to exact personalized context aware information constructs based on geo-location.  I have always been a fan of “lean back” technologies.   Technologies that essentially enable a digital remote for your life.  These types of systems have two common themes: 1) ease of use 2) The probability of usage

Predictive Content and Probability

In today’s world we are trying to create predictive, context aware systems based on the wrong models.  For today’s database architecture: 1) An item either is in the database or is not 2) A tuple either is in the query answer or is not.   This applies to all state-of-the-art data models across the board.  In a probablistic database we have a different construct altogether which is a better fit for the flow of content. For a content prediction event driven system we can assume the events are precise and have no ambiguity. Instead, it is the future event stream that is unknown and the matching of a pattern at some point in the future is predicted with some probability.   When, Where and How are the operatives for this type of predictive event, f(Wh,Wr,H) if you will.  Also note I mentioned the word stream.   I believe given the current and future infrastructures for processing we are bringing back some of the same analogies for large array signal processing frameworks.  The probablistic database models set up extremely well for these types of event processing mechanics.

For a probabilistic database we have:

  • “An item belongs to the database” is a probabilistic event.
  • “A tuple is an answer to a query” is a probabilistic event
  • Can be extended to all data models; we discuss only probabilistic relational data

Probabilistic databases distinguish between the logical data model and the physical representation of the data much like relational databases do in the ANSI-SPARC Architecture. In probabilistic databases this is even more crucial since such databases have to represent very large numbers of possible worlds, often exponential in the size of one world (a classical database).  In complex event processing systems, events from the environment are correlated and aggregated to form higher level events. Uncertainty in the events may be due to a variety of factors including imprecision in the event sensors or generators (eg streams), and corruption of the communication channel possibly dropping events, which can be measured with entropy metrics.  These attributes lend themselves well to fusion systems and the social stream architectures.   Given we are looking at heterogenous data sources that set up for collisions and data source integrity these types of databases hold great promise.    In addition many of these types of database architectures build upon Finite State Machine mechanics for event processing in operating systems.  Of further interest the data is usually imprecise.

Probalistic Databases address types of imprecision whereas:

  • Data is precise, query answers are imprecise
  • User has limited understanding of the data
  • User has limited understanding of the schema
  • User has personal preferences

Notice a “trend” here?  This sets up very well for content flow predictions.  In addition these types of systems hold well for principled semantics for complex queries.  This provides context for the queries where the data is usually imprecise.  Data integration and data hygiene are paramount in social stream systems.  Where data accuracy is important most companies spend 85% of workload cleansing data.  We could use probabilistic information to reason about soundness, completeness, and overlap of sources (think linked data here).  I have listed some of the main sources of research in Probabilistic databases herewith.  As far as I know there are no publicly commercial applications as of yet for this technology.  My bet is we will see some very soon integrated with some of the other NoSQL like technologies.

For a list of current research projects see:

Until Then,

Go Big Or Go Home!


Responsibility For The Responsible!

Hello all:

I trust this finds everyone well.  We have all the usual worldwide things going on: disasters, wars etc and then in the valley we have huge funding events happening e.g. Color getting $41M. I was thinking ’bout some lessons lived and learned whilst reflecting on past lives and a couple of items came out of the “wet-ware” subconscious: 1) Do The Right Thing 2) Make Decisions Like Your Paycheck Is Irrelevant. (style note: after all these years of Mac programming and working at Apple I still use object style capitalization).  Whilst these item do not appear to be tech worthy or money making – someone said I should teach a Tech Psychology 101 class.

Do The Right Thing

A very successful person in the tech industry who at this point I consider close to iconoclastic said,  “Just Do The Right Thing.”  I am not going to get into the metaphysical aspects of Right/Wrong here suffice to say what this person was making a comment on was have a clear conscience and do not be biased as to the correct technical and business decision.  In the long run you will benefit and you can always look back and know that what you did as a technical professional was well and just.  This was coming during a very complicated and involved several people with agendas.  For those who are not in the tech industry you would be amazed at how emotional coders, architects and program directors are with respect to some of these decisions.  Then I think that nowadays they are very close to rockstars so what not be emotional?  That said when you have a difficult decision to make “Do The Right Thing.”

Make Decisions Like Your PayCheck Is Irrelevant

Once again someone well known in the industry was walking with me at a conference trade show and due to the level of their success and the amount of well tech power they possessed I used this time to ask questions.  I asked them what is consistent in your success?  Without missing a beat this person said, “Work like your paycheck doesn’t matter.”  Of course your probably saying if this person is that successful they can afford it.  Well dear readers they were not talking about money they were talking about reputation.  Money is temporary.  Reputation lasts a long time – possibly until the big sleep.  It took me a while to truly get my head around this because well time is money and money is time.  Yet the more I pondered this statement the more it made sense (and cents).  Time went on and I started espousing some of these tenets.  Eventually I came upon a situation where I knew one of my colleagues was going into a meeting that could change the face of I/O connectors (think USB and FireWire wars) and I told him say what you know and operate like you do not need a paycheck.  He came out of the meeting and said you know I will always operate in that manner.  It completely freed me from figuring out what to do in high pressure situations.

Responsibility For The Responsible!

First and foremost one thing I do wish is I could have met the person who espoused this statement.  I actually came very close when I was in The Valley circa 93′.  Yet that is a story for another time and place.  The two previous items roll up into this tenets.  For those that are changing the world via technology we have the power to create and destroy.  For some this is the game because well they have made millions and billions, just as the artist who can create to destroy which is part of the art.  Yet even though we create technologies that change the very social fabric of humanity we are still dealing with the human form on a daily basis (caveat emptor for better or worse) and as such we have decisions that will live with us. Conflict is a given.  It is going to happen.  Humans adore war.  Many a people in the tech industry have destroyed many an art – books could be written on the amount of code and product that hit the trash and <delete>.  Part and parcel of this is the aspect of doing what your told instead of being responsible and doing the right thing.  Software is the most scalable industry period.  There may never be another industry as scalable (smart grid maybe but not yet).  The multi-cast nature of changing entire groups of people via one application or a widget is phenomenal not to mention the monetary reward.  As I am fond of saying – Idea2Bank as fast as possible yet this comes with some aspect of Responsibility For The Responsible.  There are those that can and those that cannot its a very stratifying industry.

At the end of the day do what you know – say what you think – create the software you know is true.  Software only knows brutal honesty – either it works or it doesnt and you should be a reflection of that creation.

Until Then,

Go Big Or Go Home!