Harking back to the old rule of development projects of  “two out of three”, when the three metrics are fast, cheap, and quality.  On any time limited project you only can plan to achieve two metrics, never all three. Initially I like to dive in and start with fast and cheap and work my way towards quality as necessary.  So, we’ll start with the quick and dirty approach to calling external libraries from Excel.

Project Setup

You must have a version of .NET Framework of 2.0 or greater.  The latest .NET version is free and easily downloaded from Microsoft.

You’ll also need:

• Excel 97 or later.
• External library assemblies that you need to access from Excel. In our case we will use Centerspace’s NMath.dll and NMathStats.dll.
• A freeware product called ExcelDNA written by Govert van Drimmelen that can be downloaded at Excel-DNA .

The first order of business is to unpack the downloaded file, ExcelDNA.zip, into a working directory.  For our example, we will use CenterSpaceExcel as our directory name. After unpacking you should have two folders Distribution and Source in our CenterSpaceExcel directory.  Inside the Distribution folder locate the file ExcelDNA.xll and rename it to NMathExcel.xll.

We now need to locate in the same directory the file ExcelDna.dna and rename it to NMathExcel.dna.  Then using notepad, or your favorite code editor, and open the file NMathExcel.dna.

You should see the following code:

 _
	    	Function NPower(x as double, y As double) As double
			NPower = NMath.NMathFunctions.PowFunction(x, y)
		End Function

		 _
		Function NRand() As double
			dim rand As New NMath.RandGenMTwist
			NRand = rand.Next53BitRes()
		End Function

		 _
		Function NBinomDist(NSuccess As Int32, NTrials As Int32, Prob As double, Cumul As Boolean) As double
			dim nbin As New Stats.BinomialDistribution
			nbin.N = NTrials
			nbin.P = Prob
			IF Cumul
				NBinomDist = nbin.CDF(NSuccess)
			Else
				NBinomDist = nbin.PDF(NSuccess)
			End If
		End Function

		Function NDoubleMatrixRand(rsize As integer, csize As integer, RandLBx As integer, RandUBy As integer) As Object(,)
			dim rng As New NMath.RandGenUniform(RandLBx,RandUBy)
			Rng.Reset(&H124)
			dim TempA As New NMath.DoubleMatrix(rsize, csize, Rng)
			NDoubleMatrixRand = NCopyArray(TempA, rsize, csize)

		End Function

		Function NCopyArray(IMatrix As Object, rsize As integer, csize As integer) As Object(,)
			dim i As Integer
			dim j As Integer
			dim OArray(rsize, csize) As Object
			for i = 0 to rsize - 1
			   for j = 0 to csize - 1
				OArray(i,j) = IMatrix(i,j)
			   next j
			next i
			NCopyArray = OArray
		End Function		

	End Module
]]>

We now have created the VB code to call our CenterSpace Math and Statistics libraries with the following five functions.

1. The first function shows a simple math library call to the Power function which takes a number x and raises it to the y power and returns the value.
2. The second function shows a call to obtain a fast random number from the math library.  Since we want a new number each time the spreadsheet is re-calculated we have made the function volatile.
3. The third function call shows how to set values that need to be accessed by a function in our .NET assemble; in this case, the Binomial Distribution.
4. The fourth function demonstrates the creation of a DoubleMatrix that is the filled with random uniformly distributed numbers.
5. The fifth function is a helper sub-routine to transfer data across the com interface.

Test our setup in Excel

Open Excel and move your cursor the Tools menu item.  Usually towards the bottom of the drop down menu you will find the selection Add-Ins.  After selecting Add-Ins, you see the pop-up window with the option to select Microsoft supplied Add-ins.  Choose the Browse option and go to the working directory we created at the beginning.  In our case, this will be the CenterSpaceExcel directory.  Next select the Distribution folder and you should see the renamed file: NMathExcel.xll.  Select it and you should now see the following screen.

Make sure NMathExcel is checked and click OK. If you get an error and this point it is probably due to a typo in the DNA file, otherwise you will get the expected new sheet ready for entry.

Select an empty cell and then select from the menu bar Insert then from the pulldown Function.  You should see the following pop-up.

At the bottom of the category pull down you should see our NMathExcel Functions;  Select it and you should have these options.:

If we choose NPower, we will get the next screen,

I arbitrarily typed the value of 3.2 for x and 3.327 for y.  You can see the result of 47.9329301 before selecting OK.

Select OK and Excel will insert the value into the cell.  Select another blank cell and this time choose our NRand() function.  You will notice there is no opportunity to enter values and finish by selecting OK.  At this point you should see a number between 0 and 1 in the cell.  Each time you press F9 (sheet recalc) a new random number will appear.  If we had not made this function volatile the number would not change unless you edit the cell.

To test our Binomial Distribution function, again we will select a new cell and use the insert function option to insert the NBinomDist function with the following values.

At this point we have made successful calls into both of CenterSpace’s NMath and NMath Stats .NET math libraries.

In our fourth example, we will see how Excel handles matrices and look at issues passing array arguments across the COM interface.  Excel 2003 was limited to a maximum of 60,000 cells in an array, but Excel 2007 was expanded to handle 2 million.  Excel has some quirky ways of displaying matrices, and I’ll cover the in’s and out’s of these quirks.

We have written the basic code to set up a function called NDoubleMatrixRand for the purpose of creating a matrix with supplied dimensions and filled with uniform Random numbers over a specified distribution.  We will select another blank cell and again go to insert function and this time choose NDoubleMatrixRand.  Suppose we want to create a 6×6 matrix filled with random numbers between -2 and 2.  Our input will look like the following screen.

Notice the equal sign in the middle right of the above screen is equal to {-0.994818527251482,-0.08

Values inclosed in curly brackets that are separated by commas indicates that an matrix was actually created, but you can see the Formula result is only displaying a partial value due to display size. At this point when you select OK, you will have a cell with a single value.  Here is where the fun begins.  Start at the cell and drag a 6×6 range as shown in the following screen.

Now get your fingers limbered. Here is where it gets a bit obscure – do exactly as follows.

• Press the F2 key.  (pressing F2 may be  optional but is recommended by Excel as the cell leaves the edit mode)
• Press and hold the Ctrl key followed by
• pressing and holding the Shift key followed by
• pressing the Enter key

and presto chango!  You should see a screen like this.

Notice that your cell’s formula is now enclosed in { }, indicating to Excel that the contained formula is an array function.  This is the only way to get matrices displayed.  Also, if you try to edit this cell you will get an error that changes are not allowed.  If  you want to change the dimensions simply reference the values from another cell when you create the function.

The fifth function NCopyArray copies the library matrix across the COM bridge into an Excel array object.  As I stated in the beginning this would be a quick and dirty approach and would leave room for improvement.

Summary

In my next post, I will provide the above code in C# and add more function calls with some matrices with hopefully an improved approach to NCopyArray.  Future posts will include creating a packaged XLL and a more complex example such as curve fitting.

Since time is our most precious asset, being able to quickly access complex math functions with a general purpose tool like Excel should save time and money!

At CenterSpace, we are interested if this blog is helpful and if there is a need for more examples of how our libraries can be accessed by Excel.  Let us know what areas are of interest to you.

Mike  Magee

Thanks and Resources
Also, a special thanks to Govert van Drimmelen for writing a wonderful tool such as ExcelDNA.

The post Calling External .NET Libraries from Excel appeared first on CenterSpace.

With their ongoing partnership, CenterSpace Software and Nevron have teamed up to create some free code examples for creating Shewart charts.

Quality Chart Types

Statistical quality control charts can be generally divided into two categories, those for tracking discrete attribute variables (e.g. a pass/fail test), and those for tracking continuous process variables (e.g. pipe diameter, temperature).

The statistical modeling language, “R”, provides a package (qcc) for creating these and other statistical process control charts. This R package was created by Luca Scrucca and is actively maintained and can be found in CRAN repository.

These two images demonstrate the standard look of the ‘c’ and ‘u’ attribute quality control chart. Some typical chart features include the highlighting of out-of-control data points and time varying upper and lower control limits. The charts generated by the R qcc package have served as our standard for recreating these in the .NET / C# development environment. The real world data used in our examples below was copied from the qcc package so direct comparisons can be made.

Creating a Quality Chart with .NET

To integrate these quality controls charts into a .NET/C# data driven quality monitoring application, we need both a statistical analysis library and a visualization tool that can manage the special chart style demanded by quality control engineers. CenterSpace, in partnership with Nevron, has created an extensible example application to build these types of specialized charts. Once you have these free helper classes, building an attribute u-chart is as simple or simpler than prototyping charts in R.

    public void UChart()
    {

      // u-Chart sample data
      // This data-set was copied from the 'dyedcloth' data set packaged with
      // the R-package qcc by Luca Scrucca
      //
      // Example Data Description
      // In a textile finishing plant, dyed cloth is inspected for the occurrence of
      // defects per 50 square meters.
      // The data on ten rolls of cloth are presented
      //    x number of nonconformities per 50 square meters (inspection units)
      //    samplesize number of inspection units in roll (variable sample size
      DoubleVector x =
        new DoubleVector(14, 12, 20, 11, 7, 10, 21, 16, 19, 23);
      DoubleVector samplesize =
        new DoubleVector(10.0, 8.0, 13.0, 10.0, 9.5, 10.0, 12.0, 10.5, 12.0, 12.5);

      // This builds the statistical information for the drawing the chart.
      IAttributeChartStats stats_u = new Stats_u(x, samplesize);

      // Build and display the Nevron u-Chart visualization
      NevronControlChart.AutoRefresh = true;
      NevronControlChart.Clear();
      AttributeChart cChart =
        new AttributeChart(stats_u, this.NevronControlChart);

    }

This code creates a u-Chart that looks like this below.

For those familiar with the aforementioned R-package qcc, these .NET/C# classes follow the same R naming convention for the particular chart statistics objects, but with an improved object model. So as seen in this example, the u-chart statistics are contained in a class named Stats_u, similar to the R stats.u command. Each of these statistical chart objects implements either an IAttributeChartStats or an IVariableChartStats interface, which is used by the chart generating class (AttributeChart) as seen in the last line of the code above.

Building control charts boils down to three steps using these example classes.

1. Build the necessary data vectors.
2. Build the desired chart’s statistics object,
   e.g. IAttributeChartStats Stats = new Stats_c(DoubleVector data);
3. Show chart using Nevrons .NET chart control,
   e.g. new AttributeChart(IAttributeChartStats Stats, NChartControl Chart)

Free Example Code

The example code now available on github can currently create all four essential attribute quality control charts, as seen below.

To download and run these examples just navigate to our Nevron / CenterSpace github repository and either click on the “Download Source” button in the upper right-hand corner and download either a .zip or .tar file of the project, or just clone the repository. For those unfamiliar with git, git is a source code control system designed specifically for collaborative projects such as this one. To clone the project, after installing git, simply type at your command prompt:

 git clone git@github.com:MilenMetodiev/CenterSpaceNevronExamples.git

This will create a clone of this project code at your current drive location in a directory call “CenterSpaceNevronExamples”.

Other Quality Control Charts and Future Development

Currently we have only implemented the attribute control charts. Other common quality system charts including EWMA (exponential weighted moving average), Pareto, and CumSum (cumulative sum) have not been implemented in this example, but can be using the same tool set and class patterns established in this example. If you would like help or need any assistance in getting the project running or extending this to other chart types, drop us an email .

Happy Computing,
-Paul

Resources

• The table above is adapted from the Wikipedia control chart article.
• “qcc: An R package for quality control charting and statistical process control”, R News, Volume 4/1 , June 2004.
• The standard qcc documentation from the CRAN project was very helpful with this project.

The post Statistical Quality Control Charts appeared first on CenterSpace.

In this post, the last in the series, we’ll look at how NMath provides a Monte Carlo method for performing multiple non-negative matrix factorization (NMF) clusterings using different random starting conditions, and combining the results.

NMF uses an iterative algorithm with random starting values for W and H. (See Part IV for more information on NMF.) This, coupled with the fact that the factorization is not unique, means that if you cluster the columns of V multiple times, you may get different final clusterings. The consensus matrix is a way to average multiple clusterings, to produce a probability estimate that any pair of columns will be clustered together.

To compute the consensus matrix, the columns of V are clustered using NMF n times. Each clustering yields a connectivity matrix. Recall that the connectivity matrix is a symmetric matrix whose i, jth entry is 1 if columns i and j of V are clustered together, and 0 if they are not. The consensus matrix is also a symmetric matrix, whose i, jth entry is formed by taking the average of the i, jth entries of the n connectivity matrices. The i, jth entry of a consensus matrix may be considered a “probability” that columns i and j belong to the same cluster.

NMath Stats provides class NMFConsensusMatrix for computing a consensus matrix. NMFConsensusMatrix is parameterized on the NMF update algorithm to use. Additional constructor parameters specify the matrix to factor, the order k of the NMF factorization (the number of columns in W), and the number of clustering runs. The consensus matrix is computed at construction time, so be aware that this may be an expensive operation.

For example, the following C# code creates a consensus matrix for 100 runs, clustering the scotch data (loaded into a dataframe in Part I) into four clusters:

int k = 4;
int numberOfRuns = 100;
NMFConsensusMatrix<NMFDivergenceUpdate> consensusMatrix =
  new NMFConsensusMatrix<NMFDivergenceUpdate>(
    data.ToDoubleMatrix().Transpose(),
    k,
    numberOfRuns);

Console.WriteLine("{0} runs out of {1} converged.",
  consensusMatrix.NumberOfConvergedRuns, numberOfRuns);

The output is:

100 runs out of 100 converged.

NMFConsensusMatrix provides a standard indexer for getting the element value at a specified row and column in the consensus matrix. For instance, one of the goals of Young et al. was to identify single malts that are particularly good representatives of each cluster. This information could be used, for example, to purchase a representative sampling of scotches. As described in Part IV, they reported that these whiskies were the closest to each flavor profile:

• Glendronach and Macallan
• Tomatin and Speyburn
• AnCnoc and Miltonduff
• Ardbeg and Clynelish

The consensus matrix reveals, however, that the pairings are not equally strong:

Console.WriteLine("Probability that Glendronach is clustered with Macallan = {0}",
  consensusMatrix[data.IndexOfKey("Glendronach"), data.IndexOfKey("Macallan")]);
Console.WriteLine("Probability that Tomatin is clustered with Speyburn = {0}",
  consensusMatrix[data.IndexOfKey("Tomatin"), data.IndexOfKey("Speyburn")]);
Console.WriteLine("Probability that AnCnoc is clustered with Miltonduff = {0}",
  consensusMatrix[data.IndexOfKey("AnCnoc"), data.IndexOfKey("Miltonduff")]);
Console.WriteLine("Probability that Ardbeg is clustered with Clynelish = {0}",
  consensusMatrix[data.IndexOfKey("Ardbeg"), data.IndexOfKey("Clynelish")]);

The output is:

Probability that Glendronach is clustered with Macallan = 1
Probability that Tomatin is clustered with Speyburn = 0.4
Probability that AnCnoc is clustered with Miltonduff = 0.86
Probability that Ardbeg is clustered with Clynelish = 1

Thus, although Glendronach and Macallan are clustered together in all 100 runs, Tomatin and Speyburn are only clustered together 40% of the time.

A consensus matrix, C, can itself be used to cluster objects, by perform a hierarchical cluster analysis using the distance function:

For example, this C# code creates an hierarchical cluster analysis using this distance function, then cuts the tree at the level of four clusters, printing out the cluster members:

DoubleMatrix colNumbers = new DoubleMatrix(consensusMatrix.Order, 1, 0, 1);
string[] names = data.StringRowKeys;

Distance.Function distance =
  delegate(DoubleVector data1, DoubleVector data2)
  {
    int i = (int)data1[0];
    int j = (int)data2[0];
    return 1.0 - consensusMatrix[i, j];
  };

ClusterAnalysis ca = new ClusterAnalysis(colNumbers, distance, Linkage.WardFunction);

int k = 4;
ClusterSet cs = ca.CutTree(k);
for (int clusterNumber = 0; clusterNumber < cs.NumberOfClusters; clusterNumber++)
{
  int[] members = cs.Cluster(clusterNumber);
  Console.Write("Objects in cluster {0}: ", clusterNumber);
  for (int i = 0; i < members.Length; i++)
  {
    Console.Write("{0} ", names[members[i]]);
  }
  Console.WriteLine("\n");
}

The output is:

Objects in cluster 0:
Aberfeldy Auchroisk Balmenach Dailuaine Glendronach
Glendullan Glenfarclas Glenrothes Glenturret Macallan
Mortlach RoyalLochnagar Tomore 

Objects in cluster 1:
Aberlour ArranIsleOf Belvenie BenNevis Benriach Benromach
Bladnoch BlairAthol Bowmore Craigallechie Dalmore
Dalwhinnie Deanston GlenElgin GlenGarioch GlenKeith
GlenOrd Glenkinchie Glenlivet Glenlossie Inchgower
Knochando Linkwood OldFettercairn RoyalBrackla
Speyburn Teaninich Tomatin Tomintoul Tullibardine 

Objects in cluster 2:
AnCnoc Ardmore Auchentoshan Aultmore Benrinnes
Bunnahabhain Cardhu Craigganmore Dufftown Edradour
GlenGrant GlenMoray GlenSpey Glenallachie Glenfiddich
Glengoyne Glenmorangie Loch Lomond Longmorn
Mannochmore Miltonduff Scapa Speyside Strathisla
Strathmill Tamdhu Tamnavulin Tobermory 

Objects in cluster 3:
Ardbeg Balblair Bruichladdich Caol Ila Clynelish
GlenDeveronMacduff GlenScotia Highland Park
Isle of Jura Lagavulin Laphroig Oban OldPulteney
Springbank Talisker

Once again using the cluster assignments to color the objects in the plane of the first two principal components, we can see the grouping represented by the consensus matrix (k=4).

Well, this concludes are tour through the NMath clustering functionality. Techniques such as principal component analysis, k-means clustering, hierarchical cluster analysis, and non-negative matrix factorization can all be applied to data such as these to explore various clusterings. Choosing among these approaches is ultimately a matter of domain knowledge and performance requirements. Is it appropriate to cluster based on distance in the original space, or should dimension reduction be applied? If dimension reduction is used, are negative component parameters meaningful? Are there sufficient computational resource available to construct a complete hierarchical cluster tree, or should a k-means approach be used? If an hierarchical cluster tree is computed, what distance and linkage function should be used? NMath provides a powerful, flexible set of clustering tools for data mining and data analysis.

Ken

References

Young, S.S., Fogel, P., Hawkins, D. M. (unpublished manuscript). “Clustering Scotch Whiskies using Non-Negative Matrix Factorization”. Retrieved December 15, 2009 from http://niss.org/sites/default/files/ScotchWhisky.pdf.

The post Cluster Analysis, Part V: Monte Carlo NMF appeared first on CenterSpace.

In this post, we’ll cluster the scotches using non-negative matrix factorization (NMF). NMF approximately factors a matrix V into two matrices, W and H:

If V in an n x m matrix, then NMF can be used to approximately factor V into an n x r matrix W and an r x m matrix H. Usually r is chosen to be much smaller than either m or n, for dimension reduction. Thus, each column of V is approximated by a linear combination of the columns of W, with the coefficients being the corresponding column H. This extracts underlying features of the data as basis vectors in W, which can then be used for identification, clustering, and compression.

Earlier in this series, we used principal component analysis (PCA) as a means of dimension reduction for the purposes of visualizing the scotch data. NMF differs from PCA in two important respects:

1. NMF enforces the constraint that the factors W and H must be non-negative-that is, all elements must be equal to or greater than zero. By not allowing negative entries in W and H, NMF enables a non-subtractive combination of the parts to form a whole, and in some contexts, more meaningful basis vectors. In the scotch data, for example, what would it mean for a scotch to have a negative value for a flavor charactistic?
2. NMF does not require the basis vectors to be orthogonal. If we are using NMF to extract meaningful underlying components of the data, there is no a priori reason to require the components to be orthogonal.

Let’s begin by reproducing the NMF analysis of the scotch data presented in Young et al.. The authors performed NMF with r=4, to identify four major flavor factors in scotch whiskies, and then asked whether there are single malts that appear to be relatively pure embodiments of these four flavor profiles.

NMath Stats provides class NMFClustering for performing data clustering using iterative nonnegative matrix factorization (NMF), where each iteration step produces a new W and H. At each iteration, each column v of V is placed into a cluster corresponding to the column w of W which has the largest coefficient in H. That is, column v of V is placed in cluster i if the entry hij in H is the largest entry in column hj of H. Results are returned as an adjacency matrix whose i, jth value is 1 if columns i and j of V are in the same cluster, and 0 if they are not. Iteration stops when the clustering of the columns of the matrix V stabilizes.

NMFClustering is parameterized on the NMF update algorithm to use. For instance:

NMFClustering<NMFDivergenceUpdate> nmf =
  new NMFClustering<NMFDivergenceUpdate>();

This specifies the divergence update algorithm, which minimizes a divergence functional related to the Poisson likelihood of generating V from W and H. (For more information, see Brunet, Jean-Philippe et al. , 2004.)

The Factor() method performs the actual iterative factorization. The following C# code clusters the scotch data (loaded into a dataframe in Part I) into four clusters:

int k = 4;

// specify starting conditions (optional)
int seed = 1973;
RandGenUniform rnd = new RandGenUniform(seed);
DoubleMatrix starting_W = new DoubleMatrix(data.Cols, k, rnd);
DoubleMatrix starting_H = new DoubleMatrix(k, data.Rows, rnd);

nmf.Factor(data.ToDoubleMatrix().Transpose(),
           k,
           starting_W,
           starting_H);
Console.WriteLine("Factorization converged in {0} iterations.\n",
                   nmf.Iterations);

There are a couple things to note in this code:

• By default, NMFact uses random starting values for W and H. This, coupled with the fact that the factorization is not unique, means that if you cluster the columns of V multiple times, you may get different final clusterings. In order to reproduce the results in Young et al. the code above specifies a particular random seed for the initial conditions.
• The scotch data needs to be transposed before clustering, since NMFClustering requires each object to be clustered to be a column in the input matrix.

The output is:

Factorization converged in 530 iterations.

We can examine the four flavor factors (columns of W) to see what linear combination of the original flavor characteristics each represents. The following code orders each factor, normalized so the largest value is 1.0, similar to the data shown in Table 1 of Young et al.:

ReproduceTable1(nmf.W, data.ColumnHeaders);

private static void ReproduceTable1(DoubleMatrix W,
  object[] rowKeys)
{
  // normalize
  for (int i = 0; i < W.Cols; i++)
  {
    W[Slice.All, i] /= NMathFunctions.MaxValue(W.Col(i));
  }

  // Create data frame to hold W
  string[] factorNames = GetFactorNames(W.Cols);
  DataFrame df_W = new DataFrame(W, factorNames);
  df_W.SetRowKeys(rowKeys);

  // Print out sorted columns
  for (int i = 0; i < df_W.Cols; i++)
  {
    df_W.SortRows(new int[] { i },
                         new SortingType[] { SortingType.Descending });
    Console.WriteLine(df_W[Slice.All, new Slice(i, 1)]);
    Console.WriteLine();
  }
  Console.WriteLine();
}

The output is:

#	Factor 0
Fruity	1.0000
Floral	0.8681
Sweetness	0.8292
Malty	0.6568
Nutty	0.5855
Body	0.4295
Smoky	0.2805
Honey	0.2395
Spicy	0.0000
Winey	0.0000
Tobacco	0.0000
Medicinal	0.0000 

#	Factor 1
Winey	1.0000
Body	0.6951
Nutty	0.5078
Sweetness	0.4257
Honey	0.3517
Malty	0.3301
Fruity	0.2949
Smoky	0.2631
Spicy	0.0000
Floral	0.0000
Tobacco	0.0000
Medicinal	0.0000 

#	Factor 2
Spicy	1.0000
Honey	0.4885
Sweetness	0.4697
Floral	0.4301
Smoky	0.3508
Malty	0.3492
Body	0.3160
Fruity	0.0036
Nutty	0.0000
Winey	0.0000
Tobacco	0.0000
Medicinal	0.0000 

#	Factor 3
Medicinal	1.0000
Smoky	0.8816
Body	0.7873
Spicy	0.3936
Sweetness	0.3375
Malty	0.3069
Nutty	0.2983
Fruity	0.2441
Tobacco	0.2128
Floral	0.0000
Winey	0.0000
Honey	0.0000

Thus:

• Factor 0 contains Fruity, Floral, and Sweetness flavors.
• Factor 1 emphasizes the Winey flavor.
• Factor 2 contains Spicy and Honey flavors.
• Factor 3 contains Medicinal and Smokey flavors.

The objects are placed into clusters corresponding to the column of W which has the largest coefficient in H. The following C# code prints out the contents of each cluster, ordered by largest coefficient, after normalizing so the sum of each component is 1.0:

ReproduceTable2(nmf.H, data.RowKeys, nmf.ClusterSet);

private static void ReproduceTable2(DoubleMatrix H, object[] rowKeys, ClusterSet cs)
{
  // normalize
  for (int i = 0; i < H.Rows; i++)
  {
    H[i, Slice.All] /= NMathFunctions.Sum(H.Row(i));
  }

  // Create data frame to hold H
  string[] factorNames = GetFactorNames(H.Rows);
  DataFrame df_H = new DataFrame(H.Transpose(), factorNames);
  df_H.SetRowKeys(rowKeys);

  // Print information on each cluster
  for (int clusterNumber = 0; clusterNumber < cs.NumberOfClusters; clusterNumber++)
  {
    int[] members = cs.Cluster(clusterNumber);
    int factor = NMathFunctions.MaxIndex(H.Col(members[0]));
    Console.WriteLine("Cluster {0} ordered by {1}: ", clusterNumber, factorNames[factor]);

    DataFrame cluster = df_H[new Subset(members), Slice.All];
    cluster.SortRows(new int[] { factor }, new SortingType[] { SortingType.Descending });

    Console.WriteLine(cluster);
    Console.WriteLine();
  }
}

The output is:

Cluster 0 ordered by Factor 1:
#	        Factor 0	Factor 1	Factor 2	Factor 3
Glendronach	0.0000	0.0567	0.0075	0.0000
Macallan	0.0085	0.0469	0.0083	0.0000
Balmenach	0.0068	0.0395	0.0123	0.0000
Dailuaine	0.0070	0.0317	0.0164	0.0000
Mortlach	0.0060	0.0316	0.0240	0.0000
Tomore	        0.0000	0.0308	0.0000	0.0000
RoyalLochnagar	0.0104	0.0287	0.0164	0.0000
Glenrothes	0.0054	0.0280	0.0081	0.0000
Glenfarclas	0.0127	0.0279	0.0164	0.0000
Auchroisk	0.0103	0.0267	0.0099	0.0000
Aberfeldy	0.0125	0.0238	0.0117	0.0000
Strathisla	0.0162	0.0229	0.0151	0.0000
Glendullan	0.0140	0.0228	0.0102	0.0000
BlairAthol	0.0111	0.0211	0.0166	0.0000
Dalmore	        0.0088	0.0208	0.0114	0.0204
Ardmore	        0.0104	0.0182	0.0118	0.0000 

Cluster 1 ordered by Factor 2:
#	        Factor 0	Factor 1	Factor 2	Factor 3
Tomatin	        0.0000	0.0170	0.0306	0.0000
Aberlour	0.0136	0.0260	0.0282	0.0000
Belvenie	0.0087	0.0123	0.0262	0.0000
GlenGarioch	0.0079	0.0086	0.0252	0.0000
Speyburn	0.0115	0.0000	0.0244	0.0000
BenNevis	0.0202	0.0000	0.0242	0.0000
Bowmore	        0.0049	0.0109	0.0225	0.0186
Inchgower	0.0104	0.0000	0.0218	0.0118
Craigallechie	0.0131	0.0098	0.0216	0.0136
Tomintoul	0.0085	0.0083	0.0214	0.0000
Benriach	0.0150	0.0000	0.0214	0.0000
Glenlivet	0.0125	0.0176	0.0205	0.0000
Glenturret	0.0080	0.0228	0.0203	0.0000
Benromach	0.0132	0.0140	0.0198	0.0000
Glenkinchie	0.0112	0.0000	0.0190	0.0000
OldFettercairn	0.0068	0.0137	0.0182	0.0160
Knochando	0.0131	0.0133	0.0179	0.0000
GlenOrd	        0.0118	0.0128	0.0175	0.0000
Glenlossie	0.0143	0.0000	0.0167	0.0000
GlenDeveronMacduff	0.0000	0.0156	0.0158	0.0216
GlenKeith	0.0108	0.0146	0.0145	0.0000
ArranIsleOf	0.0073	0.0086	0.0127	0.0125
GlenSpey	0.0086	0.0091	0.0119	0.0000 

Cluster 2 ordered by Factor 0:
#	        Factor 0	Factor 1	Factor 2	Factor 3
AnCnoc	        0.0294	0.0000	0.0000	0.0000
Miltonduff	0.0242	0.0000	0.0000	0.0000
Aultmore	0.0242	0.0000	0.0000	0.0000
Longmorn	0.0214	0.0141	0.0089	0.0000
Cardhu	        0.0204	0.0000	0.0094	0.0000
Auchentoshan	0.0203	0.0000	0.0065	0.0000
Strathmill	0.0203	0.0000	0.0125	0.0000
Edradour	0.0195	0.0172	0.0092	0.0000
Tobermory	0.0190	0.0000	0.0000	0.0000
Glenfiddich	0.0190	0.0000	0.0000	0.0000
Tamnavulin	0.0189	0.0000	0.0148	0.0000
Dufftown	0.0189	0.0000	0.0000	0.0147
Craigganmore	0.0184	0.0000	0.0030	0.0254
Speyside	0.0182	0.0138	0.0000	0.0000
Glenallachie	0.0178	0.0000	0.0108	0.0000
Dalwhinnie	0.0174	0.0000	0.0172	0.0000
GlenMoray	0.0174	0.0079	0.0157	0.0000
Tamdhu	        0.0172	0.0124	0.0000	0.0000
Glengoyne	0.0170	0.0090	0.0065	0.0000
Benrinnes	0.0158	0.0196	0.0161	0.0000
GlenElgin	0.0155	0.0107	0.0133	0.0000
Bunnahabhain	0.0148	0.0075	0.0078	0.0110
Glenmorangie	0.0143	0.0000	0.0123	0.0166
Scapa	        0.0140	0.0128	0.0089	0.0127
Bladnoch	0.0137	0.0063	0.0088	0.0000
Linkwood	0.0129	0.0165	0.0092	0.0000
Mannochmore	0.0124	0.0126	0.0081	0.0000
GlenGrant	0.0122	0.0121	0.0000	0.0000
Deanston	0.0119	0.0151	0.0122	0.0000
Loch Lomond	0.0105	0.0000	0.0094	0.0130
Tullibardine	0.0099	0.0093	0.0098	0.0138 

Cluster 3 ordered by Factor 3:
#	        Factor 0	Factor 1	Factor 2	Factor 3
Ardbeg	        0.0000	0.0000	0.0000	0.0906
Clynelish	0.0001	0.0000	0.0000	0.0855
Lagavulin	0.0000	0.0138	0.0000	0.0740
Laphroig	0.0000	0.0082	0.0000	0.0731
Talisker	0.0030	0.0000	0.0129	0.0706
Caol Ila	0.0048	0.0000	0.0019	0.0694
Oban	        0.0067	0.0000	0.0008	0.0564
OldPulteney	0.0114	0.0073	0.0000	0.0429
Isle of Jura	0.0079	0.0000	0.0059	0.0352
Balblair	0.0125	0.0000	0.0074	0.0297
Springbank	0.0000	0.0142	0.0189	0.0282
RoyalBrackla	0.0122	0.0078	0.0135	0.0276
GlenScotia	0.0096	0.0144	0.0000	0.0275
Bruichladdich	0.0100	0.0098	0.0140	0.0249
Teaninich	0.0081	0.0000	0.0111	0.0216
Highland Park	0.0050	0.0145	0.0146	0.0211

These data are very similar to those shown in Table 2 in Young et al. According to their analysis, the most representative malts in each cluster are:

• Glendronach and Macallan
• Tomatin and Speyburn
• AnCnoc and Miltonduff
• Ardbeg and Clynelish

As you can see, these scotches are at, or very near, the top of each ordered cluster in the output above.

Finally, it is interesting to view the clusters found by NMF in the same plane of the first two principal components that we have looked at previously.

If you compare this plot to that produced by k-means clustering or hierarchical cluster analysis, you can see how different the results are. We are no longer clustering based on “similarity” in the original 12-dimensional flavor space (of which this is a view). Instead, we’ve used a reduced set of synthetic dimensions which capture underlying features in the data.

In order to produce results similar to those of Young et al. we explicitly specified a random seed to the NMF process. With different seeds, somewhat different final clusterings can occur. In the final post in this series, we’ll look at how NMath provides a Monte Carlo method for performing multiple NMF clusterings using different random starting conditions, and combining the results.

Ken

References

Brunet, Jean-Philippe et al. (2004). “Metagenes and Molecular Pattern Discovery Using Matrix Factorization”, Proceedings of the National Academy of Sciences 101, no. 12 (March 23, 2004): 4164-4169.

Young, S.S., Fogel, P., Hawkins, D. M. (unpublished manuscript). “Clustering Scotch Whiskies using Non-Negative Matrix Factorization”. Retrieved December 15, 2009 from http://niss.org/sites/default/files/ScotchWhisky.pdf.

The post Cluster Analysis, Part IV: Non-negative Matrix Factorization (NMF) appeared first on CenterSpace.

In this post, we’ll cluster the scotches based on “similarity” in the original 12-dimensional flavor space using hierarchical cluster analysis. In hierarchical cluster analysis, each object is initially assigned to its own singleton cluster. The analysis then proceeds iteratively, at each stage joining the two most “similar” clusters into a new cluster, continuing until there is one overall cluster. In NMath Stats, class ClusterAnalysis performs hierarchical cluster analyses.

The clustering process is governed by two functions:

• A distance function computes the distance between individual objects. In NMath Stats, the distance function is encapsulated in a Distance.Function delegate, which takes two vectors and returns a measure of the distance (similarity) between them. Delegates are provided as static variables on class Distance for many common distance functions. You can also define your own delegate.
• A linkage function computes the distance between clusters. In NMath Stats, the linkage function is encapsulated in a Linkage.Function delegate. When two groups P and Q are united, a linkage function computes the distance between the new combined group P + Q and another group R. Delegates are provided as static variables on class Linkage for many common linkage functions. Again, you can also define your own delegate.

Based on the choice of distance and linkage function, radically different clustering can often result. Ultimately, background knowledge of the domain is required to choose between them.

In this case, we’ll use the Euclidean distance function and the Ward linkage function. The Ward linkage function computes the distance between two clusters using Ward’s method, which tends to produce compact groups of well-distributed size. Ward’s method uses an analysis of variance approach to evaluate the distances between clusters. The smaller the increase in the total within-group sum of squares as a result of joining two clusters, the closer they are. The within-group sum of squares of a cluster is defined as the sum of the squares of the distance between all objects in the cluster and the centroid of the cluster.

This code clusters the scotch data (loaded into a DataFrame in Part I), then cuts the hierarchical cluster tree at the level of four clusters:

ClusterAnalysis ca = new ClusterAnalysis(df,
    Distance.EuclideanDistance,
    Linkage.WardFunction
);
ClusterSet cs = ca.CutTree(4);

Printing out the cluster members from the cluster set, as shown in Part II, produces:

Objects in cluster 0:
Aberfeldy Aberlour Ardmore Auchroisk Balmenach Belvenie BenNevis
Benriach Benrinnes Benromach BlairAthol Dailuaine Dalmore
Deanston Edradour GlenElgin GlenKeith GlenOrd Glendronach
Glendullan Glenfarclas Glenlivet Glenrothes Glenturret Knochando
Linkwood Longmorn Macallan Mortlach OldFettercairn RoyalBrackla
RoyalLochnagar Strathisla Tullibardine 

Objects in cluster 1:
AnCnoc ArranIsleOf Auchentoshan Aultmore Bladnoch Bunnahabhain
Cardhu Craigallechie Dalwhinnie Dufftown GlenDeveronMacduff
GlenGrant GlenMoray GlenSpey Glenallachie Glenfiddich Glengoyne
Glenkinchie Glenlossie Glenmorangie Inchgower Loch Lomond
Mannochmore Miltonduff Scapa Speyburn Speyside Tamdhu Tobermory
Tomintoul Tomore 

Objects in cluster 2:
Ardbeg Caol Ila Clynelish Lagavulin Laphroig Talisker 

Objects in cluster 3:
Balblair Bowmore Bruichladdich Craigganmore GlenGarioch
GlenScotia Highland Park Isle of Jura Oban OldPulteney
Springbank Strathmill Tamnavulin Teaninich Tomatin

Coloring the objects based on cluster assignment in the plot of the first two principal components shows how similar this clustering is to the results of k-mean clustering.

Again, remember that although we’ve used dimension reduction (principal component analysis, in this case) to visualize the clustering, the clustering itself was performed based on similarity in the original 12-dimensional flavor space, not based on distance in this plane.

Because we have the entire hierarchical cluster tree, we can cut the tree at different levels. For example, into six clusters:

Objects in cluster 0:
Aberfeldy Aberlour Ardmore Auchroisk Belvenie BenNevis Benriach
Benrinnes Benromach BlairAthol Deanston Edradour GlenElgin
GlenKeith GlenOrd Glendullan Glenfarclas Glenlivet Glenrothes
Glenturret Knochando Linkwood Longmorn OldFettercairn
RoyalBrackla Strathisla Tullibardine 

Objects in cluster 1:
AnCnoc ArranIsleOf Auchentoshan Aultmore Bladnoch Bunnahabhain
Cardhu Craigallechie Dalwhinnie Dufftown GlenDeveronMacduff
GlenGrant GlenMoray GlenSpey Glenallachie Glenfiddich Glengoyne
Glenkinchie Glenlossie Glenmorangie Inchgower Loch Lomond
Mannochmore Miltonduff Scapa Speyburn Speyside Tamdhu
Tobermory Tomintoul Tomore 

Objects in cluster 2:
Ardbeg Caol Ila Clynelish Lagavulin Laphroig Talisker 

Objects in cluster 3:
Balblair Craigganmore GlenGarioch Oban Strathmill Tamnavulin
Teaninich 

Objects in cluster 4:
Balmenach Dailuaine Dalmore Glendronach Macallan Mortlach
RoyalLochnagar 

Objects in cluster 5:
Bowmore Bruichladdich GlenScotia Highland Park Isle of Jura
OldPulteney Springbank Tomatin

Coloring the objects based on cluster assignment in the plot of the first two principal components:

Clusters (1, 2) are unchanged, while clusters (0, 3) are now split into sub-clusters.

Both of the clustering techniques we’ve looked at so far–k-means and hierarchical cluster analysis–have clustered the scotches based on similarity in the original 12-dimensional flavor space. In the next post, we look at clustering using non-negative matrix factorization (NMF), which can be used to cluster the objects using a reduced set of synthetic dimensions.

Ken

The post Clustering Analysis, Part III: Hierarchical Cluster Analysis appeared first on CenterSpace.

In this post, we’ll cluster the scotches based on “similarity” in the original 12-dimensional flavor space using k-means clustering. The k-means clustering method assigns data points into k groups such that the sum of squares from points to the computed cluster centers is minimized. In NMath Stats, class KMeansClustering performs k-means clustering.

The algorithm used is that of Hartigan and Wong (1979):

• For each point, move it to another cluster if that would lower the sum of squares from points to the computed cluster centers.
• If a point is moved, immediately update the cluster centers of the two affected clusters.
• Repeat until no points are moved, or the specified maximum number of iterations is reached.

A KMeansClustering instance is constructed from a matrix or a dataframe containing numeric data. Each row in the data set represents an object to be clustered. The Cluster() method clusters the data into the specified number of clusters. The method accepts either k, the number of clusters, or a matrix of initial cluster centers:

• If k is given, a set of distinct rows in the data matrix are chosen as the initial centers using the algorithm specified by a KMeanClustering.Start enumerated value. By default, rows are chosen at random.
• If a matrix of initial cluster centers is given, k is inferred from the number of rows.

For example, this C# code clusters the scotch data (loaded into a dataframe in Part I) into four clusters:

KMeansClustering km = new KMeansClustering(df);
int k = 4;
km.Cluster(k, KMeansClustering.Start.QuickCluster);

K-means clustering requires a set of starting cluster centers to initiate the iterative algorithm. By default, rows are chosen at random from the given data. Here we employ the QuickCluster algorithm, similar to the SPSS QuickCluster function, for choosing the starting centers. (The QuickCluster algorithm proceeeds as follows: If the distance between row r and its closest center is greater than the distance between the two closest centers (m, n), then r replaces m or n, whichever is closest to r. Otherwise, if the distance between row r and its closest center (q) is greater than the distance between q and its closest center, then row r replaces q.)

The Clusters property returns a ClusterSet object, which represents a collection of objects assigned to a finite number of clusters. The following C# code prints out the members of each cluster:

ClusterSet cs = km.Clusters;
for (int i = 0; i < cs.NumberOfClusters; i++)
{
     Console.WriteLine("Cluster {0} contains:", i);
     int[] members = cs.Cluster(i);
     for (int j = 0; j < members.Length; j++)
     {
          Console.Write("{0} ", df.RowKeys[members[j]]);
     }
     Console.WriteLine("\n");
}

The output looks like this:

Cluster 0 contains:
Aberfeldy Aberlour Ardmore Auchroisk Balmenach Belvenie
BenNevis Benriach Benrinnes Benromach BlairAthol Craigallechie
Dailuaine Dalmore Deanston Edradour GlenKeith GlenOrd
Glendronach Glendullan Glenfarclas Glenlivet Glenrothes
Glenturret Knochando Linkwood Longmorn Macallan Mortlach
OldFettercairn RoyalLochnagar Scapa Strathisla 

Cluster 1 contains:
Ardbeg Caol Ila Clynelish Lagavulin Laphroig Talisker 

Cluster 2 contains:
AnCnoc Auchentoshan Aultmore Bladnoch Bunnahabhain
Cardhu Craigganmore Dalwhinnie Dufftown GlenElgin GlenGrant
GlenMoray GlenSpey Glenallachie Glenfiddich Glengoyne
Glenkinchie Glenlossie Inchgower Loch Lomond Mannochmore
Miltonduff Speyburn Speyside Strathmill Tamdhu Tamnavulin
Tobermory Tomintoul Tomore Tullibardine 

Cluster 3 contains:
ArranIsleOf Balblair Bowmore Bruichladdich GlenDeveronMacduff
GlenGarioch GlenScotia Glenmorangie Highland Park Isle of Jura
Oban OldPulteney RoyalBrackla Springbank Teaninich Tomatin

To help visualize these clusters, we can once again plot the scotches in the plane formed by the first two principal components (see Part I), which collectively account for ~50% of the variance, coloring the points based on cluster assignment.

Remember that although we’ve used dimension reduction (principal component analysis, in this case) to visualize the clustering, the clustering itself was performed based on similarity in the original 12-dimensional flavor space, not based on distance in this plane. Nonetheless, the clusters look pretty reasonable.

K-means clustering is very efficient for large data sets, but does require you to know the number of clusters, k, in advance. Also, you have no control of the similarity metric used to cluster the objects (within-cluster sum of squares). In the next post in this series, we’ll look at hierarchical cluster analysis, which constructs the entire hierarchical cluster tree, and allows you to specify the distance and linkage functions to use.

Ken

References

Hartigan, J.A. and Wong, M.A. (1979). A k-means clustering algorithm. Algorithm AS136, Appl. Stat. 28, pp. 100–108.

The post Clustering Analysis, Part II: K-Means Clustering appeared first on CenterSpace.

Cluster analysis is the assignment of a set of objects into one or more clusters based on object similarity. NMath Stats includes a variety of techniques for performing cluster analysis, which we will explore in a series of posts.

The Data Set

The data set we’ll use was created by David Wishart (2002), who classified 89 single malt scotch whiskies on a five-point scale (0-4) for 12 flavor characteristics: Body, Sweetness, Smoky, Medicinal, Tobacco, Honey, Spicy, Winey, Nutty, Malty, Fruity, Floral. Wishart provides clusterings of the whiskies into 4, 6, and 10 clusters. Young et al. (unpublished manuscript) demonstrate a further clustering into 4 clusters using non-negative matrix factorization (NMF). Both the Young et al. paper and the original data set are available here.

Visualization

To visualize the data set and clusterings, we’ll make use of the free Microsoft Chart Controls for .NET, which provide a basic set of charts. NMath is also available as a bundle with the Syncfusion Essential Studio and Nevron Chart for .NET at a substantial discount. NMath easily interoperates with most charting packages.

Getting Started

To begin, let’s load the data set into a CenterSpace.NMath.DataFrame object:

DataFrame df =
  DataFrame.Load("ScotchWhisky01.txt", true, true, ",", true);

The parameters to the Load() method specify:

• the filename containing the data
• whether the data in the file contains column headers
• whether the data in the file contains row keys
• the column delimiter
• whether to parse the column types, or treat everything as string data.

The data set includes a leading column of row ids. Let’s replace these keys with the distillery names, then remove the distillery column from the data frame:

df.SetRowKeys(data[0]);
df.RowKeyHeader = data.ColumnHeaders[0];
df.RemoveColumn(0);

The data frame now looks like this:

Distillery     Body Sweetness Smoky Medicinal Tobacco Honey Spicy Winey Nutty Malty Fruity Floral
Aberfeldy      2 2 2 0 0 2 1 2 2 2 2 2
Aberlour       3 3 1 0 0 4 3 2 2 3 3 2
AnCnoc         1 3 2 0 0 2 0 0 2 2 3 2
Ardbeg         4 1 4 4 0 0 2 0 1 2 1 0
Ardmore        2 2 2 0 0 1 1 1 2 3 1 1
ArranIsleOf    2 3 1 1 0 1 1 1 0 1 1 2
...

There are 89 rows representing each scotch, and 12 columns representing the score on each flavor characteristic.

Principal Component Analysis

Each whisky is representing as a point in a 12-dimensional flavor space. Principal component analysis (PCA) finds a smaller set of synthetic variables that capture the maximum variance in an original data set. The first principal component accounts for as much of the variability in the data as possible, and each succeeding orthogonal component accounts for as much of the remaining variability as possible. In NMath Stats, classes DoublePCA and FloatPCA perform principal component analyses. (For more information on PCA in NMath Stats, see this page.)

For example, the following C# code constructs a PCA from the whisky data set, then prints the proportion of the variance accounted for by each principal component:

DoublePCA pca = new DoublePCA(df);
Console.WriteLine("Variance Proportions = " +
  pca.VarianceProportions);
Console.WriteLine("Cumulative Variance Proportions = " +
  pca.CumulativeVarianceProportions);

The output looks like this:

Variance Proportions =
[ 0.301109794401424 0.192178864989234 0.0956019274277357
  0.0825032185621017  0.0723086445838344 0.0599231013596576
  0.0510808855222438 0.0458706422880217  0.0349809707532734
  0.0319772808383918 0.0229738209669541 0.00949084830712784 ]

Cumulative Variance Proportions =
[ 0.301109794401424 0.493288659390658 0.588890586818394
  0.671393805380496 0.74370244996433 0.803625551323988
  0.854706436846231 0.900577079134253 0.935558049887526
  0.967535330725918 0.990509151692872 1 ]

To visualize this information, we can construct a Scree plot, a simple line chart that shows the fraction of total variance in the data as explained by each principal component.

As you can see, the first two principal components account for ~50% of the variance.

The Scores property on DoublePCA gets the score matrix. The scores are the data formed by transforming the original data into the space of the principal components. For example, here we create a view of the original 12-dimensional data by plotting the first two principal components for each scotch against each other.

The synthetic dimensions themselves are not particularly meaningful. Essentially we’ve fit a plane into the original 12-dimensional flavor space which accounts for as much of the variance as possible. This can help reveal any natural clustering. In the whisky data, however, there does not appear to be any strong natural clusters–perhaps a group of outliers at the bottom of the plot, and another group at the right. Of course, the original flavor characteristics were chosen precisely to avoid any dramatic clustering.

In future posts, we’ll apply functions in NMath Stats for k-mean clustering, hierarchical cluster analysis, and non-negative matrix factorization to explore clusterings in the data.

Ken

References

Wishart, D. (2002). Whisky Classified, Choosing Single Malts by Flavor. Pavilon, London.

Young, S.S., Fogel, P., Hawkins, D. M. (unpublished manuscript). “Clustering Scotch Whiskies using Non-Negative Matrix Factorization”. Retrieved December 15, 2009 from http://niss.org/sites/default/files/ScotchWhisky.pdf.

The post Clustering Analysis, Part I: Principal Component Analysis (PCA) appeared first on CenterSpace.