• Noise removal
• Baseline adjustment
• Peak finding
• Peak modeling
• Peak statistical analysis

In this blog article we will discuss each of these activities and provide some NMath C# code on how they may be accomplished. This is big subject but the goal here is to get you started solving your spectographic data analysis problems, perhaps introduce you to a new technique, and finally to provide some helpful code snippets that can be expanded upon.

Throughout this article we will be using the following electrophoretic data set below in our code examples. This data set contains four obvious peaks and one partially convolved peak infilled with underlying white noise.

Noise Removal

Chromatographic, spectographic, fMRI or EEG data, and many other types of time series are non-stationary. This non-stationarity means that Fourier based filtering methods are ill suited to removing noise from these signals. Fortunately we can effectively apply wavelet analysis, which does not depend on signal periodicity, to suppress the signal noise without altering the signal’s phase or magnitude. Briefly, the discrete wavelet transform (DWT) can be used to recursively decompose the signal successively into details and approximations components. From a filtering perspective the signal details contain the higher frequency parts and the approximations contain the lower frequency components. As you’d expect the inverse DWT can elegantly reconstruct the original signal but, to meet our noise removal goals, the higher frequency noisy parts of the signal can be suppressed during the signal reconstruction and be effectively removed. This technique is called wavelet shrinkage and is described in more detail in an earlier blog article with references.

These results can be refined, but this starting point is seen to have successfully removed the noise but not altered the position or general shape of the peaks. Choosing the right wavelet for wavelet shrinkage is done empirically with a representative data set at hand.

public DoubleVector SuppressNoise( DoubleVector DataSet  )  
{
  var wavelet = new DoubleWavelet( Wavelet.Wavelets.D4 );
  var dwt = new DoubleDWT( DataSet.ToArray(), wavelet );
  dwt.Decompose( 5 );
  double lambdaU = dwt.ComputeThreshold( FloatDWT.ThresholdMethod.Sure, 1 );

  dwt.ThresholdAllLevels( FloatDWT.ThresholdPolicy.Soft, new double[] { lambdaU, lambdaU, lambdaU, lambdaU, lambdaU } );

  double[] reconstructedData = dwt.Reconstruct();
  var filteredData= new DoubleVector( reconstructedData );
  return filteredData;
}

With our example data set a Daubechies 4 wavelet worked well for noise removal. Note that the same threshold was applied to all DWT decomposition levels; Improved white noise suppression can be realized by adopting other thresholding strategies.

Baseline Adjustment

Dozens of methods have been developed for modeling and removing a baseline from various types of spectra data. The R package baseline has collected together a range of these techniques and can serve as a good starting point for exploration. The techniques variously use regression, iterative erosion and dilation, spectral filtering, convex hulls, or partitioning and create baseline models of lines, polynomials, or more complex curves that can then be subtracted from the raw data. (Another R package, MALDIquant, contains several more useful baseline removal techniques.) Due to the wide variety of baseline removal techniques and the lack of standards across datasets, NMath does not natively offer any baseline removal algorithms.

Example baseline modeling

The C# example baseline modeling code below uses z-scores and iterative peak suppression to create a polynomial model of the baseline. Peaks that extend beyond 1.5 z-scores are iteratively cut down by a quarter and then a polynomial is fitted to this modified data set. Once the baseline polynomial fits well and stops improving upon iterative suppression the model is returned.

private PolynomialLeastSquares findBaseLine( DoubleVector x, DoubleVector y, int PolynomialDegree )
 {
   var lsFit = new PolynomialLeastSquares( PolynomialDegree, x, y );
   var previousRSoS = 1.0;

   while ( lsFit.LeastSquaresSolution.ResidualSumOfSquares > 0.1 && Math.Abs( previousRSoS - lsFit.LeastSquaresSolution.ResidualSumOfSquares ) > 0.00001 )
   {
     // compute the Z-scores of the residues and erode data beyond 1.5 stds.
     var residues = lsFit.LeastSquaresSolution.Residuals;
     var Zscores = ( residues - NMathFunctions.Mean( residues ) ) / Math.Sqrt( NMathFunctions.Variance( residues ) );
     previousRSoS = lsFit.LeastSquaresSolution.ResidualSumOfSquares;

     y[0] = Zscores[0] > 1.5 ? 0 : y[0];
     for ( int i = 1; i < this.Length; i++ )
     {
       if ( Zscores[i] > 1.5 )
       {
         y[i] = y[i-1] / 4.0;
       }
     }
     lsFit = new PolynomialLeastSquares( PolynomialDegree, x, y );
    }
    return lsFit;
 }

This algorithm has proven reliable for estimating both 1 and 2 degree polynomial baselines with electrophoretic data sets. It is not designed to model the wandering baselines sometimes found in mass spec data. The SNIP [2] method or Asymmetric Least Squares Smoothing [1] would be better suited for those data sets.

Peak Finding

Locating peaks in a data set usually involves, at some level, finding the zero crossings of the first derivative of the signal. However, directly differentiating a signal amplifies noise and so more sophisticated indirect methods are usually employed. Savitzky-Golay polynomials are commonly used to provide high quality smoothed derivatives of a noisy signal and are widely employed with chromatographic and other data sets (See this blog article for more details).

// Code snippet for locating peaks.
var sgFilter = new SavitzkyGolayFilter( 4, 4, 2 );
DoubleVector filteredData = sgFilter.Filter( DataSet );
var rbPeakFinder = new PeakFinderRuleBased( filteredData );
rbPeakFinder.AddRule( PeakFinderRuleBased.Rules.MinHeight, 0.005 );
List<int> pkIndicies = rbPeakFinder.LocatePeakIndices();

Without thresholding many small noisy undulations are returned as peaks. Thresholding with this data set works well in separating the data peaks from the noise, however sometimes peak modeling is necessary for separating data peaks from noise when they are both present at similar scales.

Peak Modeling and Statistics

In addition to separating out false peaks, peaks are also modeled to compute various peak statistical measures such as FWHM, CV, area, or standard deviation. The Gaussian is an excellent place to start for peak modeling and for many applications this model is sufficient. However there are many other peak models including: Lorenzian, Voigt, [3] CSR, and variations on exponentially modified Gaussians (EMG’s). Many combinations, convolutions, and refinements of these models are gathered together and presented in a useful paper by DeMarko & Bombi, 2001. Their paper focused on chromatographic peaks but the models surveyed therein have wide application.

/// <summary>
/// Gaussian Func<> for trust region fitter.
/// p[0] = mean, p[1] = sigma, p[2] = baseline offset
/// </summary>
private static Func<DoubleVector, double, double> Gaussian = delegate ( DoubleVector p, double x )
{
   double a = ( 1.0 / ( p[1] * Math.Sqrt( 2.0 * Math.PI ) ) );
   return a * Math.Exp( -1 * Math.Pow( x - p[0], 2 ) / ( 2 * p[1] * p[1] ) ) + p[2];
};

Above is a Func<> representing a Gaussian that allows for some vertical offset. The TrustRegionMinimizer in NMath is one of the most powerful and flexible methods for peak fitting. Once the start and end indices of the peaks are determined, the following code snippet fits this Gaussian model to the peak’s data.

// The DoubleVector's xValues and yValues contain the peak's data.

// Pass in the model (above) to the function fitter ctor
var modelFitter = new BoundedOneVariableFunctionFitter<TrustRegionMinimizer>( Gaussian );

// Gaussian for peak finding
var lowerBounds = new DoubleVector( new double[] { xValues[0], 1.0, -0.05 } );
var upperBounds = new DoubleVector( new double[] { xValues[xValues.Length - 1], 10.0, 0.05 } );
var initialGuess = new DoubleVector( new double[] { 0.16, 6.0, 0.001 } );

// The lower and upper bounds aren't required, but are suggested.
var soln = modelFitter.Fit( xValues, yValues, initialGuess, lowerBounds, upperBounds );

// Fit statistics
var gof = new GoodnessOfFit( modelFitter, xValues, yValues, soln );

The GoodnessOfFit class is a very useful tool for peak modeling. In one line of code one gets the f-statistics for the goodness of the fit of the model along with confidence intervals for all of the model parameters. These statistics are very useful in automating the sorting out of noisy peaks from actual data peaks and of course for determining if the model is appropriate for the data at hand.

Peak Area

Computing peak areas or peak area proportions is essential in most applications of spectographic or electrophoretic data analysis. This is this a two-liner with NMath.

// The peak starts and ends at: startIndex, endIndex.
var integrator = new DiscreteDataIntegrator();
double area =  integrator.Integrate( DataSet[ new Slice( startIndex, endIndex - startIndex + 1) ] );

The DiscreteDataIntegrator defaults to integrating with cubic spline segments. Other discrete data integration methods available are trapezoidal and parabolic.

Summary

Contact us if you need help or have questions about analyzing your team’s data sets. We can quickly help you get started solving your computing problems using NMath or go deeper and accelerate your team’s application development with consulting.

Assorted References

1. Eilers, Paul & Boelens, Hans. (2005). Baseline Correction with Asymmetric Least Squares Smoothing. Unpubl. Manuscr.
2. C.G. Ryan, E. Clayton, W.L. Griffin, S.H. Sie, and D.R. Cousens. 1988. SNIP, a statistics-sensitive background treatment for the quantitative analysis of pixe spectra in geoscience applications. Nuclear Instruments and Methods in Physics Research Section B: Beam Interactions with Materials and Atoms, 34(3): 396-402.
3. García-Alvarez-Coque MC, Simó-Alfonso EF, Sanchis-Mallols JM, Baeza-Baeza JJ. A new mathematical function for describing electrophoretic peaks. Electrophoresis. 2005;26(11):2076-2085. doi:10.1002/elps.200410370

The post Chromatographic and Spectographic Data Analysis appeared first on CenterSpace.

An Introductory Example

Suppose we have the following sampled data.

The following C# code builds the test data and locates the single peak in this simple data set.

Using CenterSpace.NMath.Core;

DoubleVector d = new DoubleVector(0,-1, 1.5, 2, 3, 4, 4.5, 4, 3, 2.2, 1.0, -3.0, 0);
PeakFinderSavitzkyGolay pfa = new PeakFinderSavitzkyGolay(d, 5, 4);
pfa.LocatePeaks();

The peak data reported back by the Savitzky-Golay peak finder can be either the (x,y) location of the peak, or the index lying on or preceding the found peak.

Peak found at x:6.00, y:4.50
Peak found at index: 6

The peak finding PeakFinderSavitzkyGolay class requires three parameters: a data vector, the filter window width, and the degree of the smoothing polynomial.

A More Complex Example

To build a peak finder on top on this class for some domain specific data, we need to understand the basic parameters that control which peaks are reported. For this second example, let’s use the complex signal show below, which contains a mixture of isolated peaks, adjacent peaks, and narrow and broad peaks.

This signal also includes some very subtle peaks near x = 23.5 and x = 29. Applying the PeakFinderSavitzkyGolay class as shown below, all 15 peaks are found (the top of the first peak is off the scale in our image).

PeakFinderSavitzkyGolay pfa = new PeakFinderSavitzkyGolay(signal, 10, 5);
pfa.AbscissaInterval = 0.1;
pfa.LocatePeaks();
Console.WriteLine("Number of peaks found: " + pfa.NumberPeaks.ToString());
Console.WriteLine(String.Format("Peak found at x:{0:0.00}, y:{1:0.00}", pfa[4].X, pfa[4].Y));

The position of the fifth peak is reported to the console, using indexing notation, pfa[4].X, pfa[4].Y, on the peak finder object.

Number of peaks found: 15
Peak found at x:9.03, y:0.35

By setting the AbsciassaInterval to the signal sample rate (in this example 0.1 seconds) the class can scale the x-axis according to your units, and supply the (x, y) positions of all found peaks. If you only need the peak location down to the resolution of the sample interval, the peak finder will just report the index that either precedes or lies on the peak abscissa location. This avoids the an extra interpolation step to locate the inter-sample peak abcissa, and increases performance.

Now supposing that we want to eliminate all broad peaks and can increase the peak finders selectivity.

PeakFinderSavitzkyGolay pfa = new PeakFinderSavitzkyGolay(signal, 10, 5);
pfa.AbscissaInterval = 0.1;
pfa.SlopeSelectivity = 0.003;
pfa.LocatePeaks();

The property SlopeSelectivity defaults to zero, causing the peak finder to report all found peaks. By increasing the selectivity a hair to 0.003, the peak finder no longer reports the last three peaks. Both the two subtle peaks are eliminated along with the final broad peak near 26. The slope selectivity is simply the slope of the smoothed first derivative of the Savitzy-Golay polynomial at each zero crossing – so as it’s value is increased only the more pronounced peaks (with steeply diving smoothed first derivatives) are reported.

If we want to heavily filter the peaks and only see the peaks of the general trend line, we could increase the filter width dramatically from 10 to 80.

PeakFinderSavitzkyGolay pfa = new PeakFinderSavitzkyGolay(signal, 80, 5);
pfa.AbscissaInterval = 0.1;
pfa.SlopeSelectivity = 0.0;
pfa.LocatePeaks();

Using these parameters, we find only the four peaks that capture the low frequency variation of the signal above.

Peak found at x:7.53, y:0.34
Peak found at x:14.27, y:0.26
Peak found at x:20.28, y:0.25
Peak found at x:26.76, y:0.24

Note that the y-values here correspond to the smoothed, fitted polynomial not the actual data at the x value. This demonstrates the ability of the this peak finder to act as a low pass filter, which can be use to sort out the peaks of interest from higher frequency noise. This is why Savitzy-Golay data smoothing is often used for baseline subtraction (for example in pre-processing mass spectrometry data before peak finding).

Summary & Performance

The first example used a smoothing polynomial of degree 4, and the second example a polynomial degree of 5. Experience shows that typically a polynomial degree between 3-7 will be best suited to smooth measurement data and correctly locate peaks. However, there is no hard and fast rule, so feel free to experiment. Having said that, the polynomial degree must always be strictly less than the window width or an exception will be thrown.

polynomial degree < window width

Because, after object construction, this peak finder boils down to a convolution, it's performance is far better that many peak finders. On my 2.8Ghz Intel i7 Quad Core, I can find peaks in 3 million data points in about 80 ms, and in 30 million data that requires about 700ms. That would give us the ability to do peak finding in a real time system running with a sample rate of ~43 MHz. In a real system there would likely be other peak filtering overhead, but we would still be able to process data at a very high rate - suitable for most real-time data sampling applications.

Happy Computing

-Paul Shirkey

The post Finding Peaks in Data with NMath appeared first on CenterSpace.