McQuitty Causal Path Statistics

Breadtag Sagas ©: Author Tony,  2 September 2022

Several things coming together at once prompted me to think about Fred Emery’s ideas and my involvement in them from 1979 to the end of the 1990s.

First was involvement in doing politics differently and getting an independent senator David Pocock elected into the Australian parliament. After the election, I thought search-based focus groups could be used to develop an information-rich knowledge base and assist in community engagement.

Second, at the same time a health crisis made me think that it might be important to get together my knowledge of Fred Emery’s work.

The following articles have been an education and have re-engaged me with a set of powerful ideas around systems thinking.

The articles in order are: 1 Causal Texture (an annotated version of Emery & Trist’s famous paper), 2 The Search Conference (a description and explanation), 3 McQuitty Causal Path Analysis (a powerful statistical methodology as an adjunct to organisational change), 4 Participative Workplace Design (a description of workplace reform), and 5 An Organisational Thermometer (to measure progress in workplace reform).

The accompanying article on Q Research shows how I turned some of these ideas into a successful strategic marketing research business.

McQuitty Causal Path Analysis

This article is a necessary preliminary to the articles which follow and define organisational thermometers. An organisational thermometer is a tool to measure staff satisfaction in any largish enterprise in an ongoing way. The article is also related to all the Fred Emery associated articles, two of which precede it. These are 1 Causal Texture Paper and 2 the Search Conference. 

Main Points

• My background with Fred Emery and in statistics
• A general overview of statistical analysis
• Fred Emery’s consulting work for TIHR on consumer products
• McQuitty causal path analysis and how to do it explained
• Examples of McQuitty causal paths or roadmaps

1 My Background

I mentioned in Causal Texture and in the Search Conference that I met Fred Emery when employed at the Centre for Continuing Education (CCE) at the Australian National University (ANU) in 1979.

I was finishing up my PhD in Zoology at ANU at the time and was slightly dissatisfied with some aspects of inattentive reductionism in science at the time. When I discovered Fred’s ideas on systems thinking, it was as if I’d suddenly discovered what I was looking for and had to pursue it.

I got into Fred’s theoretical ideas and search conferencing quite quickly, which annoyed at least one person on the CCE staff. I became quite conversant with Fred’s theoretical ideas and reasonably competent at running search conferences in the next 18 months or so.

I left CCE in the second half of 1980 on a prestigious Leverhulme Vice Chancellor’s Fellowship in Continuing Education to the New University of Ulster’s continuing education annex in Derry.

Chris Duke the Director had obtained the Fellowship for me, for which I had no academic qualifications (which everyone knew). I had a terrific time in Northern Ireland in attempting community action research, despite the troubles re-emerging at the end of my time there. At the same time I also followed up on Fred’s work at the Tavistock Institute for Human Relations (TIHR).

On my way home, I was the guest of Einar Thorsrud at the Work Research Institutes in Oslo in Norway for two months, spent a couple of weeks with academics in Sweden and camped for two-and-a-half months with Rukmini Rao and peripherally PECCE (Public Enterprises Centre for Continuing Education) in New Delhi, India.

Sadly, because of my interest in Fred Emery, I was on the wrong side fence for Chris Duke (which I regretted), and a persona non grata on my return to CCE though granted a visiting fellowship for several months (no resources and no money).

Fred’s own senior fellowship was not renewed by ANU in 1979 and he became an independent scholar from then on at his home in Cook, ACT. (Although he still had full access and privileges at the ANU library.)

If the word statistics brings you out in a cold sweat just read slowly, there are no formulas or anything difficult below, just a story.

2 My Statistical Analysis

Most people are terrified of statistical analysis. Including, many academics and most graduate students in biology. This is especially sad because the giants of statistical methods in the early twentieth century tended to be biologists. I became a defacto expert in statistics in the Zoology Department at ANU while I was doing my PhD because:

1. I thought about the statistical design of what I was doing in advance of beginning.
2. I was concerned about using statistics intelligently to help to illuminate the questions I was asking.
3. I knew that I was going to have to rely on small samples. And, I used the idea of approaching the same question from different directions, based on the idea that one could get a confirmation from several parallel approaches with small samples, as well as or better than relying on one simple analysis with a large sample. I was also using non-parametric methods intelligently (don’t worry about it).
4. Various computer or calculator-style methods were suddenly available and one no longer had to calculate statistical methods manually. I still often did manual tests of important statistical procedures. And, I always put test data through the automated procedures, just to check that they were doing what I thought they were doing.

Most students and a surprisingly high proportion of academics merely did the standard Chi-squared or Student’s t-test, after they’d obtained their data. They accepted the standard probability of less than 5% on a null hypothesis (without thought) and made sure the sample size was what was recommended. They also conventionally used standard error bars on graphs to show how consistent their data was.

I still regret to today that I didn’t follow Professor SA Barnett’s advice to put standard error bars on my graphs, as they would have been more acceptable to everyone. The reason I didn’t was that I’d corrected my figures for legitimate reasons and felt that my standard errors would look rather good because of this. They would have. But, I was a purist. What an idiot!

My examiners being the standard biologists above hated my statistical methods. However, I’d at least been intelligent enough to thank Professor PAP Moran, Head of the Department of Statistics at the Research School of Social Sciences at ANU for his help in approving of my statistical approach. This kept their mouths shut and saved me a lot of grief.

3 General Statistical Analysis

Statistical design should come up front before beginning experiments. Statistics is also about ‘horses for courses’. There is probably a hierarchy of forms of statistical analysis represented in rank by:

3.1 The Australian Bureau of Statistics or similar national bodies elsewhere

The Australian Bureau of Statistics, despite defunding and manipulation by governments (e.g. redefining how unemployment is measured), is still Australia’s premier body for reliable, consistent and repeatable national statistics, for example, the census. The Commonwealth Statistician is very concerned with random sampling, error correction and reliable data.

3.2 Medical Statistics with Large Samples

Because we were in New Zealand for my birthday in 2020, we planned a birthday celebration at Lake Tabourie a couple of months later. Fortunately, it was one of those magical weekends with a large group of people to celebrate an important milestone. It was also very fortuitous in another way. A close friend had a major cancer operation a week later and within two weeks we were in our first Covid lockdown (one guest from Melbourne, on and off for many months).

Coincidentally, I received a book voucher from two guests and bought David Spiegelhalter The Art of Statistics: Learning from data 2019. Spiegelhalter’s book isn’t primarily about abstruse methods of analysis, but more about how we should approach statistics, particularly, medical statistics for societal benefit. He pushes an approach or problem solving cycle called PPDAC, going from Problem to Plan, to Data, to Analysis to Conclusion (much like my own approach above). The book covers the good and bad aspects of medical statistical analysis in the UK. It is an easy read and not too complicated for a lay audience.

3.3 Hypothesis testing

Hypothesis testing statistics do not have to be as accurate or correct as the other statistics above. They just need to be able to tease out answers to the questions one wants to ask. (But, you do have to know what you are doing. Because ‘rubbish in’, ‘rubbish out’ is a concern at all levels of statistical analysis.)

4 Fred Emery Consumer Product Consulting

Fred Emery was at the Tavistock Institute of Human Relations (TIHR) from 1957 to 1969 (not including his UNESCO Research Fellowship in 1951-1952). Whilst at the Tavistock, he had to conduct bread and butter consulting work because the TIHR was a self-funding model.

This became a problem in 1967 because the consulting arm of the Tavistock became more important than the research arm, which Fred tried to counteract briefly, but could not continue because his first wife Frances was extremely ill.

From 1960 to 1969 Fred worked on attitudinal research for Guinness, mustard, throat sweets, cigarettes, sweet sherry, Andrews liver salt, gravy, margarine, instant coffee, tea, petrol, biscuits, washing powders, whiskey, Lucozade and others. He also performed large social surveys and analyses on smoking and alcohol consumption.

Amongst his papers is one letter about a contract with TIHR over three years with Guinness worth £15,000 in 1963 (£350,000 today) a huge amount of money.

Fred did quite a degree of general research on alcohol and cigarette consumption in the UK (and Ireland) over years. Regardless, of what one thinks of this, knowing what we do today. And, I am sure that Fred was well aware of it in the 1960s. Fred was a consistently heavy drinker and smoker all his life. I have no idea what he felt about the tobacco or the alcohol research, but the TIHR certainly needed consulting money to survive.

Because of this research, Fred came in contact with some of the best statistical analysis, experts and techniques while conducting his commercial work, particularly with the British Tobacco Council. I suspect that it was through these contacts or merely from his incredibly broad reading that he became aware of the theoretical work of LL McQuitty (an American, US Airforce statistician).

5 McQuitty Causal Path Analysis

5.1 Background

According to Fred (pers. comm. & Emery 1970) McQuitty wasn’t particularly interested in the practical applications of his work, despite setting the analysis up for practitioners.

Fred Emery certainly was. From 1965 he began using McQuitty in all his consumer survey work and it helped him immeasurably in his analysis. He later extended Mcquitty work to all complex social science analysis, where a correlation matrix could be constructed from a large number of variables.

I learned McQuitty causal path analysis from Fred in 1979 and used it extensively over the next few years and also in my marketing research company Q Research. To my knowledge, Fred and I may be the only people who have ever done this regularly. Fred was a bit keener about McQuitty than I was initially, perhaps because of his starting point, his lesser consumer work may have been a bit soul destroying without it.

Fred mainly used McQuitty linkage analysis, but he also used McQuitty’s hierarchical cluster analysis as well in big projects (alcohol and smoking).

I didn’t see the need for the latter in my early McQuitty experiments (always re-analysing others’ data). I also didn’t see the need for it in my professional work on satisfaction surveys. My work on organisational thermometers never passed the preliminary stages. But, if someone takes up my suggestions in the next articles and, over time, becomes deeply involved in promoting organisational thermometers, then more in-depth McQuitty work and perhaps hierarchical cluster analysis may be useful.

5.2 Cause/Effect, Producer/Product

I was originally planning to reprint Fred’s Causal Path Analysis from Systems Thinking Volume 1981, pp 293-298, and annotate it. But, when I went back, I found it almost impenetrable. Hence I need to explain it in my own way. 

First off, cause and effect is something that we understand intuitively, but it is extremely rare in social science research. Let’s say I push you and you fall over. Push is the cause, fall over is the effect and perhaps grazed knee is the subsequent effect.

Fred in his paper uses the idea of producer/product. For example, if A, B and C are correlated. Then perhaps A implies B implies C. Now initially A—B—C, if strongly correlated, implies a relationship between them, but no arrow or direction.

Thus, with McQuitty causal path analysis there is no implication that A tends to produce B, which tends to produce C. However, if we know more about A,B,C from other work. The other information may provide the direction that A implies B implies C rather than the other alternatives.

5.3 The Strength and Weakness of McQuitty

The above is the power of McQuitty causal path analysis. If we have a complex matrix of social or behavioural variables, Mcquitty causal path analysis allows us to establish a linkage cluster or roadmap of how those variables are inter-related. Fred and others have also called these graph theoretical constructs (or mathematical graph theory). However, and this is important, McQuitty causal path analysis cannot tell you the direction of the influence. To put arrows on the Mcquitty roadmap, you must have other information outside of the analysis.

McQuitty in isolation, for example, if you are embarking on a new area and have no idea of what it all means, is not much use. You need to have some familiarity with the area you are studying. You need to have a basis of knowledge and understanding outside of the McQuitty analysis to interpret the McQuitty roadmap.

Now this is not usually a problem because one doesn’t generally construct a correlation matrix without some knowledge of what you are trying to achieve. For, example in the social sciences correlation matrices are usually constructed on the basis of questionnaires or surveys. Each variable represents a question.

One doesn’t develop a questionnaire without some preliminary hypotheses on what you are likely to find.

There are many methods of cluster analysis but they rarely present such a coherent map of what is going on as McQuitty does. (Emery 1970, gives a detailed version of all this, reproduced in Further Information below).

5.4 How to Create a McQuitty Causal Path from a Correlation Matrix

I was going to let you work it out yourself from McQuitty’s papers and Fred’s paper in Systems Thinking, but I came across a 3-page clear set of instructions (Emery 1980, reproduced in Further Information below).

In a correlation matrix, correlations can range from 0 to 1 or from 0 to -1. Fred suggests you get rid of negative correlations by reversing the question. For example, if you find a strong negative correlation between smoking and good health, then you find a strong positive correlation between smoking and poor health.

The strength of the correlation is important. Coming from a science background, I am uneasy with weak or very weak correlations, unless I can readily explain them. Social scientists seem less uneasy than I am.

An extremely strong correlation is one above .7 (and unusual in social science). .5 to .7 is a very strong correlation, .3 to .5 is a strong correlation. Under .3, I am uneasy as the value approaches .2. I am very uneasy about anything below .2 and not willing to accept anything below .1. But, this is a judgement call and dependent on one’s experience on a particular topic. (Sample size is also very relevant.)

6 Practical Examples of McQuitty Causal Paths or Roadmaps

6.1 McQuitty’s Hypothetical Example, 1964

McQuitty Hypothetical Causal Path, 1964

This is a lovely roadmap but perhaps a little too perfect. i and j are called reciprocal pairs (very closely associated) and the double arrowed lines are McQuitty’s convention for reciprocal pairs. The other arrows in the roadmap must come from other information and not from the McQuitty analysis.

6.2 Fred Emery’s Example from Lucozade Users, 1969

This was bread and butter work to fund the TIHR, yet Fred conducted such work professionally and the McQuitty roadmap (even without arrows) is self-explanatory.

Lucozade McQuitty Causal Path, 1969

Fred says the two clusters are independent (r = .03).

6.3 Education Re-analysis of a major US Study

Fred Emery in this study reanalyses a massive study that James S Coleman directed at the behest of the US Government in 1964 to 1966. In terms of resources and competencies no other nation then, or now, would have been so motivated.

The results were more radical than expected and tended to indicate that post-war efforts to improve general resources: highly qualified teachers, smaller classes, better texts, richer libraries… were insufficient in themselves.

The Coleman study was ignored for a variety of reasons. Coleman’s regression analysis brought out the main features but blurred the picture.

Because of its assumptions the Coleman regression analysis led to some stringent criticism, particularly by those who didn’t want to accept the results. But, Fred says that factor analysis, principal components analysis and the like would have been no better, whereas McQuitty causal path analysis is unassailable.

Fred re-analysed the Coleman data but for the purposes of a comparison with Australia. He selected a sample of 4000 northern white students and only looked at year 12 in this example (he was primarily interested in late high school).

The result shown is the key and one of the strongest clusters. There were other clusters.

Emery Education Paper, McQuitty Causal Path, 1979

Fred says: The arrow-heads are my interpretation to represent the most likely direction of influence; there is nothing in a mix of correlations that can do this.

The conclusion that Fred draws is that whilst school resources are important and necessary they are not sufficient and that this is a problem that has never been addressed.

(A necessary condition is required for something else to happen, but it does not guarantee that the something else happens. For example, while air is a necessary condition for human life, it is by no means a sufficient condition.)

Fred Emery also cites another study by WN Bardsley at ANU, with a smaller sample size of 374, which was also excellently constructed and showed much the same thing.

The key finding here Emery says: … is whether the parents really care!

7 Comment

I could give other examples, but these are sufficient and I will be giving further examples of McQuitty causal path analysis in my articles on organisational thermometers.

I’ve always suspected that Fred’s statistical credentials were better than mine and I believed him that McQuitty causal path analysis is much more formidable than other forms of cluster analysis. Apart from his own statistical abilities, he collaborated with others whose mathematical and statistical abilities were impeccable. He mentioned the statistical expertise of the British Tobacco Council quite often. (They were well aware of the issue of lung cancer in the 1960s and had much more statistical awareness of the implications than anyone outside the industry.)

I’m not expecting most readers to have an epiphany over McQuitty causal path analysis but will be very satisfied if one or two do.

Key Words: LL McQuitty, McQuitty causal path analysis, statistics, TIHR, Tavistock Institute of Human Relations, Tavistock, Fred Emery, Australian Bureau of Statistics, medical statistics, David Spiegelhalter, The Art of Statistics, PPDAC, Problem to Plan to Data to Analysis to Conclusion, hypothesis testing, linkage analysis, hierarchical cluster analysis, correlation matrix, cause/effect, producer/product, roadmap, arrows, graph theoretical constructs, mathematical graph theory, regression analysis, factor analysis, principal components analysis, cluster analysis, Coleman Study, James S Coleman, re-analysis, Chris Phillips, Living at Work, necessary, sufficient

Further Information

References

The Three McQuitty causal path examples

1 LL McQuitty Capabilities and Improvement of Linkage Analysis as a Clustering Method, Educational and Psychological Measurement 24: 441-456, 1964. The diagram above comes from this paper. I found it most useful. However, both McQuitty papers are easily found as PDFs in the Internet.

2 Fred Emery Lucozade Users, Tavistock Institute of Human Relations 1969/1 (Fred’s numbering). Fred whilst developing the theory of socio-technical systems, the Search Conference and conducting the Norwegian Industrial Democracy experiments still undertook these small examples of consumer attitude research with professionalism.

3 Fred Emery Opinion: Producing an Educated Community The Educational Magazine 36:42-46, 1979 (Victorian Department of Education). The source of the Coleman re-analysis and McQuitty example.

I have quite strong views on school education and tend to think that Fred’s findings here are unassailable and obvious when one ponders them. One can also understand why educators and those who provide resources for education would find the results of the Coleman Study so unpalatable.

The key finding was not as indicated in Wikipedia’s entry on James S Coleman that it is the education and economic attainment of the parents, even though they hint at the main reason a sentence or two below. Emery states quite clearly that it is the interest of the parents or the caregivers in education that motivates the student. If a student’s parents really want the student to learn, then self-learning will occur, however impoverished the learning environment.

In the current era the question, with busy parents both holding down demanding jobs, is whether the interest of parents in their children’s education has waned.  It does not matter how well-educated the parents are, or how enormous their economic attainment is, if the parents aren’t interested, the students won’t be motivated.

My school education was in the period of the Coleman Study and my parents were interested in our education. We had plenty of books at home. Nevertheless, I’ve always felt that my 12 years in the school system was wasted (and things are much worse today). I felt that I could have learned so much more in a better system (I went to relatively good primary schools and a good state high school in Canberra). Similarly, home schooling by intelligent parents (untrained in teaching) invariably produces much better results than the school system (ignoring of course socialisation). This shouldn’t be possible. I have sympathy for those who call schools child-minding centres.

Other References

FE Emery Causal Path Analysis pp 293-298, In FE Emery Ed. Systems Thinking, Vol 1 Penguin, 1981.

Fred Emery An Historical Note 1970 (Fred and Merrelyn Emery Papers, National Library of Australia, NLA, reproduced below)

Fred Emery Graphical Representation of a Matrix of Correlations 1980 (Fred and Merrelyn Emery Papers, NLA, reproduced below)

Fred Emery and Chris Phillips Living at Work, AGPS 1976. AGPS is the Australian Government Publishing Service, which means that this study was commissioned by the Australian Government. The book is an important sample survey of the Australian urban workforce in 1973. (It is available from the National Library of Australia, NLA).

Fred frequently cites the McQuitty method of causal path analysis outlined in the Appendix C Technical Notes of this book, to be consulted for explanation of the McQuitty technique. The title says it all, the technical notes are in three parts: 1 Sampling and Scaling, 2 the Representativeness of the Sample and 3 Causal Path Analysis. They are technical notes on how the survey was approached and as such, very similar to my notes on the organisational thermometer surveys conducted for ACTEW and outlined in a later article on this topic. They do not add anything to the simplicity I have tried to achieve in the current article. The McQuitty roadmap with correlation matrix in this book, however, is very interesting, another excellent example of the power of McQuitty causal paths and well-worth looking up. I will be including it in my article on organisational thermometers in the next month or so.

LL McQuitty Hierarchical Linkage Analysis for the Isolation of Types, Educational and Psychological Measurement 20: 55-67, 1960.

LL McQuitty Capabilities and Improvements of Linkage Analysis as a Clustering Method, Educational and Psychological Measurement 24: 441-456 ,1964. 

Fred Emery’s An Historical Note

Fred Emery An Historical Note 1970, P1

Fred Emery An Historical Note 1970, P2

Fred Emery An Historical Note 1970, P3

Fred Emery’s How to on Causal Path Analysis

Fred Emery, Step by Step Analysis, 1980, P1

Fred Emery, Step by Step Analysis, 1980, P2

Fred Emery, Step by Step Analysis, 1980, P3

Featured Image

The featured image is at the top of the article. The black is a hypothetical and rather too perfect example of a McQuitty roadmap. The handwriting is a note I found in the Emery papers at the National Library of Australia. The light writing is a correlation matrix about smoking. If you peer closely you can actually read some of the questions.

Written partly whilst quarantined with Covid 19, published in Canberra