Employee engagement is not a tangible thing and can’t be measured easily. However, based on the definition we choose, we then can ask a range of questions that will indicate how engaged the person is.
We have a dataset of an employee survey result. It collects the responses of 832 individuals on 34 questions. Among the dataset, 5 of them are basic demographic information, and the remaining ones are the responses on a 1 (strongly disagree) to 5 (strongly agree) response scale for a range of specific questions. The questions are divided in 9 topics, such as employee engagement towards the company, perceived organizational support (POS), organizational citizenship behavior (OCB), etc.
In this post we are interested in two topics in the dataset: employee engagement and perceived organizational support (POS). They are assessed via the following 7 questions (variables):
Employee organizational engagement:
- I share the values of this organization (Eng1 variable in the dataset).
- I am proud of this organization (Eng2 variable in the dataset).
- This organization is a big part of who I am (Eng3 variable in the dataset).
- I would recommend this organization as a place to work (Eng4 variable in the dataset).
Perceived organizational support (POS):
- This organization cares about my well-being (Pos1 variable in the dataset).
- This organization is interested in my contribution(Pos2 variable in the dataset).
- This organization is there for me when I need help(Pos3 variable in the dataset).
To validate if the questions are properly defined for each topic, we choose principal component analysis. Principal component analysis is a technique for identifying groups or clusters of variables. We apply it to the responses of the 7 questions to verify if the questions are sorted in the right groupings.
The following is the result. We can see that the questions for each group do receive similar scores.
Principal Components Analysis Call: principal(r = na.omit(selected.survey), nfactors = 2, rotate = "varimax") Standardized loadings (pattern matrix) based upon correlation matrix RC1 RC2 h2 u2 com Eng1 0.84 0.11 0.72 0.28 1.0 Eng2 0.89 0.07 0.80 0.20 1.0 Eng3 0.87 0.15 0.78 0.22 1.1 Eng4 0.90 0.17 0.83 0.17 1.1 Pos1 0.18 0.69 0.51 0.49 1.1 Pos2 0.04 0.88 0.77 0.23 1.0 Pos3 0.08 0.90 0.81 0.19 1.0 RC1 RC2 SS loadings 3.11 2.12 Proportion Var 0.44 0.30 Cumulative Var 0.44 0.75 Proportion Explained 0.60 0.40 Cumulative Proportion 0.60 1.00 Mean item complexity = 1 Test of the hypothesis that 2 components are sufficient. The root mean square of the residuals (RMSR) is 0.07 with the empirical chi square 182.48 with prob < 3.1e-35 Fit based upon off diagonal values = 0.97
After applying the factor analysis, we can also test the reliability of the responses to the questions within the same topic. The logic is that the respondent’s responses to a specific set of survey items tend to follow a similar pattern. Therefore, although there will be a variation in how any individual responds to a set of questions, there should be a consistency in the responses if the survey items measure the same topic.
The statistic to measure the reliability of the responses is Cronbach’s alpha which is a value ranges from 0 to 1. If Cronbach’s alpha is 0.7 or above for a set of responses, it is considered a good or acceptable level of variance.
For the reliability test, we use a four-item measure to test organizational citizenship behavior (OCB). The four questions are as follows:
- I work harder than my job requires (Ocb1 variable in the dataset).
- I put a huge amount of effort into my job (Ocb2 variable in the dataset).
- I help out my teammates (Ocb3 variable in the dataset).
- I go the extra miles (Ocb4 variable in the dataset).
Below is the output:
Reliability analysis Call: alpha(x = ocb.survey) raw_alpha std.alpha G6(smc) average_r S/N ase mean sd 0.68 0.73 0.7 0.4 2.7 0.019 2.3 0.77 lower alpha upper 95% confidence boundaries 0.64 0.68 0.72 Reliability if an item is dropped: raw_alpha std.alpha G6(smc) average_r S/N alpha se Ocb1 0.60 0.65 0.62 0.39 1.9 0.025 Ocb2 0.56 0.61 0.53 0.34 1.6 0.026 Ocb3 0.53 0.59 0.51 0.32 1.4 0.027 Ocb4 0.78 0.78 0.72 0.55 3.6 0.013 Item statistics n raw.r std.r r.cor r.drop mean sd Ocb1 797 0.71 0.76 0.62 0.53 1.8 0.82 Ocb2 791 0.75 0.80 0.75 0.57 2.2 0.91 Ocb3 791 0.79 0.82 0.78 0.59 2.4 1.03 Ocb4 792 0.69 0.59 0.35 0.30 3.0 1.42 Non missing response frequency for each item 1 2 3 4 5 miss Ocb1 0.44 0.39 0.14 0.03 0.00 0.04 Ocb2 0.24 0.43 0.26 0.05 0.02 0.05 Ocb3 0.22 0.36 0.31 0.08 0.04 0.05 Ocb4 0.20 0.20 0.20 0.20 0.21 0.05
We found that the alpha value for this measure is only 0.68, and it could be better. As we see in the next section – “Reliability if an item is dropped”, we know that if we remove the last question, the reliability of the responses can get a lot better. Maybe it’s because that the last question – “I go the extra mile” is slightly ambiguous and is interpreted differently by different people. Therefore, we can keep the first three questions because they form a coherent scale to measure organizational citizenship behavior.