What’s being tested and what’s being learnt? A contribution to lessons learned evaluation methods for community-based sustainability initiatives.
DOI:
https://doi.org/10.29015/cerem.596Keywords:
lessons learned, evaluation, community development, sustainability policy, project managementAbstract
Aim:
There is little good practice guidance with respect to methods and skills for conducting lessons learned evaluations of community-based development projects. In this paper we utilise a mixed methods approach to evaluate the lessons learned by the team members and stakeholders of a funded five year ‘test-and-learn’ sustainability initiative. The approach combines a statistical and a qualitative thematic analysis of transcribed textual data and presents an analytic framework with which to track the lessons learned by community development projects.
Design/ Research methods:
A mixed methods approach combining text and sentiment mining complemented by a qualitative thematic analysis is applied to the same data collected from stakeholder responses to an on-line survey and the transcribed audio recordings of four focus groups in which stakeholders participated.
Conclusions/ findings:
Employing replicable tools, augmented by qualitative research methods, provide a framework for a systematic approach to elicit and capture lessons learned by a sustainable community development project. These bear on how project activities, from engagement to supporting the local food economy, have been experienced by stakeholders and their learning acquired over the course of the project. Implications for future project design and funding options are considered.
Originality/ value of the article:
Despite the evident value of its contribution to improving project design and funding options, the evaluation of lessons learned in community-based sustainability work remains under-researched. The combined use of text and sentiment mining techniques with qualitative thematic analysis on the same data offers an original contribution to research in this field.
References
Anbari, F. T., Carayannis, E. G. and Voetsch, R. J. (2008) ‘Post-project reviews as a key project management competence’, Technovation, 28(10), pp. 633–643. doi: 10.1016/j.technovation.2007.12.001.
Argyris, C. and Schön, D. (1992) ‘What is an organization that it may learn?’, On organizational learning, pp. 8–29.
Bartsch, V., Ebers, M. and Maurer, I. (2013) ‘Learning in project-based organizations: The role of project teams’ social capital for overcoming barriers to learning’, International Journal of Project Management, 31(2), pp. 239–251. doi: 10.1016/j.ijproman.2012.06.009.
Big Lottery Fund (2012) £12m lifeline for vulnerable facing fuel and food poverty, Big Lottery Fund: News and Events. Available at: https://www.biglotteryfund.org.uk/global-content/press-releases/england/archive/12m-lifeline-for-vulnerable-facing-fuel-and-food-poverty (Accessed: 3 April 2016).
Boisot, M. H. (1999) Knowledge assets: Securing competitive advantage in the information economy. New York: Oxford University Press.
Braun, V. and Clarke, V. (2006) ‘Using thematic analysis in psychology’, Qualitative Research in Psychology, 3(2), pp. 77–101. doi: 10.1191/1478088706qp063oa.
Bruner, J. (1986) Actual minds, possible worlds. Cambridge, Mass.: Harvard University Press.
Bryman, A. (2012) Social research methods. 4th edn. New York: Oxford University Press.
Carrillo, P., Ruikar, K. and Fuller, P. (2013) ‘When will we learn? Improving lessons learned practice in construction’, International Journal of Project Management, 31(4), pp. 567–578. doi: 10.1016/j.ijproman.2012.10.005.
Davenport, T. H. and Prusak, L. (1998) Working knowledge: How organizations manage what they know. Boston, MA: Harvard Business School Press.
Duffield, S. M. and Whitty, S. J. (2016) ‘Application of the Systemic Lessons Learned Knowledge model for Organisational Learning through Projects’, International Journal of Project Management, 34(7), pp. 1280–1293. doi: 10.1016/j.ijproman.2016.07.001.
Feinerer, I. and Hornik, K. (2014) ‘tm: Text Mining Package.’ Available at: http://cran.r-project.org/package=tm.
Feinerer, I., Hornik, K. and Meyer, D. (2008) ‘Text Mining Infrastructure in R’, Journal Of Statistical Software, 25(5), pp. 1–54. Available at: http://www.jstatsoft.org/v25/i05.
Grimmer, J. and Stewart, B. M. (2013) ‘Text as Data : The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts’, Political Analysis, pp. 1–31. doi: 10.1093/pan/mps028.
Guba, E. G. and Lincoln, Y. S. (1989) Fourth generation evaluation. Newbury Park, CA: Sage Publications.
Hahn, U. and Mani, I. (2000) ‘The challenges of automatic summarization’, Computer, November, pp. 29–36.
Hillard, D., Purpura, S. and Wilkerson, J. (2007) ‘Computer-Assisted Topic Classification for Mixed-Methods Social Science Research’, Journal of Information Technology & Politics, 4(4), pp. 31–46. doi: 10.1080/19331680801975367.
Jockers, M. (2017) ‘Package “syuzhet”: Extracts Sentiment and Sentiment-Derived Plot Arcs from Text’. Available at: https://cran.r-project.org/web/packages/syuzhet/syuzhet.pdf.
Kayser, V. and Blind, K. (2017) ‘Extending the knowledge base of foresight: The contribution of text mining’, Technological Forecasting & Social Change, 116, pp. 208–215. doi: 10.1016/j.techfore.2016.10.017.
Kumar, B. S. and Ravi, V. (2016) ‘A survey of the applications of text mining in financial domain’, Knowledge-Based Systems, 114, pp. 128–147. doi: 10.1016/j.knosys.2016.10.003.
Laver, M., Benoit, K. and Garry, J. (2003) ‘Extracting Policy Positions from Political Texts Using Words as Data’, The American Political Science Review, 97(2), pp. 311–331.
Levitt, B. and March, J. G. (1988) ‘Organizational Learning’, Annual Review of Sociology, 14, pp. 319–340. doi: Doi 10.1146/Annurev.So.14.080188.001535.
Lindner, F. and Wald, A. (2011) ‘Success factors of knowledge management in temporary organizations’, International Journal of Project Management, 29(7), pp. 877–888. doi: 10.1016/j.ijproman.2010.09.003.
Love, P. E. D., Teo, P., Davidson, M., Cumming, S. and Morrison, J. (2016) ‘Building absorptive capacity in an alliance: Process improvement through lessons learned’, International Journal of Project Management, 34(7), pp. 1123–1137. doi: 10.1016/j.ijproman.2016.05.010.
Makrakis, V. and Kostoulas-Makrakis, N. (2016) ‘Bridging the qualitative – quantitative divide: Experiences from conducting a mixed methods evaluation in the RUCAS programme’, Evaluation and Program Planning, 54, pp. 144–151. doi: 10.1016/j.evalprogplan.2015.07.008.
Maurer, I., Bartsch, V. and Ebers, M. (2011) ‘The Value of Intra-organizational Social Capital: How it Fosters Knowledge Transfer, Innovation Performance, and Growth’, Organization Studies, 32(2), pp. 157–185. doi: 10.1177/0170840610394301.
McClory, S., Read, M. and Labib, A. (2017) ‘Conceptualising the lessons-learned process in project management: Towards a triple-loop learning framework’, International Journal of Project Management, 35(7), pp. 1322–1335. doi: 10.1016/j.ijproman.2017.05.006.
Meaney, C., Moineddin, R., Voruganti, T., Ann, M., Brien, O., Krueger, P. and Sullivan, F. (2016) ‘Text mining describes the use of statistical and epidemiological methods in published medical research’, Journal of Clinical Epidemiology, 74, pp. 124–132. doi: 10.1016/j.jclinepi.2015.10.020.
Mohammad, S. M. and Turney, P. D. (2013) ‘Crowdsourcing a Word–Emotion Association Lexicon’, Computational Intelligence, 29(3), pp. 436–465.
Moustaghfir, K. and Schiuma, G. (2013) ‘Knowledge, learning, and innovation: research and perspectives’, Journal of Knowledge Management, 17(4), pp. 495–510. doi: 10.1108/JKM-04-2013-0141.
Newton, J., Franklin, A., Middleton, J. and Marsden, T. (2012) ‘(Re-)negotiating access: The politics of researching skills and knowledge for “sustainable communities”’, Geoforum, 43(3), pp. 585–594. doi: 10.1016/j.geoforum.2011.12.003.
Nonaka, I. and Takeuchi, H. (1995) The knowledge-creating company: How Japanese companies create the dynamics of innovation. New York: Oxford University Press.
Office of Government Commerce [OGC] (2009) Managing Successful Projects with PRINCE2. London: TSO (The Stationery Office).
Patton, M. Q. (1997) Utilization-focused evaluation. 3rd edn. Thousand Oaks, CA: Sage Publications.
Powell, J. H. and Bradford, J. P. (2000) ‘Targeting intelligence gathering in a dynamic competitive environment’, International Journal of Information Management, 20(3), pp. 181–195. doi: 10.1016/S0268-4012(00)00004-9.
Putnam, R. D. (2000) Bowling alone: The collapse and revival of American community. New York: Simon and Schuster.
R Core Team (2014) ‘R: A language and environment for statistical computing.’ Vienna, Austria: Foundation for Statistical Computing. Available at: http://www.r-project.org/.
Rennie, D. L. (2012) ‘Qualitative research as methodical hermeneutics.’, Psychological Methods, 17(3), pp. 385–398. doi: 10.1037/a0029250.
Rittel, H. W. J. and Webber, M. M. (1973) ‘Dilemmas in a General Theory of Planning’, Policy Sciences, 4(2), pp. 155–169. doi: 10.1007/BF01405730.
Rolstadås, A., Tommelein, I., Morten Schiefloe, P. and Ballard, G. (2014) ‘Understanding project success through analysis of project management approach’, International Journal of Managing Projects in Business, 7, pp. 638–660. doi: 10.1108/IJMPB-09-2013-0048.
Saif, H., He, Y. and Alani, H. (2012) ‘Alleviating data sparsity for twitter sentiment analysis’, CEUR Workshop Proceedings, 838, pp. 2–9.
Schein, E. H. (1996) ‘Three Cultures of Management: The Key to Organizational Learning’, Sloan Management Review, 38(1), p. 9.
Sutcliffe, K. M. and Weber, K. (2003) ‘The High Cost of Accurate Knowledge’, Harvard Business Review, 81(5), pp. 74–82. doi: 10.1109/EMR.2003.24900.
Thomas, W. H. (2015) The basics of project evaluation and lessons learned. 2nd edn. Boca Raton, FL: CRC Press.
Ur-Rahman, N. and Harding, J. A. (2012) ‘Expert Systems with Applications Textual data mining for industrial knowledge management and text classification: A business oriented approach’, Expert Systems With Applications, 39(5), pp. 4729–4739. doi: 10.1016/j.eswa.2011.09.124.
Williams, T. (2003) ‘Learning from projects’, The Journal of the Operational Research Society, 54, pp. 443–451. doi: 10.1057/palgrave.jors.2601549.
Downloads
Published
Issue
Section
License
The aim of CEREM is to make scientific work available in accordance with the principle of open access. The rules mentioned below are important, as they enable CEREM and its publisher, the WSB Merito University in Wroclaw, to distribute the scientific work to a wide public while complying with specific legal requirements, at the same time protecting the rights of the authors.
The author transfers to the WSB Merito University in Wroclaw, free of charge and without territorial limitations, with all proprietary copyrights to the said piece of work in the understanding of the act of 4th February 1994 on copyrights and derivative rights (Journal of Laws of 1994, no. 24, item 83, as amended) on an exclusivity basis, i.e. the rights to:
1. Make the piece of work in question available via the Digital Library established by the WSB Merito University in Wroclaw.
2. Produce, record and reproduce in multiple copies the piece of work using any techniques whatsoever, including printing, reprography, magnetic recording and digital processing, and particularly its reproduction by recording on CDs and similar data carriers,
3. Use fragments of the piece of work for promotional purposes in publications, promotional materials, the Internet and Intranet type networks managed by the WSB Merito University in Wroclaw.
4. Store the piece of work into computer databases managed by the WSB Merito University in Wroclaw.
5. Copy and reproduce the piece of work using photo-mechanic technologies other than those commonly known at the time of the signature hereof (photocopies, Xerox copies etc.),
6. Process the piece of work, transferring it into an electronic form, and distribute it on the Internet without limitations.
