CASA Test Framework Primer
Summary
This document is a discussion on how the test framework for CASA is deployed. It's intended audience is developers who need to learn, or are otherwise curious about how the CASA test framework is deployed. Links to references are included largely to flesh out details that are a bit too low level for this document.
Software Test Infrastructure Technologies
CASA uses primarily a couple of test framework technologies: the Robot Framework and PyUnit. The Robot Framework is a general test framework that can be used for Python or Java. CASA uses it for Python. Please refer to http://robotframework.org/ as an introduction to this technology. The more technical User Guide can be found here: http://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html. PyUnit is the standard unit testing framework for Python. It is used for what in CASA are considered functional tests (as opposed to regression tests). A helpful website to gain some background on PyUnit is: http://pyunit.sourceforge.net/.
Gradle is the technology used to build CASA. Gradle is written in Groovy. The Gradle User Guide is here: https://docs.gradle.org/current/userguide/userguide.html. You can find an overview of Gradle in the User Guide. For some more background on Groovy: http://www.groovy-lang.org/documentation.html.
Gradle can be run from the command line (using the build.gradle command), at which point it will run Groovy to build and run the tests. The output from this will be both the log.html and report.html pages to show test results in your favorite browser. Pybot can also be run on the command line (when run in the same directory as the HTML files in the pybot subdirectory). Likewise the output from Pybot will be both the log.html and report.html test results.
Continuous Integration Technology Employed
Though this document is defining the automated test frameworks being employed by CASA, it is worthwhile to briefly mention the technology which is using these during the build and test process. Currently CASA is using Atlassian's Bamboo to build and test the software. You can read more about Bamboo here: https://confluence.atlassian.com/bamboo/bamboo-documentation-home-289276551.html. How CASA uses Bamboo is outlined here: https://casa.nrao.edu/casadocs/stable/casa-processes/build-and-test-workflow. CASA's Project Wallboard is at: https://open-bamboo.nrao.edu/telemetry.action?filter=project&projectKey=CASA. This outlines build and test results for the various plans defined for CASA. Note - this requires authentication using your NRAO credentials.
The sequence of events is that Bamboo initiates Gradle, which in turn executes Groovy scripts, which kicks off Pybot. Pybot then runs the tests dictated by the html files, and using the CasaRegression.py script, that finally runs runUnittest.py and runRegression.py scripts, executing the tests in the system, and finally producing the reports with the test results.
Regression and Unit (Functional) Tests
Regression Tests
The file which dictates how regression tests are run is: runRegressionTest.py. It currently can be found at ~/gcwrap/python/scripts/regressions/admin/runRegressionTest.py. An example of how to run your regression test is:
Where testname is the testname_regression.py. Running this will only produce terminal line messages/errors: no reports are produced running this command.
Unit (Functional) Tests
There is documentation covering Functional Tests: http://www.eso.org/~scastro/ALMA/CASAUnitTests.htm. For the most part, this documentation is pertinent.
Automated Tests Defined
An article that fleshes out the details of how tests are broken down is available: Automated Test Definitions
Requesting Tests Be Added or Removed
If you've written your test, and would like it included in the framework, directions are here: https://safe.nrao.edu/wiki/bin/view/Software/CASA/AddingTestToAutomatedFramework
If you want an existing test removed from the framework, please give this a look: https://safe.nrao.edu/wiki/bin/view/Software/CASA/RemovingATestFromTheAutomatedFramework