In the longer run the goal is to have our own Selenium testing infrastructure (which include the dashboard), but for now we have decided to use Saucelabs which provide Selenium infrastructure where we can set up testing for the tiki.org sites.
Pascal will contact Saucelabs to get it setup.
Also, we will need to do:
1) Document how to record tests using Selenium IDE (a Firefox plugin) to record tests. (Alain assigned)
2) Document where to share tests. Wiki pages will be fine for now but the idea is to have them with SVN but see next point (Alain)
3) Document how to get them into a "test suite" which will be run by the Saucelabs testing infrastructure (Alain)
4) Start creating tests for the tiki.org sites, especially doc or dev. (Let's get everybody together for this).
5) Longer term roadmap to have tests committed together with features in SVN (but since this would require getting instances to a certain configuration and with test content to run the tests from, it is dependent on the Configuration Management and Systems Orchestration project to be done first.
6) Once the above is setup, Amette will figure our a way to automate SVN bisect to identify track what commit is behind which error.
7) Perhaps this can automatically be run on another machine on any error and the results inserted into a tracker on quality.tiki.org. Nelson will have coordinate with LPH how to insert it automatically into the specified tracker, probably by calling the AJAX tiki server,
- Have several highly customized instances of Tiki that they setup for customers.
- Avoid breaking their customer's sites when upgrading to trunk.
Make sure that use scenarios which are critical for their customers continue to work on trunk.
Need to do:
- Write tests on a customer's instance to capture that customer's key use cases.
- Setup a server that will run those custom tests once a day, and notify them when it fails one of them.
- Have certain parts of Tiki for which they are kind of "custodians".
- Create and modify code that might break someone else's code
- Know when someone breaks "their" code.
- Know when they break "someone else's" code
- Write "generic" tests (i.e. not tied to a particular customer instance) to make sure that "their code" will keep working in the future.
- Want some server somewhere to run those tests on trunk once a day, and notify them when it fails one of them.
- When modifying code in a part of Tiki, run a "short" (say 5 mins) test suite that covers the portions of the code that are most likely to be affected by the change.
- Knows Tiki very well, and may be site admin on several sites.
- Is not a programmer and may not have SSH access
- Avoid breaking his sites, when upgrading to trunk
- Help the Tiki community, eventhough he is not a programmer
- Write tests on the sites he maintains, that captures the use cases he and his users care about.
- Have access to a server somewhere that can run those tests on trunk and notify him when one of them fails
- May not be able to set it up himself.
- Write "generic" tests (i.e. not tied to a particular customer instance) on a test instance of Tiki.
- May not be able to set one up himself.