With most of the WebVTT Release 0.2 work resting on the creation of a C parser, I’ve been doing some more testing with Travis CI to prepare for an automatic build system once everything is complete. In the meantime, we have a toy C parser that Ralph Giles (rillian) created and a bunch of .vtt sample files that can be validated using the webvtt module and a python script. As a start, I decided to see if I could get the python script to validate the sample files on the Travis CI machine.
Because the webvtt module is not installed, I have to run
npm install webvtt before running the python validation script. Travis CI does its magic and runs it for me, and a couple of lines after, the
make check-js command to execute the python validation script executes swimmingly. The script displays the validation results of the test files, and the Travis CI returns a “Build script exited with: 0″ to signify success.
This seems promising, but the python script had reported failing tests. I wondered if the return code for the script was reversed, so I quickly fixed the failing tests and re-pushed to my GitHub repo. The script reports 0 failed tests, but Travis CI still returns a successful 0. It looks like the python script returns the same thing regardless of whether everything passes or something failed. Guess that’ll have to be fixed so Travis CI can report a failure.
The next thing I could play with was Fuzz testing; Vince Lee (Lynart) had did some work with zzuf and wrote a simple shell script. In his GitHub repo, he had even added a directory that contained the zzuf package to build. Without hesitation, I attempted to get Travis CI to build zzuf from here…but then I realized, why not just install it instead? Sure enough, Travis CI’s docs mentioned the use of sudo apt-get, so I added
sudo apt-get update and
sudo apt-get install zzuf to my .travis.yml file. I was now on my way to running that shell script to fuzz the test files!
I had to modify the shell script a little so that it would accept command line arguments instead of waiting for input from the user. But a perplexing problem occurred, and I ran into a permission denied message:
-bash: ./fuzzv2.sh: Permission denied. That’s strange, I can’t run bash scripts? I immediately ran to Google and funnily enough, discovered a Japanese blog post where the developer ran a
chmod 777 statement to change the permissions on his script before executing it. Pretty strange that one has to do that, but I followed along and got the fuzz script working and generated fuzzed test files. I guess the next step is to throw these fuzzed files into our soon-to-be-developed C parsers and see what breaks.
Another little neat thing I found was that you can have GitHub report the Travis CI build status of your repo by embedding an image in the README file. I couldn’t get this to work with the standard README format with the command documented in Travis CI’s guide, but a README.md markdown file worked just fine with their example.
Once those C parsers are completed, we can combine the sample test files and fuzzed files to automate the entire build and test process. So every time there’s a change to anything, whether it’s the parser or a test file, there’s no need to manually build the parser, create fuzzed files of the test files, and then run the fuzzed and test files through the parser. And if anything has failed or something is broken, we’ll know as soon as possible.
My current .travis.yml file looks like this (with some random pwd/which/cat statements thrown in to see what happens):
# Change this to your needs
- npm install webvtt
- sudo apt-get update
- sudo apt-get install zzuf
- which zzuf
- which bash
- which sh
script: make check-js
- cd ./fuzz
- chmod -R 777 ./fuzzv2.sh
- ./fuzzv2.sh 0.1
- cat ./fuzzedFiles/good.tc_1009_missing_line_between_cues.vtt.fuzzed.vtt
What it should look like in the future is the installation of webvtt and zzuf, building the C parser, creating fuzzed files, and then running the test and fuzzed files against the built parser.
EDIT: you can see my Travis CI page here: https://travis-ci.org/#!/mafidchao/webvtt. My GitHub repo with the most recent branch can be found here: https://github.com/mafidchao/webvtt/tree/tci-lynart_fuzzer.