Controlling complexity is the essence of computer programming. (Brian Kernigan)
Let me remind you, I have been working with Syslog-Ng on their release automation project as part of GSoC’16. You can get more information about the project by lurking over here I will try to summarize my experiences during GSoC’16 and the deliverables.
Breaking down the project
Clone the repository and generate a changelog using a changelog generation algorithm. This is probably the toughest and most time consuming part.
releasebranch from the
masterand add commits to the branch doing version increments.
syslog-ng has a rich set of dependencies and building of
distribution tarballhas to be done in a docker environment.
After generation of the distribution tarball; a
debian source packageis generated using the previously generated distribution tarball.
The debian source package is then uploaded to OBS for generating the debian packages for different linux distributions.
A pull request is then sent to the
masterbranch from the
releasebranch and the release process is said to be complete.
The Sequence Diagram describes the project structure:
** (*) Indicates multiple methods **
I created syslog-ng-autorel for providing the above functionality. The
README.md file has the usage instructions. Go! get your hands dirty with it
Challenges and Roadblocks
I encountered lot of roadblocks during the project and some away from the computer world too. I suffered a full body infection that effected my senses. At one point, it became very difficult to cope with things and I was thinking to leave the project. Thanks to my awesome mentors, Laszlo Budai and Laszlo Varady. They let me take some time off and were more than supportive. Coming back to the project, these are the roadblocks I would like to highlight:
I was pretty fast at implementing things but I always have the habit of making a mess of the architecture. As they say, Understand the underlying business logic and abstraction first and then start implementing
Without digging into the architectural aspects, I preferred jumping directly into implementations. This technique has many pain points. As you get to understand the architectural aspects over time, your code suffers heavy refactoring. Moreover, your mentors need to step up into your code trying to figure out what’s going on. And then, Er? Eh! What? No.
We must understand that they don’t have all the time in the world and have full time jobs too. So, having a blue print before hand is crucial. It kicks the code refactoring aspect out of picture serving us students well and at the same time saves time of our mentors too.We can take the help of Sequence Diagrams, UML Diagrams etc. to communicate to mentors our ideas about architecture and at the same time take feedback. It’s a true Agile Software Development process.
I come from a scripting background with very less experience in professional coding experience in compiled languages. In my opinion, one should write some code in programming languages like
C++etc. because they are more structured and gives you required exposure to practice writing SOLID code. So, why SOLID ? SOLID principles let you mimic your architecture into implementation in a way that facilitates easy code refactoring, dependency changes in code etc. overtime.
Changelog generation is the process of generating detailed log between two releases incorporating the merged pull requests, fixed issues, bugs, enhancements etc. Why changelog, go here. We need a changelog now I used the following algorithm for generating the changelog:
- Find the list of commits between two releases.
- syslog-ng enforces commit patterns. For eg.
Merge pull request #1159 from lbudai/f/fix-elastic-mod-http-slf4jWith the help of regex we can have the list of all
pull request ids. Similarly, I can find the list of
- Some of these
pull requestsmay be linked to
fixed issues.I used a Python client library Pygithub for interacting with the Github API. So, the
linked issuesare found out.
- For other issues we can parse the commit message:
Fixed #416 queue full message
- Classification of the issues is done into
- Finally we have four types of changelog entries:
Merged Pull( Pull request not linked to an issue or a local checkout )
Fixed Issue( Uncategorized fixed issue )
Finally, a changelog is generated carrying the contributor info for each changelog entry. You can see the output of the tool here
I was using
Python3for development of the project. One of the task was to upload the Debian source package to OBS as I mentioned earlier. I tried using osc. It’s own source code is return with backward compatibility and supports
Python3but one of the dependencies
m2cryptois not Python3 compatible. This module is used to establish a SSL connection to OBS. So, I adopted a solution that involves spawning an independent
Python2process for handling OBS upload.
Apart from adding some cool tool-chains to my arsenal, I found developing a good relationship with my mentors as the best experience of the GSoC project. I wouldn’t have learned a single thing without their constant support. They are just awesome Let’s get back to the new things I learned:
- Controlling docker with Python
- Practice of writing code with SOLID principles
- Knowledge of Debian packaging
- GIT internals and sound knowledge of Github
- I think I got better at System design/architecture
I am planning to work with syslog-ng in the coming months. The goal of the GSoC’16 project was to develop building blocks for the release pipeline of syslog-ng. Let’s see if I can integrate these modules into a release pipeline. Apart from it, I am planning to create a web interface to provide analytics for the project.
Contributing to open source software is fun and it always will be. Keep digging and stay tuned !