Writing a Custom Concourse Resource — Overview
In this article I am going to give a brief overview about resources and link to my GitHub repository with a working Concourse resource written in Python. I will also write subsequent articles going more in-depth about each stage of resource. I will update this article with links to the check, in, and out articles.
Writing a Concourse resource is not terribly difficult, but instructions on how to do so are generally incomplete. I have managed to stitch together a couple of resource using these tutorials and GitHub repository as guidance:
- Concourse-CI Official : Implementing a Resource Type
- VMWare : Developing a Custom Concourse Resource
- Shin Myung Yoon : Writing a Custom Resource for Concourse — Detecting Pull Request Close/Merge Events
- Yoshiyuki Kato : How to create an original Concourse Resource
- cf-platform-eng : cf-platform-eng/concourse-pypi-resource
The resource I wrote reads and writes to/from MongoDB. I was not too concerned with giving the resource too many features since this is just a tutorial, but if you want to improve upon my work feel free to fork my repository. I would love to see what you can do!
Note: I wrote my resource in Python and my example code snippets will reflect that.
Getting Familiar with Concourse Resources
Concourse resources can be simply described as a Docker image with three executables: check, in, and out in the /opt/resource/ directory. How you decide to define these executables is up to you as long as you also create a Docker image that can run your code. My scripts are written in Python because it is a very simple language and I find the language very enjoyable to write in.
I like to think of Concourse resource types in terms of Abstract types from Object Oriented Programming. An abstract type is blueprint for how you should be writing a type of object. There are methods with some default logic that you can override, but it is not always necessary. This does not hold 100% for Concourse resources, but when you write your Dockerfile you will need to COPY in something for check, in, and out and make them executable.
Say you wanted to write a resource that only fetched data from a site but did not write anywhere. You still need to provide an out, but it could be something as simple as:
Overview of the Resource
Before diving into the specifics about the MongoDB resource I need explain the context it is being used in.
Expected Output of a Resource Stage
Concourse looks for an output in standard out. In Python, this is done by using print with its default value for the file parameter. This is often done with a statement like
If you want to have your resource print something upon execution, you will need to print it to a different stream, like sys.stderr. This is something that I gathered from cf-platform-eng/concourse-pypi-resource’s common.py and expanded upon:
Even with this knowledge, you may soon realize that check is not very verbose. Even if you print something to standard out, you will not see it unless the resource fails. I managed to induce a failure by providing the wrong credentials to the database for you see:
You can see that the first line of the standard error is something I included in my code, but would not normally be visible without that script failing.
About the check
The check is performed before anything can use the resource. This can be seen if you click on a resource block (smaller blocks on the left-hand side of a task).
check is used to detect new versions of something that the resource is designed to interact with. For example the git resource type monitors for new git hashes. And my mongo-resource monitors for new ObjectIDs.
About the in
in is performed in two situations:
- When a job is triggered, the resource is an input for the job, and the
checkhas returned something. You do not need to use the output of thecheckin the action you use, but the typical resource will use whatcheckfound. Even thetimeresource type writes will provide you with a timestamp you could use in a task within a job it is an input for. - After an
out. I will explain this in the section About theout.
The example pipeline is set up such that each time the check finds a new ObjectID in MongoDB, it triggers a new task in my do the thing job. Once do the thing starts, is pulls in common-image, this-repo, and the resource mongo-database into the job and the task grabs its own inputs (note: Concourse will prevent a pipeline from flying if the resource is not used.) In the example pipeline we use all the inputs to the job.
The in uses the ObjectID (version) passed from the check to look up the corresponding entry in the database and save the json result to a file named <ObjectID>.json and prints the version back out to standard out.
About the out
Regardless of what the example pipeline is doing in its task the out takes whatever is in the path (directory or file) (in this case the mongo-resource resource directory) and tries to either update or insert anything it finds into the database.
The out declares it is done by outputting a version just like check and in. This is because the put calls upon in again once it finishes out to ensure that the output arrived at its intended destination.
Where I leave you for now
I have written separate articles on how to write the check, in, and out and linked them below:
- Writing a Custom Concourse Resource — Overview
- Writing a Custom Concourse Resource — The
check - Writing a Custom Concourse Resource — The
in - Writing a Custom Concourse Resource — The
out - apjansing/mongo-resource
Please feel free to reach out to me with any questions. These articles are my first few and I would love it if I could get some constructive criticism about my writing or if I should elaborate more. I wrote these articles because I had to find resources from several sites just to start writing something and wants to lessen that burden for someone who could write a better resource than me!
