As someone who likes developing little utilities in my free time, there is always the urge to want to improve upon things. Naturally, then, I am lead towards the topic of Automation.
For many things in my Home lab, automation is key in having things running smoothly - and as I provide services not just for myself, but also my friends, the more I can automate, the better.
One of the most common tools I use for these sorts of things is Python - a language that is designed to be easy to write, and well-suited for automation-related tasks.
I have written Python scripts for things like automatically reaching out to APIs, optical character recognition (OCR) and even for keeping track of my kettle brewing!
Suffice to say, it’s a very powerful language, but it comes at a cost - specifically, time. Writing new python code for relatively menial tasks can take up a lot of time that I could otherwise spend working on other projects.
This is where Huginn comes in.
Huginn
Huginn is a very powerful system designed for building agents to monitor web-related activity and run automations based on that information. This includes monitoring RSS feeds, web scraping and a lot more.
My main use case for Huginn originally was to automate the updating of a calendar I have in my community Guilded server (come say hi!) with the release dates of any films or TV shows that were due to release digitally that week. And it’s great - it allows me to see all the releases dates in one place in an easy-to-read format in an application I already use day-in, day-out.
But that’s not what I want to talk about today - today, I want to talk about how I set up a ‘News Feed’ for one of my favourite games, Final Fantasy XIV Online, using Huginn.
The Old Way
Originally, in order to get posts to appear in Guilded, I wrote a Python script that would reach out to the Lodestone News API and save the latest post’s ID to a file to use as a ‘unique’ check. Every 5 minutes or so, it would reach out to the API, grab the latest post, and publish it to the Webhook under the adorably-named pseudonym “Gobbiebot” (a reference to the Goblin race in FFXIV).
Sounds straightforward, right? Well, not exactly.
What ended up happening is that if multiple posts of the same ‘category’ were posted, they could be missed by the time the system next checked. On top of that, often you would end up in situations where the same post would be sent multiple times, which just made things confusing.
I thought about going back to the Python script and re-writing it, but I figured - “hey, I’m already using Huginn for Guilded-related stuff, why not do this with it too?”. So I did.
The New Way
With Huginn, I could replace the entire script with just 3 agents, with very minimal JSON configurations.
RSS Agent
The first thing I did was move away from the manual polling and instead switched to using the RSS feed from the site - this meant that Huginn could do the checks for uniqueness and also send out each new post as a separate event.
Source:
{
"expected_update_period_in_days": "1",
"clean": "false",
"url": "https://lodestonenews.com/feed/eu.xml"
}
Event Formatting Agent
This next section was important - by default, the incoming data has some HTML characters in it (namely, <br> tags) as well as some already existing fields like content. The content field is what gets sent as the text for a webhook, but in that case it would mean sending the same text in the actual message in Guilded as well as in the embedded message. So, we need to filter that out to remove the content and also fix the HTML tags.
The last step is to make sure that there is always an image present - by default, Guilded will not accept an empty (or null) parameter for the image field of an embed message. So, on the off-chance that there isn’t one, I use the standard FFXIV post image as a placeholder.
Source:
{
"instructions": {
"content": "",
"title": "{{title}}",
"url": "{{url}}",
"description": "{{description | regex_replace: \"<br>\", \"\\n\" }}",
"image": "{% unless image %}https://img.finalfantasyxiv.com/t/459f3dba0a883b0f3a528f371f637f9d3b1fc47b.png?1672045170?16717619431671761943{% else %}{{image}}{% endunless %}",
"categories": "{{categories.first}}"
},
"matchers": [
],
"mode": "clean"
}
Post Agent
This is the last piece to the puzzle - this agent will wait for the incoming event (containing the newly-formatted information) and build the final webhook response object. This includes all the formatting changes we’ve made.
Source:
{
"post_url": "{% credential guilded_lodestone_news_webhook_url %}",
"expected_receive_period_in_days": "1",
"content_type": "json",
"method": "post",
"payload": {
"embeds": [
{
"title": "{{title}}",
"url": "{{url}}",
"image": {
"url": "{{image}}"
},
"footer": {
"text": "Category: {{categories}}"
},
"description": "{{description}}",
"content": "null"
}
]
},
"headers": {
},
"emit_events": "true",
"no_merge": "false",
"output_mode": "clean"
}
Round Up
In the end, not only does Huginn handle this task better than my Python script, but it’s also significantly easier to maintain - if anything changes with the formatting of the API, I can quickly adjust the parameters in the JSON configuration for the agents, and it’ll be up and running again in no time.
If you’d like a copy of this automation, you can find it on my GitHub here, along with any other automations I may post in the future.