Automating Solar Panel Price Comparison Data Updates with GitHub Actions

[zaryabahmedd]

Why are you starting this discussion?

Question

What GitHub Actions topic or product is this about?

General

Discussion Details

Hi everyone,

I'm working on a solar energy information website where we regularly update solar panel pricing data, specifications, and market trends for consumers in Pakistan.

Currently, we're manually updating our price comparison tables and product listings, which is time-consuming. I'm exploring how to use GitHub Actions to automate parts of this workflow.
What I'm trying to accomplish:

Automatically fetch and update solar panel pricing data from various suppliers
Schedule weekly price comparison updates
Auto-generate markdown files with updated product specifications
Deploy changes to our static site automatically

My questions:

What's the best approach to schedule data scraping/API calls using GitHub Actions workflows?
How can I store and version control pricing data efficiently in the repository?
Are there any rate limiting concerns I should be aware of when running frequent automated updates?
Has anyone built similar price comparison automation workflows?

I'm relatively new to CI/CD pipelines, so any guidance, workflow examples, or best practices would be greatly appreciated!
Thanks in advance for your help!

[deadlinecode]

While GitHub Action Pipelines can be used for this i would recommend running something like a cronjob on your server (if you have a server that you also have shell access to or similar). Not only because its easier but also more secure. However this is totally up to you and depends on the infrastructure you have setup for your website. But if you have a typical setup where you have one server that hosts your website + maybe a database to store prices or even just the HTML for your website most of the time its easier and more secure to just have a cronjob running on that server that fetches and calculates the data and directly places it into the database or the HTML files.

Otherwise you would have to setup some type of access for GitHub pipelines since they run on GitHubs own servers and need a way to upload or update data or files on your server.

There is also the option of hosting your own GitHub pipeline on your server but i think this would be a little bit overkill / too complicated for your use case.

Also note that you have only 2.000 free runner minutes per month after that you will need to pay GitHub for running your pipelines.

If you need any more guidance on how to start with anything of what i mentioned feel free to ask :)

[Siluryan]

It is great to see you applying automation to something as practical as solar energy pricing! Transitioning from manual updates to a CI/CD approach will save you a massive amount of time, especially with a volatile market like Pakistan's.

For your first question about scheduling, the best approach is to use the schedule event in your workflow, which uses POSIX cron syntax. You can set it to run every Monday at a specific time, like cron: '0 9 * * 1'. This keeps the scraping completely hands-off. For the actual scraping, you can use a Python script with libraries like BeautifulSoup or Selenium, or even a simple Node.js script. You would just call this script as a "step" in your GitHub Action after the checkout phase.

When it comes to storing and versioning that pricing data, since you are dealing with tables, JSON or CSV files are your best friends. You can have your script save the fetched data into a data/prices.json file. The "magic" happens at the end of your workflow: you use a "Git Commit and Push" action to automatically commit those changes back to your repository. This gives you a built-in historical record of every price change over time, which is incredibly valuable for showing market trends to your users.

Regarding rate limits, GitHub Actions gives you plenty of runtime (up to 2,000 minutes per month on free accounts), which is more than enough for weekly updates. The real concern is usually the source websites you are scraping. To avoid getting your workflow's IP blocked, it is best practice to add small delays between requests in your script and ensure you aren't hitting their servers too aggressively. If you're using official APIs, just keep an eye on their specific Tier limits.

A common workflow for this starts with the schedule trigger, followed by a python-setup step, then running your scrape_prices.py script, and finishing with a step that commits the new data and triggers a rebuild of your static site.

[github-actions[bot]]

🕒 Discussion Activity Reminder 🕒

This Discussion has been labeled as dormant by an automated system for having no activity in the last 60 days. Please consider one the following actions:

1️⃣ Close as Out of Date: If the topic is no longer relevant, close the Discussion as out of date at the bottom of the page.

2️⃣ Provide More Information: Share additional details or context — or let the community know if you've found a solution on your own.

3️⃣ Mark a Reply as Answer: If your question has been answered by a reply, mark the most helpful reply as the solution.

Note: This dormant notification will only apply to Discussions with the Question label. To learn more, see our recent announcement.

Thank you for helping bring this Discussion to a resolution! 💬