Tag: Scraping

Posted on

Lego Deals

This application displays the current prices of Lego sets at major retailers. Every hour, a scheduled task scrapes 29 retailer websites for their discounted Lego sets, and those prices are saved to a database. I started collecting the hourly price history in December 2022 and it continues to run today. I’m able to set alerts via Telegram when the price of a particular set drops below a given threshold. The application also keeps track of which sets I already own, which ones I am interested in, and which ones I never want to see. I can switch to a view that only shows sets I care about.

Lego set details, like number of pieces, are pulled from Rebrickable’s daily dataset. Once per day, a scheduled task downloads the latest Rebrickable dataset and adds the new entries to my database. MSRP is scraped from the Lego website and the BestBuy API. BestBuy just happens to have the best API of all the retailers. And it’s FREE!

Another page in the application displays Ebay buy-it-now, free shipping listings. Due to inconsistency with Ebay data, these prices are not tracked in my database. Ebay deals are displayed on a totally separate page from the retailer listings; however, the Ebay page can be filtered by the same criteria.

The backend, dotnet API and Angular application are hosted on a local IIS server. Several scheduled tasks, like MSRP updates, Rebrickable datasets, alerts, etc. run on the same server. The MySQL database is hosted on Bluehost.

Posted on

Scrape rendered HTML with .NET6 C#

using HtmlAgilityPack;
using OpenQA.Selenium;
using OpenQA.Selenium.Chrome;
using OpenQA.Selenium.PhantomJS;

namespace Christopher.Snay.Sample.Services.Scrapers
{
	internal class ChromeScraper : IChromeScraper
	{
		public HtmlDocument ScrapeHtml(Uri url)
		{
			return ScrapeHtml(url.ToString());
		}

		public HtmlDocument ScrapeHtml(string url)
		{
			HtmlDocument doc = new();

			using (PhantomJSDriverService driverService = PhantomJSDriverService.CreateDefaultService())
			{
				driverService.HideCommandPromptWindow = true;
				driverService.LoadImages = false;
				driverService.IgnoreSslErrors = true;

				driverService.Start();

				using IWebDriver driver = new ChromeDriver();

				driver.Navigate().GoToUrl(url);

				Thread.Sleep(3000);

				doc.LoadHtml(driver.PageSource);
			}

			return doc;
		}
	}

	public interface IChromeScraper
	{
		HtmlDocument ScrapeHtml(string url);
		HtmlDocument ScrapeHtml(Uri url);
	}
}

Requirements – The following executables must be in /bin

  • phantonjs.exe
  • chrome.exe
  • chromedriver.exe
  • + the chrome portable binaries directory, currently named \107.0.5304.88
<!-- To copy directly to bin without being placed in a sub-folder -->
<ItemGroup>
	<ContentWithTargetPath Include="Assets\phantomjs.exe">
		<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
		<TargetPath>phantomjs.exe</TargetPath>
	</ContentWithTargetPath>
	<ContentWithTargetPath Include="Assets\chrome.exe">
		<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
		<TargetPath>chrome.exe</TargetPath>
	</ContentWithTargetPath>
	<ContentWithTargetPath Include="Assets\chromedriver.exe">
		<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
		<TargetPath>chromedriver.exe</TargetPath>
	</ContentWithTargetPath>

	<None Include="Assets\phantomjs.exe" />
	<None Include="Assets\chrome.exe" />
	<None Include="Assets\chromedriver.exe" />
</ItemGroup>

I’ve used Chrome portable to avoid having to install Chrome. If Chrome is installed, the chrome.exe steps can probably be skipped.