Parse WebSite With Go for Processing the Azure .Net Core Web Service Result in Microservice

10 min readSep 18, 2020

This article will perform some operations on the data of products sold on an online shopping site and save them to the DB.

“Getting information off the internet is like taking a drink from a firehose.” — Mitch Kapor

We will get new product data with different currency prices by using .Net Core web API. But we will convert all product prices to ₺ currency and save on to the SqlDB. So we will parse a webpage and get Dolar, Euro, and British Pound currency value from the HTML and convert it to the ₺ Turkish Liras with Go lang. For all these processes, we will use Microservices. For improving the performance, we will use the Redis Cache. If there is no any Currency data in Redis, we will put all product data that we get from the .Net Core service to the RabbbitMQ. After all, we will get every product data one by one from the consumer and convert all product’s price currency to ₺ Turkish Liras with Go lang. We will save current currency data to Redis for one minute and insert all product data with the created date to the SQL DB.

Demo Project Video

1-).Net Core Product Web API Service

Create a WebApi “GoService”:

Add RabbitMQ and Newtonsoft package.

dotnet add package RabbitMQ.Client
dotnet add package Newtonsoft.Json
dotnet add package Swashbuckle.AspNetCore
dotnet add package Swashbuckle.AspNetCore.Annotations

Models/Exchange.cs: This is sold product data model.

public class Exchnage
{
   public double Price { get; set; }
   public string Name { get; set; }
   public ExchangeType ExchageType { get; set; }
   public string ExchangeName { get; set; }
}

ExchangeController()

With this Insert() Action, we will get Exchange data, connect to local RabbitMQ with 78.217. This is my modem IP. And we will put it on the one channel. We will use the “test” for the username and password from the 1881 port. We will connect to RabbitMQ like this because we will try to reach my local RabbitMQ from the remote Azure server. Because in the end, we will publish this .Net Core Web Service to the Azure.

We will put the data to the RabbitMQ because we will perform some operations on the data, and it takes so long. So this process must work behind the current operations. We will create a “product” channel on RabbitMQ and put this data on it to takes and process it by the consumer. “[SwaggerOperation(Summary = “ExchangeType is an Enum.”: We declare columns of Exchange parameter with this SwaggerOperation Attribute[].

Controller/ExchangeController.cs:

Let’s declare to config on Startup.cs:

Startup.cs: We will declare the swagger document for this service. So we add service.AddSwaggerGen() and we will add swagger declaration with c.EnableAnnotations() method. The swagger declaration is soo important. In this application, we declare Enum values of ExchangeType like above for the Insert() method.

2-) Go Web Parser

webParserGo.go: This is our main page. Firstly we will get all the products list from the SQL. So we will create a “sql.go” file and import it to “webParser.go.” We will download the “mssqldb” library from GitHub as below.

go get github.com/denisenkom/go-mssqldb

“BIG problems are best solved in small pieces.”
— Frank Sonnenberg

The packaging is everything for the GoLang. As the clean code said, we must distribute the packages by its duty with folders for readability, test, and debugging.

3-) Sql.go:

We will take the product from the queue, convert the price to Turkish Liras ₺, and save it to the SQL server.

“SqlOpen()”: We will connect to the SQL server by its IP with a shared go file. We will open and return it with its reference.

“GetSqlContent()”: We will get Product list from the DB. We will get all properties of product as an array. Name[], Price[], ExchangeType[], ExchangeValue[].
. “ctx, cancel := context.WithTimeout(context.Background(), time.Minute)”: This is the one minute Time out of the SQL execution query.
.”defer cancel() & defer stmt.close()”: It is so important for memory management.
“db.QueryContext(ctx,“select Name, Price..”)”: This is the execution of the SQL query.
. “for rows.Next() { err := rows.Scan(&_name, &_price, &_trPrice, &_exchangeType.. “: We’ll go through all the records one by one, and we will inject every property of the Product to parameters with their reference.
.”Name = append(Name, _name) Price = append(Price, _price)..”: We will append every parameter to their arrays.
.”return Name, Price, TrPrice, ExchangeType, ExchangeValue, nil”: We will return all arrays with no errors.
“InsertSqlContent()” : We will insert products, which is coming from the RabbitMQ to SQL DB.

“db.Prepare” is a prepared statement that is bound to a single database connection. The typical flow is that the client sends a SQL statement with placeholders to the server for preparation; the server responds with a statement ID. In this example, we will prepare an SQL query with Products properties placeholder with “@p1,@p2..” And finally, we will return inserted @@Identity “SCOPE_IDENTITY()”

stmt, err := db.Prepare("INSERT INTO Products2(Name,Price,TrPrice,IsActive,ExchangeType,ExchangeValue) VALUES (@p1, @p2,@p3,@p4,@p5,@p6); select ID = convert(bigint, SCOPE_IDENTITY())")

. “context.WithTimeout()”: We will set a one-minute timeout for SQL query.
.”defer cancel() & defer stmt.close()”: It is so important for memory management.
.”rows := stmt.QueryRowContext(ctx, product.Name, product.Price,..”: We will set the insert product parameter and get inserted ProductID(@@identity).

webParserGo.go(Sql):

For using Sql in Go, we will get belowe library.

go get github.com/denisenkom/go-mssqldb

We will open the SQL-DB as below. And with the “defer” keyword, we will close it when everything is finished about this DB.

“names, prices, trPrices, exchangeType, exchangeValue, err :=sql.GetSqlContent(db)”: We will get all product data from SqlDB and match them with the arrays.
“for i := range names { fmt.Println(“Product “ + strconv.Itoa(i)..”: We will print all product properties array to the screen one by one.
“rabbitMQ.ConsumeRabbitMQ(db)”: We will start to listen to the rabbitMQ for the inserted product.

4-) shared.go:

This file is used for the global declaration.

Configuration struct: We will declare Sql and RabbitMQ connection strings with Configuration struct.
AddProduct struct: This is our Product Model, which we will get from RabbitMQ and insert to SQL.
exchangeType struct: We will use this struct as an Enum. It is the ExchangeType property of the AddProduct model.
newExchangeType(): This is the constructor of exchangeType struct.

“Your website is the window of your business. Keep it fresh, keep it exciting.” —Jay Conrad Levinson

5-)parser.go :

We will parse “https://doviz.com” for getting exchanges value because there is no endpoint to calculate the Turkish lira equivalent of the incoming product price.

We will use “PuerkitBio/goquery” for the parsing a website.

go get github.com/PuerkitoBio/goquery

“res, err := client.Get(“https://www.doviz.com")”: We will get the “doviz.com” website HTML.
“doc, err := goquery.NewDocumentFromReader(res.Body)”: We will get the document for filtering the HTML by its CSS and “html-elemen type.”

data := doc.Find(".market-data .item") //We will get all the exchange value by its CSS as above.

“exchangeList = make(map[string]string, data.Length())”: This is our dictionary to keep all exchange value by its name.
“data.Each(func(i int, s *goquery.Selection) {“: We’ll go through all ”div” elements, which we parsed from the web page one by one, and filter the exchange key and value from in it.
We will filter Exchange name by its “anchor” type and “name” and “value” CSS.

name := s.Find("a .name").Text()
name = strings.Replace(name, " ", "", -1) //Gram Altın==>GramAltın
kur := s.Find("a .value").Text()

“exchangeList[name] = kur”: We will insert the parsed exchange value to the map list with its key.
“func getExchangeValueByType(key string)”: We will get an exchange value by its name.

6-) redis.go:

We will not parse the web page for every product insert. It is insane and not safe for performance. So what will we do? We will keep the above exchangeList into the Redis for one minute.

We will use go-redis/redis library for using Redis in Go.

go get github.com/go-redis/redis/

“type redisClient struct { c *redis.Client }”: This is our Redis client struct.
“var client = &redisClient{}”: We will get redisClient with reference.
“var ctx = context.Background()”: Ctx is used by the main function, initialization, tests, and as the top-level Context for incoming requests. We will call the ping command to Redis for testing.
“ExchangeRedisValue struct“: It is the data model of parsed data from the web.
“func GetRedisClient() *redisClient {“ : This is the connected Redis client. “localhost:6379” is the local Redis server. But it could be the external server.
“func (client *redisClient) SetKey and GetKey” : We will set or get exchange value by its key from the redis. We will encode and decode data by using ”encoding/json” library. => “json.Marshal() & json.Unmarshal()”

7-)rabbitMQ/consumer.go:

All main processes are calling from here. This is our microservice. We will use “streadway/amqp” for the consuming data from the RabbitMQ. And we will import all packages like “redis,sql,parser,shared.”

go get github.com/streadway/amqp

“ConsumeRabbitMQ(db *sql.DB) {“: This function will listen to the product channel on RabbitMQ.
“queue, err := amqpChannel.QueueDeclare(“product”, false, false, false, false, nil)” : We will declare the product channel, which we will listen to from rabbitMQ for an inserted product.
“messageChannel, err := amqpChannel.Consume(”: We will create the product channel.
We will listen to the “product” channel with the “go func()” command, which executes in parallel with the main Go thread. And we will use the “stopChan” channel for breaking the loop.

stopChan := make(chan bool)
go func() { ...

“for d := range messageChannel {“: is the infinity loop for listening to the message channel.
“addProduct := &shared2.AddProduct{}”: This is our product model.
“err := json.Unmarshal(d.Body, addProduct)”:We will deserialize the new coming product package to addProduct empty model.
“var redisClient = redis.GetRedisClient()”: We will get Redis client with all configuration.
“err2 := redisClient.GetKey(addProduct.ExchangeName, exchangeRedisVal)”: We will try to get the current Exchange value from the cache.
If Redis is null, we will parse to “doviz.com” for getting the current exchange value.

if exchangeRedisVal.Name == "" { 
   exchange = parser.ParseWeb(addProduct.ExchangeName)

“redisClient.SetKey(addProduct.ExchangeName, exchangeRedisVal, time.Minute*1)”: We will save the exchange value to the Redis for one minute.
“addProduct.TrPrice = addProduct.Price * exchangeValue”: We will convert the product price to Turkish Lira ₺.
We will insert the product into SQL.

res, err2 := sql2.InsertSqlContent(db, addProduct)

We will get the final all product data from the SQL.

names, prices, trPrices, exchangeType, exchangeVal, err := sql2.GetSqlContent(db)

8-) Let’s publish .Net Core Service to Azure

We will publish the .Net Core WebService to the cloud, and we will call swagger from the Azure.

We will create a new App Service on Azure. We will go to http://portal.azure.com on the browser. We will set Subscription, Resource Group, which we created before, Web App unique name (rmqservices), Running platform(.Net Core3.1), and finally, Region.

The next step is downloading the publish profile. We will use this profile for publishing the .Net Core service to the Azure on Visual Studio. And that’s all.

We will right-click the project and click the Publish item. We will select publish Profile, and finally, we will Import Profile, which we downloaded from the Azure. After all, we will publish the .Net Core project to Azure “rmqservices.azurewebsites.net.”

9-)Final Setup is Modem Wan Port Forwarding

This beloved’s code is Azure .Net Core RabbitMQ config. The hostName is my fake modem IP, and the external Port is 1881.

var factory = new ConnectionFactory()
{
    HostName = 199.59.148.82,
    UserName = "test",
    Password = "test",
    Port = 1881,
    VirtualHost = "/",
};

We have to make some arrangements for our modem for redirection from our modem to the Local RabbitMQ machine port. When we insert a product with the swagger on Azure .Net Core WebApi service, this package will come to my ModemIP and 1881 port. We will redirect to this request to 192.168.1.1, which is RabbitMQ Pc IP and, of course, port 5672, which is RabbitMQ port as bellow picture. After all, this inserted product data will subscribe to the product channel on RabbitMQ. And local Go service will consume it and start all the process. All Setup is done.

Conclusion:

In this application, we post new product data from .Net Core WebService with swagger. All new products’ currency could be changed every request. And when we insert every product, we convert the product’s price to Turkish price currency. There is no service for the getting exchange rate while converting the Product’s Price to Turkish currency. So we parsed an exchange web site for getting rates. We used microservices technologies because of this long process. Microservice allows us to work together with different technologies. Go is a powerful programming language. The packaging is so important for clean code in Golang. So We distributed all packages according to their duty. Redis, SQL, Parser, Shared all the packages are used in the “consumer.go” file. We published the .Net Core Project to Azure and redirected from the external 1881 port to the internal local RabbitMQ machine with 5672 port on the local Modem Port Forwarding setup. So we could process locally consumed product data in Go, which is subscribed by remote Azure Service.

I hope this article has given you a different perspective on how different technologies work together.

“If you have read so far, first of all, thank you for your patience and support. I welcome all of you to my blog for more!”

Source: kb.objectrocket.com, https://github.com/masnun/gopher-and-rabbit, jenicaandpatrick.com, bsilverstrim.blogspot.com, github.com/PuerkitoBio/goquery

Source Code: https://github.com/borakasmer/GoWebParser