Custom HttpClient Tool For Specific Requirements
Hi,
Today, we will talk about why and what kind of HttpClient tool we need, based on some of the requirements we need in our code development processes.
I am working in a cyber security company as a software architect. In this case, we use third-party API for calling clients to test their cyber security awareness. We call it vishing. Vishing is a social engineering attack where fraudsters manipulate victims into revealing confidential information (credentials, passwords, account numbers, proprietary data, and more) over phone calls. However, sometimes the 3rd party companies, who we work with cannot respond to our requests due to their workload, and this situation returns to us as a timeout error to us.
Goal: When we get an error we want to post data to the third-party API again, until not get an error form the API in some custom retry limit.
CustomWebClientConfig.cs: We need two simple parameters for our CustomHttpClient.
- Tries: How many times we have to try to post data to the third party api ?
- TimeOut: How many seconds we have to wait api response before return the TimeOut error ?
namespace CustomHTTPClient
{
public class CustomWebClientConfig
{
public int TimeOut { get; set; }
public int Tries { get; set; }
}
}CustomWebClient.cs: This is my custom WebClient class. I use it for my specific requests.
1-) Firstly CustomWebClient is inherited from WebClient. We will override “GetWebResponse()” and “GetWebRequest()” methods. We have three parameters. Timeout, Tries and Cookies. We talked about some of them but Cookies is new. CookieContainer is used for the client to log in to the application and maintain the cookie state for subsequent calls to the web application. CustomWebClient class has three constructors. This is the default constructor. “IOptionsSnapshot<CustomWebClientConfig>” is our WebClient config class. We will get all properties from appsettings.json
2-) These are second and third constructor of CustomWebClient.
- “CustomWebClient(IOptionsSnapshot<CustomWebClientConfig> config)”: We will get the “TimeOut” and “Tries” parameters from the config file. We will create this class instance with dependency injection. We will choose <scoped> as a class lifetime. So we will call this constructor whenever we create an instance of CustomWebClient. So we can change, WebClient working conditions in realtime by changing the config file. We added default Header parameters in this constructor. “user-agent” and “ContentType”. If we want, we can change them after the WeClient creation instance.
- Second constructor, has only timeOut parameter.
3-) The “GetWebRequest” method comes from the inherited WebClient class. We will override this method by our own business rules. We will call this method before sending a request to the Uri. “WebRequest” timeout is a very important property in our case. We will wait for the Response until the timeout property and if we don’t get a response we will repeat the same request. If we get an answer, we will put the cookies into the request’s cookieContainer.
4-) The “GetWebResponse” method comes from the inherited WebClient class too. We will override this method by our own business rules. in the try{} scope we will wait for a response, if our request expires, we will jump to the catch{} scope:
- “if (ex.Status == WebExceptionStatus.Timeout || ex.Status == WebExceptionStatus.ConnectFailure || ex.Status==WebExceptionStatus.ProtocolError)”: If we get a timeout error or url not found error we will continue our process, if not we will throw an exception.
. “if ( — Tries == 0)throw;”: If we get an expire error from the request, we will check the “Tries” property and if we don’t come to the limit, we will try again and call the GetWebResponse() method with the same request. Recursively we will send the same request to the same URL, if we get an expire error until the retry limit.
- “GetResponseStr()” is bonus method :) If you want, you can check manually whether the Url exists or not. If there is no URL like this, we will throw an exception.
- “if (ex.Status == WebExceptionStatus.ProtocolError) throw; return GetWebResponse(request);”: If we get a “ProtocolError” exception, this means there is no Url like this. So we will not try again and throw an exception. If not will try calling the “GetWebResponse()” method again with the same request.
5-) These are common methods.
- “ReadCookies”: If we get a Response from the request, we will save the response’s cookies into our CookieContainer. If we need to, we will use these cookies for subsequent calls to the web application.
- “GetResponseStr”: Not mandatory but if not satisfied with the “Protocol Error” exception, we can check whether the URL is exists or not with this method.
CustomWebClient.cs:
using Microsoft.Extensions.Options;
using System.ComponentModel;
using System.Net;
using System.Text;
using System.Threading;
namespace CustomHTTPClient
{
public class CustomWebClient : WebClient
{
public CookieContainer Cookies { get; }
public int Timeout { get; set; }
public int Tries { get; set; }
//Default Constructor
public CustomWebClient() : this(100)
{
}
public readonly IOptionsSnapshot<CustomWebClientConfig> _config;
//1-) Get Property From Config Constructor
public CustomWebClient(IOptionsSnapshot<CustomWebClientConfig> config) : this(100)
{
_config = config;
Timeout = config.Value.TimeOut;
Tries = config.Value.Tries;
Cookies = new CookieContainer();
this.Headers.Add("user-agent", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; .NET CLR 1.0.3705;)");
this.Headers.Add(HttpRequestHeader.ContentType, "application/json");
}
//2-) Get Property From Parameter Constructor
public CustomWebClient(int timeOut)
{
Timeout = timeOut;
Cookies = new CookieContainer();
Encoding = Encoding.UTF8;
}
//With CookieContainer
protected override WebRequest GetWebRequest(Uri uri)
{
WebRequest w = base.GetWebRequest(uri);
w.Timeout = Timeout * 1000;
var request = w as HttpWebRequest;
if (request != null)
{
request.CookieContainer = Cookies;
}
return request;
}
//With CookieContainer
protected override WebResponse GetWebResponse(WebRequest request)
{
try
{
WebResponse response = base.GetWebResponse(request);
ReadCookies(response);
return response;
}
catch (WebException ex)
{
if (ex.Status == WebExceptionStatus.Timeout || ex.Status == WebExceptionStatus.ConnectFailure || ex.Status==WebExceptionStatus.ProtocolError)
{
if (--Tries == 0)
throw;
//For NonExist Urls
/*var htmlStr = GetResponseStr((HttpWebRequest)request).Result;
if (!string.IsNullOrEmpty(htmlStr))
{
return GetWebResponse(request);
}
throw;*/
//---------------------------------------------
if (ex.Status == WebExceptionStatus.ProtocolError) throw;
return GetWebResponse(request);
}
else
{
throw;
}
}
}
private void ReadCookies(WebResponse r)
{
var response = r as HttpWebResponse;
if (response != null)
{
CookieCollection cookies = response.Cookies;
Cookies.Add(cookies);
}
}
//IS URL EXIST BONUS METHOD
private async Task<string> GetResponseStr(HttpWebRequest request)
{
var final_response = string.Empty;
try
{
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
StreamReader stream = new StreamReader(response.GetResponseStream());
final_response = stream.ReadToEnd();
}
catch (Exception ex)
{
//DO whatever necessary like log or sending email to notify you
}
return final_response;
}
}
}Let’s Test Our CustomWebClient
Firstly create Asp.Net Web API project.
- Add this section to the appsettings.json: We will set 3sec for the timeout value. And if we get expire error we will try 5 times.
"CustomWebClientConfig": {
"timeOut": 3,
"tries": 5
},- Add CustomWebClient class to the project. And add below codes to the “GetWebResponse()” => catch{} section. I will explain this part later.
if (Tries == 1)
{
request.Timeout = 5000;
}- program.cs: Add “CustomWebClient” tool to the project with “CustomWebClientConfig”. We add this class with “Transient” service lifetime. This means, that when we change the config, our business can be changed in real-time. For example, if we change the timeout parameter from 10sec to 5sec we wait less, for a response from a third-party URL. So we can raise application performance in real-time if we want.
//1-)Get Property From Config Constructor - RealTime
var configuration = new ConfigurationBuilder()
.SetBasePath(Directory.GetCurrentDirectory())
.AddJsonFile("appsettings.json", false, true)
.Build();
builder.Services.Configure<CustomWebClientConfig>(configuration.GetSection("CustomWebClientConfig"));
builder.Services.AddTransient<CustomWebClient>();- We will Create ExchangeController and add DummyData() [HttpPost] End Point. This is our fake third-party URL. We will send a request to this URL.
-” Thread.Sleep(3000);”: We will deliberately wait here for 3 seconds. The purpose is to test CustomWebClient expiration time and retry features.
-”jsonData”: We will return custom json data. - “Data”: We will wait for only two parameters from the third-party URL(https://localhost:7053/Exchange/dummyData). “id” and “Name”.
ExchangeController/DummyData():
[HttpPost]
[Route("dummyData")]
public JsonResult DummyData([FromBody]Data data)
{
Thread.Sleep(3000);
var jsonData = new
{
name = data.Name,
surname = "kasmer",
Id = data.Id,
isMale = true
};
return new JsonResult(jsonData);
}
}
public class Data
{
public int Id { get; set; }
public string Name { get; set; }
}ExchangeController/Post():
- “ public ExchangeController(CustomWebClient _client)”: We will get “CustomWebClient” on “ExchangeController”s Constructor with dependency injection.
- “client.BaseAddress”: This is our fake Url’s root address. And “url” is our service URL.
- “Post([FromBody] Data data)”: We will get a request for this Action with Swagger. We will send dummy Data as a parameter.
- “ client.BaseAddress = “https://localhost:7053"; var url= “Exchange/dummyData”;”: These are third-party’s URL.
- “string postData = JsonConvert.SerializeObject(data);”: We will serialize data and convert it to string.
- “string response = client.UploadString(URL, postData);”: We will use our “CustomWebClient” and post data to our fake “dummyData“ endpoint.
- “var result = JsonConvert.DeserializeObject<Data>(response); return Ok(result);”: If we get a result, we will return OkObjectResult.
[ApiController]
[Route("[controller]")]
public class ExchangeController : ControllerBase
{
CustomWebClient client;
public ExchangeController(CustomWebClient _client)
{
client = _client;
}
[HttpPost("GetExchange")]
public IActionResult Post([FromBody] Data data)
{
client.BaseAddress = "https://localhost:7053";
var url= "Exchange/dummyData";
string postData = JsonConvert.SerializeObject(data);
string response = client.UploadString(url, postData);
var result = JsonConvert.DeserializeObject<Data>(response);
return Ok(result);
}
.
.
.DEMO
We will test our WebClient Retry and TimeOut properties.
- “CustomWebClientConfig”: { “timeOut”: 3, “tries”: 5 }”: We will set CustomWebClient’s timeout parameter as 3sec and try 5 times on appsettings.json.
- “Thread.Sleep(3000)” : We will wait 3sec on “DummyData()” Action. So our CutomWebClient will throw an exception because of timeout.
3. “ if ( — Tries == 0)throw; if (Tries == 1){request.Timeout = 5000;} ”: When we get the “TimeOut” error we will retry until the tries count is 1. When we come to the last try, we will set CustomWebClient’s timeOut parameter as 5000(5sec). So when we wait 3sec in DummyData() Action, we won’t get the TimeOut error. Because after all CustomWebClient could wait until 5sec. And we will get OkObjectResult. Thus, we will manage to change the CustomWebClinet’s timeOut in real-time.
CustomWebClient/GetWebResponse():
protected override WebResponse GetWebResponse(WebRequest request)
{
.
.
.
if (ex.Status == WebExceptionStatus.Timeout || ex.Status == WebExceptionStatus.ConnectFailure || ex.Status==WebExceptionStatus.ProtocolError)
{
if (--Tries == 0)
throw;
//For Testing Added This Line
if (Tries == 1)
{
request.Timeout = 5000;
}
//----------------------------
.
.
.
Conclusion:
Finally, we have a custom WebClint tool that requests the third-party URL and waits until what we set to the timeout parameter and repeats all the processes if receive a timeout error until we set to the Tries count parameter. If you need such a tool in similar use cases, you can use it by modifying it as you wish.
See you until the next article.
“If you have read so far, first of all, thank you for your patience and support. I welcome all of you to my blog for more!”
