«

»

Oct 08

MyBot Tutorial Part 2

Here is the second part of the tutorial.

In this tutorial I will explain two methods for sending http requests. A basic method that is great to use for the first time and to do some tests, but it has a disadvantage. The other method is more advanced but that shouldn’t be a problem after the tutorial.

I have made a project that uses all the methods that I’m going to explain, you can download it here.

Let’s start by opening the file HttpHandler.cs. The methods that we are going to use for the http requests are from the class WebClient from the .Net library. But the limitation of this class is that is doesn’t handle cookies well.
To solve this we use the custom class HttpHandler that is derived from WebClient and add our own CookieContainer. This way the cookies are updated with every request that we make.

I have also added an other method to this class called clearCookies() but will explain that in part 3 of the tutorial.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Net;
using System.Collections.Specialized;
 
namespace MyBot_Tutorial
{
    class HttpHandler : WebClient
    {
        private CookieContainer m_container = new CookieContainer();
 
        protected override WebRequest GetWebRequest(Uri address)
        {
            WebRequest request = base.GetWebRequest(address);
            if (request is HttpWebRequest)
            {
                (request as HttpWebRequest).CookieContainer = m_container;
            }
            return request;
        }
 
        public void clearCookies()
        {
            m_container = new CookieContainer();
        }
    }
}

Now we can finally start with the requests. Let’s open MyBot_Tutorial.cs.

Basic

As I said before, the basic method has a disadvantage. The methods defaultGetRequest() and defaultPostRequest() are both blocking. This means that when you request a webpage or file your program will freeze till the request is completed. When you’re learning http requests or just want to try a few things it’s not really a problem but when you want to make a complete program with GUI it can be very irritating when your interface stops responding during every request.

I’ll start with the method defaultGetRequest(). First you need to know the URL you want to request. You can find the correct URL with the addon Live HTTP headers explained in part 1 of the tutorial. After that you make a variable of type Uri in which you save the URL. Finally you can make the HTTP request by calling the method DownloadString(Uri) from the HttpHandler class.
The method DownloadString(Uri) will return the response from the server in a String, which is in most cases the most useful type. You can however also choose to save it in a file or in a byte array, use then DownloadFile(Uri) or DownloadData(Uri).
It’s important that you don’t forget the try-catch statement. It will catch any webexceptions thrown by DownloadString(Uri), like invalid hostnames and problems with downloading the requested webpage.

An example of the basic POST request is the method defaultPostRequest(). The URL part of this method is exactly the same as the GET request but now we have to add content before making the request. To add the content you have to make a new variable of the type NameValueCollection. In my example I have called this variable l_Content.
The content exist of a name-value pair as explained in part 1 of the tutorial. You can add as many pairs as you want by calling

l_Content.Add(name,value).

The parameters name and value must be strings. When you have added all your name-value pairs you are ready to make the request by calling

UploadValues(l_Uri,l_Content).

Please note that this method doesn’t return a String but a Byte array. With the following code you can convert it to a string:

Encoding.Default.GetString(l_Response);

You can also find this in the example below.

/*
 * This is a standard method for making http get requests.
 * This method is blocking
 */
private void defaultGetRequest()
{
    try
    {
        String l_Url = "http://bots.uthar.nl/files/MyBotTutorial.html";
        Uri l_Uri = new Uri(l_Url);
        String l_Response = m_HttpHandler.DownloadString(l_Uri);
        textBoxResponse.Text = l_Response;
    }
    catch (Exception e)
    {
        textBoxResponse.Text = e.Message;
    }
}
 
/*
 * This is a standard method for making http post requests.
 * This method is blocking
 */
private void defaultPostRequest()
{
    try
    {
        NameValueCollection l_Content = new NameValueCollection();
        String l_Url = "http://bots.uthar.nl/files/MyBotTutorial.php";
        Uri l_Uri = new Uri(l_Url);
        //You can enter as many name/value pairs as you want.
        //In this example I will use username and password
        l_Content.Add("username", "Uthar");
        l_Content.Add("password", "123");
        byte[] l_Response = m_HttpHandler.UploadValues(l_Uri,l_Content);
        //The method UploadValues() returns a byte[]. With the following line we can convert it to a string
        textBoxResponse.Text = Encoding.Default.GetString(l_Response);
    }
    catch (Exception e)
    {
        textBoxResponse.Text = e.Message;
    }
}

This were the two basic methods. If you have any questions or want a part explained in more detail let me know in the comments.

Advanced

The advanced methods look almost exactly the same as the basic version. But the difference is that this method is non-blocking. This means that your program doesn’t freeze during a request and that you still can use your GUI without any problem. To get this working we have to make a few other methods besides the two below.

/*
 * This is a more advanced method for making http get requests.
 * This method is non-blocking
 */
private void advancedGetRequest()
{
    //There is no need for try/catch here. You need to catch exceptions with this method in the event handler.
    String l_Url = "http://bots.uthar.nl/files/MyBotTutorial.html";
    Uri l_Uri = new Uri(l_Url);
    m_HttpHandler.DownloadStringAsync(l_Uri);
}
 
/*
 * This is a more advanced method for making http post requests.
 * This method is non-blocking
 */
private void advancedPostRequest()
{
    //There is no need for try/catch here. You need to catch exceptions with this method in the event handler.
    NameValueCollection l_Content = new NameValueCollection();
    String l_Url = "http://bots.uthar.nl/files/MyBotTutorial.php";
    Uri l_Uri = new Uri(l_Url);
    //You can enter as many name/value pairs as you want.
    //In this example I will use username and password
    l_Content.Add("username", "Uthar");
    l_Content.Add("password", "123");
    m_HttpHandler.UploadValuesAsync(l_Uri, l_Content);
}

Maybe you have noticed that the advanced methods advancedGetRequest() and advancedPostRequest() don’t receive the response from the server. This is because we now use:
DownloadStringAsync(Uri), UploadValuesAsync(l_Uri,l_Content)
instead of
DownloadString(Uri), UploadValues(l_Uri,l_Content).
The two methods that end with Async don’t return a response directly but will trigger an event that will let us know when the response from the server is ready. To catch this event we have to create an event handler.

The first thing we need to do is link the event with an event handler. This is shown in the example below.

/*
 * This will add an event handler for the async http requests
 * It is only used for advancedGetRequest() and advancedPostRequest()
 */
private void initEvents()
{
    m_HttpHandler.UploadValuesCompleted += new System.Net.UploadValuesCompletedEventHandler(client_UploadValuesCompleted);
    m_HttpHandler.DownloadStringCompleted += new System.Net.DownloadStringCompletedEventHandler(client_DownloadStringCompleted);
}

The first part is the name of the event:
m_HttpHandler.UploadValuesCompleted
The second part is the event handler:
System.Net.UploadValuesCompletedEventHandler(client_UploadValuesCompleted)

The parameter of the second part, client_UploadValuesCompleted, is the name of the method we want to use for the event handler.

The two methods below are used as the event handlers and will be called automatic when the response from the server is ready. client_UploadValuesCompleted will be called when a response from a POST request is ready and client_DownloadStringCompleted when the response is from a GET request.

/*
 * This event handler will catch everything you make with DownloadStringAsync
 */
void client_DownloadStringCompleted(object sender, System.Net.DownloadStringCompletedEventArgs e)
{
    //By checking e.Error you can catch all exceptions. e.Error equals null when there aren't any exceptions.
    if (e.Error == null)
    {
        String l_Response = e.Result;
        textBoxResponse.Text = l_Response;
    }
    else
    {
        textBoxResponse.Text = e.Error.Message;
    }
}
 
/*
 * This event handler will catch everything you make with UploadValuesAsync
 */
void client_UploadValuesCompleted(object sender, System.Net.UploadValuesCompletedEventArgs e)
{
    //By checking e.Error you can catch all exceptions. e.Error equals null when there aren't any exceptions.
    if (e.Error == null)
    {
        //The method UploadValuesAsync() returns a byte[]. With the following line we can convert it to a string
        String l_Response = Encoding.Default.GetString(e.Result);
        textBoxResponse.Text = l_Response;
    }
    else
    {
        textBoxResponse.Text = e.Error.Message;
    }
}

An other difference between the basic methods is that advancedGetRequest() and advancedPostRequest() don’t have a try-catch statement to catch webexceptions. This is because the try-catch statement is moved to the event handlers, as you see in the example above. To be precise, it’s not exactly the same as a try-catch statement but with the following if-else statement you get the same result.

if (e.Error == null)
{
   //Do something
}
else
{
   //Log error message
}

If everything went well you are now able to send request and receive the response from the server, either with the basic or the advanced method.

Uhm, and now?

Now that you successfully can request any webpage and receive the response you have all the information to make a bot. The response you get from the server is the html code from the webpage you requested. In here you will find information about incoming attack, building levels, resources, units or whatever you want 🙂

If you have any questions, suggestions or want something explained in more detail let me know in the comments.

105 comments

1 ping

Skip to comment form

  1. woodz

    ok, so the first part was easy, i got that. but with the second part i couldn’t follow, it was going a bit to fast. could you give me a version of part 2 with extra info (what is what in the pictures, when to use, where to use, …). you could post it on a new webpage or something

  2. Uthar

    Hey woodz could you give me more details on what you want to know?
    Because for every method used I have written what it does and were you need it.
    Also you can download the complete solution for Visual Studio to get a better understanding of the location of each method.

  3. backtorealites

    Hello Uthar,
    Are you gonna complete your Bot Tuto Part 3? I’m now studying VB at university and gonna complete my class by the end of April 2011. Plan to work on a bot right after. Hope to be able to follow your teachings right after.

    Thanks and take care!

  4. Uthar

    I’ll probably do that some day but not any time soon. Currently busy enough with rewriting the Grepolis bot for version 2.

  5. Anonymous

    I really admire their projects. friend, could show the code name and password.
    to connect to the server Grepolis.

  6. Uthar

    @Anonymous
    You can get your own username/password by registering on the Grepolis website. If you want specific code to connect to the Grepolis server you can download the source code of the Grepolis or Grepolis 2 bot. (See download section)
    Look for a method called login()

  7. mhtsaras2

    when will you release the third part ???

  8. Uthar

    I’ll start with that in the summer vacation. You can expect a first version at the end of July.

  9. Uthar

    The third part is finally ready. Took a bit longer then expected. Was too busy with the Grepolis 2 bot.

  10. Mee

    Hey Uthar, im looking to make a LITTLE bot for a game, probably REALLY easy for someone with your knowledge.. but for me is hard even understand this tutos. =(
    Is there any way i can contact you, so u can guide me a bit on how to do?.
    The bot will be REALLY easy, it just must do one thing in a game (not grepolis).
    Can you please help me? =)
    Thx =D

    1. Uthar

      Sure, I’ll send you an email. Don’t expect me to send working parts of code that you can just copy/paste. But I can help you with questions you get during the development and give you some tips.

      1. Mee

        I DONT want u to do it for me, i want to finally learn and do something by my self =D, but seems the game im talking about its different (seems harder) and im lost =P

        1. Uthar

          Good to hear that! I just wanted to let you know in advance so that you knew what to expect.
          I just answered your email, seems you’re already making progress. If you have more questions, you know how to contact me.

  11. CuriousTinker

    Hi Uthar,
    Awesome skills, and very helpful!

    I am going through your GrepolisBot2 code, and trying to apply this tutorial to it for the login method. I can’t seem to figure out how to login using the HttpHandler. If you have any time, I would really appreciate anything you can point out.
    I call this:
    NameValueCollection l_Content = new NameValueCollection();
    String l_Url = "http://" + this.Config.Server + "/start?action=login_to_game_world";
    Uri l_Uri = new Uri(l_Url);
    l_Content.Add("world", this.Config.World);
    l_Content.Add("facebook_session", "");
    l_Content.Add("facebook_login", "");
    l_Content.Add("gift_key", "");
    l_Content.Add("name", this.Config.UserName);
    l_Content.Add("password", this.Config.UserPass);
    this.HttpHandler.UploadValuesAsync(l_Uri, l_Content);

    and use your provided client_UploadValuesCompleted code, but the response appears to just be the login/home pages again? Can this be done? Why did you opt for navigating while sending post variables in your GrepolisBot code instead of the http headers?

  12. Uthar

    You can use your idea to get it working but you need to add a few requests.
    For the login process to work you need to visit the start page first and login to the select world screen by using:

    http://en.grepolis.com
    http://en.grepolis.com/start/index?action=login_from_start_page

    After that you can login to:

    http://en.grepolis.com/start?action=login_to_game_world

    But when you request that page in your browser you’ll notice that it will automatic make two other requests after that:

    http://en27.grepolis.com/game/login?session_id=....
    http://en27.grepolis.com/game/index?login=1

    As you said I’m using the Navigate method:

    webBrowserGrepo.Navigate(l_Uri, "", l_PostDataBytes, "Content-Type: application/x-www-form-urlencoded");

    The advantage of this is that when you call:

    http://en.grepolis.com/start?action=login_to_game_world

    It will also automatically call:

    http://en27.grepolis.com/game/login?session_id=....
    http://en27.grepolis.com/game/index?login=1

    If you want to use your method you need to call those last two manual.

    1. CuriousTinker

      I couldn’t get that to work, but I have found a way to login using Javascript. The problem is, I’m trying to do it using WPF and not window forms, so there are a few things that are different.

      I am now stuck on locating the farmers locateFarmers() method. The source you have available is not your most up to date source, and has the following:
      NameValueCollection l_Content = new NameValueCollection();
      string l_Url = "http://" + l_Settings.GenServer + "/game/data?action=get&town_id=" + m_Player.Towns[m_CurrentTownIntern].TownID + "&h=" + m_H;
      Uri l_Uri = new Uri(l_Url);
      l_Content.Add("json", "{\"type\":\"map\"}");//json {"type":"map"}
      m_HttpHandler.Headers.Add("X-Requested-With", "XMLHttpRequest");
      m_HttpHandler.UploadValuesAsync(l_Uri, l_Content);
      m_HttpHandler.Headers.Remove("X-Requested-With");

      But I can’t seem to get it to work. The request (tested today) that is sent, is not just
      json{"type":"map"}
      but
      json{"types":[{"type":"map","param":{"x":10,"y":7}},{"type":"bar"}]}

      It just returns:
      885
      {“error”:”An internal error has occurred!”,”t_token”:#####,”incoming_attack_count”:0,”notifications”:[],”next_fetch_in”:300}8|10
      _srvtime1334656104

      How do you troubleshoot something like that? Did you make a method in how to find the problem?

    2. CuriousTinker

      Sorry, I have determined that it occurs with all of my requests not just locateFarmers(). All responses return that. An error, but still an updated notifications list and whether I’m being conquered or not.

      1. Uthar

        To fix those errors you should compare the server requests the browser is sending with the ones that your program is sending. Not only the URL but also the headers.
        For example a lot of requests have the “X-Requested-With” header. If you don’t add that one the request is not going to work.

        The request for the locateFarmers() has changed some time ago.

        json{"type":"map"}
        or currently
        json{"types":[{"type":"map","param":{"x":10,"y":7}},{"type":"bar"}]}

        The response of that request no longer contains any information about the farmers. You should watch the server requests your browser is making for some time and look for a request that looks like:
        /game/map_data?action=get_chunks

        There you’ll find the farmers locations.

        1. CuriousTinker

          Argh, I really cannot get anything out of this. I do still get information from the original url, and can’t see that chunks one you mentioned? I have made all the headers the same, and still getting the same error. Do you know what the x and y params passed to the map function are? They keep changing, so I am sure I’m using the wrong one:

          NameValueCollection l_Content = new NameValueCollection();
          int seconds = (int)(DateTime.UtcNow - new DateTime(1970, 1, 1)).TotalSeconds;
          string l_Url = "http://" + l_Settings.Server + "/game/data?action=get&town_id=#####&h=" + this.H + "&_" + seconds.ToString();
          Uri l_Uri = new Uri(l_Url);
          l_Content.Add("json", "{"types":[{"type":"map","param":{"x":14,"y":5}},{"type":"bar"}]}");
          this._httpHandler.Headers.Add("Host", "en29.grepolis.com");
          this._httpHandler.Headers.Add("User-Agent", "Mozilla/5.0 (Windows NT 6.1; rv:9.0.1) Gecko/20100101 Firefox/9.0.1");
          this._httpHandler.Headers.Add("Accept", "text/plain, */*; q=0.01");
          this._httpHandler.Headers.Add("Accept-Language", "en-gb,en;q=0.5");
          this._httpHandler.Headers.Add("Accept-Encoding", "gzip, deflate");
          this._httpHandler.Headers.Add("Accept-Charset", "ISO-8859-1,utf-8;q=0.7,*;q=0.7");
          this._httpHandler.Headers.Add("DNT", "1");
          //this._httpHandler.Headers.Add("Connection", "keep-alive"); <-- This returns an error, because it can't be set here
          this._httpHandler.Headers.Add("X-Requested-With", "XMLHttpRequest");
          this._httpHandler.Headers.Add("Referer", "http://en29.grepolis.com/game/index?login=1");
          this._httpHandler.UploadValuesAsync(l_Uri, l_Content);

          I do thank you for your patience. Once I get it, I am happy to send you the code if you ever want to upgrade it to WPF.

          1. Uthar

            Not every header is needed but atleast you need to have “X-Requested-With”.
            Did you already compare the request that you send with the request the browser is sending? You could use http://fiddler2.com/fiddler2/ for that.

            /game/map_data?action=get_chunks is used when your press F5 or when you move the map around.
            Regarding the x and y values. This are coordinates of the world map. They change everytime you are at another Island.
            So if you know the coordinates of your current town then you can compute the x, y values you need to use.

            And thanks for the offer. It would be nice to look at your code when you have it working.

  13. slimwoolf

    Since yesterday the 29th of August ther semms to be a change in the login procedure.

    http://en.grepolis.com/start?action=login_to_game_world seems do fail.

    I wrote an own bot but cant find the reason for that .. do you have an idea?

    1. Uthar

      They have been doing server updates the last few days. It is possible that it’s just a temporarily problem. Did you already try to clear your cache? For some reason that is always causing problems by Grepolis.

      I haven’t changed my login function and it’s still working fine. My last login request is the same as yours.

      Is action=login_to_game_world the only request you use or do you have more? Because my login function has three requests. The first one to visit the login page, the second one to login to the server selection page and the last one to login to the preferred world.

  14. slimwoolf

    i have exactly these 3 requests you described. It worked well until yesterday noon.

    I recognized that the first 2 requests now needs to be send as XMLHttpRequest (otherwize the fail also) but the last request (login_to_game_world) fails anyway with the json parameter:

    Content.Add(“world”, Server);
    Content.Add(“facebook_session”, “”);
    Content.Add(“facebook_login”, “”);
    Content.Add(“portal_sid”, “”);
    Content.Add(“name”, Player.ConnectionConfig.UserName);
    Content.Add(“password”, Player.ConnectionConfig.Password);

    1. Uthar

      You’re missing the gift key parameter. That’s the only difference I see.

  15. dat

    Hi Uthar,
    Thank you for your detailed guide and your willingness to help other newbies. In my circumstance, I want to create a bot for flash web base game. Could you please tell me is there any way to simulate the game actions and events by HTTP request?

    In the tutors, you download the string, html source, from server. However, what methods or classes should I use in case of flash game? I think the browser can send and respond to the embedded flash, so we can do the same with webclient. I really need your ideas.

    Many thanks for your help!

    1. Uthar

      Flash games also send requests to the server but they are probably encrypted, which makes them useless.

      You could make a bot for a flash game with image recognition. Then you scan for a specific pixel color to find the elements you want to click on.

      Another possibility is memory editing. This method is popular for mmorpg bots but should work fine for flash games too. It’s however a lot harder and will cost a lot of time. I haven’t written any bots like this myself so I can’t help you with this.

      1. dat

        Thank Uthar,

        The second idea isn’t good at all because we the screen resolution may change from pc to pc. Also, we have to let the bot use the whole pc when it’s running.

        For the last approach, I agree it is quite complicated. I may look back later on. Anyway, I found that someone has created bots for flash games. I think they used the HTTP request. Let say there is no encrypted. Then how can we simulate the game actions?

        I always appreciate your ideas. Thanks Uthar… I’m always looking for someone who loves making bots.

        1. Uthar

          First about the image recognition. If you use a browser like http://awesomium.com/ you can make screenshots of the webpage while your bot is minimized. It is even possible to make a screenshot of the complete webpage when the browser has scrollbars (on low resolutions).
          As far as I know this is the only browser where this is possible.

          If you want to try the http requests it works like a normal game written in php. You use some monitor tool, do some actions, and then look at your monitor tool for the requests that were made.
          If you use firefox you can for example use Firebug to monitor the http requests or you can use an extern program like Fiddler.

  16. Stjepan

    Hi Uthar,

    Thanks for your code and effort. Great work!

    I’m a little confused when I sholud pass thru several pages
    For example:
    I go to b2b.business.com
    I enter username and password
    after that fiddler gives me:
    b2b.business.com/time.php
    b2b.business.com/2ndstep.php
    and finally b2b.business.com/login.php

    My question is, how I can include in your code those additional steps (time.php & 2ndstep.php)?
    These pages are just passthru pages (PAGE1 – enter UN&PWD …. PAGE4 – is what i want), where in code i should call these 2 pages?

    1. Uthar

      You could first try if you can ignore those pages. Just continue with b2b.business.com/login.php.
      If this doesn’t work you’re probably missing some cookies that were set in the requests you skipped. In that case you really need those requests.

      There are two solutions for this. First, you can just send the same requests as fiddler did. So goto b2b.business.com and after that send the requests for b2b.business.com/time.php and b2b.business.com/2ndstep.php. Check fiddler for more details on how the requests should look like.

      The second solution is to use a browser in your bot. If you navigate to b2b.business.com and fill in your username and password (You can do that with javascript) the webbrowser will automatic call the two other requests.
      After that you can copy the cookies from the webbrowser and use them in your other requests.

  17. Jorn

    Hi Uthar,

    It’s cool to see that you’re Dutch as well (I am from Overijssel, Zwolle :)). However, I will keep this e-mail in English so other’s might learn from this.

    I’ve been learning to program in C# (I’m already experienced in OO PHP so it’s going pretty easy). I have been trying to create a bot for Travian using HTTP Requests and have been using your code as examples. Everything has been working so far but I run into a problem:

    I set my bot to the login state, I login using webBrowser.Navigate including post-data. This all works fine – the cookie gets set, it actually logs in and it then collects some data from the next page using the _DocumentCompleted event.

    In the _DocumentCompleted event handler I set the bot state to “generalinfo” and after collecting the data I tell my program to call the method StateManager(“generalinfo”);. This is basically the same principal you use with your bots. In the StateManager it calls GeneralInfo().

    In GeneralInfo() it sets a new NameValueCollection (just like you do), sets the url/uri to basically the same URL still. I then use HttpHandler.DownloadDataAsync (since it’s a GET request) on the uri.

    It gets caught by the event handler _DownloadDataCompleted which turns the encoded e.Result into a string response (also, just like you do in your bots). It then calls the ResponseManager(response); method (which also should be familiar to you) and in turn calls the GeneralInfoResponse(response) method – as that is our current state still.

    Now the problem is – meanwhile the response has changed to the log-in page again. I checked the cookieContainer, but this still contained the right cookie information (identical). I tried out what I did using my own webbrowser but it doesn’t log me out.

    Do you know what could be wrong? Am I forgetting to set certain headers which makes me disconnect? I’m basically only calling a different URL, nothing else though.

    Any help would be appreciated!

    With kind regards,
    Jorn

    1. Uthar

      You should check if you need to add the xml request header. Not sure if Travian uses that but it could result in a problem similar to yours when it’s required by the game but missing in your requests.

      1. Jorn

        Thank you for the quick reply! I have looked at the headers again but I cannot find any special or other headers that might be required. I have also tried using the XML Request Header but have been unsucessful. Could it be that the cookie get’s changed after certain page requests? How would one go about updating the cookieContainer on every request to make sure it is up-to-date?

        Your help is appreciated! 🙂

        With kind regards,
        Jorn

        P.S. I have just checked the possibility of cookie changes but this seems to not be the case. When I do the same action within the webBrowser element and I keep checking the webBrowser.Document.Cookie data it does not change.

        1. Uthar

          The webbrowser and the HttpHandler are two separate objects. After you are logged in using the webbrowser you need to copy the cookies to the HttpHandler object (Not sure if you did that already).
          For all requests you make with the HttpHandler after this the cookies will be updated automatic.

          1. Jorn

            Yes I have it set-up exactly the way you describe. I have also figured out that there is another cookie being set. However this cookie is set by the response using a Set-Cookie header. Unfortunately the webBrowser element does not seem to capture this cookie. Do you know how I could achieve this? (Manually adding this cookie makes my bot actually log-in)

            Jorn

          2. Jorn

            Hey Uthar,

            Just wanted to let you – and everyone else – know that I have figured out the problem. Apparently Travian would set HTTPONLY cookies using the response Set-Cookie header. These are “invisible” to the webBrowser control. Instead of trying to rewrite the code to not use a webBrowser control and get these headers I decided to just create a new class as was shown on MSDN to read these cookies anyway. It works great!

            Jorn

          3. Uthar

            Good to know you have it working Jorn and thanks for the explanation.

  18. Mihail

    Hi Ulthar, I have a little problem with Sending a POST request looking like that:
    /This is from fiddler/
    POST http://www.aaa/login.php HTTP/1.1
    Connection: keep-alive
    Content-Length: 346
    Cache-Control: max-age=0
    Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
    Origin: http://www.aaa.com
    User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.22 (KHTML, like Gecko) Chrome/25.0.1364.152 Safari/537.22
    Content-Type: multipart/form-data; boundary=—-WebKitFormBoundaryDQIHTmQCRHdrOhni
    Referer: http://www.aaa.com/
    Accept-Encoding: gzip,deflate,sdch
    Accept-Language: bg-BG,bg;q=0.8
    Accept-Charset: windows-1251,utf-8;q=0.7,*;q=0.3
    Cookie: PHPSESSID=65d2dfccc86477166258f6314e7353e2

    ——WebKitFormBoundaryDQIHTmQCRHdrOhni
    Content-Disposition: form-data; name=”navaction”

    login
    ——WebKitFormBoundaryDQIHTmQCRHdrOhni
    Content-Disposition: form-data; name=”UserID”

    alopolo
    ——WebKitFormBoundaryDQIHTmQCRHdrOhni
    Content-Disposition: form-data; name=”Passwort”

    ******
    ——WebKitFormBoundaryDQIHTmQCRHdrOhni–

    Ok, I understand what to do with the haeder,but i dont know how to put the information from the Boundaries.Please help me if you can.
    Thanks in advance,Mihail

    1. Uthar

      That request doesn’t make it very clear what data you need to add in the POST request. You could try it with Firefox using the FireBug addon. It will give you a better view of the POST request.

  19. Xoekong

    Hi
    i’m interesting in this topic and want to create bot in other webbrowser game but i don’t clear in function “POST” and “GET” i try to run your project but it return “The remote server returned an error: (403) Forbidden.” ? i don’t understand it can you explain me about more detail and how to use it work ?
    ps. my english skill is not good if use some word wrong i’m sorry but i’m don’t clear in

    1. Uthar

      A 403 error means that you don’t have access to view the page. You could try to make a GET request to google.com
      That should always work 🙂

      If that also gives a 403 error you need to check you security and/or proxy settings. Those settings can cause problems for outgoing web requests.

      1. Xoekong

        ok, i can get some thing from http://www.google.com

        and i try to use POST function for login game i think that is first step for make bot
        this is detail form LIVE HTTP HEADER

        http://www.demononline.in.th/html/home/index.php

        POST /html/home/index.php HTTP/1.1
        Host: http://www.demononline.in.th
        User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:20.0) Gecko/20100101 Firefox/20.0
        Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
        Accept-Language: th-th,th;q=0.8,en-us;q=0.6,en-gb;q=0.4,en;q=0.2
        Accept-Encoding: gzip, deflate
        Referer: http://www.demononline.in.th/html/home/index.php
        Cookie: DMOSESSID=CNKi3m52R-Y01gA9HABFH0
        Connection: keep-alive
        Content-Type: application/x-www-form-urlencoded
        Content-Length: 46
        txtUsername=test&hidLogin=yes&txtPassword=test

        HTTP/1.1 200 OK
        Date: Tue, 16 Apr 2013 07:32:42 GMT
        Server: Apache/2.2.16 (Debian) PHP/5.3.19-1~dotdeb.0 with Suhosin-Patch
        X-Powered-By: PHP/5.3.19-1~dotdeb.0
        Expires: Thu, 19 Nov 1981 08:52:00 GMT
        Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
        Pragma: no-cache
        Vary: Accept-Encoding
        Content-Encoding: gzip
        Content-Length: 14447
        Keep-Alive: timeout=15, max=100
        Connection: Keep-Alive
        Content-Type: text/html; charset=TIS-620

        i see it send 3 data

        txtUsername=test&hidLogin=yes&txtPassword=test

        this is my POST function

        private void defaultPostRequest()
        {

        try
        {

        NameValueCollection l_Content = new NameValueCollection();

        String l_Url = “http://www.demononline.in.th/html/home/index.php”;

        Uri l_Uri = new Uri(l_Url);

        //You can enter as many name/value pairs as you want.
        //In this example I will use username and password
        l_Content.Add(“txtUsername”,”TEST”);
        l_Content.Add(“hidLogin”, “yes”);
        l_Content.Add(“txtPassword”, “TEST”);

        byte[] l_Response = m_HttpHandler.UploadValues(l_Uri, l_Content);

        //The method UploadValues() returns a byte[]. With the following line we can convert it to a string

        txtBoxResponse.Text = Encoding.Default.GetString(l_Response);

        }

        catch (Exception e)
        {

        txtBoxResponse.Text = e.Message;

        }

        }

        when it POST already i refresh browser by “wb.Navigate(“http://www.demononline.in.th/html/home/index.php”);”
        but it not work, what should i do ?

        ps. i want to make bot and want to see real time what is bot do

        1. Uthar

          If you mean with real time that you want to see the changes in the webbrowser the moment you send a GET or POST request with the HttpHandler I can tell you that is not going to work. The HttpHandler class used in this tutorial is written so that it can be used in the background.

          The HttpHandler and webbrowser method wb.Navigate() are two different instances and don’t have effect on each other.
          Also if you login using HttpHandler and then call wb.Navigate() to view the webpage you wont be logged in the webbrowser.

          If you want to see everything in real time you should only use the webbrowser and use javascript to fill in text fields and to click on buttons.

          If you search for Invoke javascript using C#, or something similar you should find enough information.

          1. Xoekong

            Oh, thank you guy
            if i can’t see, how can i test it work or not ?

          2. Uthar

            The server response contains the html source. You saved that in txtBoxResponse.Text.
            If you look at the html source you can decide if your login was successful.

          3. Xoekong

            ok i will try,thank sir

          4. Xoekong

            i use “advancedGetRequest” but no have any response why ?

          5. Uthar

            The advanced methods work with events. If you create an event handler you will be able to see the server response.

          6. Xoekong

            i use “defaultpost” function it work but i have question again i want to make GUI for easy understand what is bot do. i want to check what is txtBoxResponse ,for example html source form login i want to use it in “if” function

            if(html source form login) txtBoxStatus.Text = “login sucess”
            or
            if(html source form attack) txtBoxStatus.Text = “Atk”

            how will i check ?

            and my game have check bot, it random 3 capital letter for typing what should i do and how to create event for Advance Function?

          7. Uthar

            You can use the Contains method. It allows you to check if a specific substring is present in your server response.
            l_Response.Contains(…)

            For the 3 capital letters you could first check if you can find the answer in the html source. This will only work if the 3 capital letters are text.
            If it’s an image you have to convert it to text and after that enter the answer. You can do this yourself or use a service like http://www.9kw.eu

          8. Xoekong

            i can’t use l_Response.Contains(“test”) it error how to use this function ?
            and 3 capital letter is image how to convert to text ?

          9. Uthar

            Then I think it’s still a byte array. Try this:
            l_Response = Encoding.Default.GetString(l_Response);
            Response.Contains(“test”);

            You can use http://www.9kw.eu/ to convert the image to text. Or you can write it yourself if you like a challence 🙂 This link should give you some ideas if you want to write it yourself:
            http://www.wausita.com/captcha/

          10. Xoekong

            challenge is so hard for me , how to use that service “http://www.9kw.eu/” i think this web is downloader program or i mistake ?

          11. Uthar

            No it is not a program you need to download. They offer an API that you can use to solve captchas (like your capital letters).
            They explain here how you can use it:
            http://www.9kw.eu/api.html

            Also they have examples in C# so that you know how to implement it.

          12. Xoekong

            i don’t understand it can you explain me ?

          13. Uthar

            I can’t explain it much better then on that page. That page describes all requests you can use and the C# example shows how you can implement it.

            But as first step you should try to save the image that contains the three capital letters. Later you need to send this image to 9kw using their API.

          14. Xoekong

            um ok, i must to send pic it in many times when i open bot it so hard lol , i have some question about that web

            i want to know function it web

            Url: http://www.9kw.eu/ Errorlog (7) Delete
            Description: API Submit | < Solve

            submit,solve,delete what is it ?

            -i try to run that program some time i can see result but some time can't see it error 0011

          15. Uthar

            On the link I gave you, you can find a list of all error codes:
            0011 Balance insufficient

            This means that you don’t have enough credits. Each time you want them to solve a captcha for you it costs 10 credits. You can get new credits by solving captchas of other users. For each captcha you solve you get 8 credits.

          16. Uthar

            To solve the captchas of other users you can go to the website and click on the third tab, it’s called “Captcha”.

          17. Xoekong

            oh i get it,i think it use computer for convert to string but use human for read and typing , it mean i have to typing in the same haha

          18. Xoekong

            i use this function but it not work ? or i use substring wrong?

            private void status_login() {
            if (txtBoxResponse.Text.Contains(“meta”)) //meta is sub string that return form txtboxresponse
            {
            txtStatus.Text = “Login success”;

            }
            else
            {
            txtStatus.Text = “Login Fail”;
            }
            }

            and how to check press enter key instead of click . i want to key username&password and then pass enter for start function “Login”

          19. Uthar

            Yes that is correct, the captchas are solved by humans and the server returns then the answer to you as a string.

            When using Contains(“meta”) you have to make sure that “meta” is unique and that it only occurs when the login was successful. A good way to do this is to compare the html source of a successful login and a failed login. Then choose a part of the html source that only occurs by the successful login.

            The Main Form has a property called “AcceptButton”, when you set this the button will be clicked when you press enter.
            Or you could add an event listener for key presses.

          20. Xoekong

            i don’t clear “AcceptButton” i use this code but it not work

            private void txtPass_KeyPress(object sender, KeyEventArgs e)
            {

            if (e.KeyCode == Keys.Enter)
            {
            MessageBox.Show(“login”);
            }
            }

          21. Xoekong

            ok ,i can check when press Enter but now i have problem about load captcha. my picturebox don’t show it but if i use wb.nevigate for login and use it recall that url it work are you have any opinion?

            this my code

            urlCaptcha = “http://www.demononline.in.th/html/map/img.php?random=67307”;
            pictureBox1.ImageLocation = urlCaptcha;

            i think it must be login early

          22. Uthar

            The picturebox is probably not working because it doesn’t have any cookies. So if you put that url in the picturebox it’s not yet logged in.
            You should however be able to get the captcha with the httpHandler that you are using for the other requests. The server response will be text only but you can save that as an image.

          23. Xoekong

            how to save that text as image?

          24. Xoekong

            when i use many AdvancePost&Get i have this problem

            NotSupportedException was unhandled by user code
            WebClient does not support concurrent I/O operations.

            how to fix it? then i create many client_Download,UploadCompleted function but i test by debug i see they execute in sequence it mean i can’t use loop or jump or it have other way?

            and about captcha i think i can solve by wb.nevigate and type it by myself 🙂

          25. Uthar

            You have to wait before the previous one has finished. If you make multiple requests at the same time you’ll get that error.

            You can always solve the captcha yourself of course. But it would be nice if the bot is able to done everything 🙂
            If you want to save the text to an image simple save it to a file with the correct image extension.

          26. Xoekong

            how to wait that finish ? at now i create many client_Download,UploadCompleted Function but it run in sequence it not trigger when have response from server

          27. Uthar

            Missed you reply somehow, sorry for the delay 😉

            You can call the next request when the events “UploadValuesCompleted” or “DownloadStringCompleted” are triggered.

          28. Xoekong

            oh, ok now i want to take user-agent how to declare it ?
            i think
            “m_HttpHandler.UserAgent = Mozilla/5.0 (Windows; U; Windows NT 6.1; en-GB; rv:1.9.2.20) Gecko/20110803 Firefox/3.6.20” but it can’t

          29. Uthar

            This is done in the HttpHandler class directly. Just below the line that updates the cookie container:

            (request as HttpWebRequest).UserAgent = l_Settings.AdvUserAgent;

          30. Xoekong

            how to test it work or not ?
            and i have problem when i stop my bot i see this “Cannot access a disposed object. Object name: ‘Form1′”
            what is it ?

          31. Uthar

            You can make a request with your program to “http://www.myip.nl/” or some similar site. The site show some information about your browser, including the user-agent.

            That error is probably caused by an event that is triggered after you click the close button. Maybe some timer that you use? or one of the web requests that wasn’t finished yet?

          32. Xoekong

            ok sir. at now i think my bot is work . thank you for teach me and answer my question. 😉

          33. Uthar

            Ok, nice work 🙂 Good luck with the further development.

        2. Xoekong

          i try to run my bot at office but they use proxy and it response “The remote server returned an error: (407) Proxy Authentication Required.” how to set about proxy ?

          1. Xoekong

            ok i can do it.if i use web.navigate and use small size can i focus on some area when open it ?

  20. waky

    I have got a question, i use yout sniff http request and when i send attack i see this in the sniffer:

    “http://es39.grepolis.com/game/town_info?action=send_units&town_id=xxxx&h=xxxxxx”

    This url receibe post data.

    When i paste the url in the mozilla i see this: “There was an internal error. Please try again later.”

    I dont understa why, if i try the same with other sniffed URL work perfectly for example:
    http://es39.grepolis.com/game/menu_bubble?action=movement

    any idea? thanks for your work!

    1. waky

      its posible attack with http request?

      1. Uthar

        If you included all the required data for the POST request it will work.

        1. waky

          http://es39.grepolis.com/game/town_info?action=send_units&town_id=74&json={“slinger”:30,”id”:173,”type”:”attack”,”attacking_strategy”:”regular”,”town_id”:74,”nlreq_id”:0}

          same this? but dont work.

    2. Uthar

      That’s because there are POST and GET requests. A POST request requires additional information. When you copy the url to send attacks in mozilla the POST request is send as a GET request. Because the Grepolis server doesn’t recognize this you get an error.

      To get the “send attack” request working you need to add all the required information.

  21. waky

    But for example in this line:

    http://es39.grepolis.com/game/town_info?action=send_units&town_id=74&json={“slinger”:30,”id”:173,”type”:”attack”,”attacking_strategy”:”regular”,”town_id”:74,”nlreq_id”:0}

    what is the post and get information?

    GET -> http://es39.grepolis.com/game/town_info?action=send_units&town_id=74

    post -> &json={“slinger”:30,”id”:173,”type”:”attack”,”attacking_strategy”:”regular”,”town_id”:74,”nlreq_id”:0}

    this not is correct, no?

    1. Uthar

      You will not be able to get the POST request working by entering the url in the address bar. Did you watch your http request with a sniffer when adding the json data in the address bar? I think that will result in a GET request.

      Maybe there are addons for your browser that allow you to create POST requests but otherwise you need to write your own program. One thing I noticed you need for your POST request is the X-Requested-With header.

  22. waky

    Ok, thanks. I will try with this information.

  23. Jai

    Hey,
    First off, thanks for the great guide! It’s been really useful.

    I have a little issue with logging in to a travian server… It’s likely a really stupid error on my part but if you have any ideas that would be great.

    I’m trying to use the following code to log on:

    private void defaultPostRequest()
    {
    try
    {
    NameValueCollection l_Content = new NameValueCollection();
    String l_Url = “http://ts3.travian.co.uk/dorf1.php”;
    Uri l_Uri = new Uri(l_Url);
    //You can enter as many name/value pairs as you want.
    //In this example I will use username and password
    l_Content.Add(“name”, “derp”);
    l_Content.Add(“password”, “derp123”);
    m_HttpHandler.Headers.Add(“User-Agent”, “Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.22 (KHTML, like Gecko) Chrome/25.0.1364.172 Safari/537.22”);
    byte[] l_Response = m_HttpHandler.UploadValues(l_Uri,l_Content);
    //The method UploadValues() returns a byte[]. With the following line we can convert it to a string
    textBoxResponse.Text = Encoding.Default.GetString(l_Response);
    }
    catch (Exception e)
    {
    textBoxResponse.Text = e.Message;
    }
    }

    But when the response comes back it is still on the login page. The password field seems to have been filled in but not the name field? HTML snippet below:

    Password:

    Any ideas you have would be highly appreciated!
    J

    1. Uthar

      When I look at the html request I see 5 parameters and you have only 2. I haven’t made anything for travian so I’m not sure they are all necessarily but I assume they are.

  24. Anonymous

    Good day Uthar,

    I have looked at your code in MyBot_Tutorial. I have 1 question for now, what is the exact difference between the default and advanced method in sending and getting data?

    1. Uthar

      The default method uses blocking calls, meaning the thread halts until the request is answered by the server. During this time you can’t do anything else.

      The advanced method uses interrupts. After sending the request the current thread is not blocked, you’ll receive an interrupt to let you know when the server answers your request.

  25. Ed

    When I request page and try to get its source I get “‹” characters. I can decode them in fiddler2 but how I should do that in my code (actually yurs code)?

    1. Uthar

      Not sure if your message above still shows the correct characters? WordPress often removes special characters from messages as precaution.
      If not you can use http://pastebin.com or something similar and post the link here.

      Regarding the code, all requests are returned as plain text, you can just put them in a string. Or did I misunderstood your question?

  26. Ed

    To make more clear there is pic. I get this insted of page source code. http://i.imgur.com/03NiOC1.png

    Some pages give me their plain text and some give those characters. Also in fiddler shows those characters but when I mark request and click “decode selected session” then it shows source code just fine.
    Thanks for you fast response

    1. Uthar

      Maybe it is a datastream instead of a single response? If you want you can send me the project files, I’ll send you an email.
      I should have some time on Monday to look at it.

      1. Ed

        I think you missunderstood me.

        I believe that ” m_HttpHandler.Headers.Add(“Accept-Encoding”, “gzip, deflate”);” this gives me encoded text gzip or deflate. When I remove this header then I get plain text but without special characters and so on. Yeah I can keep this header deleted but I want to have it. Maybe it will be more safer and “bot/program” will be more undetected. Or thats not true? If site admin checks that I get into site with one header then he will know that I browse his site not from normal browser..

        1. Ed

          I added “(request as HttpWebRequest).AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;” in HttpHandler and seems that I can get readable text and keep headers.

          1. Uthar

            Good to know you found a solution. Regarding your earlier question, I think it’s unlikely they check for all headers. Normally I add only the headers that are needed for the bot to function properly. For example in Grepolis the X-Requested-With header, without that you will not get the correct response.
            However now that you have it working you should just keep the header to be safe.

          2. Ed

            Im trying to add all headers now because I need to look that Im 100% browsing from browser. Site owner will check which requests I send then he might notice that I dont send few requests which normal browsers would send. Now Im having problems adding cookies in my requests. I see that there is cointainer but how I should add cookies into that container and use those cookies where needed

          3. Uthar

            The cookies are added automatically. Only when cookies are specified as http-only it will not work because they have restricted access.
            http://en.wikipedia.org/wiki/HTTP_cookie#HttpOnly_cookie

  27. Ed

    If they are added automatically then why I cant see them In fiddler when I do requests?

    1. Uthar

      I’ll have to check the tutorial source code but I though I implemented it there already to save the cookies. I will do this Monday, maybe earlier if I find some free time 🙂

      In the mean time, could you check the server response with fiddler to make sure there aren’t any http-only cookies.

      1. Ed

        Seems that it shows cookies but there is bunch of other problems.. When my request adds cookies then .KeepAlive header disappears.. Its so hard to set them properly.. And sometimes headers go in random order.. Damn

        1. Uthar

          You really do not need to worry about the order of the headers, nobody is going to notice this.

          But I have no idea what could cause you header to disappear.

          1. Ed

            At first I thought that fiddler hides KeepAlive header but seems that this is bug in class or something like that. I did some testings wihout fiddler on. I used this site which shows which headers I sent https://bayden.com/echo.aspx So for the first time opening site there is KeepAlive header but when I request site another time (or it can be navigating links..) KeepAlive header disappears.

  1. MyBot Tutorial Part 1 » Bots

    […] MyBot Tutorial Part 2 » […]

Leave a Reply

Your email address will not be published.

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

WordPress SEO fine-tune by Meta SEO Pack from Poradnik Webmastera